This task is about attempting to reduce the memory requirement for FWD clients. Customers are currently staying within 256mb of GUI FWD client and 64mb for appservers. The goal would be to check if this is scalable and how we can reduce this requirement especially for environments with many clients.

One idea to start with is the analysis of the font table memory requirement in FWD client. This is ~60MB and forces the FWD client to require at least 128MB to be viable. In the context of tens/hundreds/thousands of clients, this adds up to lots of GB memory. The question is whether we can share memory in between clients to reduce redundancy (e.g. font table if read-only).

Alexandru, for web clients, the Font objects are only being used for getting the FontMetrics.
We can easily reduce the web client memory requirement just by storing these metrics instead.

The main problem on this approach is that the FontMetrics stores a Font reference, which avoids Font instances to be erased by GC.

There is an alternative for this is using fontbox, which is just a pure ttf parser. Extracts the font stats which we need (charsWidth, getHeight and getWidths).

We can easily reduce the web client memory requirement just by storing these metrics instead.

This sounds reasonable.

There is an alternative for this is using fontbox, which is just a pure ttf parser. Extracts the font stats which we need (charsWidth, getHeight and getWidths).

I don't this is trivial. There are cases where we measure texts on Java client side in order to provide the right metrics. The size of a text can't be inferred using the metrics of a font. It shall be rendered and measured. There are very specific edge cases like kerning. I am not sure if this is the case ONLY for web-scaled (which is happening for several clients that use GUI).

For a bit of contexts: because of the AA in the browser that can't be disabled, we are sometimes forces to measure the size of a text at client side and send that metric to Web in order to do the scaling. There is also some effort to improve this in #9842. You can look at #9842-21 and #9842-63 (+ whole discussion in #9842). In reality, we need to store only the fonts that use web scaling (as dictated by directory.xml), but otherwise I am not sure if we ever need the fonts. I mean, we need them to be sent to the web client in order to render such texts with the right font, but this doesn't require the Java client to retain them forever.

Alexandru Lungu wrote:

We can easily reduce the web client memory requirement just by storing these metrics instead.

This sounds reasonable.

There is an alternative for this is using fontbox, which is just a pure ttf parser. Extracts the font stats which we need (charsWidth, getHeight and getWidths).

I don't this is trivial. There are cases where we measure texts on Java client side in order to provide the right metrics. The size of a text can't be inferred using the metrics of a font. It shall be rendered and measured. There are very specific edge cases like kerning. I am not sure if this is the case ONLY for web-scaled (which is happening for several clients that use GUI).

The problem is not that you need to load the Java AWT Font and render it.
When you render the font, the FontDesignMetrics has a cache mechanism (static map) which keeps your Font instances loaded in memory.
BTW, the Font stores in the Heap memory, just the metadata. The rest of the stuff like font glyphs, are stored in off-heap memory.
I have tried to not use FontDesignMetrics and use pure glyphs, but they are cached like FontDesignMetrics. You can't control the memory.

BTW, there are also two maps (text metrics and font metrics) that are loaded from the server. They contain data extracted from Windows to help FWD compute metrics faster based on Windows metrics. This is basically aids cold times by storing all sizes of static labels, so one can render a screen fast, without computing the metrics for each and one text. We usually need this. In the past, we also relied on JS metrics, but the cold time to get text metrics for each and one label was a performance killer. I think Paula knows the task where we discussed this (regarding font size approximation).

For a bit of context, in performance efforts, we always strive to get better times even for memory expense. Now we have the inverted scenario. So, whatever we want to do here, we need to ensure we don't downgrade performance. If we do, it shouldn't be general or noticeable. I would prefer to have something like:

• shared memory between Java clients
• compressed data structured that are seldomly used.
• offloading data to disk if not needed for a long period of time.
• offload more data in JS: this can be interesting as customer's browser is a bit of a free lunch in terms of memory. Instead of having 256MB of client * 100 clients on the deployment machine, we can rather have 128MV * 100 clients on the deployment and off-load the 128MB extra to each and one customer. Each customer browser may afford extra 128MB, but on the VM we can't afford extra 128MB because this multiplies with number of users.
  • Of course, we can't cache things on JS and expect to leverage that cache in Java client as the latency would be a nightmare. But maybe some logic can be transferred to Web so that some structures scopes are bound only to the Web client.
  • otherwise, maybe we can offload some information on the server, where we can share in between clients. But again, we need to avoid high latency.
• I have a feeling that text metrics may have lots of redundant data, so maybe we can work on that more.

Also, very important: we need to have the memory consumption predictable. We can't afford having thin clients initially that get fatter over time. Setting the right memory to the VM would be a nightmare without deterministic memory usage from the clients. If there is a chance to reach 200MV with a client, than this is the maximum memory we should intake in our computation. This should be optimized.

I have a feeling that text metrics may have lots of redundant data, so maybe we can work on that more.

Using something like a TRIE or PATRICIA tree to store the metrics. This will avoid having lots of duplicated string prefixes.

EDIT: instead of storing:

My Account -> metrics
My Account: -> metrics

we can store

My Account -> metrics
           -> : -> metrics

this alone will optimize 20 bytes as an example. The longer the text the better.

I have completely abandon the idea of Font loading from Web FWD client because it is too tedious to work with Font and FontDesignMetrics.

What I have done right now is to precalculate these metrics from FWD server (leading, ascend, descend, each character width, height) and send them to FWD client. The data transmission between FWD server and client is almost instance. The memory consumption is 256KB per font, but I think this is reasonable.
I am refining the implementation and see how it is working.

Also, since the Font definition is no longer required after calling ws.createFont, we can just change FontDetails.fontDefinition field to null. These font definitions takes about 10MB. Also, there is a map from fontManager, which takes about 25MB which I think I should do about it.

The off-heap memory mostly comes from rendering the images, and it is about 100MB. I will also investigate this.
We can also trim the unused jar files because they are being loaded somehow in memory. I don't know much details about that.

each character width

This phrase is a red flag for me. This is something that is not usable in kerning fonts. Characters have different widgets depending on neighbor letters. Is this used only for fixed fonts?

Also, since the Font definition is no longer required after calling ws.createFont, we can just change FontDetails.fontDefinition field to null. These font definitions takes about 10MB. Also, there is a map from fontManager, which takes about 25MB which I think I should do about it.

While it looks like you get a good technical grip on the issue, please take into consideration the efforts done to have pixel-perfect texts. There was a non-trivial effort to achieve this, sometimes using webScaled mode. Any solution dropping this level of accuracy is should be carefully analyzed.

I have a feeling that text metrics may have lots of redundant data, so maybe we can work on that more.

Please take a look on this matter as well. On disk, the text-metrics.xml has 24MB (for a customer, indented XML).

Alexandru Lungu wrote:

each character width

This phrase is a red flag for me. This is something that is not usable in kerning fonts. Characters have different widgets depending on neighbor letters. Is this used only for fixed fonts?

This is something to consider.

Alexandru Lungu wrote:

Also, since the Font definition is no longer required after calling ws.createFont, we can just change FontDetails.fontDefinition field to null. These font definitions takes about 10MB. Also, there is a map from fontManager, which takes about 25MB which I think I should do about it.

While it looks like you get a good technical grip on the issue, please take into consideration the efforts done to have pixel-perfect texts. There was a non-trivial effort to achieve this, sometimes using webScaled mode. Any solution dropping this level of accuracy is should be carefully analyzed.

Alexandru Lungu wrote:

I have a feeling that text metrics may have lots of redundant data, so maybe we can work on that more.

Please take a look on this matter as well. On disk, the text-metrics.xml has 24MB (for a customer, indented XML).

Ok, this is good to know. I will look how these are being currently used.

I have looked into text-metrics.xml and how it is being used (served as a cache, to provide fast and accurate results for text metrics). The FontDesignMetrics still gets created even if we have 100% coverage.

For the approach #11326-10, storing each character width is a solid idea because most of the Java AWT fonts computes the text width by character width (FontMetrics.stringWidth) as SUM(each char width)).
I don't see any place where the font kerning or any font text attribute is being enabled (which could break this idea).

With this change and with FontDetails.fontDefinition + FontManager optimization, I have managed to squeeze 20 MB of client memory. Heap memory baseline after GC is now 71 MB.

I will start to investigate why the offheap memory is still 101 MB.

The FontDesignMetrics still gets created even if we have 100% coverage.

What do you mean by 100% coverage? Mind that text-metics covers only static texts. There may be labels that are rendered using variables (aka dynamically). These should still use on the fly metrics.

most of the Java AWT fonts

Can you list some of these?

https://www.microsoft.com/en-us/microsoft-365-life-hacks/presentations/different-types-of-fonts-and-how-to-choose-one

This is a relevant blog for serif vs sans-serif vs monospace. What you describe is monospace. In FWD, we describe these as "fixed". We have different formulae for fixed vs non-fixed fonts. The non-fixed fonts (aka serif/sans-serif) are way more common in the applications we converts (MS Sans-Serif, Segoe UI, etc.).

I don't see any place where the font kerning or any font text attribute is being enabled (which could break this idea).

Check https://docs.oracle.com/javase/8/docs/api/java/awt/FontMetrics.html#stringWidth and:

Note that the advance of a String is not necessarily the sum of the advances of its characters.

also https://docs.oracle.com/javase/8/docs/api/java/awt/font/TextAttribute.html#KERNING

The default advances of single characters are not appropriate for some character sequences, for example "To" or "AWAY".

Note how AW has a common horizontal rendering space.

Alexandru Lungu wrote:

The FontDesignMetrics still gets created even if we have 100% coverage.

What do you mean by 100% coverage? Mind that text-metics covers only static texts. There may be labels that are rendered using variables (aka dynamically). These should still use on the fly metrics.

Not necessarily the FontDesignMetrics, but at least one Font object is being created when calling createFont.

Alexandru Lungu wrote:

most of the Java AWT fonts

Can you list some of these?

These which does not have the following options when creating/defining the font (according to the JDK 17 source code):
- CHAR_REPLACEMENT, FOREGROUND, BACKGROUND, UNDERLINE, STRIKETHROUGH, STRIKETHROUGH, RUN_DIRECTION, ... KERNING, ...

In FWD, we use only use UNDERLINE, which I need to think about this.

Alexandru Lungu wrote:

https://www.microsoft.com/en-us/microsoft-365-life-hacks/presentations/different-types-of-fonts-and-how-to-choose-one

This is a relevant blog for serif vs sans-serif vs monospace. What you describe is monospace. In FWD, we describe these as "fixed". We have different formulae for fixed vs non-fixed fonts. The non-fixed fonts (aka serif/sans-serif) are way more common in the applications we converts (MS Sans-Serif, Segoe UI, etc.).

No, what I have discussed is not monospace. The letters do not have the same amount of space.

Alexandru Lungu wrote:

I don't see any place where the font kerning or any font text attribute is being enabled (which could break this idea).

Check https://docs.oracle.com/javase/8/docs/api/java/awt/FontMetrics.html#stringWidth and:

[...]

also https://docs.oracle.com/javase/8/docs/api/java/awt/font/TextAttribute.html#KERNING

[...]

Note how AW has a common horizontal rendering space.

I have looked closely in the JVM 17 implementation, which looks like this: FontDesignMetrics.stringWidthFontDesignMetrics.stringWidth

   public int charsWidth(char[] data, int off, int len) {

        float width = 0;
        if (font.hasLayoutAttributes()) {
            if (len == 0) {
                return 0;
            }
            String str = new String(data, off, len);
            width = new TextLayout(str, font, frc).getAdvance();
        } else {
            /* Explicit test needed to satisfy superclass spec */
            if (len < 0) {
                throw new IndexOutOfBoundsException("len="+len);
            }
            int limit = off + len;
            for (int i=off; i < limit; i++) {
                char ch = data[i];
                if (ch < 0x100) {
                    width += getLatinCharWidth(ch);
                } else if (FontUtilities.isNonSimpleChar(ch)) {
                    String str = new String(data, off, len);
                    width = new TextLayout(str, font, frc).getAdvance();
                    break;
                } else {
                    width += handleCharWidth(ch);
                }
            }
        }

        return (int) (0.5 + width);
    }

The case which needs attention is when we have UNDERLINE and non-ascii chars.

Teodor Gorghe wrote:

We can also trim the unused jar files because they are being loaded somehow in memory. I don't know much details about that.

Greg, we can create a gradle task which copies in a lib.client folder only the jar dependencies to run the FWD clients. Otherwise, all jars are 'loaded into memory' when loading a Java class, to find it.

There is Software_Dependencies which we can use as a starting point.

PS: the image is generated through Linux Editor with Liberation Sans 48. You can replicate using most of the common Windows fonts (Helvetica, Times New Roman, Segoe, Sans Serif, etc.)

Alexandru Lungu wrote:

No, what I have discussed is not monospace. The letters do not have the same amount of space.

Sorry for the confusion. Indeed, this is not related to monospace. Yet, SUM(each char width) is not correct for most fonts. That is what I was trying to point out.

AVAVAVAVAV
AAAAAVVVVV

have different widths. In first case, most of the V overlap the A.

In FWD, for AWT fonts, we have the KERNING disabled.
After a discussion with Alex, the conclusion is that FWD requires KERNING, but I will do some 4GL experiments. There is no KERNING feature in the 4GL language, but there might be something related to the Windows Fonts and how they are rendered. This is a bug which I have didn't considered.

For reducing the memory consumption, the approach which is the most reliable and it does not kill the performance that much is by storing these fonts in the filesystem and reduce the memory stored in the FontDetails (fontDefinition) and customFontData (I think FS cache is fast).

Teodor Gorghe wrote:

After a discussion with Alex, the conclusion is that FWD requires KERNING, but I will do some 4GL experiments.

Something else to point: if width(string) is not the same as sum(width(char-of-string)), CARET positioning will break. I can't find the exact example from back when implementing this, but I'm pretty sure in FWD we need exact width of a string.

We discussed this in #9842

4GL does not kern the fonts (I have checked with default MS Sans Serif), default 4GL font.

I will test with other fonts.

Indeed, I can confirm the results. Can you also try with a bigger font (close to 30/40)?
It is weird tho. I am most certain we discussed this in the past for different customers and concluded that kerning is a big deal when comes down to font measurement.

Teodor, if we can rely on the fact that OE is never using kerning (maybe hints the Win GDI to not enable it), then we can make our lives easier. That would be great.
AWT already doesn't use it.
What about Web, is kerning automatically enabled on web? Can we disabled it?

In FWD, for AWT fonts, we have the KERNING disabled.
After a discussion with Alex, the conclusion is that FWD requires KERNING, but I will do some 4GL experiments. There is no KERNING feature in the 4GL language, but there might be something related to the Windows Fonts and how they are rendered. This is a bug which I have didn't considered.

I'm surprised by this. I don't expect that the 4GL would expose kerning to the developer, but I did expect that the WIN32/GDI32 font support implicitly uses kerning in all of its calculations.

Well, maybe MS Sans Serif doesn't have kerning (most probably it doesn't). Can you also test Segoe UI?

Well, considering that Java AWT in FWD doesn't have the kerning mode set, will these be rendered correctly?

These are rendered correctly, but I think the problem comes on fill-in/editor, when positioning the CARET. I am testing this right now.

We somehow mitigated this case with some workaround. Let me check if enabling the kerning does something and the text selection length wobble gets fixed.

Teodor Gorghe wrote:

I have noticed that Arial font actually uses kerning:

Is this from FWD or 4GL?

4GL.

Teodor Gorghe wrote:

4GL.

Please post the test and progress.ini (or the font-table).

4GL code4GL code

DEFINE VARIABLE cString1 AS CHARACTER INITIAL "AVAVAVAVAV" NO-UNDO.
DEFINE VARIABLE cString2 AS CHARACTER INITIAL "AAAAAVVVVV" NO-UNDO.

DEFINE VARIABLE iLen1    AS INTEGER   NO-UNDO.
DEFINE VARIABLE iLen2    AS INTEGER   NO-UNDO.
DEFINE VARIABLE iDiff    AS INTEGER   NO-UNDO.

ASSIGN 
    iLen1 = LENGTH(cString1)
    iLen2 = LENGTH(cString2)
    iDiff = ABSOLUTE(iLen1 - iLen2).

DISPLAY 
    "String 1: " cString1 FORMAT "x(15)" SKIP
    "String 2: " cString2 FORMAT "x(15)" SKIP
    WITH FRAME f-results TITLE "String Length Comparison" SIDE-LABELS.

progress.iniprogress.ini

[Startup]
V6Display=no
;ImmediateDisplay=yes
;MultitaskingInterval=100
DefaultFont=Arial, size=40
DefaultFixedFont=Arial, size=40
DLC=C:\Progress\OpenEdge
Use-3D-Size=Yes
PROPATH=.,C:\Progress\OpenEdge\gui,C:\Progress\OpenEdge,C:\Progress\OpenEdge\bin

[Colors]
;******************************************************************************
; THE DEFINITION OF COLOR 0 THROUGH 15 IS PRIVATE TO THE PROGRESS ADE.  
; MODIFYING COLORS 0 THROUGH 15 MAY PREVENT THE PROGRESS ADE FROM RUNNING.
; The following color definitions correspond to the ADE standards.
;  0 to 15  - reserved
color0=0,0,0
color1=0,0,128
color2=0,128,0
color3=0,128,128
color4=128,0,0
color5=128,0,128
color6=128,128,0
color7=128,128,128
color8=192,192,192
color9=0,0,255
color10=0,255,0
color11=0,255,255
color12=255,0,0
color13=255,0,255
color14=255,255,0
color15=255,255,255

NORMAL=0,15
INPUT=15,0
MESSAGES=15,1

[Default Window]
;x=
;y=
;rows=
;columns=

[fonts]
;******************************************************************************
; THE DEFINITION OF FONT 0 THROUGH 7 IS PRIVATE TO THE PROGRESS ADE.  
; MODIFYING FONTS 0 THROUGH 7 MAY PREVENT THE PROGRESS ADE FROM RUNNING.
; The following fonts definitions correspond to the ADE standards.
;       ?  - DefaultFont from Startup Section
;       0  - DefaultFixedFont from Startup Section (1 char per PPU)  
;       1  - Proportional System Font
;       2  - Editor Font for 4GL program entry
;       3  - TTY Simulator (should be fixed)
;       4  - Dynamically-sized widgets, eg status-line, selection-list
;       5  - Static widgets, eg. combo-boxes
;       6  - Dynamic, bold (TranMan2)
;       7  - Reserved
font0=Arial, size=32
font1=Arial, size=32
font2=Arial, size=32
font3=Arial, size=32
font4=Arial, size=32
font5=Arial, size=32
font6=Arial, size=32, bold
font7=Arial, size=32

[WinChar Startup]
;DLC=
;PROCFG=
;PROMSGS=
;PROPATH=

;******************************************************************************
; The "SingleLineBorder" setting allows you to define the characters used to
; draw all borders or "boxes" in character mode.  
; The right side of the equals sign "=" must be a comma-delimited list of
; eight numbers, corresponding to the values of the characters in your
; -cpstream character set.  The order of values MUST BE the following:
;
;    top, bottom, left, right, top left corner, top right corner, 
;    bottom right corner, bottom left corner
;
;SingleLineBorder=45,45,124,124,43,43,43,43

;******************************************************************************
; The "SysCheckmark" setting allows you to define the character used to mark
; a selection in various widgets, e.g., browser, selection list.  The right
; side of the equals sign "=" must be a single numeric character value.
;
SysCheckmark=251
DLC=C:\Progress\OpenEdge
PROPATH=.,C:\Progress\OpenEdge\tty

;******************************************************************************
; The "BrowseRowMarker" setting allows you to define the character used for
; the browse widget's row marker.  The right side of the equals sign "=" must
; be a single numeric character value.
;
;BrowseRowMarker=42

[WinChar Colors]
color0=WHITE/BLUE          "NORMAL" 
color1=BLACK/GRAY          "INPUT, UNDERLINE"   
color2=BLACK/GRAY          "MESSAGES, REVERSE" 
color3=BLUE/WHITE          "HIGHLITE, HELP" 
color4=BLINK-RED/WHITE     "URGENT" 

;NORMAL=WHITE/BLUE
;INPUT=BLACK/GRAY 
;MESSAGES=BLACK/GRAY 

[WinChar Default Window]
;rows=25
;rows=50

[WinChar Keys]
;GO=F1,CTRL-X

[ProADE]
;DividerFgColor=15
;DividerBgColor=1
;OKBoxFgColor=1
;OKBoxBgColor=8
;FillinFgColor=0
;FillinBgColor=8
;Editor4GLFgColor=DEFAULT
;Editor4GLBgColor=DEFAULT
;Editor4GLFgSmallColor=0
;Editor4GLBgSmallColor=8
;FixedFont=0
;StandardFont=1
;Editor4GLFont=2
;EditorTabStop=4

[Proedit]
SaveSettings=yes
BufList=
ExitWarning=yes
SaveBufList=no
MinimizeBeforeRun=no
RestoreAfterRun=yes
PauseAfterRun=yes
AutoCleanup=yes
MostRecentlyUsedFileList=yes
MRUEntries=4
EditorFont=2
AutoSaveCls=no

[RBStartup]
;RBSTARTUP=

[ReportBuilderSaveList]
ActionBar=1
FormatBar=1
FieldNames=0
SortFieldNames=0
ShowHiddenTables=0
PasteArgs=1
ShHzRul=1
ShVtRul=1
ShGrd=1
Maximize=0
NotFirstTime=0
Library=
Report=

[ReportBuilderPreferences]
InstantRep=2
PromptFields=1
ScrBarHz=1
ScrBarVt=1
PromptConnection=0

[ReportBuilderDefaults]
DefMeas=1
PgLM=1
BotM=50
LeftM=50
RightM=50
TopM=50
RulPit=10
VRulPit=10
Pts=120
SnapTo=1
Color=0
WidthRep=1
WidthColor=8
FaceName=Arial
LogF=No
LogT=Yes
LibDir=
DatabaseDir=
ImgDir=
UDFDirectory=
MemExt=TXT
ImgExt=BMP
ImageQuality=0

FWD renders this correctly both on WEB and Swing, but the problem comes when computing the metrics. It uses the font without KERNING and the metrics are calculated by the fast path, char by char.

• looked it closely and the Swing client renders the text without KERNING

Serban I wonder if in #9842 there is also an instability due to the fact that Swing canvas doesn't account for kerning. In other words, maybe what we though about right metrics for the texts in webscale was still off because the kerning was not used. What if webscale would actually use the Swing metrics but with kerning active. Would that provide a better non-skewed display of the text in general? Webscale is made to avoid AA kicking in, but I think the webscale algorithm is not considering kerning, so the text are rendered with Web kerning, but widened because Java measured the text without it.

Can we have an experiment on top of #9842 with it? I would expect that texts would look better. It isn't related to caret, but rather on the text look and feel.

Serban, you can take 11326a and test if you need. The text kerning has been enabled and also I have managed to achieve low memory consumption.

Right now, I am working on fixing the text selection regression on Swing, which was already on web client.
It is a bug in AbstractGuiImpl.drawTextSelection which splits the text in three parts and it draws them individually. This is not right when the kerning is being used.

It is a bug in AbstractGuiImpl.drawTextSelection which splits the text in three parts and it draws them individually. This is not right when the kerning is being used.

We had this issue for a long time and we didn't manage to find any proper solution by now. There is a very verbose discussion in #9842 (please read it carefully) about what metrics should be used for substrings.

• Reverted AWT font kerning.

I will create a separate task for that because there is more Swing font and JS font calculations to be considered.

The second most effective memory optimization is by reducing the jar files from the classpath (there are lots of jar files from server which are not being used on client side, #11362).

Right now, I will do a last look into the off-heap memory. Also, I am checking if -Xss256k is safe to use.

The off-heap memory usage is quite large, more than 64M, even after font optimization.

What I have discovered today is that AppCDS actually helps reducing the metaspace usage (which is about 60-80% of off-heap memory) by sharing the loaded classes in the shared OS memory.

I have replaced ReflectASM with pure Java MethodHandle API and the number of shared classes has increased.

Baseline metaspace: ~64 MB.
Non-shared AppCDS (with ReflectASM): 28.5 MB.
Non-shared AppCDS with ReflectASM removed: 13.54MB.

This means that ~50 MB which are the shared loaded classes, are now shared across the FWD client instances.

This is a great result! I've been wanting to move to a method handle approach for some time now (see #6819-7 and following). Do you have any measurement of the runtime performance of your change (compared to the original version)?

Prior experiments from #10071 showed that performance with JDK17 Reflection API is the same.

I have done a smoke test and I don't see any noticeable difference.

The reason that I chose MethodHandles over Java Reflection API is because Java Reflection API creates lots of proxies (which they are not gonna get into the archive) until exceeds the inflation threshold.

Teodor, where can I see these changes?

I will commit here, right now I am working to put the javadocs and the history entries.

Committed revision 16512 on task branch 11326a:
- Removed ReflectASM completely. Added some small performance improvements.

When the web client JVM state is warn, the code cache will get to about 25 MB+, which can't be shared across JVM processes.

AppCDS shares only the class definitions, but if we want to also share the compiled code cache, we need to use JDK 25 Project Layden AOT or the alternative from GraalVM.

When the web client JVM state is warn, the code cache will get to about 25 MB+, which can't be shared across JVM processes.

Constantin, I know that you had recent work on #11271 and discussion on #11371. My biggest concern ATM is that the memory demand may spike in certain use-cases (e.g. using redirected terminal). If a report is generated using frames, isn't the report stored entirely in memory (i.e. Fill-Ins representing cells, texts representing labels and so on)? I see that the effort of #11326 is to reduce baseline memory consumption, which is great. But I think we need to also be carefully about potentially large screens / redirected terminals that may create lots of widgets within the client. Also, there are places where we use I/O (e.g. filesystem access, memptr, streams, native invokers, etc.). All of these may require memory spikes. I just want to make sure we also assess these. As long as they are not server-side, then we need to have a second pass through these places and check that everything is buffered / cached / streamed / etc. to not risk bringing big data chunk in memory.

Client is single-threaded, so we need to worry about only one thread doing all this work, but that thread may have several streams opened, memptr handled, and so on, right? How can we assess this?

Alexandru Lungu wrote:

Constantin, I know that you had recent work on #11271 and discussion on #11371.

This is only for batch clients. Interactive clients will not be able to use #11371.

My biggest concern ATM is that the memory demand may spike in certain use-cases (e.g. using redirected terminal). If a report is generated using frames, isn't the report stored entirely in memory (i.e. Fill-Ins representing cells, texts representing labels and so on)?

Redirected terminal in FWD is limited to 128x512 lines x cols. So it can't go beyond that. And this is allocated once per terminal.

I see that the effort of #11326 is to reduce baseline memory consumption, which is great. But I think we need to also be carefully about potentially large screens / redirected terminals that may create lots of widgets within the client.

The frame registry can be huge for very complex screens. For interactive clients, we can't let it sit on 64MB, as frames will not be able to fit most likely. But that's on a case by case basis to determine what's the right heap size.

Also, there are places where we use I/O (e.g. filesystem access, memptr, streams, native invokers, etc.). All of these may require memory spikes. I just want to make sure we also assess these. As long as they are not server-side, then we need to have a second pass through these places and check that everything is buffered / cached / streamed / etc. to not risk bringing big data chunk in memory.

See #11327 - that needs to be fixed to not transfer 'in bulk' the memptr content or files, when i.e. COPY-LOB is used.

Client is single-threaded, so we need to worry about only one thread doing all this work, but that thread may have several streams opened, memptr handled, and so on, right? How can we assess this?

• for memptr, nothing stays in JVM after the work is done.
• for redirected terminal, multiple streams can be opened at a time

BTW, the memory referenced by a memptr is allocated off-heap while that variable is in scope on the call stack.

Constantin Asofiei wrote:

The frame registry can be huge for very complex screens. For interactive clients, we can't let it sit on 64MB, as frames will not be able to fit most likely. But that's on a case by case basis to determine what's the right heap size.

The frame registry is stored on server side?
Currently, the base heap memory usage is 45 MB (with the VirtualScreenImpl optimization). I will do the experiment with the jar files in a separate directory to see the difference.

Regarding memory spikes, right now, the client seems fine when using 128 MB heap size. I didn't saw any crashes or OutOfMemory issues.

Teodor Gorghe wrote:

Constantin Asofiei wrote:

The frame registry can be huge for very complex screens. For interactive clients, we can't let it sit on 64MB, as frames will not be able to fit most likely. But that's on a case by case basis to determine what's the right heap size.

The frame registry is stored on server side?

UI widgets have parallel structures on both server and client-side. On FWD client, we have WindowManager$WorkArea.windows with all the window and their UI components (the entire UI tree), plus WidgetRegistry class. For example, in customer applications you can open many windows and each one with very complex frames (nested frames, many widgets ,etc). All of this stays in memory on the FWD client.

Regarding memory spikes, right now, the client seems fine when using 128 MB heap size. I didn't saw any crashes or OutOfMemory issues.

From #11214, I understood we need to support ~2.7k sessions at once. With 128MB, this means ~350GB to be used, only by heap. Having it in 64MB would halven this to ~170GB which I think is a bit more acceptable. Of course, these is a lot of state and we can't let the client operate in 4MB :) but if we can squeeze it in 64MB, then I think we can be more than happy about it. 128MB is still tricky.

I'm not sure I understand this. The FWD client can have multiple streams/memptr/etc, where one is used at a time, but:

Sorry for my poor phrasing. I meant that the FWD client can have multiple steams/memptr even if single-threaded. Even if you work with a stream at that right time, you can still have some latent streams hanging around in memory. As for the "buts", I did not looked into these yet. It was just my intuition that some things may be memory consuming apart from the actual widgets.

The frame registry can be huge for very complex screens. For interactive clients, we can't let it sit on 64MB, as frames will not be able to fit most likely. But that's on a case by case basis to determine what's the right heap size.

Well, having more windows opened at the same time can lead to multiple complex screens, right? So technically speaking, one user can open 100 windows and break the client.

Alexandru Lungu wrote:

Regarding memory spikes, right now, the client seems fine when using 128 MB heap size. I didn't saw any crashes or OutOfMemory issues.

From #11214, I understood we need to support ~2.7k sessions at once. With 128MB, this means ~350GB to be used, only by heap. Having it in 64MB would halven this to ~170GB which I think is a bit more acceptable. Of course, these is a lot of state and we can't let the client operate in 4MB :) but if we can squeeze it in 64MB, then I think we can be more than happy about it. 128MB is still tricky.

128 MB is a safe max memory limit, but I should test what is the maximum window count until the client crashes.

Constantin, about the printer font, I have found it. On Windows 11/10, the default font is Arial with size 11, which is the default printer font.
The Arial font is a Microsoft proprietary font and the most similar font is Liberation Sans.
When using new Font(Font.SANS_SERIF, Font.PLAIN, 11), on linux it will resolve to Liberation Sans and on Windows to Arial. I think this is the most appropriate replacement for FWD.

Team,

Also please note there is other tasks (#10183, #10286) that has memory usage optimization changes that were rejected from trunk. But I still have no exact answer why. I have no issues with the changes on my local systems.

Thanks Eugenie. I will take a look into 10183b spawn.c changes.

I have tried to take the p2j client dependencies only, there were couple of issues while running the appservers clients, which I have fixed. The GUI client works well, but the web client crashes right now and I need to look why.
I have some build.gradle changes which I will put here once I get everything fully working.

Alexandru, your memory consumption estimation is wrong.
We need to also take to account the off-heap memory consumption. Baseline trunk uses 101 MB off-heap memory. I have managed to get it to ~45 MB per client with AppCDS + changes from here. AOT will decrease it to just ~20 MB per client (which also does some heap optimization). The FWD client uses about 6 MB off-heap memory and the rest is just JVM overhead.
The garbage collector also uses a lot of memory. Last time, I saw that GC1 took about 60 MB memory. The best alternative is serial garbage collector, which has almost 0 memory overhead.

Eugenie, I have looked into 10183b changes, which avoid forking the JVM instance. I don't see anything that can cause spawn issue.
I think the issue comes from a GLIBC version mismatch or Java version. The issue was a long time ago and I don't have the problematic binary which I can check it with ldd.

The changes from 11326a does not depend from 10183b, but it is good to know that there is 50 MB memory per client which is kept by the spawn process.

Alexandru, your memory consumption estimation is wrong.
We need to also take to account the off-heap memory consumption

Of course, that is why I wrote "~350GB to be used, only by heap". This is only for the heap metric. There are plenty to be optimized for the customer in discussion - off-heap being one of them. #11214-55 has a view on a production system and how memory is distributed. The point is that #11326 is about the memory consumption of FWD clients, but the server stores a non-trivial state for each of this clients that may skyrocket. Even if the client will work only within 128MB or 64MB, the server-side state may not fit in this limit (session caches, variables, frames, buffers, temp-tables nevertheless).

Even if we optimize the client off-heap memory from 40MB to 30MB (which is good ofc), this will be relatively irrelevant in relation with the server-side state. So we should use this task to reach an acceptable size for the client, so we can change focus to server asap. Doing micro-optimzations will have no benefit considering the server will require hundreds of gb ram to handle these sessions. Also, mind that the OG task (#11215) was running clients with 1gb only heap. having it run in 128mb heap + off-heap is already a huge improvement.

So, having clients run in 64MB heap + 32MB off-heap for example would be enough for this first iteration. The problem will aggressively shift to the server.

I have done some measurements and memory consumption for the first client is around 300 MB (server + client). For the next clients, memory consumption per client is 250 MB, with 128 MB heap size.

I am considering to analyze the server-size memory consumption because I have reached to a point where the idle memory usage on client is near optimal.

is comment #5742-12 relevant here (is that improvement implemented)?

Hi Dean!

Note #5742-12 shows the initial results for font-metrics caching. They are implemented and I can confirm that this is still working on trunk and 11326a.
I can't exactly tell right now every detail because I have focused mostly on the client heap memory usage.

Current optimizations from 11326a are related with what we do with font definition after we send it to the browser through websocket, small optimization when fetching custom fonts. These lowers down the client base heap memory usage from 90 MB to about 60 MB. There is no server side change which increases the memory usage and the performance should be about the same.
There are also some other changes which it will be coming soon on this branch, related with some unexpected Java AWT quirks (eg. linux implementation of new Font("System") and GraphicsEnvironment.getAvailableFontFamilyNames() which loads every installed font from your system and caches them in memory).

Fonts now take about 1 MB in the client heap.

Hi Teodor,
wow - great work. really looking forward to getting this all out.

Eugenie Lyzenko wrote:

Teodor Gorghe wrote:

Eugenie, I have looked into 10183b changes, which avoid forking the JVM instance. I don't see anything that can cause spawn issue.
I think the issue comes from a GLIBC version mismatch or Java version. The issue was a long time ago and I don't have the problematic binary which I can check it with ldd.

The changes from 11326a does not depend from 10183b, but it is good to know that there is 50 MB memory per client which is kept by the spawn process.

As far as I know 10183b change did undo for spawn improvements. The 10286a has modified code.

If we want to save ~50MB of RAM per every client session we need it. And it is safe, I used it locally for approx half of the year and have no problems with.

I have attached the improved spawn.c. Do we need this change?

The only condition to use this fix: the FWD build should be compiled exactly on the same system that is planned as target for deploy. Yes, it is GLIBC/NCURSES compatibility requirement.

I have analyzed server-side memory consumption. I have optimized a little bit the FontTable font metrics memory consumption (which can free up to 300-400 MB).

Other than that, there is no anything that can be optimized on server-side.
About Session cache, I have thought for it to be some sort of SoftReference, which will allow GC to reclaim the memory on demand. This idea cannot be applied for every cached DMO because we also store the changes there.

I have also tested JDK 25 AOT, which showed a huge metaspace memory consumption difference (AppCDS shared class definition, but AOT also shared the native code which normally gets compiled on runtime by JIT). Client metaspace is 3M (from 12M).

-XX:+UseCompactObjectHeaders has optimized server heap memory by 28%. However, there is no memory improvement on client side.

Teodor Gorghe wrote:

Committed revision 16513 on task branch 11326a:

• Optimized client side and server side memory.

Constantin, can you take look into the changes?
I think that I have reached to a stage where the memory consumption is minimal.

Please rebase first.

Rebased 11326a to trunk revision 16525.

Teodor, after adding javadoc to the new fields in VirtualScreenImpl, please put it into testing. I don't see any issues.

• appcds I assume collects only class files loaded by the JVM (from jars, not dynamic)
• is the .jsa file compatible between JVM vendors and versions?
• if is compatible, can we create a 'best of' .jsa file during FWD build, and included it automatically in the FWD Client's JVM args?

It collects both classes loaded statically and these loaded dynamically. Class types built dynamically during runtime are not being loaded. I have didn't saw such stuff happen on FWD client.

jsa archives are not compatible between vendors and versions. The lst file is just a plain text file, and contains the classes that needs to be included in the jsa archive. This file also contains internal JVM classes.

Please note that jsa archives depends heavily by the JVM args which you provide. The classpath, GC, etc. needs to be the same.

Teodor Gorghe wrote:

Please note that jsa archives depends heavily by the JVM args which you provide. The classpath, GC, etc. needs to be the same.

This part worries me - what happens if these configs are changed? Does it just ignore the .jsa archive? Something else?

Constantin Asofiei wrote:

Teodor Gorghe wrote:

Please note that jsa archives depends heavily by the JVM args which you provide. The classpath, GC, etc. needs to be the same.

This part worries me - what happens if these configs are changed? Does it just ignore the .jsa archive? Something else?

jsa archive load fails and the application fails to start.

Hi Greg,
Could we discuss this in this afternoon's meeting?

I think the discussion on sso and lyra will be quick? i'd like to understand the above both in terms of timing and impact.

Also, could we pull it forward 30 mins?

OK

• if 11326a passed testing, this can be merged to trunk (I assume we want this in 9866c, as these are FWD changes)
• provide details of what is needed (in docker script changes terms, directory.xml, etc) so that the lib.client folder (with FWD-client only jar dependencies) is used at runtime when spawning clients, and any other application script changes
• continue working on #11327

• please post an example of a .lst file
• a tool which runs through p2j.jar (selected packages like p2j/ui, p2j/util), gets the dependencies on some depth level (5-10-20?) from other jars (plus JDK) and creates the .lst file. This can be done via bytecode analysis tools, the Constant Pool has all the info we need to determine static dependencies.
• using this .lst file and using the docker configured JVM arguments for the FWD client, we will create the jsa archive
• will automatically inject in directory.xml the settings for appcds, and to any other client launch script, as needed, when enabled at the image build
• also, this must run with Java 17

There is no need to 'warm up' the FWD Client JVM by executing customer scenarios, as this is not using converted code, and even if there are customer app UI widgets, those are small and can stay in the heap (or it can be included as 'additional entry point jar' when building the .lst file to 'share them').

Teodor Gorghe wrote:

Example of .lst file.

Thanks, so this can be generated programmatically and not via appcds.

Teodor Gorghe wrote:

This is how a generated .lst file looks like

That is standard JVM method signatures. How having the methods help?

These JVM signatures are extracted from dependencies and from p2j bytecode analysis.

We are planing to use static AppCDS, where the archives should also contain stuff from JVM.
There is Dynamic AppCDS, where the archives does not have JVM dependencies, but we can't build these archives from lst.

• the are some dependencies which leads to loading the persist package (eg. BrowseGuiImpl -> BrowseWidget (referenced from a static constant) -> import static com.goldencode.p2j.persist.P2JQuery.* -> ...), or these which are referenced by ErrorManager). I have ended up adding two types of parameters for that bytecode generator: stop class/package (it stops searching when it encounters a class matching this filter) and excluding filter (if it reaches there, don't add the class into the listing file and also scan it).
• I got a shared/total loaded classes ratio of 38%, which is a little bit better than the one with ReflectASM. The ratio is so low because there are lots of lambda functions which are being initialized from PayloadSerializer (about 40% of non-shared classes), some lambdas from internal JDK implementation of Spring (20%), but there are also some libraries from internal JDK, like xerces which is impossible to find it the referenced from bytecode analysis.

• we generate a base JVM listing file by just running FWD client without any network parameter:
  

  java -XX:DumpLoadedClassList=jdk_base_classes.lst -cp ../lib.client com.goldencode.p2j.main.ClientDriver
  

• generate lst file from static analysis
• merge these two listing file

I have also done some PayloadSerializer changes, which are very important.

With these two combined, I have managed to get the same metaspace consumption as with the jsa from a normal AppCDS pipeline.

Teodor, please mind the effort in #9939 to get a fwd-client.jar archive that contains only the client required classes. Thus, any dependency from client to server will be fixed.

Teodor Gorghe wrote:

Where should I put this class?

I should put in FWD, or I should create a separate project?

Please add std headers and javadoc. It should not have the AGPL license part. So the header would be:

/*
** Module   : MethodCallAnalyzer.java
** Abstract : <something_here>
**
** Copyright (c) 2026, Golden Code Development Corporation.
** ALL RIGHTS RESERVED. Use is subject to license terms.
**
**           Golden Code Development Corporation
**                      CONFIDENTIAL
**
** -#- -I- --Date-- -----------------------------Description------------------------------
** 001 TG  ...
*/

Then you can add it to the project I just created at ~/secure/code/p2j_repo/tools/dev_helpers.

There are a range of different ideas discussed in this task, which is pretty confusing to understand status.

Teodor, would you please post the full list of open items to work, organized in 3 categories:

1. Low Risk AND Quick to Implement
2. Medium Risk OR Low Risk But Takes a Bit More Work
3. High Risk OR Bigger Work Needed

This will help us organize the plan for getting these merged.

• Changes from below are already in 11326a and were reviewed by Constantin.
• removal of ReflectASM, Jetty SpreadsheetWebApp configuration to avoid searching through entire classpath
• refactored getCustomFontData to avoid loading the entire font table on client-side
• PRINT_FONT fix to avoid loading a font with invalid name, which causes loading 2029 fonts
• LegacyFontMetrics rework to reduce server-side memory consumption
• FontHelper.hasLocalFont font caching in a temporary file to avoid calling getAvailableFontFamilyNames(), which loads all your fonts from the system
• GuiWebDriver optimization to reclaim unused memory
• VirtualScreenImpl refactorization to improve memory usage
• WebHtmlBrowser changes which avoids loading the entire tika dependency in memory

• Client AppCDS integration. I have already done the .lst building, which has proved good results. What is left to work: integrate MethodCallAnalyzer and lst building into the FWD docker image building process.
• Configuring the lib.client on the projects which needs memory optimization for FWD client (this is a medium-risk change because I don't know which dependency might be missing, I have already included these which I have made FWD client to start up).
• PayloadSerializer refactorization to reduce the number of lambda functions (which fills the metaspace memory). I have already done this, but I need to double check if I have missed something before commiting to a branch.

• p2j.jar modularization (Alexandru is already working on this), does not provide a substantial memory reduction because of #11326-62.

Committed revision 1 on ~/secure/code/p2j_repo/tools/dev_helpers:
- Added bytecode project.

• run this tool to generate the .lst file
• run the program to generate the archive
• automatically configure the directory.xml clientConfig/jvmArgs to have the arguments to use the appcds archive

Do we want this to be optional (via a flag at the docker image build script)?

Teodor: the tool needs to also specify the .lst file name and path where it will be saved.

• PayloadSerializer changes to optimize metaspace memory usage.

Teodor Gorghe wrote:

Committed revision 16531 on task branch 11326a:

• PayloadSerializer changes to optimize metaspace memory usage.

• please fix the formatting (curly braces on their own line, implements/extends on their own line, etc). PAYLOAD_CORE is OK, I mean in the other types.
• cache the cls.getComponentType() and cls.getDeclaredConstructor() types

Teodor Gorghe wrote:

I don't see a reason why I should cache cls.getComponentType().

[...]

That's a method call, a native call and a field access. We just remove these trips, maybe JVM will JIT-compile this code and be fast, but replacing it with a field avoids these method and native calls.

OK, let's focus on getting the items in category 1 (+ the .lst tool) into 11326a.

Do we want this to be optional (via a flag at the docker image build script)?

Probably.

Constantin Asofiei wrote:

Teodor Gorghe wrote:

I don't see a reason why I should cache cls.getComponentType().

[...]

That's a method call, a native call and a field access. We just remove these trips, maybe JVM will JIT-compile this code and be fast, but replacing it with a field avoids these method and native calls.

Ok, the changes are in 11326/r16532.

Committed revision 16533 on task branch 11326a:
- Added LstExporter

Uncommitted revision 1 on ~/secure/code/p2j_repo/tools/dev_helpers because I have moved it to 11326a/r16533.

• please fix the 'module' header (now has com.goldencode.bytecode.LstExporter.java)
• there are some lines longer than 110 limit, please fix these

11326a is OK to be in test.

Thanks.
I have committed the changes into 11326a/r16534.

Smoke-testing a customer app with 11326a passed.

ChUI regression testing has passed.
I will continue testing ETF and the remaining customer apps.

Regression testing has passed (ETF and two other customer applications).

There was just one regression related with PayloadSerializer, which was fixed in 11326a/16535

Teodor Gorghe wrote:

There was just one regression related with PayloadSerializer, which was fixed in 11326a/16535

I don't understand what the problem was. Can you detail a little?

The problem came from spawn JVM, which tried to load EventDefinition (throws ClassNotFoundException)
EventDefinition has RoaringBitmap dependency, which is not available from spawn p2j.jar.
In the previous optimization, I have added cls.getDeclaredConstructor() in Cls constructor, which JVM load the class in memory before providing the Constructor object.
new Cls(EventDefinition.class) is called on static initialization of PAYLOAD_CLASSES, which triggered a crash in Message.writeExternal.

I have fixed this issue by moving cls.getDeclaredConstructor() call into Cls.get(), which lazily loads EventDefinition only when we need that.

Teodor Gorghe wrote:

The problem came from spawn JVM, which tried to load EventDefinition (throws ClassNotFoundException)
EventDefinition has RoaringBitmap dependency, which is not available from spawn p2j.jar.
In the previous optimization, I have added cls.getDeclaredConstructor() in Cls constructor, which JVM load the class in memory before providing the Constructor object.
new Cls(EventDefinition.class) is called on static initialization of PAYLOAD_CLASSES, which triggered a crash in Message.writeExternal.

I have fixed this issue by moving cls.getDeclaredConstructor() call into Cls.get(), which lazily loads EventDefinition only when we need that.

Please add a comment explaining this, where cls.getDeclaredConstructor() gets called . So it doesn't get changed in the future.

I have added the comment.

Teodor Gorghe wrote:

I have added the comment.

Thank you. Any other testing required?

I don't think (I have also tested with the lib.client).

Teodor Gorghe wrote:

Branch 11326a was merged into trunk as rev. 16557 and archived.

Moved this task to WIP because we need to migrate customer application setup to use lib.client.

Do you mean the current trunk 16557 should not work with big M application? Because it really does not work.

Teodor, Eugenie is right; there was a 'typo':

=== modified file 'src/com/goldencode/p2j/main/BaseClientBuilderOptions.java'
--- old/src/com/goldencode/p2j/main/BaseClientBuilderOptions.java       2026-04-20 06:04:16 +0000
+++ new/src/com/goldencode/p2j/main/BaseClientBuilderOptions.java       2026-05-07 17:27:58 +0000
@@ -541,7 +541,7 @@
          sb.append(cp);
          for (int i = 0; i < jars.length; i++)
          {
-            String extraJar = libPath + File.pathSeparator + jars[i] + FileExtensions.JAR_POSTFIX;
+            String extraJar = libPath + File.separator + jars[i] + FileExtensions.JAR_POSTFIX;
             File jarFile = new File(extraJar);
             if (jarFile.exists())
             {

Teodor Gorghe wrote:

Moved this task to WIP because we need to migrate customer application setup to use lib.client.

There are also the docker changes to integrate appcds for FWD clients, using Hotel GUI as a demo.

Eugenie, please do some smoke-tests with 11326b rev 16559 (has the fix from #11326-123), if is OK I'll merge it ASAP.

Constantin Asofiei wrote:

Eugenie, please do some smoke-tests with 11326b rev 16559 (has the fix from #11326-123), if is OK I'll merge it ASAP.

OK. I'll checking out this branch now. But this can take a time. The alternative can be if you attach the modified file here and I will test it over recent trunk. This can save a bit of time.

The patch is in note #11326-122 that's the only change separator instead of pathSeparator

Constantin Asofiei wrote:

The patch is in note #11326-122 that's the only change separator instead of pathSeparator

Your fix works fine. All is back again. Please merge into trunk when possible.

Branch 11326b was merged to trunk rev 16559 and archived.

• added deploy.appcds ant task, which creates the .lst file using LstExporter. It also creates a .jsa archive, which is used for local setups. These files are stored in deploy/client/appcds, which are being included in deploy archive, which can be used for building the docker image later on.
• changed the hotel_gui dockerfile to build appcds file .jsa.
• changed directory.xml.template to use AppCDS archive by default.

I am continuing implementing the lib.client after doing some investigation on #11452.

Also, the runtime JVM args are hardcoded in build.xml. There are important because classpath is important, JSA archive also depends by the type of use garbage collector, etc.
One idea which I have is getting this information from directory.xml.

I have focused building one client JSA archive, using JVM parameters which are compatible between each other, but if multiple JSA files are required, build.properties and the dockerfile needs to be changed.

Teodor Gorghe wrote:

One idea which I have is getting this information from directory.xml.

Yes, we need this. But we need to consider classpath-extra-jars and exactly how FWD builds the classpath and passes it to the spawner.

Also, the directory.xml needs to be prepared to use the appcds config via jvmArgs.

I have focused building one client JSA archive, using JVM parameters which are compatible between each other, but if multiple JSA files are required, build.properties and the dockerfile needs to be changed.

This is a good point: we can have multiple types of clients spawned via the directory (appserver, GUI, ChUI, batch processes), which have their own classpath or JVM settings. Can we build a tool which gets the directory.xml and does everything for all clients (i.e. walks through all clientConfig nodes and resolves the JVM args and classpath, and automatically adds the appcds config)?

Constantin Asofiei wrote:

Teodor Gorghe wrote:

One idea which I have is getting this information from directory.xml.

Yes, we need this. But we need to consider classpath-extra-jars and exactly how FWD builds the classpath and passes it to the spawner.

Also, the directory.xml needs to be prepared to use the appcds config via jvmArgs.

I have focused building one client JSA archive, using JVM parameters which are compatible between each other, but if multiple JSA files are required, build.properties and the dockerfile needs to be changed.

This is a good point: we can have multiple types of clients spawned via the directory (appserver, GUI, ChUI, batch processes), which have their own classpath or JVM settings. Can we build a tool which gets the directory.xml and does everything for all clients (i.e. walks through all clientConfig nodes and resolves the JVM args and classpath, and automatically adds the appcds config)?

Yes, I can build a simple tool gets the configuration for each type of client.

The directory.xml is already using the JVM args since I have changed directory.xml.template to use AppCDS (./docker/docker_prepare.sh is only required to update to the new configuration)

deleted

• Created a tool in FWD, called JvmArgsExtractor.java, which extracts the client parameters from directory.xml.
• Modified the changes from #11326-130 to integrate this tool.
• I have changed deploy.appcds ant task and moved most of it into deploy/server/deploy_appcds.sh for flexibility for each project. I have also created a version for powershell, which is in deploy/server/deploy_appcds.ps1. Ant task is still there, which uses these two scripts.
• Added support for classpath_extra_jars and support for libPath on prepare_dir.sh.

The local setup didn't changed that much, the new ant task is a dependency of deploy-prepare task.

For docker setup, I have it working, but there is a problem which I will fix it. /opt/hotel/client/appcds is not persisted like how /opt/hotel/etc and /opt/hotel/logs are. These appcds files does not belong to the docker image because we need the directory.xml, which contains the JVM args.
I think I will move this into etc or create a new volume mapping in host deploy, called appcds, which maps to /opt/hotel/appcds.

Constantin, I have done the lib.client change in deploy.prepare task for hotel_gui (which takes from p2j/dist/client, p2j.jar, p2j.dll and libp2j.so).

The tricky part comes for docker, which requires changes for FWD docker image. I am planning to take the dist archive for client and extract into /opt/fwd-deploy/client/lib.
Right now, for everything, we use /opt/fwd-deploy/convert/build/lib for both server/client/convert.

Teodor, 11326c looks OK.

Thanks.
I have found some issues while parsing directory.xml from #11472, and I would need to rethink everything about JvmArgsExtractor.java.
In directory.xml, you can provide a more complex structure like server/<serverID>/clientConfig/jvmArgs, server/<serverID>/<client_type>/.../jvmArgs, server/default/clientConfig/jvmArgs, /server/default/runtime/default/.../jvmArgs, etc.

Currently, it uses XPath for getting the JVM args.
I was thinking to replace the current approach with one which uses the already existing code (eg. load directory.xml using DirectoryService and do the scan using it), which already handles default values. I need to take in consideration about multiple server ids and the precedence rules from Utils.getDirectoryNodeWorker. Maybe I can extend the XPaths.

Teodor Gorghe wrote:

I was thinking to replace the current approach with one which uses the already existing code (eg. load directory.xml using DirectoryService and do the scan using it), which already handles default values. I need to take in consideration about multiple server ids and the precedence rules from Utils.getDirectoryNodeWorker. Maybe I can extend the XPaths.

This would be better, but I'm not sure how you create the context for a specific user. Because you can have /server/default/runtime/<user>/clientConfig .

• Improved implementation for support of client types.

=== modified file 'docker/docker-compose_create_pg_cluster.yml'
--- old/docker/docker-compose_create_pg_cluster.yml    2026-01-21 23:26:06 +0000
+++ new/docker/docker-compose_create_pg_cluster.yml    2026-05-15 05:24:16 +0000
@@ -1,19 +1,19 @@
 services:

   hotel_db:
-    image: hotel_gui_postgres:latest
+    image: hotel_gui_pg14_server:451_11326c-16563_20260514a
     container_name: hotel_db
     command: >
       bash -c " 
         cd /tmp &&
-        sudo fwd_create_pg_cluster.sh --pgdata /pgdata/fwdcluster --rolefile-path /docker-entrypoint-initdb.d
+        sudo fwd_create_pg_cluster.sh --pgdata ${PGDATA:-/opt/hotel/db/fwdcluster}
       " 
     ports:
       - ${PGPORT:-5432}:${PGPORT:-5432}
     volumes:
-      - hotel_db_postgres:/pgdata:delegated
+      - hotel_db_postgres:/opt/hotel/db:delegated
     environment: 
-      PGDATA: /pgdata/fwdcluster
+      PGDATA: /opt/hotel/db/fwdcluster
       PGPORT: ${PGPORT:-5432}

• The latest version of image should be used, and the name of the image comes from docker/build_docker.sh. Did that change?
• The mount points are generic inside the container to the volume, so they don't need to include the application info.
• Please make sure to remove --rolefile-path /docker-entrypoint-initdb.d from the command-line. It's not in your addition.

docker/docker-compose_create_pg_cluster.yml changes should be reverted. This was an attempt to create a pg cluster, but I have figured out since then how to use.

Thanks a lot Roger!
After #11326c gets to trunk, I will commit these changes to hotel_gui, after the testing is done.

For directory.xml template files, I will change -Xshare:on to -Xshare:auto to make the clients to start even if there is no valid archive in that directory.
This eliminates a variable when debugging the application.

I think the prepare scripts for Windows they also needs to be changed (I have forgot to include in the patch).

Testing was done on 4 projects, without any issue.
I am merging this now.

Branch 11326c was merged into trunk as rev. 16575 and archived.

Teodor Gorghe wrote:

Committed the changes into hotel_gui revision 454.

If you want the base 24.04 Docker images to contain this version, you must have the version on xfer updated as well, since that is where it is pulled from. So don't forget to make the request, unless you can do it yourself.

rfb@xfersrv01:~$ update_app.sh
+N  deploy/server/deploy_appcds.ps1                                                                                                                                                                             
+N  deploy/server/deploy_appcds.sh
 M* build.xml
 M* deploy/server/directory.xml.template
 M* deploy/server/prepare_dir.ps1
 M  deploy/server/prepare_dir.sh
 M* docker/docker-compose.yml
 M* docker/docker-compose_client.yml
 M  docker/docker-compose_prepare.yml
 M* docker/docker-compose_server.yml
 M  docker/docker_prepare.sh
 M  docker/repo/entrypoint/docker-entrypoint_prepare.sh
 M* docker/run_docker.ps1
 M  docker/run_docker.sh
 M  json_template.sh
All changes applied successfully.                                                                                                                                                                               
Updated to revision 454 of branch sftp://rfb@devsrv01/opt/secure/code/p2j_repo/samples/hotel_gui

Show update_app.shShow update_app.sh

b="hotel_gui" 
[ ! -z "$1" ] && b="$1" 
p=sftp://rfb@devsrv01/opt/secure/code/p2j_repo/samples/$b/
path="/opt/fwd/$b" 
[ "$b" == "hotel_chui" ] && p=sftp://rfb@devsrv01/opt/secure/code/p2j_repo/samples/hotel/
[ "$b" == "hotel" ] && path="/opt/fwd/hotel_chui" 
if [ -d "$path" ]; then
   cd $path
   bzr bind $p
   bzr update
   bzr revert > /dev/null 2>&1
   bzr unbind
   sudo chown root:users -R .
   sudo find . -type d -exec chmod g+s '{}' \; && sudo chmod 0775 . -R
   cd
else
   echo "$path" not found.
fi

Ok, thanks for letting me know!

Teodor Gorghe wrote:

Committed the changes into hotel_gui revision 454.

I made matching changes in hotel revision 204. Can you review them?

I know that all the thinning down that Teodor has done for this task is good... I could use some thinning myself! But I'm curious about whether or not the same attention is being paid to the "server" set of jars (files built into the ./dist/server directory) as these (the ./dist/client directory). The docker images that are built for applications leverage the archives, and deploy them according to the Docker image being built. For now, all the images utilized the ./dist/client/convert set of jars, because there were issue running applications with the others. I believe with this task, the ./dist/client set of files is more reliable, and should be used to build runtime Docker images for applications, since we want them to be "thin".

So should the ./dist/client jars be used for server and client images? (Server being the image that starts the server process and has access to the directory.xml) or should the server use the ./dist/server jars?

It is hard to provide an answer. What I know is that the server image spawns FWD client instances (so it should have the client dist files), it should have the server dist files (obvious), but the FWD server also does a conversion during runtime (eg. dynamic queries, dynamic tables, etc), so it should contain the convert dist set. I don't know what is exactly the difference between convert and server, which I think I shall take a look into build.gradle.

I will review the hotel changes tomorrow (it's almost 10 PM on my timezone).

• build.xml:deploy.prepare task does not contain the part which copies the client jars into lib.client.

There is something which I forgot to include in hotel_gui project, which is the change of including client_lib_path into JSON template files.

Should I do this change as well?

I have tested the SIGNATURE widget, which now it works.
Since there is just one dependency change, introducing bcel on FWD client, which already was on FWD server, I don't think there is anything which needs to be tested since this dependency introduces apache commons lang3, which is already included in the project.

Teodor Gorghe wrote:

I have tested the SIGNATURE widget, which now it works.
Since there is just one dependency change, introducing bcel on FWD client, which already was on FWD server, I don't think there is anything which needs to be tested since this dependency introduces apache commons lang3, which is already included in the project.

Sooo, is there anything else left to test? May I place this in the merge queue?

I don't see anything else what I can test. This needs full regression testing?

Teodor Gorghe wrote:

I don't see anything else what I can test. This needs full regression testing?

Constantin?

Teodor, have you looked at the build of the FWD Docker containers which are used as the basis for the application Docker images? I had been utilizing the container project, but that is deprecated for the tools/docker/docker_build.sh in FWD (most recent in 9709e branch). The basic premise is that the appropriate versions of dist ZIP files are used to position FWD jars in the /opt/fwd-deploy directories:

# Step 6/11
RUN (pushd /tmp/repo; ./deploy_fwd.sh /opt/fwd-deploy "$DEPLOY_LIST" -f; popd)

An example of the dist directory after including archive in the gradlew command:

-rw-rw---- 1 rfb rfb  23085742 Jun  9 17:42 fwd_deploy-admin_4.0.0_p2j_9709e_16636_20260609.zip
-rw-rw---- 1 rfb rfb  93619568 Jun  9 17:42 fwd_deploy-client_4.0.0_p2j_9709e_16636_20260609.zip
-rw-rw---- 1 rfb rfb 331107810 Jun  9 17:42 fwd_deploy-convert_4.0.0_p2j_9709e_16636_20260609.zip
-rw-rw---- 1 rfb rfb  37625273 Jun  9 17:44 fwd_deploy-docs_4.0.0_p2j_9709e_16636_20260609.zip
-rw-rw---- 1 rfb rfb    171682 Jun  9 17:44 fwd_deploy-native_4.0.0_p2j_9709e_16636_20260609.zip
-rw-rw---- 1 rfb rfb 311860088 Jun  9 17:44 fwd_deploy-server_4.0.0_p2j_9709e_16636_20260609.zip
-rw-rw---- 1 rfb rfb  25237910 Jun  9 17:44 fwd_deploy-spawner_4.0.0_p2j_9709e_16636_20260609.zip

So if you were to use the "convert spawner" options to deploy_fwd.sh you would end up with the below in your docker image:

rfb@fwddev_40_ubuntu_2404_jdk17_9709e_latest_container:~/projects/fwd/9709e$ ls -l /opt
total 12
lrwxrwxrwx 1 root root   23 May 14 10:52 fwd -> /opt/fwd-deploy/convert
drwxr-xr-x 1 rfb  rfb  4096 May 14 10:51 fwd-deploy
drwxr-xr-x 2 root root 4096 May 14 10:52 spawner
rfb@fwddev_40_ubuntu_2404_jdk17_9709e_latest_container:~/projects/fwd/9709e$ ls -l /opt/fwd-deploy/
total 16
drwxr-xr-x 1 rfb rfb 4096 May 14 10:51 convert
drwxr-xr-x 1 rfb rfb 4096 May 14 10:51 spawner
rfb@fwddev_40_ubuntu_2404_jdk17_9709e_latest_container:~/projects/fwd/9709e$ ls -l /opt/fwd-deploy/convert/
total 68
drwxr-xr-x 1 rfb rfb  4096 Feb  1  1980 build
-rwxr-xr-x 1 rfb rfb 21881 May 14 10:51 import.sh
drwxr-xr-x 1 rfb rfb  4096 Feb  1  1980 locale
drwxr-xr-x 1 rfb rfb  4096 Feb  1  1980 rules
drwxr-xr-x 1 rfb rfb  4096 Feb  1  1980 src
drwxr-xr-x 1 rfb rfb  4096 Feb  1  1980 tools
-rw-r--r-- 1 rfb rfb   200 May 14 10:51 version.properties

The FWD_LIB environment variable defaults to /opt/fwd in the Docker images, which is a symlink to /opt/fwd-deploy/convert. This makes FWD available for Docker images that come FROM the FWD Docker images. For example, the Hotel GUI Docker files in ./docker:

-rw-rw---- 1 rfb rfb  2945 Apr  8 18:34 fwd_4.0_ubuntu_client_Dockerfile
-rw-rw---- 1 rfb rfb  4081 Apr  8 18:38 fwd_4.0_ubuntu_server_Dockerfile
-rw-rw---- 1 rfb rfb  4086 Apr  8 18:36 fwd_4.0_ubuntu_pg_server_Dockerfile

Is this task to match up with building the proper FWD docker image that would contain the client archive so memory of FWD in that image would be lessened, or is this task making the set of jars dynamic so that use of a FWD client would build the list of jars as needed? I'm wondering if this Dockerfile setup is where building the image so you can symlink to /opt/fwd-deploy/client for the location of the jars is "moot" now.

Roger Borrello wrote:

Is this task to match up with building the proper FWD docker image that would contain the client archive so memory of FWD in that image would be lessened, or is this task making the set of jars dynamic so that use of a FWD client would build the list of jars as needed? I'm wondering if this Dockerfile setup is where building the image so you can symlink to /opt/fwd-deploy/client for the location of the jars is "moot" now.

I have looked at both container FWD docker images (22.04 and 24.04) and the docker file from FWD tools/docker, which is based on Ubuntu 24.04.

AFAIK, container docker file uses ALL for ./deploy_fwd.sh and tools/docker just client convert spawner.

By adding client to the the DEPLOY_LIST will increase the FWD docker image with about 500 MB.

Right now, we use just 1 base FWD docker image for client, server images. For server, right now, we need both client and convert images, due to the nature how FWD works, server needs client images because it spawns clients in the same container.

For client, we can use just have a FWD client base image, to reduce the total size of container/client application image.

The main purpose for this task is to reduce RAM memory usage of FWD clients. It is important because high RAM memory consumption is a huge bottleneck for the converted applications because they can't run more than just a few clients with a medium sized machine. Even if there are just 20-30 MB RAM which are being freed up per client when using AppCDS with client jars, this is worth if you run for example, 40+ clients on a 32GB machine.

Currently there is no distinguishing between Docker images that are including various levels of PostGreSQL client, so I've had to add that to the image name. So fwd_4.0_ubuntu_24.04_jdk17 might have included PG14, PG15, PG16, or PG17. So I had to add that information to the name... but that name now collides with the fwd_4.0_ubuntu_24.04_pgNN_jdk17 name that was indicative of the server image. This means I have to update the naming convention to either include client in the client image, so fwd_4.0_ubuntu_24.04_client_pgNN_jdk17 would indicate a FWD client image with the PGNN client installed. The trickle-down effect is that any scripts would need updating to indicate the correct name, although there aren't that many that are specifically utilizing client images, so that is a minimal impact.

Teodor, #9709-971 contains some info related to the FWD docker images. The Hotel apps, as well as any customer apps, would probably be relying upon the new client images that are built by the tools/docker/docker_build.sh process. I will probably replace the container projects build_fwd_24.04.sh script with this one, since it is more complete.

All the project build scripts may need to be modified to use this smaller docker image.

Radu Apetrii wrote:

Teodor Gorghe wrote:

I don't see anything else what I can test. This needs full regression testing?

Constantin?

I would like to revive this note. Alex/Constantin: do you think there is any more testing required for 11326d?

Teodor, where do you envision deploy_appcds.sh to be invoked (besides build.xml)? Should this be done at server startup? Or is this purely a build-time thing? We can't utilize build.xml in circumstances, like Docker image build.

ant deploy.appcds for local setup or docker prepare in case of docker environments.

Should be invoked everytime when there is a jvm arg change or a change on a jar file.

There is no support for client.sh AppCDS yet. Top priority was web clients, appservers and batch-process.

Teodor Gorghe wrote:

There is no support for client.sh AppCDS yet. Top priority was web clients, appservers and batch-process.

client.sh is a major method for launching batch processes.

I have thought that they were being launcher by ClientSpawner, my bad.

Claude is trying to convince me that the LstExporter isn't necessary, and rather the manifest of the p2j.jar should be utilized. Here is a report.

AppCDS Client Memory & Startup Findings¶

Comparison of the FWD ChUI client launched with AppCDS versus without it, plus the rework that made AppCDS a net win for both the command-line client and the server-spawned webClient.

Measured with deploy/client/appcds_mem.sh (command-line client) and deploy/client/appcds_mem_pid.sh (snapshot an already-running webClient by PID). Both sample /proc/<pid>/smaps_rollup and confirm via /proc/<pid>/maps whether the .jsa is actually mapped.

Outcome¶

After the rework, the appcds configuration is faster to start and lighter, and its per-client advantage grows as clients are added:

The appcds peak RSS fell from 270 MB (bloated archive) to 178 MB once the class list was trimmed -- see "The journey" below.

Setup¶

• Launcher: client-terminal.sh (ChUI entry point)
• appcds: default client.sh -- classpath <libPath>/p2j.jar (whose manifest Class-Path pulls in the curated client/lib set) + -XX:SharedArchiveFile=.../default.jsa
• no-appcds: client.sh --no-appcds -- full FWD discovery classpath (~361 jars), no archive (a stable baseline; its config never changed across runs, so small run-to-run differences are measurement noise)
• Memory in MB; ToSettle = wall-time to the last RSS increase (startup/warmup proxy); ARCHIVE = whether the .jsa is mapped.

The journey: bloated -> lean¶

Before -- explicit classpath + static class list (bloated)¶

The archive was dumped against an explicit ~169-entry classpath and a class list that concatenated LstExporter static reachability with the runtime dump:

Resulting default.jsa: 144 MB. Measurements:

Root cause of the loss: a ChUI client actually loads ~2,000 classes, but the archive held ~20,000. A static CDS archive eagerly loads its archived app-loader classes during VM init, so appcds materialized ~18k extra classes' runtime metadata as committed PrivDirty (~+57 MB/client) that the lazily-loading no-appcds run never created. Cross-process sharing helped (SharedClean rose with client count) but could not offset the eager-load bloat.

After -- manifest classpath + runtime-only class list (lean)¶

The archive is now dumped against <libPath>/p2j.jar (letting the jar manifest expand the classpath) and the class list is the runtime dump only (~2,007 classes). Measurements:

Per-client scaling (the cross-process sharing payoff)¶

appcds's per-client footprint drops faster as clients are added (158->134, -25) than no-appcds (168->149, -19), because the lean archive's pages are shared across the JVMs (SharedClean up, PrivClean ~0). That is the benefit AppCDS is designed for, now compounding the single-client win instead of being buried under eager-load bloat.

The fix in deploy_appcds.sh¶

1. Dump against the manifest classpath. client_cp is the directory.xml value <libPath>/p2j.jar. p2j.jar's MANIFEST.MF Class-Path entry lists every sibling jar, so this single entry IS the full client classpath -- and it is the exact form the server hands the spawned webClient. Dumping against this form is what lets BOTH client types map one archive (see webClient note).
2. Runtime-only class list by default. The list is the -XX:DumpLoadedClassList output (~2,007 classes), not the static walk.
3. .lst SPI sanitize. LstExporter copied META-INF/services/* entries verbatim, including legal inline comments (e.g. ecj's javax.tools.JavaCompiler: org.eclipse.jdt...EclipseCompiler #Eclipse compiler). The -Xshare:dump class-list parser rejects a class name with a trailing comment, so a sed strips inline #... while leaving full-line comments intact.

Why INCLUDE_STATIC exists (and defaults to off)¶

There are two ways to build the class list, with a real trade-off:

• runtime_lst (-XX:DumpLoadedClassList) -- exactly the classes one real client startup loads (~2,007). Tight and low-memory, but only covers the code paths exercised during that capture. Application screens/features not hit during the dump are simply not in the archive (they still load fine at runtime, just without the CDS speedup).
• base_lst (LstExporter static reachability) -- every class transitively reachable from the client roots across the whole classpath (~18,039). Broad coverage that catches classes a single idle startup never touches, but it is wildly over-inclusive on a fat classpath.

The static set is what caused the 144 MB / +57 MB-per-client bloat above (eager loading of archived classes at VM init). So the choice was:

• delete LstExporter -> lose the broad-coverage option entirely, or
• leave it on -> ship the bloat by default, or
• make it opt-in -> keep both behaviors available.

INCLUDE_STATIC takes the third path. Default false gives the lean, runtime-only archive -- the configuration that produced the measured win above. Setting INCLUDE_STATIC=true re-adds the static set for cases where maximal startup coverage is worth the memory cost (e.g. you want features not exercised during the dump pre-archived too, ideally after driving real screens during the DumpLoadedClassList step).

Implementation note: under the manifest classpath (-cp p2j.jar), LstExporter cannot see the manifest-referenced jars (java.class.path lists only p2j.jar), so when enabled the INCLUDE_STATIC branch builds an explicit analysis classpath from the lib dir's jars -- separate from the (manifest) dump classpath.

INCLUDE_STATIC trade-off (measured)¶

The lean-default decision was confirmed with real numbers on both client types.

webClient (single client, snapshot by PID):

The static archive costs ~+109 MB on the webClient -- higher PrivClean (more archived pages read in) and PrivDirty (more eagerly-loaded class metadata).

Command-line client, 5 concurrent clients:

With the static archive, command-line appcds is ~+296 MB (~+59 MB/client) over no-appcds -- back to a memory loss. The only thing static improves over lean is startup (4.7s vs 3.9s, ~0.8s), and that is a poor exchange for ~+74 MB/client: lean already captures the bulk of the startup win (3.9s vs ~7s for no-appcds) while staying under no-appcds on memory.

Conclusion: lean (runtime-only) is the right default for both client types. Use INCLUDE_STATIC=true only when startup latency is paramount and RAM is abundant. (ToSettle varies with system load while many clients launch, so trust within-run deltas more than across-run ones; the memory columns are steadier.)

Why this also fixes the webClient¶

The webClient is spawned by the server; getcp.sh/.cpath never apply to it. The server builds its command from directory.xml clientConfig, and BaseClientBuilderOptions.addJarsToClasspath() defaults the classpath to <libPath>/p2j.jar. Confirmed from a live webClient command line:

java -Xshare:auto -XX:SharedArchiveFile=../appcds/client/default.jsa -Xmx128m \
     -Djava.library.path=/opt/fwd-deploy/client/lib \
     -classpath /opt/fwd-deploy/client/lib/p2j.jar \
     com.goldencode.p2j.main.ClientDriver ...

So the webClient runs -cp p2j.jar (manifest-expanded) -- never an explicit list. Previously the archive was dumped against the explicit ~169-entry classpath, so the webClient's classpath did not match and -Xshare:auto silently skipped the archive (verified: no .jsa in /proc/<pid>/maps). Dumping against <libPath>/p2j.jar now matches the webClient automatically -- no directory.xml classpath injection needed.

How to reproduce¶

1. Re-dump: cd deploy/server && ./deploy_appcds.sh /opt/fwd-deploy/client/lib/p2j.jar (set INCLUDE_STATIC=true first only if you want the broad static archive).
2. Command-line client: cd deploy/client && ./appcds_mem.sh -- -i 1 and ./appcds_mem.sh --clients 5 -- -i 1.
3. webClient: start server, open https://localhost:8443/chui, find the PID (pgrep -fa com.goldencode.p2j.main.ClientDriver), then ./appcds_mem_pid.sh --reset appcds <pid>; reconfigure/bounce for the baseline and ./appcds_mem_pid.sh no-appcds <newpid>.

Caveats¶

• Part of the absolute memory gap is the curated ~167-jar classpath vs the full ~361-jar no-appcds discovery, not purely the archive. The pure-CDS contribution is clearest in ToSettle and in the growing per-client SharedClean/Pss advantage.
• The command-line client now runs the same thin classpath as the webClient (p2j.jar + manifest, no hotel.jar/fwdspi.jar); it ran fine (STATUS=ok). If a deployment needs the app jars client-side, append them after .cpath in getcp.sh --appcds (CDS only requires the dump classpath to be a prefix).

I think that deploy_appcds.sh LstExporter class/package list can be fine-tuned even further to find a good balance between archive size and percentage of shared class.

Pure AppCDS generates about 4k classes and I think I can target to achieve this value.

I think there is something wrong on your setup. How the LstExporter has generated lst with 20k classes? When I have implemented this, I have tested it and it was around 8k classes (archive size was ~50-60 MB).

I checked in revision 206 of hotel so it is more reviewable. The JSON format contains the appcds values in their own section. Sample:

{
  "server_xml_file": "server.xml",
  "directory_xml_file": "directory.xml",
  "spawner_path": "/opt/spawner_chui/spawn",
  "client_start_dir": "/home/rfb/projects/hotel_dev/deploy/client",
  "appcds" : {
     "client_lib_path": "/opt/fwd-deploy/client/lib",
     "webclient_appcds_jvmargs": "-Xshare:auto -XX:SharedArchiveFile={archive}",
     "webclient_appcds_archive": "/home/rfb/projects/hotel_dev/deploy/appcds/client/default.jsa" 
  },
  "webclient_memory": "128m",
  "dateFormat": "mdy",
  "numberGroupSep": ",",
  "numberDecimalSep": ".",
  "p2j_entry": "login.p",
  "pkgroot": "com.goldencode.hotel",
  "propath": ".:",
  "search_path": ".",
  "path_separator": ":",
  "file_separator": "/",
  "case_sensitive": "TRUE",
  "os_user": "rfb",
  "server_log": "/home/rfb/projects/hotel_dev/deploy/logs",
  "client_log": "/home/rfb/projects/hotel_dev/deploy/logs",
  "embedded_host": "localhost",
  "admin_port": 8443,
  "dbnames": "hotel",
  "hotel": {
    "dbtype": "h2",
    "collation": "en_US@iso88591_fwd_basic",
    "dbhost": "localhost",
    "dbpath": "/home/rfb/projects/hotel_dev/deploy/db" 
  }
}

I left the memory value outside, because it is a good value to be able to configure with/without AppCDS involvement.

Roger, use this list for LstExporter and share the results:

com.goldencode.p2j.main.ClientDriver \
-com.goldencode.p2j.testengine \
-com.goldencode.p2j.uast \
-com.goldencode.p2j.soap \
-com.goldencode.p2j.security \
-com.goldencode.p2j.schema \
-com.goldencode.p2j.scheduler \
-com.goldencode.p2j.persist \
-com.goldencode.p2j.convert

Teodor Gorghe wrote:

Roger, use this list for LstExporter and share the results:

I will get to this as soon as I can.

Teodor Gorghe wrote:

Roger, use this list for LstExporter and share the results:

A run with 5 clients yielded:

Launcher: /home/rfb/projects/hotel_dev/deploy/client/client-terminal.sh
Clients per run: 5; settle: 15s; client args: (none)

MODE        #Up   PeakRSS       Rss       Pss  SharedCln  PrivClean  PrivDirty  ARCHIVE ToSettle  STATUS
                     (MB)      (MB)      (MB)       (MB)       (MB)       (MB)               (s)  
appcds        5    1411.7    1379.9    1189.8      226.8        0.1     1153.1  mapped       4.6  ok
no-appcds     5     917.1     915.2     799.0      133.9        0.0      781.3  absent       5.1  ok

Reading the table:
 - ARCHIVE=mapped means the .jsa is really mapped in; =absent means -Xshare:auto
   silently skipped it (classpath mismatch) -- then 'appcds' isn't using CDS at all.
 - ToSettle is wall-time to the last RSS increase (warmup proxy). CDS should make
   the 'appcds' value smaller -- that startup speedup is its main single-client win.
 - Pss splits shared pages across mappers. With --clients >1 and ARCHIVE=mapped,
   compare total Pss (and SharedClean): appcds should win as K grows, because the
   one archive is shared across all K JVMs. At K=1 appcds often ties or loses on RSS.
 - 'no-appcds' also loads the full library set, so part of any delta is footprint.

I reran after not creating the static lst:

Make sure that CDS is actually working. Check it using jcmd <PID> VM.metaspace, it seems to me that CDS is not working.

How much the archive size has been reduced? From my setup, it was reduced from 74M to 71M. The generated .lst from LstExporter should have around 4k classes.

Do you think it is reasonable to use just the listing files just from java -cp p2j.jar:... com.goldencode.p2j.main.ClientDriver?

What is RSS on a application with lots of GUI elements?

Teodor Gorghe wrote:

Make sure that CDS is actually working. Check it using jcmd <PID> VM.metaspace, it seems to me that CDS is not working.

I updated #11326-187 to include a table for a run of 5 client.sh against the non-static list (using meta data classpath).

Then I started a web-client (ChUI). Show jcmd outputShow jcmd output

jcmd 101936 VM.metaspace
101936:

Total Usage - 247 loaders, 7877 classes (2089 shared):
  Non-Class: 1012 chunks,     29.35 MB capacity,   29.28 MB (>99%) committed,    29.10 MB (>99%) used,   191.59 KB ( <1%) free,   840 bytes ( <1%) waste , deallocated: 68 blocks with 13.51 KB
      Class:  399 chunks,      4.22 MB capacity,    4.16 MB ( 99%) committed,     4.01 MB ( 95%) used,   148.26 KB (  3%) free,    48 bytes ( <1%) waste , deallocated: 140 blocks with 32.16 KB
       Both: 1411 chunks,     33.56 MB capacity,   33.44 MB (>99%) committed,    33.11 MB ( 99%) used,   339.85 KB ( <1%) free,   888 bytes ( <1%) waste , deallocated: 208 blocks with 45.67 KB

Virtual space:
  Non-class space:       64.00 MB reserved,      29.31 MB ( 46%) committed,  1 nodes.
      Class space:        1.00 GB reserved,       4.19 MB ( <1%) committed,  1 nodes.
             Both:        1.06 GB reserved,      33.50 MB (  3%) committed. 

Chunk freelists:
   Non-Class:

 16m: (none)
  8m: (none)
  4m: (none)
  2m:    2, capacity=4.00 MB, committed=0 bytes (  0%)
  1m: (none)
512k:    2, capacity=1.00 MB, committed=0 bytes (  0%)
256k: (none)
128k:    2, capacity=256.00 KB, committed=0 bytes (  0%)
 64k: (none)
 32k: (none)
 16k:    2, capacity=32.00 KB, committed=0 bytes (  0%)
  8k:    2, capacity=16.00 KB, committed=0 bytes (  0%)
  4k:    2, capacity=8.00 KB, committed=0 bytes (  0%)
  2k:    2, capacity=4.00 KB, committed=0 bytes (  0%)
  1k: (none)
Total word size: 5.31 MB, committed: 0 bytes (  0%)

       Class:

 16m: (none)
  8m:    1, capacity=8.00 MB, committed=0 bytes (  0%)
  4m: (none)
  2m:    1, capacity=2.00 MB, committed=0 bytes (  0%)
  1m:    1, capacity=1.00 MB, committed=0 bytes (  0%)
512k:    1, capacity=512.00 KB, committed=0 bytes (  0%)
256k:    1, capacity=256.00 KB, committed=0 bytes (  0%)
128k: (none)
 64k: (none)
 32k:    1, capacity=32.00 KB, committed=0 bytes (  0%)
 16k: (none)
  8k: (none)
  4k: (none)
  2k: (none)
  1k: (none)
Total word size: 11.78 MB, committed: 0 bytes (  0%)

        Both:

 16m: (none)
  8m:    1, capacity=8.00 MB, committed=0 bytes (  0%)
  4m: (none)
  2m:    3, capacity=6.00 MB, committed=0 bytes (  0%)
  1m:    1, capacity=1.00 MB, committed=0 bytes (  0%)
512k:    3, capacity=1.50 MB, committed=0 bytes (  0%)
256k:    1, capacity=256.00 KB, committed=0 bytes (  0%)
128k:    2, capacity=256.00 KB, committed=0 bytes (  0%)
 64k: (none)
 32k:    1, capacity=32.00 KB, committed=0 bytes (  0%)
 16k:    2, capacity=32.00 KB, committed=0 bytes (  0%)
  8k:    2, capacity=16.00 KB, committed=0 bytes (  0%)
  4k:    2, capacity=8.00 KB, committed=0 bytes (  0%)
  2k:    2, capacity=4.00 KB, committed=0 bytes (  0%)
  1k: (none)
Total word size: 17.09 MB, committed: 0 bytes (  0%)

Waste (unused committed space):(percentages refer to total committed size 33.50 MB):
        Waste in chunks in use:    888 bytes ( <1%)
        Free in chunks in use:    339.85 KB ( <1%)
                In free chunks:      0 bytes (  0%)
Deallocated from chunks in use:     45.67 KB ( <1%) (208 blocks)
                       -total-:    386.39 KB (  1%)

chunk header pool: 1425 items, 108.19 KB.

Internal statistics:

num_allocs_failed_limit: 3.
num_arena_births: 494.
num_arena_deaths: 0.
num_vsnodes_births: 2.
num_vsnodes_deaths: 0.
num_space_committed: 536.
num_space_uncommitted: 0.
num_chunks_returned_to_freelist: 3.
num_chunks_taken_from_freelist: 1412.
num_chunk_merges: 3.
num_chunk_splits: 993.
num_chunks_enlarged: 716.
num_inconsistent_stats: 0.

Settings:
MaxMetaspaceSize: unlimited
CompressedClassSpaceSize: 1.00 GB
Initial GC threshold: 21.00 MB
Current GC threshold: 35.12 MB
CDS: on
MetaspaceReclaimPolicy: balanced
 - commit_granule_bytes: 65536.
 - commit_granule_words: 8192.
 - virtual_space_node_default_size: 8388608.
 - enlarge_chunks_in_place: 1.
 - new_chunks_are_fully_committed: 0.
 - uncommit_free_chunks: 1.
 - use_allocation_guard: 0.
 - handle_deallocations: 1.

How much the archive size has been reduced? From my setup, it was reduced from 74M to 64M. The generated .lst from LstExporter should have around 4k classes.

Without using the LstExporter the directory is below. .jsa is 21M The default.lst contains 2006 lines (no comments).

-rw-rw---- 1 rfb rfb       35 Jun 18 08:32 default.cpath
-r--r----- 1 rfb rfb 21716992 Jun 18 08:32 default.jsa
-rw------- 1 rfb rfb    98555 Jun 18 08:32 default.lst

When I run LstExporter with your revised parameters, .jsa is 165M. The .lst has 21,946 lines.

-rw-rw---- 1 rfb rfb        35 Jun 18 08:48 default.cpath
-r--r----- 1 rfb rfb 165281792 Jun 18 08:48 default.jsa
-rw-rw---- 1 rfb rfb   1062583 Jun 18 08:47 default.lst

Is that what you were looking for? Have you tried this?

Teodor Gorghe wrote:

Do you think it is reasonable to use just the listing files just from java -cp p2j.jar:... com.goldencode.p2j.main.ClientDriver?

What is RSS on a application with lots of GUI elements?

I can either port this over to hotel_gui, or patch 9866c and run the customer app... I'll start the 9866c build, as I already have this ported to the customer app.

165M is huge, I have didn't saw an archive so big which was generated by deploy_appcds.sh
This is what I ran to manually test this (in lib.client):

java -cp p2j.jar com.goldencode.util.LstExporter com.goldencode.p2j.main.ClientDriver -com.goldencode.p2j.testengine -com.goldencode.p2j.uast -com.goldencode.p2j.soap -com.goldencode.p2j.security -com.goldencode.p2j.schema -com.goldencode.p2j.scheduler -com.goldencode.p2j.persist -com.goldencode.p2j.convert -o classes.lst

Loading JARs and SPIs from classpath...
Scanning...
Done! Total classes: 4474

java -cp p2j.jar -Xshare:dump -XX:SharedClassListFile=classes.lst -XX:SharedArchiveFile=archive.jsa

[0.394s][warning][cds] Preload Warning: Cannot find antlr/CommonAST
[0.395s][warning][cds] Preload Warning: Cannot find antlr/CommonHiddenStreamToken
[0.395s][warning][cds] Preload Warning: Cannot find antlr/Token
[0.396s][warning][cds] Preload Warning: Cannot find antlr/collections/AST
[0.397s][warning][cds] Preload Warning: Cannot find com/carrotsearch/hppc/IntArrayList
[0.506s][warning][cds] Preload Warning: Cannot find com/goldencode/ast/Aast
[0.508s][warning][cds] Preload Warning: Cannot find com/goldencode/ast/AnnotatedAst
[0.509s][warning][cds] Preload Warning: Verification failed for com.goldencode.ast.AnnotatedAst$1
[0.515s][warning][cds] Preload Warning: Cannot find com/goldencode/ast/ManagedHiddenStreamToken
[0.516s][warning][cds] Preload Warning: Verification failed for com.goldencode.ast.TreeLocal
[0.542s][warning][cds] Preload Warning: Cannot find com/goldencode/p2j/admin/client/application/home/accounts/CertificateKeys
[0.564s][warning][cds] Preload Warning: Cannot find com/goldencode/p2j/admin/server/AdminServiceImpl
[0.567s][warning][cds] Preload Warning: Cannot find com/goldencode/p2j/admin/server/AuthServiceImpl
[0.584s][warning][cds] Preload Warning: Verification failed for com.goldencode.p2j.admin.server.reports.AbstractReportBuilder
[0.592s][warning][cds] Preload Warning: Verification failed for com.goldencode.p2j.admin.server.reports.AccountsReportBuilder
[0.594s][warning][cds] Preload Warning: Verification failed for com.goldencode.p2j.admin.server.reports.AppserversReportsBuilder
[0.595s][warning][cds] Preload Warning: Verification failed for com.goldencode.p2j.admin.server.reports.DefaultReportBuilder
[0.595s][warning][cds] Preload Warning: Verification failed for com.goldencode.p2j.admin.server.reports.GroupsReportsBuilder
[0.596s][warning][cds] Preload Warning: Verification failed for com.goldencode.p2j.admin.server.reports.ProcessesReportsBuilder
[0.599s][warning][cds] Preload Warning: Verification failed for com.goldencode.p2j.admin.server.reports.ReportsManager$2
[0.600s][warning][cds] Preload Warning: Verification failed for com.goldencode.p2j.admin.server.reports.UsersReportsBuilder
[0.602s][warning][cds] Preload Warning: Cannot find com/goldencode/p2j/admin/shared/AdminService
[0.603s][warning][cds] Preload Warning: Cannot find com/goldencode/p2j/admin/shared/AuthService
[0.606s][warning][cds] Preload Warning: Cannot find com/goldencode/p2j/aspects/ltw/MethodTraceController
[0.606s][warning][cds] Preload Warning: Cannot find com/goldencode/p2j/aspects/ltw/MethodTraceController$Impl
[1.031s][warning][cds] Preload Warning: Verification failed for com.goldencode.p2j.reporting.BrowseJasperReportTemplate
[1.609s][warning][cds] Preload Warning: Verification failed for com.goldencode.p2j.util.CryptoUtils
[1.610s][warning][cds] Preload Warning: Verification failed for com.goldencode.p2j.util.CryptoUtils$1
[1.611s][warning][cds] Preload Warning: Verification failed for com.goldencode.p2j.util.CryptoUtils$CipherParams
[1.631s][warning][cds] Preload Warning: Verification failed for com.goldencode.p2j.util.GUIPrinterStreamSupport
[1.864s][warning][cds] Preload Warning: Cannot find com/goldencode/p2j/xml/SaxWriterImpl$1
[1.869s][warning][cds] Preload Warning: Cannot find com/goldencode/p2j/xml/XmlAst
[1.971s][warning][cds] Preload Warning: Cannot find com/google/gwt/regexp/shared/RegExp
[1.972s][warning][cds] Preload Warning: Cannot find com/google/gwt/user/client/rpc/RemoteService
[1.972s][warning][cds] Preload Warning: Cannot find com/google/gwt/user/server/rpc/RemoteServiceServlet
[1.973s][warning][cds] Preload Warning: Cannot find com/google/gwt/view/client/Range
[2.033s][warning][cds] Preload Warning: Cannot find javanet/staxutils/IndentingXMLStreamWriter
[2.086s][warning][cds] Preload Warning: Cannot find javax/xml/bind/DatatypeConverter
[2.126s][warning][cds] Preload Warning: Verification failed for net.sf.jasperreports.engine.export.JRXlsExporter
[2.319s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/cos/COSName
[2.319s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/io/MemoryUsageSetting
[2.320s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/multipdf/PDFMergerUtility
[2.320s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/PDDocument
[2.321s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/PDDocumentCatalog
[2.322s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/PDDocumentInformation
[2.322s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/PDPage
[2.323s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/PDPageContentStream
[2.323s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/PDPageContentStream$AppendMode
[2.324s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/PDPageTree
[2.324s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/PDResources
[2.325s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/common/PDDestinationOrAction
[2.325s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/common/PDRectangle
[2.326s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/common/PDStream
[2.329s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/font/PDFont
[2.329s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/font/PDTrueTypeFont
[2.330s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/font/PDType1Font
[2.331s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/font/encoding/Encoding
[2.331s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/font/encoding/WinAnsiEncoding
[2.332s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/graphics/PDXObject
[2.333s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/pdmodel/interactive/action/PDActionJavaScript
[2.333s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/rendering/ImageType
[2.334s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/rendering/PDFRenderer
[2.335s][warning][cds] Preload Warning: Cannot find org/apache/pdfbox/util/Matrix
[2.407s][warning][cds] Preload Warning: Cannot find org/bouncycastle/crypto/CipherParameters
[2.408s][warning][cds] Preload Warning: Cannot find org/bouncycastle/crypto/Digest
[2.408s][warning][cds] Preload Warning: Cannot find org/bouncycastle/crypto/MultiBlockCipher
[2.409s][warning][cds] Preload Warning: Cannot find org/bouncycastle/crypto/PBEParametersGenerator
[2.409s][warning][cds] Preload Warning: Cannot find org/bouncycastle/crypto/digests/MD5Digest
[2.410s][warning][cds] Preload Warning: Cannot find org/bouncycastle/crypto/digests/SHA1Digest
[2.411s][warning][cds] Preload Warning: Cannot find org/bouncycastle/crypto/digests/SHA256Digest
[2.411s][warning][cds] Preload Warning: Cannot find org/bouncycastle/crypto/digests/SHA512Digest
[2.412s][warning][cds] Preload Warning: Cannot find org/bouncycastle/crypto/engines/AESEngine
[2.413s][warning][cds] Preload Warning: Cannot find org/bouncycastle/crypto/generators/PKCS5S1ParametersGenerator
[2.413s][warning][cds] Preload Warning: Cannot find org/bouncycastle/crypto/params/KeyParameter
[2.973s][warning][cds] Skipping jdk/internal/event/Event: JFR event class
[2.973s][warning][cds] Skipping com/goldencode/ast/AnnotatedAst$1: Failed verification
[2.973s][warning][cds] Skipping com/goldencode/ast/TreeLocal: Failed verification
[2.973s][warning][cds] Skipping com/goldencode/p2j/util/GUIPrinterStreamSupport: Failed verification
[2.974s][warning][cds] Skipping com/goldencode/p2j/admin/server/reports/AccountsReportBuilder: Failed verification
[2.974s][warning][cds] Skipping jdk/internal/event/SecurityProviderServiceEvent: JFR event class
[2.974s][warning][cds] Skipping com/goldencode/p2j/admin/server/reports/AbstractReportBuilder: Failed verification
[2.974s][warning][cds] Skipping com/goldencode/p2j/util/CryptoUtils$CipherParams: Failed verification
[2.974s][warning][cds] Skipping com/goldencode/p2j/util/CryptoUtils: Failed verification
[2.974s][warning][cds] Skipping net/sf/jasperreports/engine/export/JRXlsExporter: Failed verification
[2.975s][warning][cds] Skipping com/goldencode/p2j/util/CryptoUtils$1: Failed verification
[2.975s][warning][cds] Skipping com/goldencode/p2j/reporting/BrowseJasperReportTemplate: Failed verification
[2.975s][warning][cds] Skipping jdk/internal/event/X509CertificateEvent: JFR event class
[2.975s][warning][cds] Skipping org/conscrypt/OpenSSLProvider: Signed JAR
[2.975s][warning][cds] Skipping com/goldencode/p2j/admin/server/reports/UsersReportsBuilder: Failed verification
[2.975s][warning][cds] Skipping com/goldencode/p2j/admin/server/reports/ReportsManager$2: Failed verification
[2.975s][warning][cds] Skipping com/goldencode/p2j/admin/server/reports/ProcessesReportsBuilder: Failed verification
[2.975s][warning][cds] Skipping com/goldencode/p2j/admin/server/reports/AppserversReportsBuilder: Failed verification
[2.975s][warning][cds] Skipping com/goldencode/p2j/admin/server/reports/DefaultReportBuilder: Failed verification
[2.975s][warning][cds] Skipping com/goldencode/p2j/admin/server/reports/GroupsReportsBuilder: Failed verification

ls -ll *.jsa *.lst

-rw-rw---- 1 tg tg 190K Jun 18 15:55 classes.lst
-r--r----- 1 tg tg 71M Jun 18 16:00 archive.jsa

Teodor Gorghe wrote:

165M is huge, I have didn't saw an archive so big which was generated by deploy_appcds.sh
This is what I ran to manually test this (in lib.client):

I need info on lib.client. What is it used for? I don't see reference to it anywhere (besides the build.xml)

I corrected what was creating a huge archive, and ported to hotel_gui. I was recursing through all the *.jar files.

Here is a table of my runs...

• Rows accumulate across runs (server bounces between configs). Use --reset on the FIRST run of a fresh comparison to clear old rows.
• ARCHIVE = mapped confirms the .jsa is actually in use for that Client.

Notice ARCHIVE is absent for the first 2... that was intentional. I allow the --no-appcds option to client.sh, but also the prepare_dir.sh can disable in the directory.xml so that web clients can be tested. The next pair of runs were swing runs (./client.sh --swing) but they failed because of the classpath. I think that you know that the same exact classpath needs to be used at this end as was used to prepare the archive. This is the reason to add default.cpath as a generated file which contains the classpath as used in the archive generation... so there isn't any guesswork. I will have to add that back into getcp.sh and client.sh.

The last 2 are properly configured (directory.xml and classpath for client.sh).

The web client uses a different scheme for the classpath... the manifest must contain it. Is that possible to do with the ClientDriver when using the runlist created only by -XX:DumpLoadedClassList and not the static list? I'm asking because of ignorance.

I ran the Swing client test with 2 and 5 concurrent clients:

So appcds wins.

I checked in updates to both hotel and hotel GUI that are able to configure AppCDS (and not configure). The template preparation was updated, although it simply asks if you want a sample added to your project, and uses standard settings to add to the JSON.

I would need to schedule a meet with you.

Today, 10:00 AM GMT-4 time, are you available (this is 17:00 on my time)?

Teodor Gorghe wrote:

I would need to schedule a meet with you.

Today, 10:00 AM GMT-4 time, are you available (this is 17:00 on my time)?

AFAIK, Roger doesn't work on Fridays, so you'll probably have to wait until Monday.

Radu Apetrii wrote:

Teodor Gorghe wrote:

I would need to schedule a meet with you.

Today, 10:00 AM GMT-4 time, are you available (this is 17:00 on my time)?

AFAIK, Roger doesn't work on Fridays, so you'll probably have to wait until Monday.

Ok, I will wait for him for a proposal.

Roger, for now, the stats showed in last notes, convinced me that having an almost full coverage archive, is not the most efficient because not every project reaches all possible path from FWD bytecode.

For now, I think we can disable LstExporter by default and use the minimal runtime generation.
In the future, I think we can create a minimal FWD server - FWD client setup, which loads several basic widgets.
This would also be very useful when moving to Java 25 because we can use AOT, which shares the code cache across FWD clients.

Also, I have checked the hotel changes.

Is not a good time to refactor getcp.sh to generate a lightweight class path, because adding dependency jar files is not required.

Teodor, thank you for pressing on in my absence. The getcp.sh helper was created to move so much logic out of client.sh and server.sh. It needs to stay up-to-date. If we could get the complexity of the logic for finding jars into this AppCDS standard, then would could just pass the classpath over as the .cpath file does. That would be a win.

There is something bothering me about the directory.xml changes I made, and that is the libPath. I went to some effort to only include the libPath section in the directory when AppCDS is configured:

          <node class="string" name="libPath">
            <node-attribute name="value" value="{client_lib_path}"/>
          </node>

If this section can be included whether or not AppCDS is configured, that would remove some complex logic from prepare_dir.sh. I thought it safer to not introduce additional configuration items if we were not using AppCDS. (Because what would be appropriate for {client_lib_path} if that is the case?)

If there is no libPath, the server will use the deploy/lib or /opt/fwd-deploy/convert/lib libPath, depending what is the used p2j.jar for server runtime.

Teodor Gorghe wrote:

If there is no libPath, the server will use the deploy/lib or /opt/fwd-deploy/convert/lib libPath, depending what is the used p2j.jar for server runtime.

• Is it OK to have it in the directory, but not need it?
• And in that case, it wouldn't really matter what the value is set to, correct?

What do you mean by not needing?

In case of not using AppCDS, the spawned clients will still use libPath. It was designed to use for client lib jars, which load less dependencies into memory, resulting to a smaller heap memory consumption.

If there is no client_lib_path into prepare json file, both solutions are good. I have included it marked as default to ../lib because it's the most common setup.
Your solution seems more right, but adds more complexity in prepare_dir.sh script.

Of course, if that value is incorrect, clients will fail to spawn.

Teodor Gorghe wrote:

What do you mean by not needing?

When the entire library is passed in the classpath... the previous way classpaths were build.

In case of not using AppCDS, the spawned clients will still use libPath. It was designed to use for client lib jars, which load less dependencies into memory, resulting to a smaller heap memory consumption.

If there is no client_lib_path into prepare json file, both solutions are good. I have included it marked as default to ../lib because it's the most common setup.
Your solution seems more right, but adds more complexity in prepare_dir.sh script.

Since the code update in prepare_dir.sh behaves properly if no appcds information is found in the JSON, and builds the directory without any libPath, I believe we should leave it as currently implemented.

We should get #11472-37 reviewed so we can allow the customer to test ASAP.

Teodor, should fwdspi.jar be part of the archiveClient in build.gradle? In my Dockerfiles I copy it to the project lib directory:

# Step 9/45
RUN cp ${FWD_LIB}/build/lib/fwdspi.jar /opt/${PROJECT}/lib

Can you tell in which docker file did you found this issue?

Also, why FWD_LIB is the client archive? Shouldn't be the convert archive (/opt/fwd)?
For hotel_gui, in /opt/.../lib should be the converted application files. fwdspi.jar also needs to be there for H2 CollatorProvider. It is only used for FWD server/conversion.

Teodor Gorghe wrote:

Can you tell in which docker file did you found this issue?

In trying to duplicate the customer scenario in hotel_chui for simplification, I've found that the customer's docker/fwd_4.0_ubuntu_client_Dockerfile is not supposed to try to copy fwdspi.jar, as it is not used on the client. So I will be updating their Dockerfile.

Nothing to see here.

Teodor Gorghe wrote:

Also, why FWD_LIB is the client archive? Shouldn't be the convert archive (/opt/fwd)?

FWD_LIB is always set to /opt/fwd. But that symlink should be /opt/fwd-convert or /opt/fwd-client as necessary. I don't think that is an issue, because most scripts and JSON are explicitly using /opt/fwd-client.

For hotel_gui, in /opt/.../lib should be the converted application files. fwdspi.jar also needs to be there for H2 CollatorProvider. It is only used for FWD server/conversion.

Yes... found that.

Roger Borrello wrote:

Teodor Gorghe wrote:

Can you tell in which docker file did you found this issue?

In trying to duplicate the customer scenario in hotel_chui for simplification, I've found that the customer's docker/fwd_4.0_ubuntu_client_Dockerfile is not supposed to try to copy fwdspi.jar, as it is not used on the client. So I will be updating their Dockerfile.

Nothing to see here.

Ok, that makes sense.