Project

General

Profile

Bug #11327

implement size limited "chunking" for bulk data transfer between client and server

Added by Greg Shah 5 months ago. Updated 9 days ago.

Status:
Test
Priority:
Normal
Assignee:
Teodor Gorghe
Target version:
-
Start date:
Due date:
% Done:

100%

billable:
No
vendor_id:
GCD
case_num:
version_reported:
version_resolved:
reviewer:
Constantin Asofiei
production:
No
env_name:
topics:

11327.zip (282 KB) Teodor Gorghe, 04/24/2026 02:02 AM

Screenshot from 2026-07-06 10-22-51.png (335 KB) Teodor Gorghe, 07/07/2026 12:41 AM

Related issues

Related to Runtime Infrastructure - Feature #5776: reduce memory requirements for the FWD client Test
Related to User Interface - Bug #11326: Reduce memory consumption of FWD clients Test

History

#1 Updated by Greg Shah 5 months ago

As documented in #5776-4, there are use cases where we do bulk data transfer between client and server using a byte[] which is sized based on the data being transferred. If that data is very large, then the server or client might OOME as a result since there are no limits on the size.

This task should do the following:

1. Identify all the cases where this can occur.
2. Write example 4GL ABLUnit testcases for each of these locations.
3. Implement a "chunk" based data transfer process that limits the maximum size of each chunk to some configurable value (with a reasonable default). If data must be transferred in excess of that value, then we send the data in multiple chunks until it is all transferred.
4. Modify all of the locations (where this is a problem) to use this new facility.
5. Test and fix as needed.

#2 Updated by Greg Shah 5 months ago

  • Related to Feature #5776: reduce memory requirements for the FWD client added

#3 Updated by Constantin Asofiei 4 months ago

Teodor, I think we need this next.

#4 Updated by Teodor Gorghe 4 months ago

I will take a look, thanks!

#5 Updated by Constantin Asofiei 4 months ago

Greg Shah wrote:

1. Identify all the cases where this can occur.

This can be done by looking at all the network exports (ServerExports, ClientExports, MemoryDaemon, etc) where byte[] is either the return type or a parameter type. After that, back-track and create 4GL unit tests for these APIs.

#6 Updated by Teodor Gorghe 4 months ago

  • Assignee set to Teodor Gorghe

#7 Updated by Constantin Asofiei 4 months ago

The same needs to be considered if there are exports which transfer or return a longchar value - these can be huge.

#8 Updated by Greg Shah 4 months ago

  • Related to Bug #11326: Reduce memory consumption of FWD clients added

#9 Updated by Teodor Gorghe 4 months ago

I have analyzed all the cases where RMI calls have at least one byte[] or as the return type.

Methods which contains byte[] in the signature:

- ServerExports.getCustomFontData // The one introduced from #11326
- ServerExports.loadEnvironment // When loading INI file from jar

- ClientExports.setWindowIcon // when loading the icon from jar
- ClientExports.setImage // when loading the image from jar, for BUTTON widget and IMAGE widget
- ClientExports.addGlobalImage // comes for multiple sources, like Picture OCX widget, TreeList, ImageList and ButtonListGroupItem, when loading the image from jar.
- ClientExports.setWaitState // when loading the cursor icon from jar.
- ClientExports.oleGetData, ClientExports.oleSetData // OCX CFTreeView control.
- ClientExports.readClientSideURL // Called from Spreadsheet OCX LOAD-IMAGE.

- MemoryDaemon.read // ByteBucket OO implementation, BinaryData.getBytes, MemoryStream, COPY-LOB, sockets
- MemoryDaemon.write // ByteBucket OO implementation, MemoryStream, BinaryData.putBytes, COPY-LOB

- StreamHelper.checkOutStream, StreamHelper.checkInStream // comes from #11271, being used when filesystem is on server-side, but the stream is on client-side.

- FileSystem.getFileContents // being called from a Report 4GL extension widget and from X-DOCUMENT, when loading a file from jar.

I have made a couple of tests, the exception is SE.getCustomFontData (requires only configuration changes), CE.readClientSideURL (requires a SPREADSHEET component which I don't know how to initialize it), CE.oleGetData/CE.oleSetData (I need to find how to initialize OCX).

#10 Updated by Teodor Gorghe about 2 months ago

  • % Done changed from 0 to 50
  • Status changed from New to WIP

I have started to work in this task.

I have implemented a new type of stream, BinaryPayload, which wraps the byte[] resource over remote procedure call. ChunkPullInputStream is the new implementation of the stream, which allows logic like font loading, image loading, to fetch the data through communication protocol, in chunks, without materializing a full byte[].

ChunkTransferDaemon mirrors the same design as StreamDaemon, where you can get the configurable chunk_size, initiate a chunk transfer and pull from a chunk transfer.

This allows to change every byte[] from RPC method definitions with BinaryPayload, and use that object to decide if it should be used as a stream or to retrieve full byte[]:
  • there are some methods left, oleGetData, oleSetData, which I have kept to materialize full byte[], by calling BinaryPayload.drain. oleSetData enforces full byte[] because it adds into a map. oleGetData returns memptr and it needs the length before initializing the memptr. readClientSideURL also materializes full byte[] because of WorkbookWrapper.loadImage.

I have also implemented chunked based transfer for COPY-LOB, for memptr, blob. I have didn't adjusted that for longchar and clob because it would need a CharsetDecoder/Encoder.

I will commit the changes in 11327a after I finish doing some final style checks.

#11 Updated by Teodor Gorghe about 2 months ago

Committed revision 16614 on task branch 11327a:
  • Added initial work.

#12 Updated by Teodor Gorghe about 1 month ago

Task branch 11327a was rebased to trunk revision 16634. Last revision is now 16635.

#13 Updated by Constantin Asofiei about 1 month ago

Constantion Asofiei wrote:

Teodor, something we need to think about: appserver calls can be in the same JVM - so, should a byte[] be chunked or not, when transferring a memptr argument? What would be the performance impact for a chunked transfer when in the same JVM? Note at this also applies for the #11371 work - there will be clients like batch processes or MSA agents which will exist in the FWD server (no FWD client, no network transfer). Do we keep the chunked transfer or avoid it?

Greg Shah wrote:

It is worth extra effort now to avoid any overhead from this in the intra-JVM case.

#14 Updated by Teodor Gorghe about 1 month ago

I don't think the impact would be significant because memptr max size is 2 MB. It is actually 2 GB.

#15 Updated by Teodor Gorghe about 1 month ago

For intra-jvm, I have changed the assign method on memptr to avoid fully materializing a byte[] when copying from the old memptr to another.
I have tested this change with 2GB file, and it no longer crashes with OOME.

#16 Updated by Teodor Gorghe about 1 month ago

I have tested latest changes of #11327 on a big GUI project, with custom fonts and GUI elements.
I was able to run the FWD client with 64 MB heap size, which after #11326 changes I wasn't able to.


GC seems to be a little bit active, but there are also some memory leaks from radio sets or something related to caching.
Eduard/Constantin, can you provide the appserver project? I have a small change for appserver serialization and I want to test the impact.

#18 Updated by Teodor Gorghe about 1 month ago

  • % Done changed from 50 to 100
  • Status changed from WIP to Review
  • reviewer Constantin Asofiei added

#19 Updated by Teodor Gorghe about 1 month ago

  • Status changed from Review to WIP
  • % Done changed from 100 to 90

Constantin, you are right about the interaction between FWD client and websocket.
There are some places where there is a message buffer in byte[] which is allocated and not streamed.

I am fixing it right now.

#20 Updated by Teodor Gorghe about 1 month ago

  • % Done changed from 90 to 100
  • Status changed from WIP to Review

Fix is in 16639.

#21 Updated by Dean Macken about 1 month ago

Hi all,
What is the status on this one in terms of availability on trunk\9866c? it seems significant

#22 Updated by Teodor Gorghe about 1 month ago

Currently, 11327a has 5948 new lines of code. It is in review.
If the testing goes well, I think it can be merged soon to trunk.

#23 Updated by Constantin Asofiei 29 days ago

Teodor, please rebase.

#24 Updated by Teodor Gorghe 29 days ago

Task branch 11327a was rebased to trunk revision 16650. Last revision is now 16655.

#25 Updated by Constantin Asofiei 28 days ago

  • Status changed from Review to Internal Test
Teodor, I haven't found anything wrong with the approach and implementation. Next steps:
  • fix the history entry in GuiWebEmulatedWindow.java
  • document the directory configuration for these, with their default values
  • create a test plan where runtime testing is tested:
    • with the default values for these 'chunks'
    • for each application, determine what would be the proper max heap size with the default chunk size
    • testing real GUI application with file upload, images, and so on (what 'triggers' large transfer over client-side network or web).

#26 Updated by Teodor Gorghe 28 days ago

I have done the suggestion about disabling chunk transfer when chunk size parameter is 0, alongside with the history entry fix in 11327/r16656.

#27 Updated by Teodor Gorghe 28 days ago

Documented the new directory.xml parameters in Network.

#28 Updated by Teodor Gorghe 23 days ago

Constantin Asofiei wrote:

Teodor, I haven't found anything wrong with the approach and implementation. Next steps:
  • fix the history entry in GuiWebEmulatedWindow.java
  • document the directory configuration for these, with their default values
  • create a test plan where runtime testing is tested:
    • with the default values for these 'chunks'
    • for each application, determine what would be the proper max heap size with the default chunk size
    • testing real GUI application with file upload, images, and so on (what 'triggers' large transfer over client-side network or web).

With all GUI apps, the optimal heap size is now -Xmx64M (using default directory.xml parameters).

For the real GUI application, I can't find a scenario triggering loading large image, but I have found some scenarios which triggers file upload/download. None of them has regressed or crashed using -Xmx64M.
Current WorkbookWrapper readClientSideURL users does not use large files.
Went through scenarios which uses memptr and copy-lob (with large reports) and the heap memory consumption seems to be bounded.

#29 Updated by Constantin Asofiei 16 days ago

Teodor, any other testing needed?

#30 Updated by Teodor Gorghe 16 days ago

I need to make sure that copy-lob tests still works.
With my tests, copy-lob with large payload works as expected.

#31 Updated by Teodor Gorghe 16 days ago

copy-lob tests works the same as with trunk.

#32 Updated by Constantin Asofiei 15 days ago

Teodor Gorghe wrote:

copy-lob tests works the same as with trunk.

Can this be merged?

#33 Updated by Teodor Gorghe 15 days ago

Yes.

#34 Updated by Constantin Asofiei 15 days ago

  • Status changed from Internal Test to Merge Pending

Please merge 11327a now.

#35 Updated by Constantin Asofiei 15 days ago

Constantin Asofiei wrote:

Please merge 11327a now.

Sorry, you need to rebase again, forgot about Artur's pending merge.

#36 Updated by Teodor Gorghe 15 days ago

  • Status changed from Merge Pending to Test

Branch 11327a was merged into trunk as rev. 16676 and archived.

#37 Updated by Robin Harris 14 days ago

Teodor Gorghe wrote:

Branch 11327a was merged into trunk as rev. 16676 and archived.

I'm so excited about this one!!! Is it just a runtime change or does it require conversion?

#38 Updated by Teodor Gorghe 14 days ago

Runtime change only.

#39 Updated by Robin Harris 13 days ago

Roger any change we could have have an new docker image for trunk rev 16676?

#40 Updated by Teodor Gorghe 13 days ago

+Roger
Roger, can you check #11327-39?

#41 Updated by Roger Borrello 13 days ago

Robin Harris wrote:

Roger any change we could have have an new docker image for trunk rev 16676?

Sure. Check in 10 mins.

#42 Updated by James Berry 9 days ago

Hi all - this will be coming up for some Border testing very soon and i'd like to plan some tests. Aside from firing off bulky reports such as a dump of the data from Price Book, are there any areas/processes you think I would benefit from us looking at?

#43 Updated by Constantin Asofiei 9 days ago

James Berry wrote:

Hi all - this will be coming up for some Border testing very soon and i'd like to plan some tests. Aside from firing off bulky reports such as a dump of the data from Price Book, are there any areas/processes you think I would benefit from us looking at?

Anything with large data applied to files on disk (reading, writing, upload?), COPY-LOB or memptr. PS: this is a public task, so please create a task in the private project if you need to discuss this more.

#44 Updated by James Berry 9 days ago

Constantin Asofiei wrote:

Anything with large data applied to files on disk (reading, writing, upload?), COPY-LOB or memptr. PS: this is a public task, so please create a task in the private project if you need to discuss this more.

Cheers Constantin. Robin has directed me towards a different ticket to do this - thanks for the info too.

Also available in: Atom PDF