Project

General

Profile

Preloading Classes for Better Runtime Performance

Introduction

The first time a converted program is run after server startup, a Java class loader needs to load classes into the JVM's memory. This operation involves:
  1. searching the classpath and finding the classes in jar files
  2. reading the class files into memory
  3. verifying the classes
  4. running any static initializers
  5. possibly accessing application configuration, etc.

Some FWD runtime support classes will be needed as well. The first time these are used, a similar loading/verifying/initalization process takes place.

In addition, FWD by default assembles and loads certain types of classes as they are needed at runtime. These include Data Model Object (DMO) implementation classes and Java proxy objects. Both are used heavily by most converted applications.

By default, all of this work is done lazily (i.e., on demand) at runtime. Since this default behavior only loads a class when it actually is needed, it saves memory compared to preloading many classes which may or may not ever be used. However, the penalty for this lazy loading is slower performance when a program or runtime service is accessed for the first time. Once these classes are loaded and used the first time after server startup, they reside in memory, and this initial work will not need to be performed again. So, subsequent uses of the same programs and runtime support classes will be faster.

In extreme cases, the first-time execution of a program or API can be orders of magnitude slower than subsequent executions.

This document discusses some optional features in FWD which enable deterministic preloading of both application and FWD runtime classes which are likely to be needed. When an appropriate subset of classes is identified and preloaded, much of the first-time-load performance penalty described above for lazy loading can be alleviated.

Pregenerate DMO Interfaces and Classes

Data Model Objects (DMOs) are object representations of database tables (persistent and temporary). Each instance of a DMO represents a database record. The interface defines getter and setter methods for the fields of such records, as well as annotations representing various properties of the table. The FWD runtime relies on these DMOs to perform create/read/update/delete (CRUD) database operations.

The FWD conversion generates interfaces for persistent tables from the schema (*.df) files and for temp-tables from DEFINE TEMP-TABLE statements in the original 4GL code. DMOs for temp-tables defined dynamically at runtime via a CREATE TEMP-TABLE statement are generated on-the-fly by running a subset of the FWD conversion at runtime. The interfaces for these dynamic DMOs are assembled and loaded lazily, at runtime, when first needed.

When a converted application is run and static or dynamic DMO resources are needed to perform database operations for a particular table, the FWD runtime loads the associated interfaces on demand. At this time, the runtime also will assemble and load a Java class which implements each interface. This assembly and classloading has performance overhead, as noted above.

Pregenerating Data Model Object (DMO) interfaces and classes helps improve application performance by reducing the need to generate these classes lazily at runtime. This optimization can significantly decrease the time it takes to run through code paths the first time, giving a better perceived experience to the first users to use the application after a server restart.

Static DMOs

Normally, only the DMO interfaces for static DMOs are generated during the conversion process. Optionally, the conversion can be made to assemble the implementing DMO classes as well. This eliminates the overhead of generating these classes on-the-fly at runtime, leading to better performance the first time through a code path after server startup.

  1. Modify FWD build script build.xml to include:
       <!-- compile -->
   <target name="compile.post-ops" 
           description="Generates, for every DMO and DMO buffer interface, a DMO implementation class and respectively a DMO buffer proxy class file. Does NOT convert, create jars or deploy." >
      <java classname="com.goldencode.p2j.compile.PostCompilationOpsDriver" 
            fork="true" 
            failonerror="true" 
            dir="${basedir}">
         <jvmarg value="-server"/>
         <jvmarg value="-Xmx16G"/>
         <jvmarg value="-XX:-OmitStackTraceInFastThrow"/>
         <jvmarg value="-DP2J_HOME=${p2j.home}"/>
         <arg value="${build.home}/classes/"/>
         <arg value="${pkgroot}/dmo/"/>
         <arg value="${build.home}/classes/${pkgroot}/dmo/_meta"/>
         <arg value="${build.home}/classes-dmo-impls/"/>
         <arg value="${build.home}/classes-dmo-buf-proxies/"/>
         <arg value="${deploy.home}/server/proxy-cache-map-data.txt"/>
         <arg value="${build.home}/lib/${appname}_dmo_impls.jar"/>
         <arg value="${build.home}/lib/${appname}_dmo_buf_proxies.jar"/>
         <classpath refid="compile.classpath"/>
      </java>
    </target>

    The arguments need to reflect:
    - <arg value="${build.home}/classes/"/>: folders that contain DMOS, separated by : e.g., ${build.home}/classes/:${build.home}/classes.aop/:${build.home}/classes.proxy/
    - <arg value="${pkgroot}/dmo/"/>: where the DMOS are located
    - <arg value="${build.home}/classes/${pkgroot}/dmo/_meta"/>: blacklist paths to indicate which schema folders won't be processed, separated by :
    - <arg value="${build.home}/classes-dmo-impls/"/>: where to save all the DMO implementations
    - <arg value="${build.home}/classes-dmo-buf-proxies/"/>: where to save all the DMO buffer proxies
    - <arg value="${deploy.home}/server/proxy-cache-map-data.txt"/>: where to save the map file that helps associate a proxy class file name to the interfaces/classes names that were used to generate it.
    - <arg value="${build.home}/lib/${appname}_dmo_impls.jar"/>: where the jar with the DMO implementations will be saved
    - <arg value="${build.home}/lib/${appname}_dmo_buf_proxies.jar"/>: where the jar with the DMO buffer proxies will be saved
As an alternative to using ant, this script can be ran from the application root directory; arguments need to be modified as explained above.
#!/bin/bash

start_time=$(date +%s)

basedir=$(pwd)
build_home="${basedir}/build" 
lib_home="${basedir}/deploy/lib" 
cfg_home="${basedir}/cfg" 
p2j_home="." 
appname="hotel" 
pkgroot="com/goldencode/${appname}" 
deploy_home="${basedir}/deploy" 

# Set up classpath
helper=$(cd $cfg_home/..; echo $(pwd))
PATH=$helper:$PATH
cpath="" 
for f in ${lib_home}/*.jar
do
   cpath="${cpath}:${f}" 
done

# Compile post-operations
java -server -Xmx16G -XX:-OmitStackTraceInFastThrow -DP2J_HOME="${p2j_home}"\
        -cp "${cpath}" com.goldencode.p2j.compile.PostCompilationOpsDriver \
        "${build_home}/classes/" \
        "${pkgroot}/dmo/" \
        "${build_home}/classes/${pkgroot}/dmo/_meta" \
        "${build_home}/classes-dmo-impls/" \
        "${build_home}/classes-dmo-buf-proxies/" \
        "${deploy_home}/server/proxy-cache-map-data.txt" \
        "${build_home}/lib/${appname}_dmo_impls.jar" \
        "${build_home}/lib/${appname}_dmo_buf_proxies.jar" 

end_time=$(date +%s)
execution_time=$((end_time - start_time))

echo "Total time: ${execution_time} seconds"
  1. Go to the application root directory and make sure that all the *.class files of the DMO interfaces and DMO buffer interfaces exist in the specified folders.
  2. Run ant compile.post-ops or the script. This process may take about 10 minutes.
  3. Ensure that the two new JAR files are created in the specified directory, and either move them manually to deploy/lib, or with ant deploy.prepare.
  4. Update directory.xml to include the following configuration under <node class="container" name="clientConfig"> in the server/default container:
    <node class="boolean" name="use-proxies-cache">
  <node-attribute name="value" value="TRUE"/>
</node>
<node class="string" name="proxy-cache-file-path">
  <node-attribute name="value" value="proxy-cache-map-data.txt"/>
</node>

    Make sure that the proxy-cache-map-data.txt is at the specified proxy-cache-file-path either on disk or in the classpath (in a jar at that location) .

When the server starts, it will use the DMOs in the new JAR files.

Dynamic DMOS

Dynamic DMOs are generated at runtime based on application needs, which can introduce some performance overhead the first time through a code path. However, by caching and reusing these dynamically generated DMOs, performance of that first run through a code path can be significantly improved.

Unlike static DMOs, the structures of dynamic DMOs are defined by business logic at runtime, and thus are not known at application conversion time. Therefore, the pregeneration of dynamic DMOs is a two-step process:

  1. The application is started in a "collection" mode. This involves running the application through code paths which define the most essential dynamic temp-tables (those which would benefit most from pregeneration and early classloading). This step both stores the generated dynamic DMO interfaces/classes persistently and records the list of such DMOs. Theoretically, this step only needs to be run once for a particular version of the application. In practice, it can be run as many times as needed to determine and collect the best set of dynamic DMOs to be pregenerated.
  2. The second step involves a server restart in a "normal" mode. This mode will use the DMO resources created by the first step, avoiding the need to generate these resources on demand, at runtime. This becomes the common server startup mode, once the set of pregenerated DMOs to be loaded has settled.
To enable this feature:
  1. Add the necessary parameters in directory.xml under <node class="container" name="server"> <node class="container" name="default">:
    <node class="container" name="dynamic-dmos">
  <node class="boolean" name="use-persistent-cache">
    <node-attribute name="value" value="TRUE"/>
  </node>
  <node class="boolean" name="update-persistent-cache">
    <node-attribute name="value" value="TRUE"/>
  </node>
  <node class="string" name="persistent-cache-jar-path">
    <node-attribute name="value" value="persistent_cache.jar"/>
  </node>
  <node class="string" name="persistent-cache-map-path">
    <node-attribute name="value" value="dynamic_dmo_cache_data.txt"/>
  </node>
  <node class="string" name="temporary-cache-folder-path">
    <node-attribute name="value" value="temp_lib_cache"/>
  </node>
  <node class="string" name="temporary-cache-jar-path">
    <node-attribute name="value" value="lib_cache/temp_dynamic_dmos.jar"/>
  </node>
</node>

    Arguments needed:
    - dynamic-dmos/use-persistent-cache - marks whether the persistent cache should be used. Reading from persistent cache and writing to it depend on this flag. It is mandatory, had no default value.
    - dynamic-dmos/update-persistent-cache - marks whether the persistent cache jar and map file will be updated during this server run, by default it is set to false. On the first run, this needs to be set on true for everything to be generated. See notes.
    - dynamic-dmos/persistent-cache-jar-path - the path to the final jar. It is mandatory, had no default value.
    - dynamic-dmos/persistent-cache-map-path - a file containing a HashMap object, mapping a string made from a DynamicTablesHelper.CacheKey to a list of interface names used to create the DMO class needed. Default is /deploy/server/dynamic-dmo-cache-data.txt.
    - dynamic-dmos/temporary-cache-folder-path - the path to a temporary folder, created during server runtime. There will be saved any new classes generated at runtime, so at server shutdown they can be added to a jar. When the jar is created, the folder will be deleted. Default is /deploy/temp-lib-cache.
    - dynamic-dmos/temporary-cache-jar-path - the path to a temporary jar, that is created and deleted at server shutdown, used to update the persistent jar mentioned above. Default is /deploy/lib-cache/temp-dynamic-dmos.jar.
  2. Run the server as usual. For best result, replicate a common use case.
  3. Stop the server, use CTRL + C or -k option. The new JAR will be created.
  4. Move/copy the generated jar in deploy/lib.
  5. When the server will be restarted, the already generated DMOS will be used.
Notes:
  • If update-persistent-cache is TRUE, the jar and the cache map will be updated with the newly used dynamic DMOS of the current server run, if false the existent map and jar are used but not updated.
  • The persistent cache map can also be read from the class path, but only when update-persistent-cache is FALSE.
  • This parameter, update-persistent-cache, can be overridden by a bootstrap configuration: persistence:dmo:updateDynamicDMOCache=true can be added to the server start-up command in server.sh or in server.xml.
  • For every new conversion, the dynamic dmos JAR and the file at persistent-cache-map-path should be deleted.

Application Class Data Sharing (AppCDS)

Application Class Data Sharing (AppCDS) is a JVM feature introduced in Java 11, that helps reduce the startup time and memory footprint of Java applications by creating a preprocessed shared archive of classes to be preloaded.

Leveraging this feature involves a three-step process:
  1. Class List Recording: During this run of the application, the JVM records the list of classes that are loaded and saves this list to a file.
  2. Archive Creation: The JVM uses this recorded list to create a shared archive file containing the preloaded classes.
  3. Class Preloading: When starting the application, the JVM uses the shared archive to preload classes, reducing the time and resources needed for loading them individually.

These steps describe how AppCDS is configured for the FWD server. Since FWD 4.0 (trunk rev 16575), AppCDS archives can also be generated automatically for spawned FWD clients — see Configuring AppCDS for FWD Clients below.

Configuring AppCDS in FWD

If the use of Java 17 is intended, first follow Switching_Between_Java_8_and_Java_17.
To enable this three-step process, the FWD server scripts (usually located in deploy/server/server.sh) needs to be updated. The script will contain three new parameters, -x1, -x2, and -x3, for each step of the process.
  1. Initialize the following variables in the script:
    appcds=0
max_classes_default=100000
cdsparams="" 
archive_retention=3
cds_archive="cds-archive.jsa"

    As AppCDS is available for Java version greater than 11, add the detection of the version used.
    # Determine the configured JDK
#jver is 18 for java 1.8, 15 for java 1.5, 110 for java 11, 170 for java 17 etc.
jver=$(${prog} -version 2>&1 | awk -F '"' '/version/ {print $2}' | awk -F '.' '{sub("^$", "0", $2); print $1$2}')
  2. In the while loop that processes the options, make available -x1, -x2 and -x3 by modifying
    "?dyptksw1az:h:c:i:b:m:j:r:-:" to "?dyptksw1az:h:c:i:b:m:j:r:x:-:".
    Also add the option in the while loop: 
          x  ) appcds=$OPTARG
           if [[ ${!OPTIND} =~ ^[0-9]+$ ]]; then
              max_classes=${!OPTIND}
              OPTIND=$((OPTIND + 1))
           else
              max_classes=$max_classes_default
           fi
           ;;
  3. Add the function that performs rolling generation for the CDS archive:
    # Setup rolling generation for the archive
roll_cds_archive()
{
   local archive_file=$1
   local retention=$2

   if [ -f "${archive_file}.${retention}" ]; then
      rm -f "${archive_file}.${retention}" 
   fi
   for ((i=retention-1; i>=0; i--)); do
      if [ -f "${archive_file}.${i}" ]; then
         mv "${archive_file}.${i}" "${archive_file}.$((i+1))" 
      fi
   done
   if [ -f "${archive_file}" ]; then
      mv "${archive_file}" "${archive_file}.0" 
   fi
}
  4. Add this function to set up AppCDS parameters based on the value of appcds (1,2,3 - the step that is being ran) :
    # Setup Application Class Data Sharing
setup_appcds()
{
   local appcds=$1
   local max_classes=$2

   unfiltered_cds="cds-class-list-unedited.lst" 
   [ -f "cds-blacklist.lst" ] && cds_filter=$(cat "cds-blacklist.lst") || cds_filter="" 
   filtered_cds="cds-class-list-filtered.lst" 
   cds_archive="cds-archive.jsa" 
   xlog="" 
   if [ $appcds -eq 1 ]; then
      # Full recording
      xshare="-Xshare:off" 
      xxdumploaded="-XX:DumpLoadedClassList=${unfiltered_cds}" 

      # Perform rolling archive
      roll_cds_archive $cds_archive $archive_retention
   elif [ $appcds -eq 2 ]; then
      # Archive previous full recording
      if [ ! -f $unfiltered_cds ]; then
         echo "File $unfiltered_cds is missing. Make sure you started the server with parameter \"-x1\" at least once before." 
         exit 1
      else
         echo "Using class list created at $(stat -c "%w" $unfiltered_cds | cut -d '.' -f 1)".
      fi
      awk 'length($0) <= 4096' $unfiltered_cds | head -n $max_classes > $filtered_cds
      for p in $cds_filter; do
         sed -i "/$p/d" $filtered_cds
      done
      xshare="-Xshare:dump" 
      xxsharedclass="-XX:SharedClassListFile=${filtered_cds}" 
      xxsharedarchive="-XX:SharedArchiveFile=$cds_archive" 
      xlog="-Xlog:cds:file=cds.log -Xlog:class+load:file=cds-class-load.log" 
      output_redir="> /dev/null 2>&1" 
   elif [ $appcds -eq 3 ]; then
      if [ ! -f $cds_archive ]; then
         echo "Archive $cds_archive is missing. Make sure you have previously started the server with parameter \"-x2\"." 
         exit 1
      else
         echo "Using archive created at $(stat -c "%w" $cds_archive | cut -d '.' -f 1)".
      fi
      # Start with previous archived list
      xshare="-Xshare:on" 
      xxsharedarchive="-XX:SharedArchiveFile=$cds_archive" 
   fi
   # Setup the cds parameters from above
   cdsparams="${xshare} ${xxsharedarchive} ${xxdumploaded} ${xxsharedclass} ${xlog}" 
}
  5. Before running the server, call the setup_appcds function if appcds is set:
    if [[ $jver -ge 110 ]] && [[ $appcds -ge 1 && $appcds -le 3 ]]; then
   setup_appcds $appcds $max_classes
fi
  6. Modify the parameters used when starting the JVM:
    - Include cdsparams
    - Change entry point from com.goldencode.p2j.main.ServerDriver to com.goldencode.p2j.main.FwdLauncher
    - Remove -Djava.system.class.loader=com.goldencode.p2j.classloader.MultiClassLoader
    Example:
    entrypoint="com.goldencode.p2j.main.FwdLauncher" 
# run the FWD server
if [ $test -eq 1 ] ; then
   echo $prog $perf $hprof $maxheap $srvr $jmx $dtxt $agent $spi $open_api $cpath $cdsparams $entrypoint $mode $batch $cfg $port $profile "$@" 
elif [ $statcode = "y" ]; then
   $prog $perf $hprof $maxheap $srvr $jmx $dtxt $agent $spi $open_api $cpath $cdsparams $entrypoint $mode $batch $cfg $port $profile "$@" 2> /dev/null
else
   eval $prog $perf $hprof $maxheap $srvr $jmx $dtxt $agent $spi $open_api $cpath $cdsparams $entrypoint $mode $batch $cfg $port $profile "$@" $output_redir
fi

    The entry point and MultiClassLoader changes address the incompatibility of -Djava.system.class.loader with AppCDS. By using a new entry point that programmatically adds the MultiClassLoader, the custom class loader is utilized without setting it as a system property at JVM startup, thus avoiding conflicts with AppCDS.
  7. Check the successful creation of the archive at the end of the script:
    # Check if the archive was created successfully when appcds=2
if [ $appcds -eq 2 ] && [ ! -f $cds_archive ]; then
   echo "Warning: Archive $cds_archive was not created successfully." 
fi

Using AppCDS

  1. Prepare the Environment: Ensure that the FWD server script is correctly set up and that you are using Java 11 or higher.
  2. Record the class list: Execute the server script with the -x1 parameter to start recording the classes ./server.sh -x1. The server will start as normal, saving the used classes in the generated file cds-class-list-unedited.lst. Perform the typical operations and workflows that the application is expected to handle, so the recorded class list accurately reflects the classes needed for optimal performance.
  3. Create the shared archive: Execute the server script with the -x2 parameter ./server.sh -x2. You can specify the number of classes to include in the archive with ./server.sh -x2 120000. The process will generate a shared archive file, typically named cds-archive.jsa.
  4. Using the archive: Execute the server script with the -x3 parameter to start the server using the shared archive ./server.sh -x3.

By following these steps, you can leverage AppCDS to improve the startup performance of the FWD server, ensuring that frequently used classes are preloaded efficiently.

Troubleshooting AppCDS

If the archive is not created successfully, consider lowering the number of max classes using -x2 <no_classes>, with the default being 100,000. Note that the AppCDS archive is limited to 2GB.

Configuring AppCDS for FWD Clients

Beginning with FWD 4.0 (trunk rev 16575), FWD can build AppCDS archives for spawned clients automatically. This support was added primarily to simplify deployment of docker-based setups, but it benefits any deployment that spawns FWD clients.

Benefits:
  • Lower memory consumption for each client JVM, because the shared classes live in shared memory.
  • Faster client startup.

Unlike the server process described above (which records and dumps a single archive in three manual steps via -x1/-x2/-x3), the client archives are generated automatically — no human intervention is required.

What changed

Support is delivered by the following hotel_gui changes (revs 455, 456, 457):
  • A new script used to build the AppCDS archives: deploy/server/deploy_appcds.sh (a PowerShell version, deploy/server/deploy_appcds.ps1, is also provided for Windows).
  • build.xml changes, including a new deploy.appcds Ant task, plus docker-related file changes.
  • A directory.xml.template change so that spawned FWD clients use AppCDS. Rebuilding directory.xml is required to pick up the newest template changes.

Generating the client archives

Run the new Ant task from the project root:

ant deploy.appcds

For dockerized setups, use prepare entrypoint, which now uses deploy_appcds.sh.

This invokes deploy/server/deploy_appcds.sh (or deploy_appcds.ps1 on Windows), which:
  • Reads your directory.xml, extracts each type of configuration (via JvmArgsExtractor), and generates the corresponding .jsa archive files. This is fully automatic and does not require human intervention.
  • Saves the results into appcds/client (for example, /opt/hotel/appcds/client).

You can also run the script directly. Its options are:

Usage: deploy_appcds.sh [-f <directory.xml>] [-d <appcds_dir>] [path_to_p2j.jar]
   path_to_p2j.jar     = Path to p2j.jar (default="../lib/p2j.jar")
   -f | --file         = Path to directory.xml (default="../server/directory.xml")
   -d | --dir          = Path to output appcds folder (default="../appcds/client")
   -r | --runtime-only = Don't use LstExporter for static reachability (runtime-only)
   -h | --help         = Usage information

Archive naming rules

The archive names are derived by JvmArgsExtractor from your directory.xml:
  • It first looks for a specific server configuration (the archive name is prefixed with serverId_ when a specific server configuration is found); otherwise it uses the default server configuration.
  • For each client configuration, it first looks for the specific configurations (such as ChUI or GUI), then falls back to the default configuration. No suffix is added when the default configuration is used.
  • A configuration is considered to exist if at least one of two values is overridden: libPath or jvmArgs.

Enabling AppCDS in directory.xml

Make sure the jvmArgs for your client configuration in directory.xml include the AppCDS JVM arguments, pointing -XX:SharedArchiveFile at the generated .jsa:

        <node class="container" name="clientConfig">
          ...
          <node class="string" name="jvmArgs">
            <node-attribute name="value" value="-Xshare:auto -XX:SharedArchiveFile=/home/fwd/my_project/deploy/appcds/client/appcds.jsa -Xmx128m -Djava.awt.headless=true"/>
          </node>
          ...
        </node>

Remember that directory.xml must be rebuilt to pick up the directory.xml.template changes that enable AppCDS for spawned clients.

Verifying

After the archives have been generated, start the FWD server and confirm that the spawned clients use AppCDS:
  • use jps -lvm to check for existing Java processes in your system. Save the PID for the client process which you want to check if AppCDS is working.
  • run jcmd <PID> VM.metaspace (replace <PID> with your PID). Find the CDS row, which tells if AppCDS is running (example CDS: on). Check shared class ratio if the AppCDS archive actually covers a high percentage of actual loaded class (higher is better).