FWD » Core Development » Conversion Tools

The profile approach (see #6256) provides the ability to segment a larger code base into named profiles. An important objective of doing this is to be able to convert minimum sets of functionality rather than requiring the entire application to be converted at once. To make this work properly, we must be able to specify dependencies between profiles and the priority or order of conversion within profiles. This is important because converting an arbitrary profile may depend on files in one or more other profiles. If we cannot find the conversion artifacts for a dependency, then we cannot properly convert. This also matters within a profile. When all files are converting in a single batch, it is possible to convert procedures in a simple alphabetical order. When some files are converted separately, we may need control over the ordering such that we can get correct results.

Constantin suggested the following:

As I see it, you need a custom comparator to sort the file list according to some rules.

Greg: for this, a custom comparator could be configured in p2j.cfg.xml, and if that parameter exists (and if the class can be loaded), then sort the program list using that. It's just a matter to write the comparator in Java, add it to a jar and to the conversion classpath.

But I don't think this can work with the .cls files, as those need some special processing in FWD, only with the plain .p programs.

The comparator is an interesting idea. It certainly would provide a great deal of customizable control. However, it brings with it:

• Extra complexity because now we have to have extra code that is compiled and found at conversion time.
• Dependencies are hidden inside whatever Java code is written, rather than being made explicit in configuration.

For this last reason especially, I prefer to avoid the comparator approach.

For dependencies between profiles, I think we should a add a depends element which encodes one or more profiles on which this profil depends. In other words, that list of profiles must be converted first. A key idea here is this must be an acyclic directed graph. There must be no direct or indirect cycles (profiles which depend upon themselves directly or indirectly).

For ordering within a given profile, I think we can just allow the order of the file-set to define the order of conversion. This may mean that the file-set will no longer be a simple directory spec, but I think that is an acceptable trade off. Breaking out the entries and ordering them explicitly makes the result obvious since it is encoded explicitly in the configuration.

Today we just sort alphabetically, but the only real advantage to that is that it is easy for a human to see where in the overall code the processing is occurring. That is useful but not critical. The idea of dependencies must break that alphabetical ordering anyway, so there is no extra loss here.

For OO we must process the objects in their object graph order. We already do this, but we will need changes to handle it across profiles.

Greg, if I understand correctly, your suggestion is to order the non-OO file list using the include/exclude filters from the file-set in p2j.cfg.xml, for whatever profile gets converted. A problem I see here, is we use the include/exclude filters for example to exclude an entire folder and just cherry-pick some files from the excluded folder, with include filters, after that. Better said: there is already a semantics for the order of include and exclude filters in a file-set. In any case, it should not be a problem to enable sorting using the order of the include filters in the file-set.

Are profiles already supported today ? And if so, is there an explanation on how to use them ?

Greg, if I understand correctly, your suggestion is to order the non-OO file list using the include/exclude filters from the file-set in p2j.cfg.xml, for whatever profile gets converted. A problem I see here, is we use the include/exclude filters for example to exclude an entire folder and just cherry-pick some files from the excluded folder, with include filters, after that. Better said: there is already a semantics for the order of include and exclude filters in a file-set. In any case, it should not be a problem to enable sorting using the order of the include filters in the file-set.

I'm more focused on the directory and filename directives than the include and exclude filters. The excludes we don't have to worry about because those files are never converted so the order of non-conversion doesn't matter. I agree that there might be cases when we want a different order than can be expressed with a minimal file-set definition. A brute force solution to that problem is to more explicitly list directory or filename entries that otherwise would not be needed.

If we find that to be a common case, we can consider adding other syntax but for now I prefer if we can solve this using the existing approach.

Are profiles already supported today ? And if so, is there an explanation on how to use them ?

Yes, these are supported today. Unfortunately, we have not yet documented the features. You can see the discussions including some configuration syntax in #6256 and #6253.

These profiles might be something that can help me. A couple of questions :

• Can the propath include directories outside the file sets inside the profile ?
• How are these profiles processed during a conversion ? Is it in the order they are specified in the config file ? So that it has the same effect as what I did by naming the folders with number prefixes so that they are alphabetically in the correct dependency order ? ( projects with least dependencies first )

To explain this a bit, now I have folders like this :

a005zzz/src/api
a006zzz/src/impl
a007bbb/src/api
a008bbb/src/impl

The above guaranties that we can convert a007bbb/src/api separately ( as long as we also include a006 and a005 and everything above it ) and then use the result in a full conversion environment. This only works if we don't make temp table changes that conflict with projects depending on a007. This is already a big help for us, but the a unique consistent naming of DMOs would help us much more of course.

btw, for now we could live with a schema shared between the profiles.
Would the profiles name the DMOs with a namespace ? That could make the DMO names more unique ?

Temp table DMOs I mean.

I don't know why, but I can't modify my own comments in this ticket...

Can the propath include directories outside the file sets inside the profile ?

Yes, but in practical terms this is mostly useful for include files. Our OO approach still requires all referenced OO .cls files to be converted in a single batch.

How are these profiles processed during a conversion ? Is it in the order they are specified in the config file ?

No, today the code builds up an overall list of files to process and alphabetizes them all into a single list. My idea is to specify the dependencies between profiles and ensure that when you convert profile C which depends on B (and B depends on A), that we:

1. Have access to the converted code for B and A such that we can read any needed conversion details from the artifacts in the jar. OR
2. Convert the missing profiles in the proper order. If B is missing but A exists, then we convert B before C. If A is missing, then we convert A before whatever we do for B or C. And so forth...

So that it has the same effect as what I did by naming the folders with number prefixes so that they are alphabetically in the correct dependency order ? ( projects with least dependencies first )

For now it is a way to specify an app-level configuration but it doesn't cleanly resolve the various other issues of converting app by app.

Would the profiles name the DMOs with a namespace ?

It does not do this today.

That could make the DMO names more unique ?

Eric?

Greg Shah wrote:

No, today the code builds up an overall list of files to process and alphabetizes them all into a single list. My idea is to specify the dependencies between profiles and ensure that when you convert profile C which depends on B (and B depends on A), that we:

1. Have access to the converted code for B and A such that we can read any needed conversion details from the artifacts in the jar. OR
2. Convert the missing profiles in the proper order. If B is missing but A exists, then we convert B before C. If A is missing, then we convert A before whatever we do for B or C. And so forth...

This would indeed make it possible to do a conversion of a single app. Especially if you combine this with putting DMOs in the namespace of the app wher it is defined in. I think this is the right solution.

We should implement a dependency manager to track dependencies and allow cascading adds and deletes from the conversion list (see ##7180 and #7179).

The OO processing today does part of this job, for the object graph. It would be moved into this external dependency manager.

I think the best start would be to lay down the foundation, implement the profile dependencies and use the acyclic directed graph structure to confirm that there are no profiles that depend directly or indirectly on themselves.

Sounds good. All config should be in p2j.cfg.xml, primarily (maybe exclusively) in the profile sections.

Greg Shah wrote:

Sounds good. All config should be in p2j.cfg.xml, primarily (maybe exclusively) in the profile sections.

I've added the following:

   <profile name="test1" depends="test2,test3">
      <file-set filename="zfile_set.txt" />
   ...

Committed 7169a/16351-16352. Added profile configuration dependencies.

• ProfileConfig contains a set of dependency profile names which will end up being used for populating the dependencyProfiles collection after being validated.
• Configuration has getValidProfileDependencies now, a method used to populate the dependencyProfiles in ProfileConfig. This method also returns a LinkedList of profiles ordered in topological order based on which has less dependencies (this can be used in the future to determine what profiles we convert first, a good start)

1. A user will be able to define a profile dependency, but what if the same file is present in multiple profiles? Do we scan the profile fileset to check if such issues exist when we validate the profile?
2. When a profile is dependent on another one, it must be added to the conversion process even if that profile was not specified. Do we just turn the conversion into a multi-profile one or split the conversion into multiple threads?
3. We have an order for the profiles depending on the in degree from the graph in 7169, this would allow checking which profiles have the 0 in degree and convert them first, then the conversion graph is recalculated to remove converted profiles.
4. How should I decide which 0 degree profile gets converted first, maybe based on the amount of files (less -> more)?

Please create the class DependencyManager in the artifacts package. That is the class that should track the dependency graph and calculate the order of operations. The dependency manager deals with profiles, not specific artifacts.

We will implement the dependencies as a directed acyclic graph.

• Each profile is a node in the graph.
• Each dependency is an edge that points from one profile/node to one other profile/node.
• Nodes that are not referenced as a dependency by any other node:
  • If they have dependencies defined, then they would be "leaf nodes" which can only be processed at the end of a dependency chain.
  • If they have no dependencies defined, are "independent nodes" that play no part in the dependency graph.
• Nodes that are referenced as a dependency, but which themselves do not define a dependency, are "root nodes", at which all dependency operations start.
• Nodes which have dependencies defined and which themselves are defined as dependency by at least one other node, are "intermediate nodes" that make up the internal traversal of the graph.
• Graph traversal can be calculated for any list of nodes as follows:
  • API Queue<DependencyNode> DependencyManager.calculate(List<DependencyNode>)
    • The parameter is the list of nodes for which the traversal must be calculated.
    • The return value is the queue from which the caller can obtain jobs that are ready to process.
  • Iterate through the list of nodes, for each node:
    • If the node has no dependencies (independent or root nodes), then it can be added to the "may be ready to process" list and the ready flag of the node should be set to true.
    • If the node has dependencies (leaf or intermediate nodes), then:
      • The node should be added to the notification list for each of its dependency nodes.
      • The list of dependencies should be copied into a Set named pending.
      • Its ready flag should be marked false.
  • Iterate through the "may be ready to process list" and for each node:
    • Detect if the node requires processing.
      • In practice, the caller will have to provide an instance of a functional interface that reports this fact. Since the processing of a node (e.g. conversion) is handled by the caller, the calculation about whether something has been processed or not (e.g. an up to date version of the jar for those resources is available) must also be handled by the caller.
      • If the node needs processing, add it to the "ready to process" queue.
      • If the node does not need processing (i.e. it is already available), then notify all nodes in its notification list that "processing has completed".
  • When a node is notified that a dependency has had processing completed, remove that node name from the pending set. If the set is empty, mark the node's ready flag as true and add the node to the queue.
  • We should have a process that checks the graph for invalid conditions (duplications and cycles).
• The caller of calculate() will recieve a queue back. The contents are the jobs which are currently ready to process. For each job pulled from the queue, the caller will do some work (e.g. conversion) and when complete, it will call the "finished()@ method on the node, which will notify all nodes in its notification list that "processing is complete".

A user will be able to define a profile dependency, but what if the same file is present in multiple profiles? Do we scan the profile fileset to check if such issues exist when we validate the profile?

I think we should implement a restriction that if a profile is used with dependency management, then there must not be any overlap between file sets in different profiles. Every file must only be converted in one profile.

When a profile is dependent on another one, it must be added to the conversion process even if that profile was not specified. Do we just turn the conversion into a multi-profile one or split the conversion into multiple threads?

Probably both. In #3211, we are making the PatternEngine multithreaded. When that is merged to trunk, then the conversion of a single profile will map to a single ConversionDriver run which will process one profile in a multi-threaded manner.

We probably need a new DepenencyConversionDriver which would interact with the DependencyManager, call calculate() and process the returned queue. Each node in the queue would be a profile that would gets its own ConversionDriver run. Each of those should be handled in parallel up to some configured number of active jobs.

We have to discuss if the separate ConversionDriver runs are separate processes.

We have an order for the profiles depending on the in degree from the graph in 7169, this would allow checking which profiles have the 0 in degree and convert them first, then the conversion graph is recalculated to remove converted profiles.

My previous note should answer this.

How should I decide which 0 degree profile gets converted first, maybe based on the amount of files (less -> more)?

For now, I was planning a naive approach of just taking the jobs from the queue in their natural order. We can refine it later.

• DependencyNode.java contains the profile (ProfileConfig instance), three sets of DependencyNode instances representing outgoing dependencies, incoming dependencies, and pending dependencies (that did not complete processing), a flag that indicates if the node can be processed;
  • dependencies can be added, but not removed (if there are issues when adding the dependencies, then we exit the program)
  • dependencies and dependents can be retrieved as unmodifiable sets.
  • isIndependent() and isRoot() methods
  • dependencies are removed after being processed, so that the pending collection can be used to set the ready flag.
  • finished() that removes the current node from pending
• DependencyEvaluator is a functional interface which defines a boolean shouldProcess(ProfileConfig profile), I am using it in the DependencyManager as:
  

     private final DependencyEvaluator evaluator = new DependencyEvaluator()
     {
        @Override
        public boolean shouldProcess (ProfileConfig profile)
        {
           return true;
        }
     };
  

• DependencyManager.java contains a private final Map<ProfileConfig, DependencyNode> nodes = new LinkedHashMap<>();, it can also be a Set<DependencyNode>.
  • The DependencyManager is initialized with a set of ProfileConfig instances, parsed using buildGraph() (populates the nodes and also dependencies for each node). There's also a hasCycle() method to confirm that we are using a DAG.
  • calculate() that takes a list of DependencyNode instances, intializes the pending for each node and then uses the evaluator to build the return queue.

Still thinking on the implementation and making corrections based on what I understood from #7169-33 / #7169-34.

It sounds good so far. Please note that if we need to directly reference ProfileConfig then that class should not be in the FWD-specific packages (com.goldencode.p2j.*). Either it should move to some subpackage of com.goldencode or it should be abstracted as an interface that ProfileConfig will implement.

Greg Shah wrote:

It sounds good so far. Please note that if we need to directly reference ProfileConfig then that class should not be in the FWD-specific packages (com.goldencode.p2j.*). Either it should move to some subpackage of com.goldencode or it should be abstracted as an interface that ProfileConfig will implement.

Every class will go into com.goldencode.artifacts.

I committed my changes to 7169a/16353.

The next step should be to evaluate the file that will be converted by each profile and to make sure there are no conflicts, maybe split the conversion so that a jar is generated for each converted profile and use that to check when the other profiles can be processed.

• All profiles are automatically added to the graph, but what if the conversion uses just a few profiles? Should the DAG be made for the active profiles, or should each profile be included and verified that no duplicate files exist?

The dependency manager will only process profiles that are explicitly part of the list provided by the caller OR which are dependencies of those profiles.

• process the command line arguments (the profiles and modes), the ant command I used is

     <target name="convert.profile" 
             depends="init" 
             description="test">
        <record name="cvt_${LOG_STAMP}.log" action="start"/>
        <java classname="com.goldencode.p2j.convert.DependencyConversionDriver" 
              fork="true" 
              failonerror="true" 
              dir="${basedir}">
           <jvmarg value="-server"/>
           <jvmarg value="-Xmx${conversionHeap}"/>
           <jvmarg value="-XX:-OmitStackTraceInFastThrow"/>
           <jvmarg value="-DP2J_HOME=${fwd.home}"/>
           <jvmarg value="-Xdebug"/>
           <jvmarg value="-Xnoagent"/>
           <jvmarg value="-Djava.compiler=NONE"/>
           <jvmarg value="-Xrunjdwp:transport=dt_socket,address=2100,server=y,suspend=y"/>
           <arg value="-ptest1"/>
           <arg value="-ptest2"/>
           <arg value="f2+m0+cb"/>
           <classpath refid="compile.classpath"/>
        </java>
        <record name="cvt_${LOG_STAMP}.log" action="stop"/>
     </target>

• initialize the profiles (build the dependencies for each ProfileConfig and check if those exist)
• validate the used profiles (check the file list of each profile for duplicate files)

Next is the job execution, where I have to convert those profiles separately. Coming up with ideas now.

• The DependencyConversionDriver is an independent class that does not extend the TransformDriver, this is because each profile will have the ConversionDriver execute the conversion.
• The command line arguments are processed to build a SharedJobConfiguration. Each profile will require a conversion and in turn, a JobDefinition. The SharedJobConfiguration will be used to initialize the JobDefinition for each conversion that will be executed.
• Again, at the same time the arguments are processed, the ProfileConfig dependencies are built and the DependencyManager is initialized with the configuration profiles + the their dependencies depending on the provided arguments.
• Each profile will have a different job, so there will be x number of threads that will run the conversion depending on the threads available in the pool.

1. Each profile will run using the ConversionDriver and should create a jar (1 profile = 1 jar), the name of the jar should be profile.getName() + ".jar".
2. The Evaluator from #7169-35 will check if the jar exists and mark the node as finished if necessary.

I am unsure how to proceed and create the jar for each profile. The jar should include the following:

       <jar jarfile="${build.home}/lib/${appname}.jar" 
           basedir="${build.home}/classes" 
           excludes="**/*.template" 
           includes="**/${pkgroot}/**
                     dir_schema_ext.xml
                     ehcache.xml
                     *.jasper
                     **.xml
                     cfg/**
                     **/*.dict
                     **/*.p2o
                     **/*.wrx
                     **/*.png
                     **/*.jpg
                     **/*.gif
                     **/*.bmp
                     **/*.ico
                     app/**
                     ddl/**" 
           manifest="${manifest.dir}/${manifest_appname}.mf" 
           zip64Mode="as-needed" 
           index="true" />

Using a build/profiles/<profile_name> folder where each conversion is stores is the way to go at the moment.

Each profile will run using the ConversionDriver and should create a jar (1 profile = 1 jar), the name of the jar should be profile.getName() + ".jar".

Although this isn't an unreasonable approach, I think that we should not place hard limits on how the converted application is packaged into jars.

The Evaluator from #7169-35 will check if the jar exists and mark the node as finished if necessary.

I don't think we can do this solely on the basis of whether a jar exists or not. The contents matter. This is where some integration with #9638 and #9639 needs to happen.

I am unsure how to proceed and create the jar for each profile. The jar should include the following:
[...] and this means I will have to keep all the conversion files, but have them named based on the profile used.

The jar is just packaging. In the past we have left that packaging to the ant build script.

Ultimately, before we worry about how we jar things, we need to decide what determines whether a given profile has the binary results present.

Constantin: What are your thoughts here?

Greg Shah wrote:

Each profile will run using the ConversionDriver and should create a jar (1 profile = 1 jar), the name of the jar should be profile.getName() + ".jar".

Although this isn't an unreasonable approach, I think that we should not place hard limits on how the converted application is packaged into jars.

The Evaluator from #7169-35 will check if the jar exists and mark the node as finished if necessary.

I don't think we can do this solely on the basis of whether a jar exists or not. The contents matter. This is where some integration with #9638 and #9639 needs to happen.

I can make use of the incremental conversion check in the Evaluator at the moment:

                  if (force || 
                      mgr.getTreeId(file) == AstManager.INVALID_ID || 
                      ConversionData.mustConvertFile(file)         ||
                      ConversionData.containsModifiedSchemaTable(file, schemaTablesChanged))

This can be temporary, so that I can test the conversion a bit more. Profile validation works properly, I am adding the files to a collection and calculating the total length (it will throw an error if the total length doesn't match the size of the collection).

I've made the change mentioned in #7169-46 and noticed some issues with the conversion, that would be the AstManager which resets after each one.

There are a lot of issues related to the conversion database.

I committed 7169a/16354. Added DependencyConversionDriver. Splitting the conversion process is something that I need to think over more since it is not working properly and as mentioned before, #9638 and #9639 will be needed.

Rebasing has a tiny issue because trunk and 7169a both added src/com/goldencode/artifacts. I got the branch into this state:

ddf@ddfmachine:~/gcd/branches/7169a$ bzr status
removed:
  src/com/goldencode/artifacts/
  src/com/goldencode/artifacts/Artifact.java
...
added:
  src/com/goldencode/artifacts/
  src/com/goldencode/artifacts/Artifact.java
...
modified:
  src/com/goldencode/p2j/cfg/Configuration.java
  src/com/goldencode/p2j/cfg/ProfileConfig.java

I'll create 7169b instead of trying to recover this branch, I can't revert specific files that are just removed and added because of an error:

brz: ERROR: Key b'artifacts-20230506140925-1g1xofytqx1dv9cv-1' is already present in map

This is how I managed to rebase the branch:

bzr rebase ../trunk

ddf@ddfmachine:~/gcd/branches/7169a$ bzr status
added:
  src/com/goldencode/artifacts/
  src/com/goldencode/artifacts/DependencyEvaluator.java
  src/com/goldencode/artifacts/DependencyManager.java
  src/com/goldencode/artifacts/DependencyNode.java
renamed:
  src/com/goldencode/artifacts/ => src/com/goldencode/artifacts.moved/
modified:
  src/com/goldencode/p2j/cfg/Configuration.java
  src/com/goldencode/p2j/cfg/ProfileConfig.java
conflicts:
  Conflict adding file src/com/goldencode/artifacts.  Moved existing file to src/com/goldencode/artifacts.moved.

1. cp -r src/com/goldencode/artifacts src/com/goldencode/artifacts.7169a
2. bzr revert src/com/goldencode/artifacts.moved/
3. mv src/com/goldencode/artifacts.7169a/* src/com/goldencode/artifacts
4. rm -rf src/com/goldencode/artifacts.moved
5. rm -rf src/com/goldencode/artifacts.7169a
6. bzr add src/com/goldencode/artifacts/
7. bzr resolve src/com/goldencode/artifacts
8. bzr rebase-continue

If you continue to get in the next commits:
Conflict adding files to src/com/goldencode/artifacts.  Created directory. (if more files were added)

just repeat the above

Alexandru proposed to have this documented in a wiki.

I'm not exactly sure what you are trying to do with this process. It does worry me that you are going to lose all history for the src/com/goldencode/artifacts/ package and all files in it. If so, then this approach is not OK.

Greg Shah wrote:

I'm not exactly sure what you are trying to do with this process. It does worry me that you are going to lose all history for the src/com/goldencode/artifacts/ package and all files in it. If so, then this approach is not OK.

I did not lose the history of src/com/goldencode/artifacts, the issue was that trunk and 7169a added artifacts folder while files from 7169a should have only been added.

This is the status of the rebased branch after rebasing to trunk/16454:

ddf@ddfmachine:~/gcd/branches/7169a$ bzr status -r 16454..
added:
  src/com/goldencode/artifacts/DependencyEvaluator.java
  src/com/goldencode/artifacts/DependencyManager.java
  src/com/goldencode/artifacts/DependencyNode.java
  src/com/goldencode/p2j/convert/DependencyConversionDriver.java
  src/com/goldencode/p2j/convert/JobDefinition.java
modified:
  src/com/goldencode/p2j/cfg/Configuration.java
  src/com/goldencode/p2j/cfg/ProfileConfig.java
  src/com/goldencode/p2j/convert/TransformDriver.java

• The profile dependency config (depends can be used on a profile definition)
• Profile dependencies are validated
• The DependencyManager builds a dependency graph (DAG)
• The evaluator, how to conversion resolves converted profiles.

I've made a few adjustments after the last rebase and I still take to take care of a few issues with the conversiondatabase.

I've resumed the work on this task and I've successfully made a way to convert profiles and their dependencies by creating separate ConversionDriver processes for each individual profile, a conversion process of a profile will not start if there are unconverted dependencies.

• The ConversionDriver will write to the same cvtdb each time.

• Resource allocation to ensure big modules will convert properly and not run out of memory.

My goal is to have the conversion of each profile in their own folder up and running, with stable naming across all profiles.

The ConversionDriver will write to the same cvtdb each time.

The cvtdb should be common. I don't think we want multiple (per-app) cvtdbs.

Greg Shah wrote:

The ConversionDriver will write to the same cvtdb each time.

The cvtdb should be common. I don't think we want multiple (per-app) cvtdbs.

Currently we wipe the cvtdb each time we start a conversion (ConversionData.cleanAll()), even if we run incremental conversion we will end up clearing tables when we start multiple conversions. This is because ConversionData was created to handle a single conversion.

• Run ConversionDriver on different processes and have different conversion databases for each profile, we use a shared database for determining the ast ids for the registry or anything else that might be important down the line.

• If one profile conversion breaks, we will have to back out all the data for that particular profile from the database (When we fail an incremental conversion, we usually have to reconvert the whole application, so we have no code in place for backing just the 'bad/incomplete' stuff).
• Incremental conversion will take a hit in speed, since it relies a lot on the conversion database data, so if we have to run incremental conversion for multiple profiles this will be a problem.

• added:
  • src/com/goldencode/artifacts/DependencyException.java: Exception for profile dependency errors
  • src/com/goldencode/artifacts/DependencyProfile.java: an interface to avoid using ProfileConfig in the artifacts package (this may change in the future).
• modified:
  • src/com/goldencode/artifacts/ArtifactManager.java: cleanup necessary to run multiple conversions
  • src/com/goldencode/artifacts/DependencyEvaluator.java: replaced the ProfileConfig
  • src/com/goldencode/artifacts/DependencyManager.java: validates profile constraints for all profiles (cycle, calculates conversion order)
  • src/com/goldencode/artifacts/DependencyNode.java: added a flag to ensure a node is never queued twice
  • src/com/goldencode/p2j/cfg/Configuration.java: added a change from 11156a (#11156)
  • src/com/goldencode/p2j/cfg/ProfileConfig.java: Removed dependencyProfiles.
  • src/com/goldencode/p2j/convert/ConversionDriver.java: added getModes() to retrieve mods
  • src/com/goldencode/p2j/convert/DependencyConversionDriver.java: removed file-specs/call graph mode, resolves the profile filesets and checks for duplicate files, only uses incremental currently, profiles run 1 by 1 currently.
  • src/com/goldencode/p2j/convert/JobDefinition.java: updated to match trunk
  • src/com/goldencode/p2j/convert/TransformDriver.java: changes to fit the conversion, properly cleanup sources, and throw errors
  • src/com/goldencode/p2j/pattern/AstProcessorJob.java: changes to allow the conversion to work when profiles change
  • src/com/goldencode/p2j/uast/ScanDriver.java: ensure a cleanup even when the scan fails
  • src/com/goldencode/p2j/uast/SymbolResolver.java: replace the sources of a previous conversion job

Changes were committed to 7169a/16785.

My next goal is to have two profiles converted at the same time and not sequential, this will be a bit more complex to implement because I have to check what resources will need to be shared between different ConversionDrivers.

Danut: Please post a summary our discussion from yesterday's meeting.

One ConversionDriver will convert a profile, profiles that do not depend on each other will be converted separately and profiles that have dependencies will have to wait for those to be converted. To ensure a smooth conversion/incremental conversion, we are going to store data in the conversion database (the idea was to store the name of the profile converted and a "conversion number") to make sure we are able to clean up data of a failed profile, fix the code, and resume the conversion where we left at. Sharing the database between two concurrent conversions is also a problem that needs to be addressed.

Every table will have the addition of columns to record the profile and ordinal of the run. This should yield a unique key that allows any rows from a failed run to be removed, leaving the database intact, even on a failed run. The developer will be able to fix any issues and pick up where it last successfully completed a profile.

Committed 7169a/16804, allow conversion for each profile and maintain the profile/ordinal in the conversion database (allows smooth cleanup and change detection).

Rebased 7169a to latest trunk/16804, the branch is now at revision 16810.

code-review reported differences with LegacySignatures, this was already fixed in trunk/16801.

Currently running conversion in parallel for profiles is working properly (including sharing the database), I am still expanding scenario and let AI run the suite of tests while I address code-review. The suite run conversion/incremental conversion with ConversionDriver to ensure backwards compatibility (so that current conversions will not break on an update) and conversion/incremental conversion with the DependencyConversionDriver. The process was smooth and I got to the point where there were a few issues unrelated to the branch that were found (basically the runs were inconsistent with the sequential profile conversion), some of the issues:
- Stable temp-table P2O lookup (guess I did not manage to fix them up properly in #11720), this caused wrong temp-table queries (no WHOLE-INDEX used, used PreselectQuery instead of AdaptiveQuery with the index)
- Incremental reconversion generated a lower cased java class for a .cls instead of the expected class name (this could cause some issues when compiling the project)
- Duplicate AST registry entries for parent classes, this means two tree ids for the same file in any conversion where a class has a subclass (this should be a regression from #7180 because of the SourceArtifact usage).

The goal of the suite created using AI was to run conversion and ensure it is consistent after multiple runs (in parallel, incremental) and it was nice to catch those problems.

• Properly define the conversion status when running the DependencyConversionDriver (console logging the progress of the conversion in detail without mixing ConversionDriver)
• Properly define the ant target and properties, also document them in this task.
• Creating a tool to automatically split project sources into profiles which will allow smoother testing without a lot of manual work for most of the applications. This should be compatible with #5000 or at least be created as a reusable base.
• Testing multiple small projects then move on to bigger projects.
• Maintain support for ConversionDriver and ensure it continues to work.

New changes are available in 7169a/16811-16812 related to concurrent profile conversion and profile splitting (subject to change) based on the available sources.

Using the DependencyConversionDriver¶

com.goldencode.p2j.convert.DependencyConversionDriver converts a project split into configuration profiles which depend on each other. Each profile is converted by its own
ConversionDriver job, after all the profiles it depends on were converted. All the jobs run in one JVM and share the conversion database (cvt/cvtdb) and the AST registry, so a
profile sees the conversion results of the profiles it depends on and does not convert them again. With -jobs=n, up to n independent profiles are converted at the same time.

1. Define the profiles¶

Each profile is a <profile> in cfg/p2j.cfg.xml with its own <file-set>. The depends attribute lists, comma separated, the profiles it needs:

  <profile name="dep_base">
     <file-set>
        <directory recursive="false" path="./abl/dep/" spec="base_*.p" />
        <directory recursive="false" path="./abl/dep/" spec="Base*.cls" />
     </file-set>
  </profile>

  <profile name="dep_common" depends="dep_base">
     <file-set>
        <directory recursive="false" path="./abl/dep/" spec="common_*.p" />
     </file-set>
  </profile>

  <profile name="dep_app" depends="dep_common, dep_report">
     <file-set>
        <directory recursive="false" path="./abl/dep/" spec="app_*.p" />
     </file-set>
  </profile>
  

Rules, checked before any conversion starts:

• A file belongs to the file-set of one profile only; otherwise: File '...' is used by both profile 'a' and profile 'b'.
• The depends graph has no cycle, names only existing profiles, and a profile can't depend on itself.
• A profile may use only files of its own profile or of profiles it depends on, directly or transitively. For super classes and interfaces this is an error naming the fix (add the
  owner to depends, or move the class to a profile both depend on). A file of a profile already converted or up to date, found depending on a changed file of another profile, is
  reported as an undeclared dependency with a warning.
• These parameters are read once for the whole JVM, so no profile of the run may override their global value: cvtpath, name-map, basepath, output-root, oo-skeleton-path,
  datanames, codenames, error-dest, warn-dest, case-sensitive, registry, binary-ast, unix-escapes. Any other parameter (including <schema>) can be set per profile.

2. Run it¶

  java -server -Xmx4G -DP2J_HOME=. -cp "p2j/build/lib/*" \
       com.goldencode.p2j.convert.DependencyConversionDriver -d2 -Pdep_app -Pdep_solo -jobs=4 f2+m0+cb
  

• -P<name> is a profile to convert; repeat it for several. The profiles it depends on are added automatically, so -Pdep_app converts dep_base, dep_common, dep_report and
  dep_app.
• -jobs=n is the maximum number of profiles converted at the same time (default 1). All jobs run in this JVM, so size -Xmx for n jobs.
• All the other options (-Dn, -A, -R, -Tn, -I, -numThreads=n) and the mode (F0/F1/F2, M0/M1, CB) mean the same as for ConversionDriver and apply to every
  job.
• No file-specs are accepted: the profiles define the files.
• Mode F3 (call graph) is not supported, as the call graph covers the whole project; use ConversionDriver for it.

An ant target in the project's build.xml can wrap it:

  <target name="convert.dependency" depends="init">
     <java classname="com.goldencode.p2j.convert.DependencyConversionDriver" 
           fork="true" failonerror="true" dir="${basedir}">
        <jvmarg value="-Xmx${dependencyHeap}"/>
        <jvmarg value="-server"/>
        <jvmarg value="-DP2J_HOME=${fwd.home}"/>
        <arg value="-d2"/>
        <arg value="-Pdep_app"/>
        <arg value="-Pdep_solo"/>
        <arg value="-jobs=${dependencyJobs}"/>
        <arg value="f2+m0+cb"/>
        <classpath refid="compile.classpath"/>
     </java>
  </target>
  

3. Scheduling¶

• A profile starts once all the profiles it depends on finished successfully.
• Until the first job completes, jobs run one at a time if the first job drops the database (full conversion) or loads the schemas; the other jobs then reuse those schemas.
• Jobs run at the same time only if their profiles use the same schemas (the same namespaces and import files), as the schemas are kept JVM-wide.
• Inside a job, files are processed by numThreads threads as in ConversionDriver.

4. Full and incremental conversion¶

• Full (no -I): if the database holds no data of files outside the profiles of the run, the first job drops it once; otherwise each job drops only the data of its own files and
  profile, so other profiles' results are kept.
• Incremental (-I): the changes are detected once, before any job runs (changed files, includes, service configurations, schemas, deleted programs). A profile with nothing to
  convert is skipped as up to date. A file of a later profile found depending on a changed file is converted by its own profile's job. A changed schema is processed even if no profile
  needs converting.

5. Output, logs and failures¶

The console shows only the progress and a summary:

  Started conversion of profile dep_base (1/5), log cvt_dep_base_20261027_105044.log
  Finished conversion of profile dep_base (1/5)
  Skipped conversion of profile dep_solo (2/5): up to date
  ...
  Conversion completed: 4 profile(s) converted, 1 up to date.
  

• The full output of each job goes to cvt_<profile>_<start time>.log in the project root; a job which only processes changed schemas and deleted programs logs to
  cvt_schemas_<start time>.log.
• A profile fails when its job throws or a file of the profile fails to convert. The console shows the reason in one line and the stack trace is appended to the profile's log.
• The profiles depending on a failed profile are not converted; independent profiles still are. The summary lists both: Conversion failed for profile(s) [...], not converted as they
depend on them: [...].
• The exit code is 1 if the profiles are invalid or any profile fails, otherwise 0.

Using profile_split.sh¶

p2j/tools/scripts/profile_split.sh proposes a split of a project's 4GL sources into configuration profiles which depend on each other, to be converted by the
DependencyConversionDriver. It runs com.goldencode.p2j.convert.ProfileSplitter, which parses the files, builds their dependency graph and groups them into profiles. It only
writes new files: the project's configuration is not changed and nothing is converted.

1. Run it¶

Run it in the project root (the directory with cfg/p2j.cfg.xml):

  p2j/tools/scripts/profile_split.sh [-x<max_heap>] [-f<path_to_fwd_jars>] [-- <splitter options>]
  

• -x is the maximum heap of the splitter JVM (default 4g).
• -f is the directory with p2j.jar and the FWD libraries (default p2j/build/lib).
• -h shows the usage.
• Everything after -- is passed to the splitter:
  • -prefix=name is the prefix of the profile names (default p); it must start with a letter and hold only letters, digits or _.
  • The files to split, given as for ConversionDriver: -Z <file-set list>, -S <dir> "<spec>", -F <file list>, or -p<profile> (at most one) for the file-set of that profile.
    Without them, the file-set of the default profile is split; if there is none, the split stops with No files to split....
  • Other ConversionDriver options, such as -d2, are passed on to the scan. Do not pass -T<n> with n > 1: a multi-threaded scan fails for .cls files.

Examples:

  p2j/tools/scripts/profile_split.sh -- -d2
  p2j/tools/scripts/profile_split.sh -x 8g -- -d2 -prefix=app
  p2j/tools/scripts/profile_split.sh -- -d2 -S ./abl "*.p" 
  

2. How the files are grouped¶

A file depends on another one through 4GL language constructs only:

• the classes it references;
• the programs it runs with RUN, when the name is a literal;
• the files which define the NEW SHARED objects it uses (variables, temp-tables, work-tables, buffers, frames, streams, menus, queries, browses, datasets);
• the classes and procedures named by its TestSuite annotations.

A RUN VALUE(...) or a RUN from a procedure library can't be resolved and is listed in the report as an unresolved reference. Dependencies made at runtime by application code,
such as ADM2's RUN start-super-proc("adm2/x.p"), are not seen; profiles which need them must declare them by hand.

The target size of a profile, T, is 10% of the files, rounded up, at least one file. Then:

1. Files which depend on each other in a cycle form a cluster and are never split.
2. The base profile (<prefix>_1_base) holds the clusters used by more than one cluster, plus everything they depend on, whatever its size.
3. Every other cluster is used by at most one cluster, so the rest forms trees: an entry point and the files only it uses.
4. A tree larger than T is cut from its leaves up; the rest of the tree depends on the cut piece.
5. The pieces, largest first, join the first profile they fit in within T (the base first), as long as no path joins the two through a third profile; otherwise they start a new
   profile.

A profile is larger than T only when its files can't be separated. The result is checked before anything is written: every file in exactly one profile, no cycle between profiles, and
every file dependency covered by the depends of its profile, directly or transitively. The output is deterministic: the same sources give byte-identical files.

3. Output¶

• cfg/p2j.cfg.profiles.xml: a copy of cfg/p2j.cfg.xml with the proposed profiles, named <prefix>_<N>_<hint> in dependency order, each with a depends attribute and a
  <file-set> which lists its files one by one. The other profiles and all the existing file-sets are dropped, as a file may belong to one profile only. Each profile gets the
  <schema>, <include-mappings> and parameters of the default profile, except the parameters which must keep their global value in a dependency conversion (e.g. cvtpath,
  registry).
• profiles.report.md: the summary; for each profile, why its files were grouped, its size and cost (source plus include bytes), each dependency with a reference which makes it, its
  cycles and its files; the unresolved references; notes about what was dropped from the configuration.
• cvt_split_<stamp>.log: the console output of the run.
• cvt_profiles/: the scratch folder of the scan (its own AST registry and conversion database); it can be deleted after the split. The scan also rewrites post_parse_fixups.log in
  the project root.

The console ends with a summary, for example:

  Profile split:
  9 profiles for 64 files (target size 7 files, 10%), <n> unresolved references.
     p_1_base     8 files
     p_2_abl      7 files  <- p_1_base
     ...
  dependencyProfiles=-Pp_1_base -Pp_2_abl ...
  Wrote cfg/p2j.cfg.profiles.xml and profiles.report.md.

  To convert the profiles:
     cp cfg/p2j.cfg.profiles.xml cfg/p2j.cfg.xml
     java -server -Xmx<heap> -DP2J_HOME=. -cp "p2j/build/lib/*" \
          com.goldencode.p2j.convert.DependencyConversionDriver -d2 -Pp_1_base -Pp_2_abl ... -jobs=<n> f2+m0+cb
  

4. Failures¶

• If any file can't be parsed, the split fails (exit code 1) and lists the files: their dependencies are unknown, so no split is proposed. Fix them or leave them out of the files to
  split.
• When the split fails, cfg/p2j.cfg.profiles.xml and profiles.report.md are not written.

5. Using the result¶

Review profiles.report.md, then follow the printed commands: copy cfg/p2j.cfg.profiles.xml over cfg/p2j.cfg.xml (keep a copy of the original) and convert with the
DependencyConversionDriver, choosing the heap and the number of jobs.

The approach looks really good. Minor request: case-sensitive and unix-escapes should be per-profile.

Committed 7169a/16813.

Rebased 7169a to latest trunk/16826, the branch is now at revision 16835.

Greg, I have a question regarding file_sys.xml Merge-File-System-Configuration. Is it still used?

I don't think we actually use it anywhere. The file_sys.xml is indeed written and available but I think we ignore it in all our tools and project setup. I can't be 100% sure but I don't know of anyone using it.

BTW: please fix your wiki page link to avoid the URL.

Greg Shah wrote:

I don't think we actually use it anywhere. The file_sys.xml is indeed written and available but I think we ignore it in all our tools and project setup. I can't be 100% sure but I don't know of anyone using it.

BTW: please fix your wiki page link to avoid the URL.

I extracted this from one of my sessions during on a small dive:

  - Since 2018 the runtime has read these values from legacy-system instead, so following the wiki does nothing.
  - The choice: make the generator write legacy-system and fix the wiki, or remove both.

I don't think merging the file_sys.xml manually is work we want anyone to do and we moved on from it because of this.

Committed 7169a/16836.

Current changes¶

Known issues¶

Currently checking the issues above to determine if they are still correct and then I will create separate tasks.