FWD » Core Development » Conversion Tools

The profile approach (see #6256) provides the ability to segment a larger code base into named profiles. An important objective of doing this is to be able to convert minimum sets of functionality rather than requiring the entire application to be converted at once. To make this work properly, we must be able to specify dependencies between profiles and the priority or order of conversion within profiles. This is important because converting an arbitrary profile may depend on files in one or more other profiles. If we cannot find the conversion artifacts for a dependency, then we cannot properly convert. This also matters within a profile. When all files are converting in a single batch, it is possible to convert procedures in a simple alphabetical order. When some files are converted separately, we may need control over the ordering such that we can get correct results.

Constantin suggested the following:

As I see it, you need a custom comparator to sort the file list according to some rules.

Greg: for this, a custom comparator could be configured in p2j.cfg.xml, and if that parameter exists (and if the class can be loaded), then sort the program list using that. It's just a matter to write the comparator in Java, add it to a jar and to the conversion classpath.

But I don't think this can work with the .cls files, as those need some special processing in FWD, only with the plain .p programs.

The comparator is an interesting idea. It certainly would provide a great deal of customizable control. However, it brings with it:

• Extra complexity because now we have to have extra code that is compiled and found at conversion time.
• Dependencies are hidden inside whatever Java code is written, rather than being made explicit in configuration.

For this last reason especially, I prefer to avoid the comparator approach.

For dependencies between profiles, I think we should a add a depends element which encodes one or more profiles on which this profil depends. In other words, that list of profiles must be converted first. A key idea here is this must be an acyclic directed graph. There must be no direct or indirect cycles (profiles which depend upon themselves directly or indirectly).

For ordering within a given profile, I think we can just allow the order of the file-set to define the order of conversion. This may mean that the file-set will no longer be a simple directory spec, but I think that is an acceptable trade off. Breaking out the entries and ordering them explicitly makes the result obvious since it is encoded explicitly in the configuration.

Today we just sort alphabetically, but the only real advantage to that is that it is easy for a human to see where in the overall code the processing is occurring. That is useful but not critical. The idea of dependencies must break that alphabetical ordering anyway, so there is no extra loss here.

For OO we must process the objects in their object graph order. We already do this, but we will need changes to handle it across profiles.

Greg, if I understand correctly, your suggestion is to order the non-OO file list using the include/exclude filters from the file-set in p2j.cfg.xml, for whatever profile gets converted. A problem I see here, is we use the include/exclude filters for example to exclude an entire folder and just cherry-pick some files from the excluded folder, with include filters, after that. Better said: there is already a semantics for the order of include and exclude filters in a file-set. In any case, it should not be a problem to enable sorting using the order of the include filters in the file-set.

Are profiles already supported today ? And if so, is there an explanation on how to use them ?

Greg, if I understand correctly, your suggestion is to order the non-OO file list using the include/exclude filters from the file-set in p2j.cfg.xml, for whatever profile gets converted. A problem I see here, is we use the include/exclude filters for example to exclude an entire folder and just cherry-pick some files from the excluded folder, with include filters, after that. Better said: there is already a semantics for the order of include and exclude filters in a file-set. In any case, it should not be a problem to enable sorting using the order of the include filters in the file-set.

I'm more focused on the directory and filename directives than the include and exclude filters. The excludes we don't have to worry about because those files are never converted so the order of non-conversion doesn't matter. I agree that there might be cases when we want a different order than can be expressed with a minimal file-set definition. A brute force solution to that problem is to more explicitly list directory or filename entries that otherwise would not be needed.

If we find that to be a common case, we can consider adding other syntax but for now I prefer if we can solve this using the existing approach.

Are profiles already supported today ? And if so, is there an explanation on how to use them ?

Yes, these are supported today. Unfortunately, we have not yet documented the features. You can see the discussions including some configuration syntax in #6256 and #6253.

These profiles might be something that can help me. A couple of questions :

• Can the propath include directories outside the file sets inside the profile ?
• How are these profiles processed during a conversion ? Is it in the order they are specified in the config file ? So that it has the same effect as what I did by naming the folders with number prefixes so that they are alphabetically in the correct dependency order ? ( projects with least dependencies first )

To explain this a bit, now I have folders like this :

a005zzz/src/api
a006zzz/src/impl
a007bbb/src/api
a008bbb/src/impl

The above guaranties that we can convert a007bbb/src/api separately ( as long as we also include a006 and a005 and everything above it ) and then use the result in a full conversion environment. This only works if we don't make temp table changes that conflict with projects depending on a007. This is already a big help for us, but the a unique consistent naming of DMOs would help us much more of course.

btw, for now we could live with a schema shared between the profiles.
Would the profiles name the DMOs with a namespace ? That could make the DMO names more unique ?

Temp table DMOs I mean.

I don't know why, but I can't modify my own comments in this ticket...

Can the propath include directories outside the file sets inside the profile ?

Yes, but in practical terms this is mostly useful for include files. Our OO approach still requires all referenced OO .cls files to be converted in a single batch.

How are these profiles processed during a conversion ? Is it in the order they are specified in the config file ?

No, today the code builds up an overall list of files to process and alphabetizes them all into a single list. My idea is to specify the dependencies between profiles and ensure that when you convert profile C which depends on B (and B depends on A), that we:

1. Have access to the converted code for B and A such that we can read any needed conversion details from the artifacts in the jar. OR
2. Convert the missing profiles in the proper order. If B is missing but A exists, then we convert B before C. If A is missing, then we convert A before whatever we do for B or C. And so forth...

So that it has the same effect as what I did by naming the folders with number prefixes so that they are alphabetically in the correct dependency order ? ( projects with least dependencies first )

For now it is a way to specify an app-level configuration but it doesn't cleanly resolve the various other issues of converting app by app.

Would the profiles name the DMOs with a namespace ?

It does not do this today.

That could make the DMO names more unique ?

Eric?

Greg Shah wrote:

No, today the code builds up an overall list of files to process and alphabetizes them all into a single list. My idea is to specify the dependencies between profiles and ensure that when you convert profile C which depends on B (and B depends on A), that we:

1. Have access to the converted code for B and A such that we can read any needed conversion details from the artifacts in the jar. OR
2. Convert the missing profiles in the proper order. If B is missing but A exists, then we convert B before C. If A is missing, then we convert A before whatever we do for B or C. And so forth...

This would indeed make it possible to do a conversion of a single app. Especially if you combine this with putting DMOs in the namespace of the app wher it is defined in. I think this is the right solution.

We should implement a dependency manager to track dependencies and allow cascading adds and deletes from the conversion list (see ##7180 and #7179).

The OO processing today does part of this job, for the object graph. It would be moved into this external dependency manager.

I think the best start would be to lay down the foundation, implement the profile dependencies and use the acyclic directed graph structure to confirm that there are no profiles that depend directly or indirectly on themselves.

Sounds good. All config should be in p2j.cfg.xml, primarily (maybe exclusively) in the profile sections.

Greg Shah wrote:

Sounds good. All config should be in p2j.cfg.xml, primarily (maybe exclusively) in the profile sections.

I've added the following:

   <profile name="test1" depends="test2,test3">
      <file-set filename="zfile_set.txt" />
   ...

Committed 7169a/16351-16352. Added profile configuration dependencies.

• ProfileConfig contains a set of dependency profile names which will end up being used for populating the dependencyProfiles collection after being validated.
• Configuration has getValidProfileDependencies now, a method used to populate the dependencyProfiles in ProfileConfig. This method also returns a LinkedList of profiles ordered in topological order based on which has less dependencies (this can be used in the future to determine what profiles we convert first, a good start)

1. A user will be able to define a profile dependency, but what if the same file is present in multiple profiles? Do we scan the profile fileset to check if such issues exist when we validate the profile?
2. When a profile is dependent on another one, it must be added to the conversion process even if that profile was not specified. Do we just turn the conversion into a multi-profile one or split the conversion into multiple threads?
3. We have an order for the profiles depending on the in degree from the graph in 7169, this would allow checking which profiles have the 0 in degree and convert them first, then the conversion graph is recalculated to remove converted profiles.
4. How should I decide which 0 degree profile gets converted first, maybe based on the amount of files (less -> more)?

Please create the class DependencyManager in the artifacts package. That is the class that should track the dependency graph and calculate the order of operations. The dependency manager deals with profiles, not specific artifacts.

We will implement the dependencies as a directed acyclic graph.

• Each profile is a node in the graph.
• Each dependency is an edge that points from one profile/node to one other profile/node.
• Nodes that are not referenced as a dependency by any other node:
  • If they have dependencies defined, then they would be "leaf nodes" which can only be processed at the end of a dependency chain.
  • If they have no dependencies defined, are "independent nodes" that play no part in the dependency graph.
• Nodes that are referenced as a dependency, but which themselves do not define a dependency, are "root nodes", at which all dependency operations start.
• Nodes which have dependencies defined and which themselves are defined as dependency by at least one other node, are "intermediate nodes" that make up the internal traversal of the graph.
• Graph traversal can be calculated for any list of nodes as follows:
  • API Queue<DependencyNode> DependencyManager.calculate(List<DependencyNode>)
    • The parameter is the list of nodes for which the traversal must be calculated.
    • The return value is the queue from which the caller can obtain jobs that are ready to process.
  • Iterate through the list of nodes, for each node:
    • If the node has no dependencies (independent or root nodes), then it can be added to the "may be ready to process" list and the ready flag of the node should be set to true.
    • If the node has dependencies (leaf or intermediate nodes), then:
      • The node should be added to the notification list for each of its dependency nodes.
      • The list of dependencies should be copied into a Set named pending.
      • Its ready flag should be marked false.
  • Iterate through the "may be ready to process list" and for each node:
    • Detect if the node requires processing.
      • In practice, the caller will have to provide an instance of a functional interface that reports this fact. Since the processing of a node (e.g. conversion) is handled by the caller, the calculation about whether something has been processed or not (e.g. an up to date version of the jar for those resources is available) must also be handled by the caller.
      • If the node needs processing, add it to the "ready to process" queue.
      • If the node does not need processing (i.e. it is already available), then notify all nodes in its notification list that "processing has completed".
  • When a node is notified that a dependency has had processing completed, remove that node name from the pending set. If the set is empty, mark the node's ready flag as true and add the node to the queue.
  • We should have a process that checks the graph for invalid conditions (duplications and cycles).
• The caller of calculate() will recieve a queue back. The contents are the jobs which are currently ready to process. For each job pulled from the queue, the caller will do some work (e.g. conversion) and when complete, it will call the "finished()@ method on the node, which will notify all nodes in its notification list that "processing is complete".

A user will be able to define a profile dependency, but what if the same file is present in multiple profiles? Do we scan the profile fileset to check if such issues exist when we validate the profile?

I think we should implement a restriction that if a profile is used with dependency management, then there must not be any overlap between file sets in different profiles. Every file must only be converted in one profile.

When a profile is dependent on another one, it must be added to the conversion process even if that profile was not specified. Do we just turn the conversion into a multi-profile one or split the conversion into multiple threads?

Probably both. In #3211, we are making the PatternEngine multithreaded. When that is merged to trunk, then the conversion of a single profile will map to a single ConversionDriver run which will process one profile in a multi-threaded manner.

We probably need a new DepenencyConversionDriver which would interact with the DependencyManager, call calculate() and process the returned queue. Each node in the queue would be a profile that would gets its own ConversionDriver run. Each of those should be handled in parallel up to some configured number of active jobs.

We have to discuss if the separate ConversionDriver runs are separate processes.

We have an order for the profiles depending on the in degree from the graph in 7169, this would allow checking which profiles have the 0 in degree and convert them first, then the conversion graph is recalculated to remove converted profiles.

My previous note should answer this.

How should I decide which 0 degree profile gets converted first, maybe based on the amount of files (less -> more)?

For now, I was planning a naive approach of just taking the jobs from the queue in their natural order. We can refine it later.

• DependencyNode.java contains the profile (ProfileConfig instance), three sets of DependencyNode instances representing outgoing dependencies, incoming dependencies, and pending dependencies (that did not complete processing), a flag that indicates if the node can be processed;
  • dependencies can be added, but not removed (if there are issues when adding the dependencies, then we exit the program)
  • dependencies and dependents can be retrieved as unmodifiable sets.
  • isIndependent() and isRoot() methods
  • dependencies are removed after being processed, so that the pending collection can be used to set the ready flag.
  • finished() that removes the current node from pending
• DependencyEvaluator is a functional interface which defines a boolean shouldProcess(ProfileConfig profile), I am using it in the DependencyManager as:
  

     private final DependencyEvaluator evaluator = new DependencyEvaluator()
     {
        @Override
        public boolean shouldProcess (ProfileConfig profile)
        {
           return true;
        }
     };
  

• DependencyManager.java contains a private final Map<ProfileConfig, DependencyNode> nodes = new LinkedHashMap<>();, it can also be a Set<DependencyNode>.
  • The DependencyManager is initialized with a set of ProfileConfig instances, parsed using buildGraph() (populates the nodes and also dependencies for each node). There's also a hasCycle() method to confirm that we are using a DAG.
  • calculate() that takes a list of DependencyNode instances, intializes the pending for each node and then uses the evaluator to build the return queue.

Still thinking on the implementation and making corrections based on what I understood from #7169-33 / #7169-34.

It sounds good so far. Please note that if we need to directly reference ProfileConfig then that class should not be in the FWD-specific packages (com.goldencode.p2j.*). Either it should move to some subpackage of com.goldencode or it should be abstracted as an interface that ProfileConfig will implement.

Greg Shah wrote:

It sounds good so far. Please note that if we need to directly reference ProfileConfig then that class should not be in the FWD-specific packages (com.goldencode.p2j.*). Either it should move to some subpackage of com.goldencode or it should be abstracted as an interface that ProfileConfig will implement.

Every class will go into com.goldencode.artifacts.

I committed my changes to 7169a/16353.

The next step should be to evaluate the file that will be converted by each profile and to make sure there are no conflicts, maybe split the conversion so that a jar is generated for each converted profile and use that to check when the other profiles can be processed.

• All profiles are automatically added to the graph, but what if the conversion uses just a few profiles? Should the DAG be made for the active profiles, or should each profile be included and verified that no duplicate files exist?

The dependency manager will only process profiles that are explicitly part of the list provided by the caller OR which are dependencies of those profiles.

• process the command line arguments (the profiles and modes), the ant command I used is

     <target name="convert.profile" 
             depends="init" 
             description="test">
        <record name="cvt_${LOG_STAMP}.log" action="start"/>
        <java classname="com.goldencode.p2j.convert.DependencyConversionDriver" 
              fork="true" 
              failonerror="true" 
              dir="${basedir}">
           <jvmarg value="-server"/>
           <jvmarg value="-Xmx${conversionHeap}"/>
           <jvmarg value="-XX:-OmitStackTraceInFastThrow"/>
           <jvmarg value="-DP2J_HOME=${fwd.home}"/>
           <jvmarg value="-Xdebug"/>
           <jvmarg value="-Xnoagent"/>
           <jvmarg value="-Djava.compiler=NONE"/>
           <jvmarg value="-Xrunjdwp:transport=dt_socket,address=2100,server=y,suspend=y"/>
           <arg value="-ptest1"/>
           <arg value="-ptest2"/>
           <arg value="f2+m0+cb"/>
           <classpath refid="compile.classpath"/>
        </java>
        <record name="cvt_${LOG_STAMP}.log" action="stop"/>
     </target>

• initialize the profiles (build the dependencies for each ProfileConfig and check if those exist)
• validate the used profiles (check the file list of each profile for duplicate files)

Next is the job execution, where I have to convert those profiles separately. Coming up with ideas now.

• The DependencyConversionDriver is an independent class that does not extend the TransformDriver, this is because each profile will have the ConversionDriver execute the conversion.
• The command line arguments are processed to build a SharedJobConfiguration. Each profile will require a conversion and in turn, a JobDefinition. The SharedJobConfiguration will be used to initialize the JobDefinition for each conversion that will be executed.
• Again, at the same time the arguments are processed, the ProfileConfig dependencies are built and the DependencyManager is initialized with the configuration profiles + the their dependencies depending on the provided arguments.
• Each profile will have a different job, so there will be x number of threads that will run the conversion depending on the threads available in the pool.

1. Each profile will run using the ConversionDriver and should create a jar (1 profile = 1 jar), the name of the jar should be profile.getName() + ".jar".
2. The Evaluator from #7169-35 will check if the jar exists and mark the node as finished if necessary.

I am unsure how to proceed and create the jar for each profile. The jar should include the following:

       <jar jarfile="${build.home}/lib/${appname}.jar" 
           basedir="${build.home}/classes" 
           excludes="**/*.template" 
           includes="**/${pkgroot}/**
                     dir_schema_ext.xml
                     ehcache.xml
                     *.jasper
                     **.xml
                     cfg/**
                     **/*.dict
                     **/*.p2o
                     **/*.wrx
                     **/*.png
                     **/*.jpg
                     **/*.gif
                     **/*.bmp
                     **/*.ico
                     app/**
                     ddl/**" 
           manifest="${manifest.dir}/${manifest_appname}.mf" 
           zip64Mode="as-needed" 
           index="true" />

Using a build/profiles/<profile_name> folder where each conversion is stores is the way to go at the moment.

Each profile will run using the ConversionDriver and should create a jar (1 profile = 1 jar), the name of the jar should be profile.getName() + ".jar".

Although this isn't an unreasonable approach, I think that we should not place hard limits on how the converted application is packaged into jars.

The Evaluator from #7169-35 will check if the jar exists and mark the node as finished if necessary.

I don't think we can do this solely on the basis of whether a jar exists or not. The contents matter. This is where some integration with #9638 and #9639 needs to happen.

I am unsure how to proceed and create the jar for each profile. The jar should include the following:
[...] and this means I will have to keep all the conversion files, but have them named based on the profile used.

The jar is just packaging. In the past we have left that packaging to the ant build script.

Ultimately, before we worry about how we jar things, we need to decide what determines whether a given profile has the binary results present.

Constantin: What are your thoughts here?

Greg Shah wrote:

Each profile will run using the ConversionDriver and should create a jar (1 profile = 1 jar), the name of the jar should be profile.getName() + ".jar".

Although this isn't an unreasonable approach, I think that we should not place hard limits on how the converted application is packaged into jars.

The Evaluator from #7169-35 will check if the jar exists and mark the node as finished if necessary.

I don't think we can do this solely on the basis of whether a jar exists or not. The contents matter. This is where some integration with #9638 and #9639 needs to happen.

I can make use of the incremental conversion check in the Evaluator at the moment:

                  if (force || 
                      mgr.getTreeId(file) == AstManager.INVALID_ID || 
                      ConversionData.mustConvertFile(file)         ||
                      ConversionData.containsModifiedSchemaTable(file, schemaTablesChanged))

This can be temporary, so that I can test the conversion a bit more. Profile validation works properly, I am adding the files to a collection and calculating the total length (it will throw an error if the total length doesn't match the size of the collection).

I've made the change mentioned in #7169-46 and noticed some issues with the conversion, that would be the AstManager which resets after each one.

There are a lot of issues related to the conversion database.

I committed 7169a/16354. Added DependencyConversionDriver. Splitting the conversion process is something that I need to think over more since it is not working properly and as mentioned before, #9638 and #9639 will be needed.

Rebasing has a tiny issue because trunk and 7169a both added src/com/goldencode/artifacts. I got the branch into this state:

ddf@ddfmachine:~/gcd/branches/7169a$ bzr status
removed:
  src/com/goldencode/artifacts/
  src/com/goldencode/artifacts/Artifact.java
...
added:
  src/com/goldencode/artifacts/
  src/com/goldencode/artifacts/Artifact.java
...
modified:
  src/com/goldencode/p2j/cfg/Configuration.java
  src/com/goldencode/p2j/cfg/ProfileConfig.java

I'll create 7169b instead of trying to recover this branch, I can't revert specific files that are just removed and added because of an error:

brz: ERROR: Key b'artifacts-20230506140925-1g1xofytqx1dv9cv-1' is already present in map

This is how I managed to rebase the branch:

bzr rebase ../trunk

ddf@ddfmachine:~/gcd/branches/7169a$ bzr status
added:
  src/com/goldencode/artifacts/
  src/com/goldencode/artifacts/DependencyEvaluator.java
  src/com/goldencode/artifacts/DependencyManager.java
  src/com/goldencode/artifacts/DependencyNode.java
renamed:
  src/com/goldencode/artifacts/ => src/com/goldencode/artifacts.moved/
modified:
  src/com/goldencode/p2j/cfg/Configuration.java
  src/com/goldencode/p2j/cfg/ProfileConfig.java
conflicts:
  Conflict adding file src/com/goldencode/artifacts.  Moved existing file to src/com/goldencode/artifacts.moved.

1. cp -r src/com/goldencode/artifacts src/com/goldencode/artifacts.7169a
2. bzr revert src/com/goldencode/artifacts.moved/
3. mv src/com/goldencode/artifacts.7169a/* src/com/goldencode/artifacts
4. rm -rf src/com/goldencode/artifacts.moved
5. rm -rf src/com/goldencode/artifacts.7169a
6. bzr add src/com/goldencode/artifacts/
7. bzr resolve src/com/goldencode/artifacts
8. bzr rebase-continue

If you continue to get in the next commits:
Conflict adding files to src/com/goldencode/artifacts.  Created directory. (if more files were added)

just repeat the above

Alexandru proposed to have this documented in a wiki.

I'm not exactly sure what you are trying to do with this process. It does worry me that you are going to lose all history for the src/com/goldencode/artifacts/ package and all files in it. If so, then this approach is not OK.

Greg Shah wrote:

I'm not exactly sure what you are trying to do with this process. It does worry me that you are going to lose all history for the src/com/goldencode/artifacts/ package and all files in it. If so, then this approach is not OK.

I did not lose the history of src/com/goldencode/artifacts, the issue was that trunk and 7169a added artifacts folder while files from 7169a should have only been added.

This is the status of the rebased branch after rebasing to trunk/16454:

ddf@ddfmachine:~/gcd/branches/7169a$ bzr status -r 16454..
added:
  src/com/goldencode/artifacts/DependencyEvaluator.java
  src/com/goldencode/artifacts/DependencyManager.java
  src/com/goldencode/artifacts/DependencyNode.java
  src/com/goldencode/p2j/convert/DependencyConversionDriver.java
  src/com/goldencode/p2j/convert/JobDefinition.java
modified:
  src/com/goldencode/p2j/cfg/Configuration.java
  src/com/goldencode/p2j/cfg/ProfileConfig.java
  src/com/goldencode/p2j/convert/TransformDriver.java

• The profile dependency config (depends can be used on a profile definition)
• Profile dependencies are validated
• The DependencyManager builds a dependency graph (DAG)
• The evaluator, how to conversion resolves converted profiles.

I've made a few adjustments after the last rebase and I still take to take care of a few issues with the conversiondatabase.