FWD » Core Development » Conversion Tools

To translate the above configuration requirements into a sample p2j.cfg.xml, I understand it should look something like this:

<?xml version="1.0"?>

<!-- P2J main configuration -->
<cfg>

   <global>

      <!-- default conversion tools "internal" values -->
      <parameter name="P2J_RULES"        value="${P2J_HOME}/p2j/rules" />
      <parameter name="patpath"          value=".:${P2J_HOME}/pattern:${P2J_RULES}/include:${P2J_RULES}:" />
      <parameter name="registry"         value="./cfg/registry.xml" />
      <parameter name="rootlist"         value="./cfg/default_rootlist.xml" />    
      <parameter name="matchlist"        value="./cfg/default_matchlist.xml" />
      <parameter name="datanames"        value="./cfg/default_datanames.xml" />

      <!-- default values from the original system on which Progress 4GL ran -->
      <parameter name="propath"          value="${P2J_HOME}:${P2J_HOME}/abl/default/path/1:${P2J_HOME}/abl/default/path/2:${P2J_HOME}/abl/default/path/3:" />
      <parameter name="basepath"         value="./abl" />
      <parameter name="include-spec"     value="*.[fhiv]" />
      <parameter name="oo-skeleton-path" value="./skeleton" />

      <!-- conversion output -->
      <parameter name="output-root"      value="./src" />
      <parameter name="pkgroot"          value="com.acme" />
      <parameter name="merge-point"      value="/server/default/runtime/default/" />
      <parameter name="comments"         value="true" />
      <parameter name="foreign-keys"     value="false" />
      <parameter name="opsys"            value="UNIX" />

      <!-- default file-set -->
      <file-set>
         <directory recursive="true" path="./abl/" spec="(*.[pPwWtT]|*.cls)" />
         <exclude filter="./abl/possenet/*" />
      </file-set>

   </global>

   <schema>

      <namespace name="acme" default="true" />
      <namespace name="standard" default="true" />

      <metadata name="standard">
         <table name="_area" />
         <table name="_db" />
         <table name="_file" />
         <table name="_file-trig" />
         <table name="_field" />
         <table name="_field-trig" />
         <table name="_index" />
         <table name="_index-field" />
         <table name="_user" />
         <table name="_connect" />
         <table name="_myconnection" />
         <table name="_database-feature" />
         <table name="_filelist" />
         <table name="_startup" />
         <table name="_lock" />
         <table name="_usertablestat" />
         <table name="_sequence" />
      </metadata>
   </schema>

   <profile name="module1">

      <file-set>
         <directory recursive="true" path="./abl/common/" spec="(*.[pPwWtT]|*.cls)" />
         <directory recursive="true" path="./abl/module1/oo/" spec="*.cls" />
         <directory recursive="true" path="./abl/module1/legacy/" spec="*.[pPwW]" />
         <directory recursive="true" path="./abl/module1/trig/" spec="*.[tT]" />
         <exclude filter="./abl/module1/legacy/broken/*" />
      </file-set>

      <parameter name="merge-point"      value="/server/default/runtime/module1/" />
      <parameter name="propath"          value="${P2J_HOME}:${P2J_HOME}/abl/common/:${P2J_HOME}/abl/module1:${P2J_HOME}/abl/some/other/path:" />

      <schema>
         <namespace name="module1" default="true" />
         <namespace name="standard" />
      </schema>

   </profile>

   <profile name="module2">

      <file-set>
         <directory recursive="true" path="./abl/common/" spec="(*.[pPwWtT]|*.cls)" />
         <directory recursive="true" path="./abl/module2/main/" spec="(*.[pPwWtT]|*.cls)" />
      </file-set>

      <parameter name="datanames"        value="./some/other/location/module2_datanames.xml" />
      <parameter name="propath"          value="${P2J_HOME}:${P2J_HOME}/abl/common/:${P2J_HOME}/abl/module2:${P2J_HOME}/abl/module2/main:" />

      <schema>
         <namespace name="module2" default="true" />
         <namespace name="standard" />
      </schema>

   </profile>

   <profile name="module2-test">

      <file-set>
         <directory recursive="true" path="./abl/common/" spec="(*.[pPwWtT]|*.cls)" />
         <directory recursive="true" path="./abl/module2/main/" spec="(*.[pPwWtT]|*.cls)" />
         <directory recursive="true" path="./abl/module2/test/" spec="*.[pP]" />
      </file-set>

      <parameter name="propath"          value="${P2J_HOME}:${P2J_HOME}/abl/common/:${P2J_HOME}/abl/module2/main:${P2J_HOME}/abl/module2/test:" />

      <schema>
         <namespace name="module2_unittest" default="true" />
         <namespace name="standard" />
      </schema>

   </profile>

</cfg>

Is this the right vision?

It seems like some of the global parameters don't make sense to override (e.g., registry, oo-skeleton-path, etc.).

I'm assuming (for now) that profile *.df files will be found in ./data/<profile-name>/.

At this moment the structure of the configuration file is:

<cfg>
  <global>...</global>

  <schema>
    <!-- unnamed profile -->
    <namespace name="ns_0.1"/>
    <namespace name="ns_0.2"/>
    <!-- end of unnamed profile -->

    <profile name="p1">
       <namespace name="ns_1.1"/>
       <namespace name="ns_1.2"/>
    </profile>
    ...
    <profile  name="pk">
       <namespace name="ns_k.1"/>
       <namespace name="ns_k.2"/>
    </profile>

    <metadata name="standard" />
  </schema>
</cfg>

• a set of property-es specific to respective profile,
• the schema sub-node with a set of namespaces and
• the set of files to be processed.

The metadata should be still common and defined only once.

Is this the right vision?

Mostly, yes. The part I don't understand is why there is a schema node inside the profile. I would not change anything. Just allow the namespace elements to be directly in the profile element.

It seems like some of the global parameters don't make sense to override (e.g., registry, oo-skeleton-path, etc.).

Agreed.

I'm assuming (for now) that profile *.df files will be found in ./data/<profile-name>/.

No, I don't think this is the case. Ovidiu already added support for specifying a custom path for the .df. This should be used unchanged. In other words, we don't need any changes to how the namespace nodes are processed in a profile.

Greg Shah wrote:

I'm assuming (for now) that profile *.df files will be found in ./data/<profile-name>/.

No, I don't think this is the case. Ovidiu already added support for specifying a custom path for the .df. This should be used unchanged. In other words, we don't need any changes to how the namespace nodes are processed in a profile.

I think that is a good idea for the default location: if importFile is not specified then the default value should be ./data/<profile-name>/<namespace>.df. For the unnamed profile the .df will be located directly into ./data/, by default.

Yes, the syntax is OK.

• there is the 'global' profile which is used implicitly
• the 'global' profile also sets defaults for any explicit profile
• each profile has the same syntax as the 'global' profile (so the namespace nodes would be added in a schema node)
• when reading a profile, first the 'global' configuration is created and after that the profile is just merged in the 'global' configuration

This way, we can work with the same Configuration instance, and other FWD conversion code will require no changes.

• 'merge' mode, where the profiles are merged and everything is converted together.
• individual mode, where FWD runs conversion for each specified profile, sequentially (same way as you would run individual ConversionDriver commands in a script, to run each profile).

When running in single profile mode, the profile will need to overwrite default parameter and file-set nodes, and merge namespace nodes to the global schema, correct?

each profile has the same syntax as the 'global' profile (so the namespace nodes would be added in a schema node)

This is a change from the current approach. I guess the benefit of consistency may outweigh the short term interest in less change. Let's go ahead.

I'm good with the other ideas. I think the multi-profile individual mode will not be used too often. I only see 2 scenarios:

• Can't run the entire project at once because of some kind of memory constraint. This would be a kind of bug that needs to be fixed anyway, so I don't see this as a long term need.
• Early runs of the conversion where we want to allow a wide range of the project to be checked without aborting on each problem. I think we are going to make conversion more tolerant of failures, but still this could have some benefit.

Is there some other use case I'm missing?

When running in single profile mode, the profile will need to overwrite default parameter and file-set nodes,

Correct.

and merge namespace nodes to the global schema, correct?

We already handle the namespace parts today for this mode. Do we need to change it?

At least the syntax in note 5 is no longer valid, right?

Right.

The profile node has only name attribute in #6256-8.

The SchemaConfig$Profile has a isDefault, which suggests that there can be a isDefault attribute at the profile node. Do we need this? I can't find any reference in redmine related to it, but there is SchemaConfig.getDefaultProfile code which uses this to determine the default profile.

The namespace should assume default="true" so that it doesn't have to be duplicated. But it should be possible to define default="false" to establish a logial database but only load it for specific files (via directory or file-level hints). We support this today. Initially, Ovidiu's changes required default="true" to be explicit but he changed that to be assumed since it is the common case.

Greg Shah wrote:

The namespace should assume default="true" so that it doesn't have to be duplicated. But it should be possible to define default="false" to establish a logial database but only load it for specific files (via directory or file-level hints). We support this today. Initially, Ovidiu's changes required default="true" to be explicit but he changed that to be assumed since it is the common case.

I'm not talking about the namespace. I'm talking about the schema/profile node. Current code assumes there can be a <profile default = "true"/> attribute.

The SchemaConfig$Profile has a isDefault, which suggests that there can be a isDefault attribute at the profile node. Do we need this? I can't find any reference in redmine related to it, but there is SchemaConfig.getDefaultProfile code which uses this to determine the default profile.

Sorry, I was thinking about the namespace. I don't recall the reason for this.

I don't think there is a hard requirement for that at the profile level.

Do you see a need to make the profile/schema node optional? In this case, it would be inherited from the global config.

PS: I'm not working on the multi-profile mode (either merge or list), this was just an idea.

Do you see a need to make the profile/schema node optional? In this case, it would be inherited from the global config.

Yes, this makes sense.

PS: I'm not working on the multi-profile mode (either merge or list), this was just an idea.

The multi-profile merge mode is needed as part of this task. The multi-profile individual mode is a useful addition, but is not required.

Greg Shah wrote:

The multi-profile merge mode is needed as part of this task.

Understood.

Greg Shah wrote:

The SchemaConfig$Profile has a isDefault, which suggests that there can be a isDefault attribute at the profile node. Do we need this? I can't find any reference in redmine related to it, but there is SchemaConfig.getDefaultProfile code which uses this to determine the default profile.

Sorry, I was thinking about the namespace. I don't recall the reason for this.

• if a profile is set as default, then automatically load the default profile (be it runtime or conversion).
• if multiple profiles are set as default, abend
• if no profile is set as default, let the global configuration be the default, unless one is specified at command line. Currently it uses the first 'schema/profile', a 'random' choice, if no default profile is set. I don't like this, it makes more sense to default to the global config. Let the user explicitly choose what needs to run.

Constantin Asofiei wrote:

• if no profile is set as default, let the global configuration be the default, unless one is specified at command line. Currently it uses the first 'schema/profile', a 'random' choice, if no default profile is set. I don't like this, it makes more sense to default to the global config. Let the user explicitly choose what needs to run.

Another addition: if a profile is set as default, command line can still override it.

It is a good plan.

Greg Shah wrote:

The multi-profile merge mode is needed as part of this task.

This will be only for conversion. Runtime part is tricky, especially the schema part.

A useful addition (medium term, not immediately) would be to be able to load multiple DF files into a single schema namespace. Rather than concatenating the DF files together manually or maintaining a script to do so, the SchemaLoader could be configured to parse multiple DF files into a single, logical schema. This could be represented as one or more import-file elements as children of the namespace element, rather than the current importFile attribute of namespace. We could leave the importFile attribute intact for backward compatibility or deprecate it, but the preferred way to define a schema namespace would be with the new import-file element. If no importFile attribute or import-file child element is defined, the namespace would default to the current approach of just assuming the name attribute of the namespace element indicates the root name of the DF file, located in the data subdirectory of the project.

For example:

<namespace name="federated">
   <import-file path="[optional/relative/location/of/]shard1.df" />
   <import-file path="[optional/different/relative/location/of/]shard2.df" />
</namespace>

Eric: I think the shard idea makes good sense. We have multiple customers using this approach in OE today. The idea should probably be in its own task.

• multiple profiles are allowed (only for conversion time), and they can be specified via -pprofile1 -pprofile2 at command line or as default=true at the p2j.cfg.xml profile
• if no command-line -p argument, then p2j.cfg.xml is checked for one or more default profiles. If none set, the global config is used.
• if one or more profiles are determined as default, start with the pristine global config, and merge all the profiles into this:
  • parameters are overwritten, in the order of how the profiles are defined in p2j.cfg.xml
  • the file-set used is the one computed from the profiles - each profile computes its files individually, and an explicit list of files is returned. The global file-set is not used!
• schema namespaces are merged from all profiles, and are used instead of the global schema. WARNING: if there is a namespace clash and the schema is not the same for all namespaces sharing this name (over all used profiles), then ... weird things will happen.
• if a profile has no schema defined, then there will be no schema used for that profile (I do not default to the global schema)

When I said the multiple profile merge was needed, I meant a different idea.

Each profile can be run standalone. But when you run multuple profiles as a batch, we need the settings to remain intact. The files specified for the profile should be parsed/converted with the same settings as if the profile was run standalone. But the overall list of files processed will be the aggregate of all the profiles selected.

parameters are overwritten, in the order of how the profiles are defined in p2j.cfg.xml

No, we don't want this. The settings for a given file will depend on the specific profile it is in + the global settings, that is all.

schema namespaces are merged from all profiles,

No, this won't work. The idea is the same as for the rest of the parameters. The files specified for that profile must get just the schemata that are specified in that profile.

Greg Shah wrote:

The files specified for the profile should be parsed/converted with the same settings as if the profile was run standalone.

What if a file is part of multiple profiles? I think we see this in a current project. Also, if there is any namespace (or schema file) conflict, it will not work.

What you describe can't be achieved easily... it means switching the conversion context for each file, to set the schema, propath settings, and more that I can't think about without digging into the code.

What if a file is part of multiple profiles?

The profiles are intended to be mutually-exclusive. We can define a rule that the first profile wins (and print a warning), if a file is specified more than once.

What you describe can't be achieved easily... it means switching the conversion context for each file, to set the schema, propath settings,

Yes, I understand. That is the intention of this task.

My idea was to add an extra layer at the ConversionDriver level, which would associate the profile with some sub-set of total file list. As each file is processed, the proper profile would be used and all schemata/configuration would come from that profile.

Greg Shah wrote:

My idea was to add an extra layer at the ConversionDriver level, which would associate the profile with some sub-set of total file list. As each file is processed, the proper profile would be used and all schemata/configuration would come from that profile.

Thanks, I think I may be able to make it work for the front phase. But the conversion part may be tricky.

We can defer any conversion issues until a second phase of work. The front end part is what is needed this week.

Greg Shah wrote:

We can defer any conversion issues until a second phase of work. The front end part is what is needed this week.

I think it works, kind of ugly IMO, but with minimal code for the front phase. I don't see a way to switch profiles without explicitly 'injecting' code like 'cfg.withFileProfile(file)', which does all the underlying work of switching the Configuration and SchemaConfig.

I'll test and commit tomorrow.

1. a merged configuration with all schemas in one database, named main_db
2. a multi configuration with all schemas spread into multiple databases, named main_db (which is a different schema than the above), multi_1, multi_2, and multi_3

And there is a "common" database between the 2 named common_db.

• data/merged which contains just the main_db
• data/multi which contains the main_db, multi_1, multi_2, and multi_3
• data/merged_multi_common which contains the common_db

What would the p2j.cfg.xml configuration look like? Right now it looks like:

   <schema>
      <profile name="merged" default="true">
         <namespace name="standard" />

         <namespace
            name="common_db" 
            default="false" >
            <parameter name="ddl-dialects" value="postgresql" />
         </namespace>
         <namespace
            name="main_db" 
            default="true" >
            <parameter name="ddl-dialects" value="postgresql" />
         </namespace>
      </profile>

      <profile name="multi" default="false">
         <namespace name="standard" />

         <namespace
            name="common_db" 
            default="false" >
            <parameter name="ddl-dialects" value="postgresql" />
         </namespace>
         <namespace
            name="main_db" 
            default="true" >
            <parameter name="ddl-dialects" value="postgresql" />
         </namespace>
         <namespace
            name="multi_1" 
            default="true" >
            <parameter name="ddl-dialects" value="postgresql" />
         </namespace>
         <namespace
            name="multi_2" 
            default="true" >
            <parameter name="ddl-dialects" value="postgresql" />
         </namespace>
         <namespace
            name="multi_3" 
            default="true" >
            <parameter name="ddl-dialects" value="postgresql" />
         </namespace>
      </profile>

My build.xml goes through the effort of copying the correct set of DF files from either data/merged or data/multi to the data/ directory to be alongside the common_db.df file, which is kept in that location.

I'd like to just place the DF files as mentioned, and leave them be during the conversion process. What should my configuration look like?

As far as the dump files, there is a singular collection of all the dump files, and there are symbolic links to the same collection for dump/multi_1/, dump/multi_2/, and dump/multi_3/ so I don't believe that aspect of configuration should change.

Our initial support for multi-profile conversion needs to be enhanced. FWD supports conversion of separate profiles, but it does not allow merging all these converted profiles (with one or more of them representing an application module) and run them together, in the same FWD server.

• #6407 is part of the solution. (name_map.xml in 'extension jars')
• We need to handle the fact that per profile state like registry.xml and all the .p2o/.schema needs to be merged in some way.
• Different profiles may reference overlapping sets of OO 4GL classes. We probably need some concept of which profile is authoritative for these classes so that they can be only converted once. Doing so is a problem because the only mapping we have from 4GL OO features to Java OO features is calculated during parsing/conversion itself but the dependent projects must have access to this in order to convert. The current approach converts overlapping OO 4GL classes multiple times and has no way to merge them into a single result.

The objective here is to have two or more completely different converted applications which may or may not use the same schema/database, and we want to:

• Convert and compile each one independently, with the minimum dependencies on the others.
• Run them in the same FWD server at runtime with no duplication of classes or conversion artifacts/state.

As one customer notes: in Java this can be accomplished easily by just referencing a jar in your development environment. For that to work, we would have to package up enough of the conversion artifacts/state to allow it to be referenced in another "application". Doing this would make very large organizations able to split their development across tens or even hundreds of different development groups without each one having to convert the entire set of all dependent applications. This is a critical objective.

What would be required is that the dependencies would be converted first. Then the jar(s) (which have the converted code) would need to be available. Crucially, some amount of conversion knowledge about how the 4GL dependencies converted would be required. For example, we would need to know how each externally visible method and data member mapped to the converted version. Some amount of this (maybe all?) could be read from our class and method annotations (e.g. LegacySignature).

There is a problem when the 'exclude' filter is used to not add files to conversion. During pre-scan phase, the filter is not checked, but instead a physical file on disk (based on the configured PROPATH). And, that file is used (if found), even if is excluded via the profile 'exclude' filter.