FWD » Core Development » Conversion Tools

Incremental conversion for OpenClient proxy Java code is not possible at this time. If a 4GL program configured as proxy in the .xpxg files is included in the incremental conversion, it will override any OpenClient Java code for the program's (Sub)AppObject.

Dănuț Filimon wrote:

I got the open_client_example project from Constantin and did a simple test. After conversion, I modified ./abl/subtype/type1/proc1.p and incremental conversion modified ./scr.proxy/com/goldencode/testcases/proxy/TestType1.java and ./scr.proxy/com/goldencode/testcases/proxy2/Test2Type1.java. The problem is that the java file for the proc1.p is not modified during incremental conversion, but the SubAppObject(s) are modified (the procedure is specified in the .xpxg file so it should be right).

The proc1.p java file from the src folder is actually updated, but the one from the src.proxy is not.

Constantin, can you explain what the actual issue is and what is the expected behavior?

The issue I think was that when running incremental conversion, the OpenClient proxy generation in FWD was not including all other programs part of .xpxg - thus the OpenClient generated via incremental was wrong.

If FWD now includes automatically all programs from .xpxg when generating the OpenClient, and all .java for the OpenClient proxy code is generated correctly, then that is good.

Otherwise, a question: if there are changes in the SOAP, REST, web or proxy/.xpxg resources, does FWD now to incrementally convert them? I mean something like you add/remove a file for REST/SOAP/.xpxg, without changing the 4GL code.

Constantin Asofiei wrote:

Otherwise, a question: if there are changes in the SOAP, REST, web or proxy/.xpxg resources, does FWD now to incrementally convert them? I mean something like you add/remove a file for REST/SOAP/.xpxg, without changing the 4GL code.

Modifying/deleting .xpxg will not run the incremental conversion, not sure about soap/rest/web.

Dănuț Filimon wrote:

Constantin Asofiei wrote:

Otherwise, a question: if there are changes in the SOAP, REST, web or proxy/.xpxg resources, does FWD now to incrementally convert them? I mean something like you add/remove a file for REST/SOAP/.xpxg, without changing the 4GL code.

Modifying/deleting .xpxg will not run the incremental conversion, not sure about soap/rest/web.

That is definitely something for us to fix.

There's resolveProxyProgramsConfiguration() method that is called in middle() when initializing a ServiceSupport worker. rules/convert/brew.xml uses it. Tracking modified files during incremental is possible, but the same can't be said about deleted files. I will probably have to write something similar to #6083.

ServiceSupport.resolveRestConfiguration for REST, resolveWebHandlers for WEB, initialize for SOAP. Those 3 will have to be checked. All of those will need to be registered in the file_signature conversion table and checked.

Greg, should soap/rest/web/proxy files be checked right at the start of the conversion or should we wait until the last possible moment? Personally I don't like the idea of having these files all over the place.

Greg, should soap/rest/web/proxy files be checked right at the start of the conversion or should we wait until the last possible moment?

We need the full list of files to convert at the start of conversion, just like we calculate today. There is no option to wait until later.

Danut, while we are at it: the .jast for the FWD OpenClient proxy programs is still in abl/ and not in cvt/ folder.

Constantin Asofiei wrote:

Danut, while we are at it: the .jast for the FWD OpenClient proxy programs is still in abl/ and not in cvt/ folder.

This is generated in ConversionDriver.generateProxyPrograms() by

            processTrees("Generate Proxy Client Programs - pass #" + pass + " of " + size,
                         "convert/proxy_programs",
                         codenames,
                         false,
                         proxyAsts,
                         debug);

      public JavaAst createJavaFile(String filename)
      {
         JavaAst result = createAst(COMPILE_UNIT, "compilation unit", null, null, -1);
         filename = Configuration.getPathToConversionFolder(filename); // added this line
         result.brainwash(filename);

         return result;
      }

Working on getting all files right at the start and continuing from there.

• LegacyProxyConfig.java has an additional property: parent - the .xpxg file path.
• ConversionDriver.generateProxyPrograms() will check the .xpxg file and see if it must be converted along with the configured procedures. It will also save the new hash after converting the proxies.
• ServiceSupport.verifyConversionFiles(); returns a set containing procedures of proxies (maybe soap/rest/web too) that need to be included in the conversion (so an additional mustConvertFile() call here).
• added ServiceSupport methods to initialize soap/rest/web/proxy files, moved the necessary properties to WorkArea. WorkArea also contains the original file names for soap/rest/web/proxy so that I can have a starting point and check the files. I might be able to remove this and just use the new LegacyProxyConfig property.

I looked over the REST operations, procedures and internal procedures are used and some of them only have their name (not the full path). I am not sure how to correlate the .paar with the right procedure/internal procedure that needs to be converted when the .paar is modified.

Dănuț Filimon wrote:

I looked over the REST operations, procedures and internal procedures are used and some of them only have their name (not the full path). I am not sure how to correlate the .paar with the right procedure/internal procedure that needs to be converted when the .paar is modified.

I checked the propath and it does have one for the folder I need, I can just use FileOperationsWorker.findFile() to get the normalized path of the procedure that I need.

Incremental conversion is working, I plan to cleanup the implementation and see if there are any improvements that I can make, commit and then work on deleted files.

Committed 6629a/16251. Initial implementation for incremental conversion of proxy/soap/rest/web program configurations.

• modified src/com/goldencode/p2j/convert/ConversionDriver.java
  • Added saveServicePrograms() to save the file hash for the rest and web program configuration because those do not have a specific method to be processed, the procedures used are part of the full conversion.
  • Saved the file hash for soap in generateWsdls().
  • Check if the proxy file must be converted before adding it to proxyFileSets (ensures proxyFiles only contains proxy files that need to be converted and that the proxy source file has it's file hash updated).
• modified src/com/goldencode/p2j/convert/TransformDriver.java
  • Schema Fixups is an unrelated change, but related to #10265 (no files to convert, but this step was still being executed).
  • executeJob() calls ServiceSupport.verifyServiceFiles() that returns a set of procedure files used by the configured program files. When the proxy/soap/rest/web program configuration is changes, all the associated program files will need to be reconverted, even if ConversionData.mustConvertFile(file) returns false.
• modified src/com/goldencode/p2j/pattern/LegacyProxyConfig.java
  • Added sourceFile property, used in the ConversionDriver to determine the proxy source file that needs to be checked (mustConvertFile(), saveFileHash()).
• modified src/com/goldencode/p2j/pattern/RestOperation.java
  • Added sourceFile property, used in ServiceSupport.verifyServiceFiles() to determine what file needs to be reconverted.
• modified src/com/goldencode/p2j/pattern/ServiceSupport.java
  • Moved operations, webHandlers, soapConfigs to the WorkArea.
  • Added proxyFiles, soapFiles, restFiles, webFiles to WorkArea (and getters where it was necessary). All of them are sets that will contain the program configuration filenames.
  • Initialization methods for services had to be modified since the properties were moved to WorkArea.
  • Added verifyServiceFiles(), checks all program configurations and returns a set of service files that need to be reconverted. A modified program configuration means all associated procedure files need to be reconverted.
  • Most of the files are just the name and rely on the propath to be found, this is why I used FileOperationsWorker.findFile() to get the actual path and assign the sourceFile.
  • Adjusted methods since operations, webHandlers, soapConfigs can be null.
• modified src/com/goldencode/p2j/pattern/SoapConfig.java
  • Added getOperations(), used in ServiceSupport.verifyServiceFiles().
• modified src/com/goldencode/p2j/pattern/SoapOperation.java
  • Added sourceFile property.
• modified src/com/goldencode/p2j/pattern/WebHandlerConfig.java
  • Added clsPath property.
• modified src/com/goldencode/p2j/uast/JavaPatternWorker.java
  • Solves #6629-12.

Dănuț Filimon wrote:

One issue with deleting a program configuration file is that deleting it will not cause all associated program files to be deleted as well, it will just need to reconvert them. To delete a program configuration I need to know what files the configuration was referencing because it will no longer be initialized, this will require a conversion table to store information about services.

I am mostly done with the support for deleting files when the program configuration is deleted. I'll have to test deleting the proxy configuration a bit more because I don't want to miss a scenario.

Danut, I think the changes in 6629a are good. One issue I'm not sure how to improve: when deleting a service file, to me it seems that getServiceFilesToReconvert delets the record from the services table too early - if the parse does not complete, then the associated 4GL programs remain marked as completed in the conversion db, while their service gets 'deleted'.

Can we consider to delete the record with the hash for all these files, so that these files can be properly re-marked only once the re-parse/etc finishes?

Constantin Asofiei wrote:

Danut, I think the changes in 6629a are good. One issue I'm not sure how to improve: when deleting a service file, to me it seems that getServiceFilesToReconvert delets the record from the services table too early - if the parse does not complete, then the associated 4GL programs remain marked as completed in the conversion db, while their service gets 'deleted'.

Can we consider to delete the record with the hash for all these files, so that these files can be properly re-marked only once the re-parse/etc finishes?

We can do this in clean(), just delete the services entry using the source_file (current file).

Constantin Asofiei wrote:

Danut, please put it in conversion testing. I've reviewed this some time ago but forgot to post.

I am rebasing 6629a to pickup the recent 7180a changes that got into trunk, I expect some compatibility changes and I will request a review if the logic will have to change.

Rebased 6629a to latest trunk/16442, the branch is now at revision 16445.

I fixed most of the changes to use artifacts, only need to retest the examples I have and then commit.

Found the following error while testing a customer:

     [java] java.lang.NullPointerException
     [java]     at java.base/java.io.File.<init>(File.java:278)
     [java]     at com.goldencode.artifacts.ArtifactManager.normalizeFilename(ArtifactManager.java:1242)
     [java]     at com.goldencode.artifacts.ArtifactManager.normalizeFilename(ArtifactManager.java:1221)
     [java]     at com.goldencode.artifacts.ArtifactManager.addArtifact(ArtifactManager.java:723)
     [java]     at com.goldencode.p2j.pattern.ServiceSupport.resolveWebHandlers(ServiceSupport.java:1012)
     [java]     at com.goldencode.p2j.pattern.ServiceSupport.initializeWebConfigurations(ServiceSupport.java:322)
     [java]     at com.goldencode.p2j.pattern.ServiceSupport.verifyServiceArtifacts(ServiceSupport.java:393)
     [java]     at com.goldencode.p2j.convert.TransformDriver.executeJob(TransformDriver.java:1068)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.main(ConversionDriver.java:1410)
     [java]

I also have to rebase and pick up trunk/16450

Rebased 6629a to trunk/16453, the branch is now at revision 16457.

Committed 6629a/16457. Removed initializeWebConfigurations() calls.

I tested conversions for ChUI and a small customer app, found no issues.

Constantin: Is there any additional testing needed or can this merge?

Tested a customer application and there's an issue with:

                  try
                  {
                     String sourceFile = FileOperationsWorker.findFile(name + "." + ext, null);
                     restOp.sourceArtifact = ArtifactManager.addArtifact(sourceFile);
                  }
                  catch (ConfigurationException e)
                  {
                     LOG.log(Level.WARNING,
                             "Could not find file within the configured PROPATH for " + name);
                  }

Dănuț Filimon wrote:

Tested a customer application and there's an issue with:
[...] The expected_file.p can't be found because the propath mentions folder1/folder2/ but the file is actually in folder1/folder2/folder3/folder4. The implementation is based on the file being found through the propath, having to look into all of the folders to get the right file is not correct. This is important because we have to use the same artifacts to save information to the conversion database and use it during incremental.

Committed a small NPE fix, there was also a wrong path being looked up for rest. 6629a/16459.

• remove WebHandlerConfig - I think I forgot to delete it via #11213
• JavaPatternWorker - has no functional change?

Otherwise, you mention wrong path being looked up for rest - where is this fix?

Constantin Asofiei wrote:

Otherwise, you mention wrong path being looked up for rest - where is this fix?

It is this little bit:

@@ -1306,7 +1327,15 @@
                   restOp.source = name;
                   try
                   {
-                     String sourceFile = FileOperationsWorker.findFile(name + "." + ext, null);
+                     String file = folder +
+                                   (folder.isEmpty() || folder.endsWith("/") ? "" : "/") + name +
+                                   (ext.isEmpty() ? "" : ".") + ext;
+                     String sourceFile = FileOperationsWorker.findFile(file, null);
+                     if (sourceFile == null)
+                     {
+                        LOG.log(Level.WARNING, "Could not find file within the configured PROPATH for " + file);
+                        continue;
+                     }
                      restOp.sourceArtifact = ArtifactManager.addArtifact(sourceFile);
                   }
                   catch (ConfigurationException e)

Dănuț Filimon wrote:

Constantin Asofiei wrote:

Otherwise, you mention wrong path being looked up for rest - where is this fix?

It is this little bit:
[...]

Thanks, I understand it now.

6629a/16460 is available, I will be rebasing the branch now.

Rebased 6629a to latest trunk/16505, the branch is now at revision 16512.

Dănuț Filimon wrote:

Rebased 6629a to latest trunk/16505, the branch is now at revision 16512.

Please do conversion and (some) runtime testing.

Only two files failed conversion, the error is:

     [java] Caused by: java.lang.NullPointerException: Cannot invoke "java.util.List.size()" because "this.val$service.legacyParams" is null
     [java]     at com.goldencode.p2j.pattern.ServiceSupport$Library$1.parameters(ServiceSupport.java:2347)
     [java]     at com.goldencode.p2j.pattern.ServiceSupport$Library.getParameters(ServiceSupport.java:2204)
     [java]     at com.goldencode.expr.CE13799.execute(Unknown Source)
     [java]     at com.goldencode.expr.Expression.execute(Expression.java:398)
     [java]     ... 44 more

I am currently investigating, as I did not make any changes to that code.

There are a few parameters which are not initialized properly, in @SupportService.resolveService() there should be a rest operation found for a specific operation so that the parameters are added, but this does not happen.

Dănuț Filimon wrote:

There are a few parameters which are not initialized properly, in @SupportService.resolveService() there should be a rest operation found for a specific operation so that the parameters are added, but this does not happen.

I found the issue, there's a <wsad:Name/> node which specified a path for a cls file, but that path is using dots instead of "/". The code from #6629-38 is the key, the file should also do name.contains(".") ? name.replace(".", "/") : name for the path to be correct and parameters to be assigned.

Committed 6629a/16513. Incremental conversion fixed the remaining files and the project compiled.

Testing went well, I am waiting for the runtime results from Artur.

Runtime testing on the customer project can be done only with the custom branch patched with the changes from 6629a. This particular customer did not receive the latest version of standard.df and build.xml and the associated changes, this makes it complicated to convert and runtime test using trunk.

Artur Școlnic wrote:

Runtime testing on the customer project can be done only with the custom branch patched with the changes from 6629a. This particular customer did not receive the latest version of standard.df and build.xml and the associated changes, this makes it complicated to convert and runtime test using trunk.

This will require other revisions from trunk, including #7180 changes.