FWD » Core Development » Conversion Tools

From #10536-27:

I don't know when this was regressed, but if you have two tables named Foo_BARSomething and Foo_BAR_Something in a .df file, and you try to convert, only a .java DMO is generated; this is because the list of AST files gets built as case-insensitive; this patch forces case-sensitivity for DMO and POJO .jast files.

--- src/com/goldencode/p2j/convert/ConversionDriver.java
+++ src/com/goldencode/p2j/convert/ConversionDriver.java
@@ -989,7 +989,8 @@
       JavaPatternWorker.disableNewFileTracking();
       String[] dmoList = JavaPatternWorker.getPersistedFiles();

-      return new ExplicitFileList(dmoList);
+      // these are OS files, need to be case-sensitive
+      return new ExplicitFileList(dmoList, true);
    }

    /**
@@ -1035,7 +1036,8 @@
       // add generated JASTs to the overall list (used with brew later)
       JavaPatternWorker.disableNewFileTracking();

-      return new ExplicitFileList(JavaPatternWorker.getPersistedFiles());
+      // these are OS files, need to be case-sensitive
+      return new ExplicitFileList(JavaPatternWorker.getPersistedFiles(), true);
    }

    /**

which is a side-effect of <action>dmos = create("java.util.HashSet")</action> in p2o.xml is a case-sensitive set - it should be case-insensitive for Windows OS.

This task is meant to check the OS-filename generated for artifacts like DMOs, POJOs, frames, menus, etc, if FWD is aware of the OS where the name is generated - on Windows, collisions related to case-sensitivity must not happen.

This task is meant to check the OS-filename generated for artifacts like DMOs, POJOs, frames, menus, etc, if FWD is aware of the OS where the name is generated - on Windows, collisions related to case-sensitivity must not happen.

To clarify, we should always generate the same results regardless of whether we are running on Linux or Windows. The conversion output, including the exact same list of classes etc... should be identical. The OS on which conversion runs should not change the output of the conversion.

In the case of Windows, with its case-insensitive filesystem, we must ensure that two or more inputs do not create the same output filename. But this same behavior must always be implemented, no matter on which platform the conversion runs.

I've made a similar patch to #10626-2 in #9917, it was for the frames and menus (explained the issue in #9917-10, the patch is in #9917-13). The fix reached trunk in revision 16139.

Dănuț Filimon wrote:

I've made a similar patch to #10626-2 in #9917, it was for the frames and menus (explained the issue in #9917-10, the patch is in #9917-13). The fix reached trunk in revision 16139.

That patch does not solve the core issue with this task - the DMO names when converted on Windows will still collide. We need to go 'deeper' where the names of the DMO is calculated, and ensure that is unique case-insensitively.

And double-check the other .java artifacts (even converted .java code for 4GL files) that they are unique case-insensitively, and not dependent on the OS where the conversion is ran.

We use opsys to determine what OS is being used (default is Unix OS). opsys can be set in p2j.cfg.xml, so there's a way to check if Windows is used in the rules. NameConverterWorker is used to create the dmo names, so I think NameConverter.convert() should take an additional parameter for the case where the name needs be case-sensitive or not.

We shouldn't change the output based on the OPSYS setting. The output should be the same always (safe for case-insensitive filesystems).

Good, opsys is out of the question.

I converted the two tables, Foo_BARSomething and Foo_BAR_Something, and got a FooBarsomething dmo. The Foo_BAR_Something was split as foo_ bar_ something and then each token was replaced to get FooBarSomething (for the class type, the table type resulted in foo_bar_something). Foo_BARSomething was split as foo_ barsomething and I got FooBarsomething. The dmo names are generated correctly and the patch from #10626-2 fixes the import.

I have a windows machine setup for converting hotel, so I'll use that to check for other collisions.

We need to generate the same .java names regardless of OS - so I think that collection needs to be case-insensitive, regardless of OS.

Constantin Asofiei wrote:

We need to generate the same .java names regardless of OS - so I think that collection needs to be case-insensitive, regardless of OS.

Indeed, having different outputs between OS could cause some issues when troubleshooting other issues in the future. Is there any way to declare a case insensitive HashMap in the rules (I mean using com.goldencode.util.CaseInsensitiveHashMap)?

Dănuț Filimon wrote:

Constantin Asofiei wrote:

We need to generate the same .java names regardless of OS - so I think that collection needs to be case-insensitive, regardless of OS.

Indeed, having different outputs between OS could cause some issues when troubleshooting other issues in the future. Is there any way to declare a case insensitive HashMap in the rules (I mean using com.goldencode.util.CaseInsensitiveHashMap)?

Yes, just replace the type in the create statement with the case-insensitive map or set implementation in FWD.

Constantin Asofiei wrote:

Dănuț Filimon wrote:

Constantin Asofiei wrote:

We need to generate the same .java names regardless of OS - so I think that collection needs to be case-insensitive, regardless of OS.

Indeed, having different outputs between OS could cause some issues when troubleshooting other issues in the future. Is there any way to declare a case insensitive HashMap in the rules (I mean using com.goldencode.util.CaseInsensitiveHashMap)?

Yes, just replace the type in the create statement with the case-insensitive map or set implementation in FWD.

Perfect, I'll just have to implement the contains() method for CaseInsensitiveHashMap in this scenariom then check if there are any changes to the functions used in the rules regarding the new map.

Dănuț Filimon wrote:

Perfect, I'll just have to implement the contains() method for CaseInsensitiveHashMap in this scenariom then check if there are any changes to the functions used in the rules regarding the new map.

Let me make a correction, I should use CaseInsensitiveHashSet, which I totally missed until now.

I setup and tested POJOs on hotel and found no issues, the hotel.p2o ends up using the correct class names (FooBarSomething and FooBarsomething1) and the pojos are created using those two class names.

Looking at menus and frames now.

The issue with menus starts with this piece of code:

            <action>extends = sprintf("%s%s", clspfx, "Definition")</action>
            <action>cname = classname</action>
            <action>fname = execLib("gen_fname", cname)</action>

            <action>printfln("Creating separate file: %s %s", cname, fname)</action>

            <rule>lastFile != null
               <action>execLib("gen_add_widgets", pnames, wtypes)</action>
               <action>persistJavaFile(lastFile)</action>
            </rule>

            <action>createJavaFile(fname, true)</action>

            <action>lastFile = fname</action>           

            <action>pkgname = #(java.lang.String) execLib("gen_pkg")</action>

            <action>createJavaAst(java.kw_package, pkgname, javaRootId)</action>

            <action>createImport("com.goldencode.p2j.ui.*")</action>

            <action>putNote("javaname", cname)</action>

            <action>
               tw.graft("class_def_ext",
                        null,
                        javaRootId,
                        "classname", cname,
                        "pkgname", pkgname,
                        "access" , "public",
                        "extends", extends)
            </action>

This bit of code:

         <action>
            classname = sprintf("%s%s%s",
                                baseClass,
                                names.convert(getNoteString("name"), names.class), 
                                "Menu")
         </action>

My initial idea was to add an index at the end of the name:

            <rule>menuNames.contains(classname)
               <action>nextMenuSuffix = execLib("getUniqueSuffix", nextMenuSuffix, classname, menuNames)</action>
               <action>classname = sprintf("%s%d", classname, nextMenuSuffix)</action>
            </rule>
            <action>menuNames.add(classname)</action>

I thought of using another set of rules to store a case sensitive map that maps the namings, but I don't think this is the best solution.

I discussed with Octavian and he mentioned I could use createString2StringMap to get a map across other rules. I have an idea where I will store the classname as a key (which s case-sensitive) and then I will calculate the key based on the existing values from the map (put those in a case insensitive set and check new postfix values until it can't be found in the set). This looks a bit expensive when I think there can be thousands of menus generated for an application, so I am trying to find something faster/less expensive.

Committed 10626a/16221. Generates names for menus, sub-menus, dmos, pojos, frames that do not collide in case-insensitive file-systems.

Is this ready for review? What is the % Done?

Greg Shah wrote:

Is this ready for review? What is the % Done?

I looked into the ChUI project for some other cases where names are built and found procedure with names like <name>Debug, <name>Debug0 and other suffixes like SeDate, FIn, FPrinter. I am trying to understand if this will be an issue and what I need to fix next.

Dănuț Filimon wrote:

Greg Shah wrote:

Is this ready for review? What is the % Done?

I looked into the ChUI project for some other cases where names are built and found procedure with names like <name>Debug, <name>Debug0 and other suffixes like SeDate, FIn, FPrinter. I am trying to understand if this will be an issue and what I need to fix next.

I didn't find any place where this might be generated, so 10626a can be reviewed as is.

Code Review Task Branch 10626a Revision 16221

I don't have a problem with the changes. Please help me understand why frame_generator and frame_construction don't need to be backed by the H2Map implementation.

Greg Shah wrote:

Code Review Task Branch 10626a Revision 16221

I don't have a problem with the changes. Please help me understand why frame_generator and frame_construction don't need to be backed by the H2Map implementation.

Because frame names are created in frame_scoping.rules and the framePool Map from frame_generator.xml is populated with the javaname created in the mentioned rules set. Since there's a single place where the frame name is created, there's no need to store an additional H2Map.

I did a conversion with a customer application and forwarded the jars to Artur for testing, where he found 3 tests failing during harness. I used trunk/16220 for the conversion.

Dănuț Filimon wrote:

I did a conversion with a customer application and forwarded the jars to Artur for testing, where he found 3 tests failing during harness. I used trunk/16220 for the conversion.

I'll redo the conversion using 10164_main and see if the issue is the same.

The bare-metal machine for the application that failed was updated recently and I've been running conversions on it with no issues. I plan to retest the conversion and add another application to the list.

Rebased 10626a to latest trunk/16312, the branch is now at revision 16313.

All testing passed! 10626a can be merged!