Developing 4GL Code During and After Conversion - FWD - Golden Code Redmine

Developing 4GL Code During and After Conversion¶

Developing 4GL Code During and After Conversion

This chapter explores two separate, but related concepts:

managing the requirement for change to your Progress 4GL source code base if the project is expected to be able to reconvert in the future without losing any updates or integrated Java features.
continuing development in the Progress 4GL after a conversion project is completed.

FWD is designed to fully enable both of these activities. The primary limitations at this time relate to the fact that FWD does not yet offer a full IDE for 4GL development. This IDE support is under development. When that is available, no external editor/debugger will be needed for 4GL development.

This chapter discusses approaches and points out the known limitations to integrating new or modified Progress 4GL code into a converted application.

Managing Change During a Conversion Project¶

The needs of an organization are unlikely to stand still during the conversion of a production application. Defects must be fixed, change requests addressed. Until the converted system is in production, the existing system must be serviced. This requires close coordination and cooperation among the team which maintains and enhances the existing system, and the team which converts the application.

There is no reason to limit 4GL development during the conversion project itself. Work on the conversion can occur in parallel and periodically any changes can be picked up and converted. This can be daily, weekly, monthly... or whatever.

Be sure to limit any new or modified 4GL source code to the use of language features that are already supported and/or are already planned to be supported by FWD. Even if FWD has support for a specific language statement, built-in function or other feature, it is possible for that support to be partial. For example, there are frame options, widget methods and attributes or other details of a given feature that may not yet be supported, even if the majority of that feature is supported. When in doubt, check the FWD Conversion Reference or the FWD source code itself to determine if a given feature is supported.

Post-Conversion Progress 4GL Development¶

A primary goal of converting a Progress 4GL application to Java using the FWD technology is usually to make a clean break from the OpenEdge environment. This can be done with FWD without abandoning 4GL development. Long term development can be 100% 4GL, 100% Java or (most often) some mixture of 4GL and Java development. Over time, if it makes sense, files/modules or other parts of the code can be shifted into Java development. As this happens, these files/modules are removed from the conversion list. However, all the rest of the code can continue to be converted using the automated conversion process. The result is that organizations can shift their development processes/languages/tools on a completely different schedule from the shift to FWD. Considering the great value of the existing 4GL development team, it is almost always the correct decisiont to continue maintaining the Progress 4GL code base using 4GL development, at least for some time. Since it is increasingly difficult to hire/find new 4GL developers, it is also very likely that adding Java resources and shifting some development (especially of new code) to Java makes very good sense. This will also make it easier to leverage the full power of the Java platform, which is substantially greater than what can be done in the 4GL in OpenEdge.

Choosing to Modify the Source Code in 4GL or Java¶

When needing to make a change to a converted application, you have choose whether you should change the original 4GL code (and reconvert) or whether you should edit the converted Java code. Even in a scenario where Java is the preferred development environment, the following are reasons to modify the 4GL (and reconvert):

The code must run on both a Progress 4GL environment and the Java-based FWD environment.
The changes are relatively small (e.g. fixing some easy bug).
The changes require an intrusive modification of the block structure of an existing 4GL file.
The changes cannot be isolated in a reasonable number of procedures (in order to add Java code to a converted 4GL code via conversion hints you have to isolate your code into procedures).

It is possible that some 4GL source code changes will not be honored in the resulting reconverted source code. This can occur in a variety of cases. For example, if the conversion detects that a particular portion of code is unreachable or otherwise “dead”, that code will be dropped. Likewise, it is possible to customize the conversion project such that code in particular files is replaced in a way that will not directly convert the actual 4GL code.

The FWD conversion will be updated over time to improve refactoring, optimization, formatting and to generally make the resulting generated code cleaner, smaller, more readable and more maintainable. In addition, should there be any defects in the conversion process which leave behind currently unidentified problems in the code, FWD updates could fix these issues. For this reason, there is an important advantage of being able to re-run conversion over time. So long as this is possible, running conversion with improved FWD code applies the conversion improvements and fixes across the entire project with no manual effort. Keeping this open as a possibility does require some extra consideration, but this is well worth the effort.

Write the changes in Java when:

All future development will be done exclusively in Java AND there is no interest in ever running the conversion using an improved future version of FWD.
The functionality can be isolated in new external procedures. The resulting Java code most likely will be more readable and will have better performance than the converted 4GL code. There are techniques that can be used to call this new code from converted 4GL code without breaking the ability to re-convert the 4GL code.
There is a need to use functionality or frameworks that only exist in Java.
The functionality would be more easily and/or simply written in Java.

Techniques for Integrating Java Code into Reconverted 4GL Code¶

There are many advantages to being able to re-convert a project over time (see the section above). However, there are also many advantages to being able to write new/arbitrary Java code and to integrate that into the converted application. The following techniques can be used to accomplish both objectives.

Direct Java Access¶

Starting in FWD v4, it is possible to use the OO 4GL syntax for Direct Java Access. This means that Java objects and classes can be directly referenced/instantiated from 4GL code and any member data or methods are fully available. See that linked document for details. This is the recommended approach for utilizing Java code from converted 4GL code.

Including Java Classes in the Converted Project¶

Java classes can be included into project for the following purposes:

adding new Java-written external procedures (represented by Java classes);
adding new Java-written functions (represented by Java static functions);
replacing any converted classes with modified ones;
adding any additional Java classes.

In order to use these classes they should be available in the classpath of the target FWD server. The easiest way to do it is to include these files into application's jar. Follow these steps to do it:

Java classes should physically reside in some folder other that src. A common place for them is the srcnew directory (create it at the same level as the src directory).
If you are going to use Java classes as external procedures called from the converted code, these classes should logically reside under the root package of the project (which matches pkgroot parameter into p2j.cfg.xml). For the examples below the root package is com.company.project and files should reside under srcnew/@com/company/project@
directory
You should modify build.xml script in order to copy files from srcnew to src before compiling project classes:

<!-- Add new parameter which points to the srcnew directory. -->
<property name="srcnew.home"   value="${basedir}/srcnew" />
...
<!-- Modify “prepare” target in order to copy all files from srcnew to src before
     compiling files into src. -->
<target name="prepare">
...
   <copy todir="${src.home}">
      <fileset dir="${srcnew.home}" />
   </copy>
</target>

Calling a Java “External Procedure”¶

Consider you've added new Java-written external procedure (i.e. new .java file). Let's name it NewProcedure.java and placed it under com.company.project.util package (where com.company.project is the root folder of the project). In order for it to be treated as an external procedure, it must have a void execute() method. An example of such procedure:

package com.company.project.util;

import com.goldencode.p2j.util.*;
import static com.goldencode.p2j.util.BlockManager.*;

public class NewProcedure
{
   public void execute()
   {
      externalProcedure(new Block()
      {
         public void body()
         {
            // procedure code
         }
      });
   }
}

Procedure with parameters will look in this way:

package com.company.project.util;

import com.goldencode.p2j.util.*;
import com.goldencode.p2j.persist.*;
import project.dmo.*;
import static com.goldencode.p2j.util.BlockManager.*;

public class NewProcedure
{
   public void execute(final character p1, final integer p2)
   {
      externalProcedure(new Block()
      {
         public void init()
         {
            // register all parameters
            TransactionManager.register(p1, p2);
         }

         public void body()
         {
            // procedure code
         }
      });
   }
}

In order to run this procedure from 4GL code you have to do the following steps:

Add the mapping of the Java class to a virtual 4GL procedure to cfg/name_map_merge.xml file:
```
<class-mapping jname="util.NewProcedure" pname="util/new-procedure.p"/>
```
The Progress 4GL name (pname) of the virtual procedure can be arbitrary, in does not need to have any relationship to the Java name (jname). On the other hand, the Java name must exactly match the new class.

Then run this virtual procedure from 4GL code, edit the 4GL in any location that must invoke the functionality, using the defined pname instead of the Java name:

run util/new-procedure.p.

or, with parameters:

def var p1 as character.
def var p2 as integer.
...
run util/new-procedure.p (p1, p2).

These calls will result in the following Java code after conversion:

import com.company.project.util.NewProcedure;
...
NewProcedure newProcedure = new NewProcedure();
newProcedure.execute();

or, with parameters

import com.company.project.util.NewProcedure;
...
NewProcedure newProcedure = new NewProcedure();
newProcedure.execute(p1, p2);

Calling a Java Method¶

You can call a Java-written static function from 4GL program. Consider you have the static function character NewProcedure.newFunction(integer val). Follow these steps to call this function from a 4GL program:

Declare a virtual function with the same return type and the same parameters as the target Java function has:
```
function new-function returns character (val as integer):
   return ?.
end.
```

Add to pattern/customer_specific_annotations_prep.rules to the walk-rules section rule that will convert the calls of virtual 4GL functions to the calls of Java function:

<walk-rules>
   <!-- this node represents a function call and the node text equals the
        name of our function (i.e. this is our function)  -->
   <rule>evalLib("function_calls") and
         text.equalsIgnoreCase("new-function")

      <!-- change the call of the virtual function to the call of
           Java function -->
      <action>copy.setType(prog.customer_specific)</action>
      <action>copy.setText("NewProcedure.newFunction")</action>

      <!-- if the class which contains the target Java function is outside of the
           current package, then we should add appropriate import statement -->
      <action>copy.putAnnotation("import", "com.company.project.util.*")</action>
   </rule>
   ...
</walk-rules>

Call the virtual function from 4GL code where it is needed:

def var str as char.
def var num as integer.
...
str = new-function(num).

It will result in the following converted code:

import com.company.project.util.*;
...
str.assign(NewProcedure.newFunction(num));

Using Standard Tools in customer_specific_annotations_prep File¶

The convert/standard_customer_specific_tools.rules file exposes some functions which help working with or creating prog.customer_specific nodes. The next paragraphs will describe all the available tools exposed by the standard_customer_specific_tools file and their associated annotations, if any.

When using the standard tools in the rules defined by the customer_specific_annotations_prep file, one must expose the functions which define these tools to the current rule set. For this, add the following lines under the root node, in the customer_specific_annotations_prep file:

<!-- include the standard tools library -->
<include name="convert/standard_customer_specific_tools" />

Another library with common APIs is defined by the include/common-progress.rules file, which exposes various helper functions which work with a progress AST. To include it, add the following lines under the root node, in the customer_specific_annotations_prep file:

<!-- include common expression libraries -->
<include name="common-progress" />

Besides working with prog.customer_specific nodes, the customer_specific_annotations_prep file provides a way to do other types of processing, such as dropping frame properties which are not yet supported or removing obsolete program calls, as the following examples demonstrate:

Example 1:

define button but no-focus.
def var i as int.
form i but with frame f1 3d.

TRPL Rules:

<rule>type == prog.kw_no_focus and
      parent.type == prog.define_button
  <action>copy.remove()</action>
</rule>

Converted Code:

public interface ToolsTestF1
extends CommonFrame
{
   ...
   public static class ToolsTestF1Def
   extends WidgetList
   {
      FillInWidget i = new FillInWidget();

      ButtonWidget but = new ButtonWidget();

      public void setup(CommonFrame frame)
      {
         frame.setDown(1);
         i.setDataType("integer");
         but.setLabel("but");
         i.setLabel("i");
      }

      {
         addWidget("i", "i", i);
         addWidget("but", "", but);
      }
   }
}

Details:

Assuming FWD does not support the no-focus property for buttons, we can automatically drop it at conversion time.

Example 2:

message userid("test").
message userid.

TRPL Rules:

<rule>
   type == prog.func_char                   and
   isNote("oldtype")                        and
   getNoteLong("oldtype") == prog.kw_userid and
   isNote("builtin")                        and
   getNoteBoolean("builtin")                and
   this.numImmediateChildren > 0
   <action>copy.getChildAt(0).remove()</action>
</rule>

Converted Code:

message(SecurityOps.getUserId());
message(SecurityOps.getUserId());

Details:

As the 4GL's USERID builtin function is automatically converted to a SecurityOps.getUserId() call and FWD does not support the USERID's schema parameter, we need to automatically drop the function's parameter. Without the above rules in the customer_specific_annotations_prep file, the converted code would look like:

message(SecurityOps.getUserIdFromDB(new character("test")));
message(SecurityOps.getUserId());

which is not useful, as SecurityOps.getUserIdFromDB always returns an empty string.

“assign_wrapper” annotation and add_assign_wrapper tool¶

The assign_wrapper annotation marks any prog.customer_specific node so that it will be converted to a 4GL-style assignment, using the assign API for its 4GL compatible data type. The left-side value of the assignment will be the field/variable named by the text in the prog.customer_specific node; the right-side of the assignment will be a reference to an AST which defines the instance field or variable, given by the refid annotation.

Assuming the lastref is a reference for the new prog.customer_specific node, the following TRPL code is needed to set the proper annotations:

<action>lastref.putAnnotation("assign_wrapper", true)</action>
<action>lastref.putAnnotation("refid",          refid)</action>

where refid is the ID of the AST node which holds the expression/variable/field.

This annotation has an associated API, add_assign_wrapper, which groups together all the rules needed to create and annotate a prog.customer_specific node. The parameters of this API are:

target represents the AST node to which the new node will be added (a com.goldencode.ast.Aast instance)
idx represents the position in target's children where the new node will be added (a java.lang.Integer value).
refid is a java.lang.Long value which represents the id of the node which holds the expression/variable which will be set as parameter to the assign wrapper node.
innerGetter is a java.lang.String value which, when set to a non-null value, replaces the right-side of the 4GL-style assignment with a Java static method call.

This API returns a com.goldencode.ast.Aast instance, a reference to the new prog.customer_specific node with the assign_wrapper annotation.

“assign” annotation and add_assign tool¶

The assign annotation marks any prog.customer_specific node so that it will be converted to a Java-style assignment, using Java's assign (=) operator. The left-side value of the assignment will be the field/variable named by the text in the prog.customer_specific node; the right-side of the assignment will be a reference to an AST which defines the expression, field or variable, given by the refid annotation.

Assuming the lastref is a prog.customer_specific node, the following TRPL code is needed to set the proper annotations:

<action>lastref.putAnnotation("assign", true)</action>
<action>lastref.putAnnotation("refid",  refid)</action>

where refid is the ID of the AST node which holds the expression/variable/field.

This annotation has an associated API, add_assign, which groups together all the rules needed to create and annotate a prog.customer_specific node. The parameters of this API are:

target represents the AST node to which the new node will be added (a com.goldencode.ast.Aast instance)
idx represents the position in target's children where the new node will be added (a java.lang.Integer value).
refid is a java.lang.Long value which represents the id of the node which holds the expression/variable which will be set as parameter to the assign wrapper node.

This API returns a com.goldencode.ast.Aast instance, a reference to the new prog.customer_specific node with the assign annotation.

“vardef” annotation and add_vardef tool¶

The vardef annotation marks any prog.customer_specific node so that it will be converted to a definition of an instance field for the associated external procedure class. Assuming the lastref node is a prog.customer_specific node, the following TRPL code is needed to set the proper annotations:

<action>lastref.putAnnotation("vardef",    true)</action>
<action>lastref.putAnnotation("classname", <classname>)</action>
<action>lastref.putAnnotation("javaname",  <javaname>)</action>
<action>lastref.putAnnotation("def_init",  <def_init>)</action>
<action>lastref.putAnnotation("import", <import>)</action>

where the <...> parameters have the same meaning as the parameters for this annotation's associated API (which creates a prog.customer_specific node with all the needed details), add_vardef:

target represents the AST node to which the new node will be added (a com.goldencode.ast.Aast instance)
idx represents the position in target's children where the new node will be added (a java.lang.Integer value).
import is the package/class import required to have access to the variable's data type. This is an optional parameter, and must be a java.lang.String value or null.
classname represents the new variable's type (a java.lang.String value), with or without the package.
javaname represents the instance field name (a java.lang.String value).
def_init, if set to true, will initiate the instance field with an object created using the default constructor; else, the instance field will be assigned to null.

This API returns a com.goldencode.ast.Aast instance, a reference to the new prog.customer_specific node with the vardef annotation.

“varref” annotation and add_varref tool¶

The varref annotation marks any prog.customer_specific node to emit a variable/instance field reference. Assuming the lastref node is a prog.customer_specific node, the following TRPL code is needed to set the proper annotations:

<action>lastref.putAnnotation("varref", true)</action>
<action>lastref.putAnnotation("refid",  <refid>)</action>

where the <refid> parameter has the same meaning as the refid parameter for this annotation's associated API (which creates a prog.customer_specific node with all the needed details), add_vardef:

target represents the AST node to which the new node will be added (a com.goldencode.ast.Aast instance)
idx represents the position in target's children where the new node will be added (a java.lang.Integer value).
refid is a java.lang.Long value which represents the id of the node which holds the member variable which will be referenced.

This API returns a com.goldencode.ast.Aast instance, a reference to the new prog.customer_specific node with the varref annotation.

How to Use the varref, vardef, assign and assign_wrapper¶

This example demonstrates how to use the varref, vardef, assign and assign_wrapper annotations to replace all user-name variable references which are used directly in a IF clause with the user name determined from the SecurityManager.getUserId(). Assuming we have the following snippet of 4GL code:

def shared var user-name as char.
...
if user-name = “admin”
then ...

we need to end up with this kind of converted code:

import com.goldencode.p2j.security.SecurityManager;
...
public class Test
{
   SecurityManager _securitymanager_ = null;

   character _username_ = new character();
   ...

   public void execute()
   {
      externalProcedure(new Block()
      {
         public void body()
         {
            _securitymanager_ = SecurityManager.getInstance();
            _username_.assign(_securitymanager_.getUserId());
            if (_isEqual(_username_, "admin"))
            {
               message("you have access");
            }
         }
      });
   }
}

Looking at the Java code, we notice that the following does not exist in the original 4GL code, and we need to construct it by hand:

_securitymanager_ and _username_ variable definitions;
initialize _securitymanager_ variable to SecurityManager.getInstance()@;
save the user ID to _username_ variable;
replace the has-admin-access reference with a _username_ reference.

The first thing we need to do is make sure we are processing the correct node, the if has-admin-access statement, via the following test:

<rule>type == prog.var_char and
      text == "user-name"   and
      upPath(“KW_IF/EXPRESSION/EQUALS”)
...
</rule>

which checks if we are on a logical variable named has-admin-access and this is a child of a KW_IF node. All other rules will be enclosed in this rule.

Second, we need to manufacture the _securitymanager_ variable definition, which will be held by a prog.customer_specific node, and initialize it. For this, we can use the add_vardef API:

   <!-- define the _securitymanager_ variable and set it to null -->
   <action>
      lastref = execLib("add_vardef",
                        copy.parent,
                        copy.parent.indexPos,
                        "com.goldencode.p2j.security.SecurityManager",
                        "SecurityManager",
                        "_securitymanager_",
                        false)
   </action>
   <!-- save the ID of the newly created node -->
   <action>smid = lastref.id</action>

and the add_assign API to initialize it:

   <!-- add the left-side of the Java-style assignment -->
   <action>
      lastref = execLib("add_assign",
                        copy.parent,
                        copy.parent.indexPos,
                        smid)
   </action>
   <!-- attach the SecurityManager.getInstance as the right-side of the assignment -->
   <action>
      lastref = createProgressAst(prog.customer_specific,
                                  "SecurityManager.getInstance",
                                  lastref,
                                  0)
   </action>

where copy.parent and copy.parent.indexPos was used so that the variable definition and the assignment will be added just before the KW_IF node. Also, lastref is a variable which holds a reference to the new AST node and smid is the ID of the AST node which holds the _securitymanager_ variable definition.

Third, we need to declare the _username_ variable:

   <!-- define the _username_ variable and instantiate it -->
   <action>
      lastref = execLib("add_vardef",
                        copy.parent,
                        copy.parent.indexPos,
                        null,
                        "character",
                        "_username_",
                        true)
   </action>
   <!-- save the ID of the newly created node -->
   <action>uid = lastref.id</action>

and save the user ID as returned by the _securitymanager_.getUserId call via an add_assign_wrapper call:

   <!-- build the right-side part of the assignment -->
   <action>
      lastref = execLib("add_assign_wrapper",
                        copy.parent,
                        copy.parent.indexPos,
                        uid,
                        null)
   </action>

   <!-- the left side (the parameter for the assign API) is a
        _securitymanager_.getUserId call -->
   <action>
      lastref = createProgressAst(prog.customer_specific,
                                  "getUserId",
                                  lastref,
                                  0)
   </action>
   <action>lastref.putAnnotation("nonstatic", true)</action>
   <action>lastref.putAnnotation("needs_parm", true)</action>
   <action>lastref.putAnnotation("refid", smid)</action>

Finally, we need to replace the user-name variable reference with a _username_ variable reference:

   <!-- replace user-name with _username_ reference -->
   <action>
      lastref = execLib("replace_with_customer_specific",
                        copy, "_username_", null, "character")
   </action>

where the replace_with_customer_specific tool was used to replace the current node with a prog.customer_specific node. This node needs to be explicitly annotated with varref and refid annotations:

   <action>
      lastref.putAnnotation("varref", true)
   </action>
   <action>
      lastref.putAnnotation("refid", uid)
   </action>

where uid is the ID of the AST node which declares the _username_ variable.

To conclude, the entire TRPL code which performs this task is:

<rule>type == prog.var_char and
      text == "user-name"   and
      upPath(“KW_IF”)

   <variable name="lastref" type="com.goldencode.ast.Aast" />
   <variable name="smid" type="java.lang.Long" />
   <variable name="uid" type="java.lang.Long" />

   <!-- 1. define the _securitymanager_ variable and set it to null -->
   <action>
      lastref = execLib("add_vardef",
                        copy.parent,
                        copy.parent.indexPos,
                        "com.goldencode.p2j.security.SecurityManager",
                        "SecurityManager",
                        "_securitymanager_",
                        false)
   </action>
   <!-- save the ID of the newly created node -->
   <action>smid = lastref.id</action>

   <!-- 2. add the left-side of the Java-style assignment -->
   <action>
      lastref = execLib("add_assign",
                        copy.parent,
                        copy.parent.indexPos,
                        smid)
   </action>
   <!-- attach the SecurityManager.getInstance as the right-side of the assignment -->
   <action>
      lastref = createProgressAst(prog.customer_specific,
                                  "SecurityManager.getInstance",
                                  lastref,
                                  0)
   </action>

   <!-- 3. define the _username_ variable and instantiate it -->
   <action>
      lastref = execLib("add_vardef",
                        copy.parent,
                        copy.parent.indexPos,
                        null,
                        "character",
                        "_username_",
                        true)
   </action>
   <!-- save the ID of the newly created node -->
   <action>uid = lastref.id</action>

   <!-- build the right-side part of the assignment -->
   <action>
      lastref = execLib("add_assign_wrapper",
                        copy.parent,
                        copy.parent.indexPos,
                        uid,
                        null)
   </action>

   <!-- the left side (the parameter for the assign API) is a
        _securitymanager_.getUserId call -->
   <action>
      lastref = createProgressAst(prog.customer_specific,
                                  "getUserId",
                                  lastref,
                                  0)
   </action>
   <action>lastref.putAnnotation("nonstatic", true)</action>
   <action>lastref.putAnnotation("needs_parm", true)</action>
   <action>lastref.putAnnotation("refid", smid)</action>

   <!-- 4. replace user-name with _username_ reference -->
   <action>
      lastref = execLib("replace_with_customer_specific",
                        copy, "_username_", null, "character")
   </action>
   <action>
      lastref.putAnnotation("varref", true)
   </action>
   <action>
      lastref.putAnnotation("refid", uid)
   </action>
</rule>

“constructor” annotation and add_constructor tool¶

The constructor annotation marks any prog.customer_specific node to emit a constructor to instantiate a new object of the given type. Assuming the lastref node is a prog.customer_specific node, the following TRPL code is needed to set the proper annotations:

<action>lastref.putAnnotation("constructor", true)</action>
<action>lastref.putAnnotation("import",  <import>)</action>

where the <import> parameter has the same meaning as the import parameter for this annotation's associated API, add_constructor, which creates a prog.customer_specific node with all the needed details:

target represents the AST node to which the new node will be added (a com.goldencode.ast.Aast instance)
idx represents the position in target's children where the new node will be added (a java.lang.Integer value).
classname represents the new object's type (a java.lang.String value representing the class name).
import is the package/class import required to have access to this type. This is an optional parameter, and must be a java.lang.String value or null.

This API returns a com.goldencode.ast.Aast instance, a reference to the new prog.customer_specific node with the varref annotation.

Example:

def var i as int.
RUN foo.p.
RUN bar.p(input i).

TRPL Rules:

01: <rule>type == prog.statement and downPath("KW_RUN/FILENAME")
02:    <!-- get the filename node -->
03:    <action>
04:      ref = copy.getChildAt(0).getImmediateChild(prog.filename, null)
05:    </action>
06:
07:    <rule>ref.text.equalsIgnoreCase("foo.p") or
08:          ref.text.equalsIgnoreCase("bar.p")
09:
10:      <!-- change statement node -->
11:      <action>copy.setType(prog.customer_specific)</action>
12:      <action>putNote("constructor", true)</action>
13:
14:      <!-- set the c'tor, depending on which program -->
15:      <rule>ref.text.equalsIgnoreCase("foo.p")
16:        <action>copy.setText("FooOverwritten")</action>
17:      </rule>
18:      <rule>ref.text.equalsIgnoreCase("bar.p")
19:        <action>copy.setText("BarOverwritten")</action>
20:      </rule>
21:
22:      <!-- add imports as neeed -->
23:      <action>putNote("import", "com.company.project.*")</action>
24:
25:      <!-- check if there are parameters -->
26:      <action>
27:   ref = copy.getChildAt(0).getImmediateChild(prog.lparens, null)
28:      </action>
29:
30:      <!-- remove the statement's children, to leave only the new node  -->
31:      <action>copy.getChildAt(0).remove()</action>
32:
33:      <!-- save the parameters so they will be emitted for the c'tor-->
34:      <rule>ref != null
35:         <!-- it is enough to link only the first parameter,
36:              as the next ones will automatically follow -->
37:         <action>copy.addChild(ref.getFirstChild())</action>
38:      </rule>
39:    </rule>
40: </rule>

Converted Code:

import com.company.project.*;
...
new FooOverwritten();
new BarOverwritten(i);

Details:

When the above TRPL code is placed in the customer_specific_annotations_prep file, it will convert all RUN foo.p and RUN bar.p statements to an automatic invocation of a specific constructor. The constructor for those specific classes may run another external procedure or execute some other code.

The TRPL rules perform the following tasks:

line 01 makes sure we are on a RUN program statement.
lines 02 to 05 get the program name targeted by the RUN statement.
lines 07 and 08 checks if we are on the correct program (foo.p and bar.p).
line 11 changes the type of the current node from prog.statement to prog.customer_specific.
line 12 marks the node with the constructor annotation.
lines 15 to 17 change the text of the node to the class name which replaces the foo.p program, FooOverwritten.
lines 18 to 20 change the text of the node to the class name which replaces the bar .p program, BarOverwritten.
line 23 adds the needed import, com.company.project.*.
lines 25 to 28 check if there are any parameters for the RUN statements. If any, they would be kept in a prog.lparens child node for the prog.kw_run node, and a reference to this node will be saved.
line 31 removes all the children for the current node, as this node is now a prog.customer_specific node.
lines 33 to 38 append all the parameters (if any) to the current node. Only the first parameter node is required to be added as a child to the prog.customer_specific node, as the next ones will be linked to the first one. Also, note that the prog.lparens node is not added as a child, instead its first child (the first parameter) is added.

“import” annotation¶

Whenever a prog.customer_specific node is encountered and this annotation is set, an import statement will be automatically added by the standard tools (if the node was not already handled by the customer tools). If lastref is a prog.customer_specific node, to set the required import for this node use the following TRPL code:

<action>lastref.putAnnotation("import", <package_or_class>)</action>

where <package_or_class> is a specific package name or a full class name.

Standard Conversion of prog.customer_specific Nodes¶

If a prog.customer_specific node is encountered with none of the assign_wrapper, assign, vardef, varref or constructor annotations set, the node will be automatically converted to a Java static or instance method call. The actual method name will be taken from the node's text, which is set when the prog.customer_specific node is created, as in:

<variable name="lastref" type="com.goldencode.ast.Aast" />
<action>
   lastref = createProgressAst(prog.customer_specific, <methodname>, <target>, <idx>)
</action>

where:

<methodname> is the static or instance method name.
<target> is the parent AST to which the new node will be attached.
<idx> is the 0-based position in the target's AST children where the new node will be attached. If not specified, this is assumed to be -1 which creates the node as the last child.

To distinguish between static and non-static method calls, any instance method call must be annotated with the nonstatic annotation. In addition to this annotation, the needs_parm annotation must be set and also the refid annotation must be set and point to the variable definition which references the object used for the instance call. Assuming lastref is a prog.customer_specific node, the TRPL code looks like:

<action>lastref.putAnnotation("nonstatic", true)</action>
<action>lastref.putAnnotation("needs_parm", true)</action>
<action>lastref.putAnnotation("refid", <refid>)</action>

When converting to static method calls, if the prog.customer_specific node has a needs_parm annotation, it means the method call requires explicit parameters set by the conversion rules. Currently, only a single parameter can be added to the static method call. This parameter will be resolved from the refid annotation of the prog.customer_specific node, as for varref and assign cases.

Example 1:

def var i as int.

function java-static-call returns char(input i as int):
   return ?.
end.

message java-static-call(input i).

function java-instance-call returns char(input i as int):
   return ?.
end.

message java-instance-call(input i).

TRPL Rules:

01: <rule>evalLib("function_calls")
02:    <rule>type == prog.func_char and text.equalsIgnoreCase("java-static-call")
03:      <action>copy.setType(prog.customer_specific)</action>
04:      <action>copy.setText("ExternalClass.javaStaticCall")</action>
05:    </rule>
06:
07:    <rule>type == prog.func_char and text.equalsIgnoreCase("java-instance-call")
08:      <action>copy.setType(prog.customer_specific)</action>
09:      <action>copy.putAnnotation(“nonstatic”, true)</action>
10       <action>copy.putAnnotation(“needs_parm”, true)</action>
11:      <action>copy.putAnnotation(“refid”,     oid)</action>
12:      <action>copy.setText("javaInstanceCall")</action>
13:    </rule>
14: </rule>
15:
16: <rule>type == prog.kw_funct       and
17:      parent.type == prog.function and
18:      isNote("name")               and
19:      (getNoteString("name").equalsIgnoreCase("java-static-call") or
20:       getNoteString("name").equalsIgnoreCase("java-instance-call"))
21:    <action>copy.parent.setHidden(true)</action>
22: </rule>

Converted Code:

message(ExternalClass.javaStaticCall(i));
message(object.javaInstanceCall(i));

Details:

When the above TRPL code is placed in the customer_specific_annotations_prep file, it will convert all java-static-call function calls to a ExternalClass.javaStaticCall call and all java-instance-call function calls to a object.javaInstaceCall call. In both cases, the function code is not written in 4GL, but instead is written directly in Java. These functions are placeholders that are not meant to ever be executed in the 4GL, but which exist only to allow the conversion to “hook in” and replace those calls with arbitrary calls to custom Java code. To be able to pass the first phase of FWD conversion, the 4GL code needs existing empty function definitions, with the same return type and parameters as the ones being invoked. As they are not needed in the final Java code, these function definitions will be hidden automatically.

The TRPL rules perform the following tasks:

line 01 makes sure we are on a function call node.
lines 02 to 05 rewrites this node to a prog.customer_specific node and changes its text from a java-static-call function call to ExternalClass.javaStaticCall call.
lines 07 to 14 rewrites this node to a prog.customer_specific node and changes its text from java-instance-call function call to a javaInstanceCall method call. In this case, the oid must point to an existing variable or instance field (line 11 sets the refid annotation, which tells the conversion rules from where to get the object reference). Note how in this case both the nonstatic and the needs_parm annotations are set.
line 16 to 22 will hide the bogus function definitions java-instance-call and java-static-call.

In this case, as we rewrite a 4GL function call to another explicit Java static or instance method call, all parameters set for the 4GL function call will emit naturally for the Java method call. If the needs_parm and the refid annotations would have been set for the prog.customer_specific node which converts to a static method call, then the single explicit parameter will be emitted first in the parameter list. Only one parameter may be specified using needs_parm.

Example 2:

def var i as int.

function java-static-call returns char(input i as int):
   return ?.
end.

message java-static-call(input i).

TRPL Rules:

01: <rule>evalLib("function_calls") and
02:       type == prog.func_char    and
03:       text.equalsIgnoreCase("java-static-call")
04:    <action>
05:       lastref = execLib("add_vardef",
06:                        copy.parent,
07:                        copy.parent.indexPos,
08:                        null,
09:                        "character",
10:                        "_test_",
11:                        true)
12:    </action>
13:    <action>copy.setType(prog.customer_specific)</action>
14:    <action>copy.setText("ExternalClass.javaStaticCall")</action>
15:    <action>copy.putAnnotation("needs_param", true)</action>
16:    <action>copy.putAnnotation("refid",       lastref.id)</action>
17: </rule>

Converted Code:

character _test_ = new character();
...
message(ExternalClass.javaStaticCall(_test_, i));

Details:

This example demonstrates how the needs_parm and refid annotations can be used to add an additional parameter to a static method call. The TRPL rules perform the following tasks:

lines 01 to 03 makes sure we are on a function call node.
lines 04 to 14 creates the new _test_ variable.
lines 13 and 14 rewrites this node to a prog.customer_specific node and changes its text from a java-static-call function call to ExternalClass.javaStaticCall call.
lines 15 and 16 add an additional parameter to the java method call, which will be emitted first.

replace_with_customer_specific tool¶

This is a special API which replaces the target node with a prog.customer_specific node, with the details given by the following parameters:

target represents the target AST node, which needs to be replaced (a com.goldencode.ast.Aast instance)
rtxt represents the text of this node (a java.lang.String value). This can be a method name, a variable name or something else, depending on the type of the node.
rimp is the package/class import required to have access to the method/variable set with the rtxt parameter. This is an optional parameter, and must be a java.lang.String value or null.
rtype represents the data type of this node's expression, set using a java.lang.String value. Usually, this is the return type or the data type of the variable represented by this node.

This API returns a com.goldencode.ast.Aast instance, a reference to the node which replaces the old one.

Example 1:

def var i as int.

function java-static-call returns char(input i as int):
   return ?.
end.

message java-static-call(input i).

TRPL Rules:

01: <rule>evalLib("function_calls")
02:    <rule>type == prog.func_char and text.equalsIgnoreCase("java-static-call")
04:       <action>
05:          execLib(“replace_with_customer_specific”,
06:                  this,
07:                  "ExternalClass.javaStaticCall",
08:                  null,
09:                  “character”)
10:       </action>
11:    </rule>
12: </rule>

Converted Code:

message(ExternalClass.javaStaticCall(i));
...
public character javaStaticCall(final integer _i)
{
   return characterFunction(new Block()
   {
      integer i = new integer(_i);

      public void body()
      {
         returnNormal(new character());
      }
   });
}

Details:

This example is converted in the same way as the example for the standard conversion of prog.customer_specific nodes paragraph. The single difference is that the replace_with_customer_specific is used to replace the current node with a custom one, used to emit the method call.

Also, as the bogus 4GL function definition is not hidden, it will be emitted in the converted code.

Example 2:

def shared var has-admin-access as logical.
...
if has-admin-access
then ...

TRPL Rules:

<rule>type == prog.var_logical and
      text == "has-admin-access" and
      upPath(“KW_IF”)
   <action>
      execLib("replace_with_customer_specific",
              copy,
              "SecurityChecks.hasAdminAccess",
              "com.company.project.security.*",
              "logical")
   </action>
</rule>

Converted Code:

if (SecurityChecks.hasAdminAccess())
{
  ...
}

Details:

Assuming we have a legacy 4GL shared variable named has-admin-access which is set on client login (reading rights from some the DB or other external resource), we can replace it with custom security checks which use the FWD security APIs to determine user's rights.

rewrite_run tool¶

This is a special tool which can rewrite RUN program statements so that another program/method will be invoked. This is useful in cases when the target 4GL program is obsolete (or will not survive the conversion, as will be replaced by some other Java API calls), such as when the application's login program is discarded and replaced with a different version, written directly in Java (so the FWD's security and authentication APIs can be used).

ref represents the AST node which holds the RUN program statement (a node com.goldencode.ast.Aast instance, of prog.statement type)
mthd represents the method to be executed (a java.lang.String value). This can be either a static or instance method, assuming the instance variable was defined previously.
import is the package/class import required to have access to the specified method. This is an optional parameter, and must be a java.lang.String value or null.

When using this API, it is mandatory to be executed only when the currently processed node satisfies the type == prog.statement and downPath("KW_RUN/FILENAME") condition (i.e. the node which holds the RUN program statement is being processed).

Once this rule is satisfied, the program's name can be found and saved using this TRPL code:

<action>
   ref = copy.getChildAt(0).getImmediateChild(prog.filename, null)
</action>

After the node where the 4GL program name is found, if this is a program of interest, we can rewrite the statement. To conclude, when executing this API, the the TRPL code in the following snippet must be used:

<rule>type == prog.statement and downPath("KW_RUN/FILENAME")
   <variable name="ref"   type="com.goldencode.ast.Aast" />

   <!-- get the filename node -->
   <action>
      ref = copy.getChildAt(0).getImmediateChild(prog.filename, null)
   </action>
   <!-- check if we are a match for our program -->
   <rule>ref.text.equalsIgnoreCase(<prog1>)
      <action>
         execLib("rewrite_run",
                 ref,
                 <method1>,
                 <import1>)
      </action>
   </rule>

   <!-- continue checking as many other programs as needed -->
   <rule>ref.text.equalsIgnoreCase(<prog2>)
      <action>
         execLib("rewrite_run",
                 ref,
                 <method2>,
                 <import2>)
      </action>
   </rule>
</rule>

In the above snippet, the <prog1> and <prog2> programs will be replaced with other calls. Once we are under the correct node (i.e. the node with the STATEMENT/KW_RUN/FILENAME downpath), we can rewrite as many RUN statements as needed. Note that the rerwrite_run can handle RUN statements with or without arguments.

Example:

def var i as int.

run util/obsolete-program.p(input i).

TRPL Rules:

01: <rule>type == prog.statement and downPath("KW_RUN/FILENAME")
02:
03:    <variable name="ref"   type="com.goldencode.ast.Aast" />
04:
05:    <!-- get the filename node -->
06:    <action>
07:       ref = copy.getChildAt(0).getImmediateChild(prog.filename, null)
08:    </action>
09:
10:    <rule>ref.text.equalsIgnoreCase("util/obsolete-program.p")
11:       <action>
12:          execLib("rewrite_run",
13:                  ref,
14:                  "OtherProgram.newMethod",
15:                  "com.compay.app.util.*")
16:       </action>
17:    </rule>
18: </rule>

Converted Code:

OtherProgram.newMethod(i);

Details:

Assuming we have an obsolete 4GL program named util/obsolete-program.p and we want to overwrite it so that another program/method is executed. In this case, we rewrite all the RUN util/obsolete-program.p to an OtherProgram.newMethod static method call.

As this API can be used only when we are processing a certain type of node, line 01 makes sure we are currently processing the RUN statement. Lines 05 to 08 get the reference to the AST node which holds the program name. If this is our program, lines 10 to 17 will rewrite the RUN statement to the specific static method call.

Modifying a Converted Java Source File¶

You can modify the converted Java version of the procedure, and this changed version will automatically overwrite the converted version before building application classes. As it was discussed into “Including Java classes into converted project” chapter, you should place the target Java file under the srcnew directory. Note that if the converted version will change, you will need to change the modified version appropriately too! This can happen if:

you've changed 4GL version of this file;
you've added / removed some temporary tables to / from the project;
not likely, but nevertheless, if FWD conversion code has changed.

Overwriting converted files is something to avoid if at all possible, because if you suppose that the converted file may be changed after a new conversion, you will have to check it, and, if it is true, you will have to merge changes manually. That means that the long term maintenance of that file will take more effort.

Modifying Data Model Objects (DMOs)¶

If you are supporting the original system, then you can add DMOs or separate fields directly to the 4GL database, export its definitions to the .df file and then put that file into the data directory of the conversion project, replacing the old file with the new table definitions. In this case, new elements will be available both in the original and converted version. If the original 4GL system is no longer available, you can manually edit the .df schema file to add the new table and/or field definitions. Be careful when editing the 4GL schema file, as you must follow the syntax used by the 4GL schema export process; details about the syntax of this file and what is supported by FWD can be found in the Schema Disposition part of the FWD Conversion Reference book.

For an existing DMO, if a new index is created or there are some property related changes, the changes will need to be reflected in the dmo_index.xml file. If the merge file is still used and the DMO is defined in it, then all the changes can be safely added to it; else, if the DMO is generated during conversion (and belongs to the dmo_index.xml file), some changes will need to be done in the 4GL schema and others will be automatically picked up during conversion, from the 4GL programs which use this permanent table.

The changes which need to be done in the 4GL schema are related to table fields (i.e. adding, removing, changing type/name, field type, case sensitivity) and table indexes. Other info - like natural joins which translate to foreign relations or runtime defined indexes - will be read from the 4GL file and, during conversion, will automatically be associated with the DMO's definition, in the dmo_index.xml file.

You also can add DMOs or separate fields to the converted project directly in Java. These new elements will not be available for the original 4GL environment. If you want to be able to reconvert the project, then you cannot modify existing DMOs using Java, only add new DMOs. If the conversion is finished and you will never reconvert the project then you can directly edit generated DMO files using Java. You are free to add new DMOs or modify existing ones. Details about how to edit generated DMOs or add new DMOs directly in Java can be found in the Managing Data Model Object (DMO) Changes section of the Integrating Hand-Written Java chapter of this book.

Full Conversion vs. Partial Conversion¶

Partial Conversion¶

If you are making changes in the original 4GL code then you may want to see how your code will work on the converted system (e.g. on a production FWD server). You can convert full project, however it may take a lot of time. In this case you can convert the project partially: only modified files. Follow these steps in order to do it:

Create a text file that will contain relative or absolute paths to the files that you want to convert. Put it in the root folder of the project (where the conversion script resides). Each file that requires reconversion should be placed on a separate line. Only .p files (not .i or .w) should be included.
You should include all programs run from the set of the modified files using run statement. Programs can be run directly from the modified files or through a chain of run calls. Consider we have modified files prog1.p and prog2.p. prog1.p runs prog3.p. prog3.p runs prog4.p. So you have to add prog1.p, prog2.p, prog3.p and prog4.p to the target list.
Let's create our sample file target-files.txt, here you can see its content:
```
src/project/util/some-file.p
src/project/other-file.p
src/project/another/third-file.p
```

Modify the bash script that is used for the project conversion:
1. Find the line where ConversionDriver is called, it will look like this:
```
java ... com.goldencode.p2j.convert.ConversionDriver -sd1 f2+m0+cb src/project "*.[pPwW]" ...
```
You should use -f[options] key and pass the file which contains the list of files that should be reconverted for file specifications. In our case the previous line should be changed in this way:
```
java ... com.goldencode.p2j.convert.ConversionDriver -fd1 f2+m0+cb target-files.txt ...
```
2. Find the lines where ant is called. E.g.:
```
ant clean
ant jar 2>&1 | tee err_ant_jar.log
```
You should comment them out:
```
# ant clean
# ant jar 2>&1 | tee err_ant_jar.log
```
That will allow you to substitute name_map.xml and dmo_index.xml and will save your time by preventing full recompile.

Copy src/<project path>/name_map.xml in some other folder.
Copy src/<project path>/dmo/dmo_index.xml in some other folder.
If modified files contain temporary tables then, most likely, compilation of the project after partial reconversion will fail because temporary DMOs (TempRecord1, TempRecord2, etc.) have through numeration for all project files, and DMOs with then new numeration will intersect DMOs with the old numeration. So, if modified tables contain temporary tables, follow these steps to avoid this problem:
1. Search for the “TempRecord%d” into rules/schema/p2o.xml and replace it with, say, “TempRecordNew%d” - after that new temporary DMOs will be named TempRecordNew1, TempRecordNew2, etc. and they will not conflict with the old DMOs.
2. Search for the “tt%d” into rules/schema/p2o.xml and replace it with, say, “ttn%d” - after that the names of the indexed of new temporary DMOs will start with “idx__ttn” instead of “idx__tt” and they will not conflict with the indexes of the old DMOs.
3. Partially reconvert the project using the conversion script.
4. Find the newly generated src/<project path>/dmo/dmo_index.xml and add the contents of the <schema impl="_temp.impl" name="_temp"> section to the same section into the old dmo_index.xml file (you've saved it). You will get something like this:
```
<schema impl="_temp.impl" name="_temp">
   <class interface="TempRecordNew1"/>
   <class interface="TempRecordNew2">
      <index name="idx__ttn2_some_field">
         <column ignore-case="true" name="some_field"/>
      </index>
   </class>
    ...
   <class interface="TempRecord1"/>
   <class interface="TempRecord2">
      <index name="idx__tt2__other_another">
         <column ignore-case="true" name="other_field"/>
         <column ignore-case="true" name="another_field"/>
    </index>
   </class>
   <class interface="TempRecord3"/>
   ...
</schema>
```
Then replace the newly generated dmo_index.xml with the modified old one.

If modified files does not contain temporary tables (i.e. you have skipped the actions in the section №5) then:
1. Partially reconvert the project using the conversion script.
2. Put copied dmo_index.xml back to its original location overwriting the newly generated.
Put copied name_map.xml back to its original location overwriting the newly generated.
Compile newly generated sources by executing ant jar.

Before performing full reconversion don't forget to undo your changes into the conversion script and p2o.xml.

If you will have any problems with partial conversion, remember that full reconversion always is a safe option to use.

Partial Conversion Tips¶

Some additional incremental conversion tips:

If changes in the converted code are relatively small, then the best approach may be the following:
1. You should have two copies of the converted code. The first copy can be called “working” - you run converted application using this copy. The other can be called “temporary” - you perform project conversion using this copy.
2. Make a change in the 4GL code of the temporary copy and reconvert changed files.
3. Add changes from the converted code to the working copy using a code merging tool. You only need to inject into working copy Java code changes that reflect your changes in 4GL code. You should preserve numbering of temp tables of the working copy. This step requires some skills and understanding of the converted code.
4. You can build and run working copy

This approach allows you to speed up the development by skipping steps №3-7 from the “straightforward” partial conversion approach (see “Partial conversion” chapter) and let you have more control over a working copy. But it is only can be performed by a programmer that understands the converted code and may not be so efficient if changes in 4GL code are complex.

Step №1 in the “Partial conversion” chapter may take a long time if there are many dependent files. Instead, you may take the name_map.xml of the fully converted project and put it instead of cfg/name_map_merge.xml.
If procedures use shared temp-tables, then it may happen that during partial reconversion the file containing temp-table definition wasn't reconverted while the file that uses this temp-table - was, or vice versa. Consider the name of the shared temp table was TempRecord1 and become TempRecord2New after reconversion. So after reconversion files may look this way:

Class1.java:

TempRecord1 someRecord = TemporaryBuffer.define(TempRecord1.class, "someRecord", false);

Class2.java

TempRecord2New someRecord = (TempRecord2New) TemporaryBuffer.useShared("someRecord");

In this case you will get a runtime exception into Class2:

java.lang.ClassCastException: com.goldencode.p2j.persist.$Proxy...
cannot be cast to com.company.project.dmo._temp.TempRecord2New
        at com.company.project.Class2.<init>(Class2.java:...)

In order to fix it, you should manually change the class name of the DMO in the reconverted code back to its original class name. In our case it will be (Class2.java):

TempRecord1 someRecord = (TempRecord1) TemporaryBuffer.useShared("someRecord");

Don't forget to recompile changed file(s) and restart the application.

Project

General

Profile

FWD

Wiki