Project

General

Profile

Development Environment Setup

Introduction

FWD is both a set of automated conversion tools as well as a runtime used to provide an environment and set of ABL-compatible services for the proper execution of a converted application.

This chapter describes how to create a FWD development system. This requires an installation of the FWD project and enough of the related runtime components for the purpose building a development environment. This environment is suitable for converting an ABL application into a Java application. This installation described here is also suitable for the purposes of building, modifying, or enhancing the FWD technology.

With these instructions a user can establish an installation in which they can:

  • prepare a Progress 4GL application project for conversion;
  • run the analysis and reporting tools to inspect or otherwise process the 4GL source code;
  • run the automated conversion tools to translate the 4GL source into an equivalent Java application;
  • build/compile the results of conversion such that the application is ready to execute;
  • build the FWD code to allow use for conversion or runtime;
  • add, remove or otherwise modify, enhance or customize the FWD conversion or runtime code;
  • execute and test the converted application using the FWD application server and clients; and
  • source level debug the converted application and/or the FWD runtime code.

It is assumed that the FWD client, FWD application server and optionally the database server will be run on the same development system as where the FWD project is modified and built. This installation is intended to be for the convenience of the developer. It is not intended to be a production quality installation.

For production installs or or other non-developer installations, please see the Installation, Configuration and Administration Guide.

The instructions in this chapter are a superset of the Building FWD From Source chapter, the Conversion Environment Setup chapter and some of the Application Server Installation and Client Installation chapters.

System Requirements

These are the requirements for a FWD development system.

Hardware

Please see the Development Environment System Requirements for details on the hardware requirements.

Operating System

Java processes are normally platform neutral. When applications or tools are written in pure Java, it is possible for such processing to be run on any operating system or hardware platform that supports the proper level of J2SE (Java 2 Standard Edition) required. From the software point of view, you can install the client on any operating system as long as there is a JVM implementation for it.

At this time, while the vast majority of the client FWD code is written in pure Java, there is a small amount of platform-specific code (written in C) used for character mode user interface processing, process launching, native memory access and native library calls. That platform-specific code is implemented using the Java Native Interface (JNI) and is often referred to as “native” code. A customized version of that native code is included in the FWD project. This code is currently compiled as a 32-bit or 64-bit shared library for Linux (libp2j.so) or Windows (p2j.dll).

Linux Windows
FWD has been successfully run on a variety of recent Linux distributions from Ubuntu, SuSE, Red Hat and CentOS. There are no specific dependencies on any particular distribution. It is expected that the native code could be easily ported to any UNIX-compatible system. Any Linux/UNIX operating system can be used as long as exists support of NCURSES, libffi and JVM implementation for it. The client will run on any Windows OS, from XP or later. Both Windows XP and Windows 7 are well tested.

Network Requirements

Build

By default, the FWD build is written to download its dependencies (3rd party Java libraries) when needed. These will be cached locally and reused. The download is done by Gradle and it typically happens at the following times:

  • The first time that FWD is built.
  • Whenever the cache is cleared/deleted.
  • When a new FWD version requires an updated version or a new dependency.

For this download to occur, Internet access is required. Once downloaded and cached, no further Internet access is needed for building FWD.

Runtime

It is assumed that the FWD clients and the FWD application server will run on the development system itself.

It is possible to run the application server and the database server on the same system. In that case, there is typically nothing that must be done to enable the network path.

If the database server is run on a separate system and there is a firewall or router in the network path between the application server and the database server, then you must ensure that the firewall is configured to allow the related communication flows to occur.

Choosing the Appropriate JVM

The core requirement for the JVM is that it must be compatible with Java 8.

Each FWD component (conversion, application server, client, user-defined functions running at the database) can use a different JVM. These can run on the same system or different systems. They can even run on a different operating system platform. There is no dependency between JVMs of different components, since they are separate operating system processes.

The following JVM versions have been well tested:

  • Oracle JDK 1.8 on Linux
  • Oracle JDK 1.8 on Windows
  • OpenJDK 1.8 on Linux
  • OpenJDK 1.8 on Windows

The specific versions of Oracle JDK 1.8.0_101 and OpenJDK 1.8.0_111 have both been tested and have been found to work well. Generally, it is safe to use any later version of the Java 1.8 platform.

In order to build FWD and/or the converted application, you must have a JDK (Java Development Kit). It is not possible to use the JRE (Java Runtime Environment) since that is a Java distribution which lacks the necessary compiler and tools.

Only a 64-bit JDK should be used on a development system.

Installation Process

We will assume that the operating system is already installed and up-to-date.

The Linux instructions included here are written (and tested) on Ubuntu 16.04. If the target Linux system is a different distribution, the equivalent commands must be run.

The Windows instructions can be used for any Windows version from Windows XP through Windows 10.

Install the Java Virtual Machine

Please check previous paragraphs for details on how to choose the JDK to install.

Linux Windows
Make sure the bin/ directory of the JVM installation is added to PATH variable of the operating system so the application could be executed by using java command in a terminal window: the PATH for the user should be set to include the directory containing the command line executables (e.g. java or javac) of this installation. In Linux this would be the bin/ directory of the Java installation. In other words, this should be the default installation for any shell or command prompt from which the FWD technology will be built or used.

To install OpenJDK 1.8, following commands can be used:

sudo apt-get install openjdk-8-jdk openjdk-8-source
cd /usr/lib/jvm
sudo rm default-java
sudo ln -s java-8-openjdk-amd64 default-java


It is also recommended to set the user's CLASSPATH to current folder (“.”):
export CLASSPATH=.
Please note that this will only set the variable for the current terminal session. You will need to set this each time your start a new terminal session, if it is needed.

The Oracle JDK 1.8 is a well tested option for Windows. The JDK can be downloaded from http://www.oracle.com/technetwork/java/javase/downloads/index-jsp-138363.html.

If you are planning to build FWD binaries on Windows make sure to install the accompanying JRE when installing the JDK, this is the default option of the JDK's installer.

Make sure JAVA_HOME system variable exists and points to the location of the installed JDK. The variable is needed by the FWD build scripts to locate the Java build files. If not defined explicitly it would point to the location of the public JRE installed together with the JDK (or any other JRE set as the default one), which is missing the build files, and the build would fail.

Add %JAVA_HOME%/jre/bin/server to the system PATH variable. This ensures that the FWD client spawner process, spawn.exe, locates jvm.dll correctly.

Note:
Because of a bug in how the gcc C++ compiler processes paths (please see #3273 for further information), this value cannot contain spaces!

There two possible approaches to avoid this depending on the current system configuration:

1. System with no JDK yet installed. Install JDK into directory without spaces, like C:\jdk180 for example.
2. System with JDK already installed into something like "Program Files\path to JDK" containing spaces and there are the reasons to keep this configuration. in this case create symbolic link to JDK that has no spaces inside:
mklink /D SomePathWithNoSpaces "The target path\to JDK with\spaces inside" 

an example (having C:\ as current directory):
mklink /D jdk180 "Program Files\Java\jdk1.8.0_121" 

The quotation in link creation command is mandatory for mklink to properly resolve the original directory. After link has been created it must be used in JAVA_HOME and JRE_HOME environment variables instead of original path with spaces.

As alternative solution to avoid that, use the legacy 8.3 filenames for tokens that contain spaces. In a default Java installation, they are Program Files or Program Files (x86) must be replaced with PROGRA~1 and PROGRA~2 respectively, but they can differ if Windows was reinstalled. Use dir C:\ /X to see actual legacy name for each file/dir, in second to the last column. The value of this environment variable for default installation will look like this:
JAVA_HOME=C:\PROGRA~1\Java\jdk1.8.0_121

The above listed environment variables must be defined system wide so that they are available to all users. On Windows 7, you can access Environment Variables... in Control Panel/System and Security/System/Advanced System Settings, Advanced tab.

Install NCURSES Development Headers (Linux Only)

The native code on Linux/UNIX is dependent upon NCURSES. In order to properly compile and link that code, the NCURSES development headers must be installed.

In prior releases of Debian/Ubuntu distributions, the ncurses.h header file was present in a standard gcc installation, but recent versions (e.g. Ubuntu 12.10 and later) no longer provide this by default. Make sure to install the libncurses5-dev package to get the required headers.

Install it using sudo apt-get install libncurses5-dev.

Install C Compiler and Native Build Utilities

FWD includes native C code that must be built for many of the build targets. The most commonly used target is all, which builds the native components. Building the native code requires a C compiler and native code build utilities.

Linux Windows
The gcc and binutils packages can be installed using sudo apt-get install gcc (the utilities will be pulled in automatically). MinGW is the recommended compiler and tools suite for building the FWD native code on Windows. It is an open source Windows port of gcc and the other common build tools that exist on UNIX/Linux (the linker, make...). It is packaged in a manner that allows full WIN32 development. As this is not using the Cygwin compatibility environment at runtime, the native FWD code will link to the msvcrt library to get the basic C runtime routines.

Download one of the precompiled binary distributions, for your convenience the 4.9.0 version has been stored at https://proj.goldencode.com/downloads/mingw/mingw32_4.9.0_20140425.zip for the 32-bit binary or https://proj.goldencode.com/downloads/mingw/mingw64_4.9.0_20140425.zip for the 64-bit binary. These were originally found at MinGW precompiled page (mingw-w32-bin-i686-2013MMDD.7z or mingw-w64-bin-x86_64-2013MMDD.7z). Install 32-bit compiler in 32-bit OS and 64-bit compiler in 64-bit OS.

Unpack the MinGW archive, assuming the current directory is system root:
jar xf path_to_archive\mingw32_4.9.0_20140425.zip
or
jar xf path_to_archive\mingw64_4.9.0_20140425.zip.

Include the appropriate \mingwXX\bin in the PATH system variable, i.e. C:\mingw64\bin or C:\mingw32\bin, depending on the Windows architecture in use.

Configure JNI Linking

The JDK installation must be configured to allow linking JNI modules to the shared library that implements the JVM JNI interfaces.

JNI is a bidirectional interface between C/C++ code and the JVM. It is used by FWD:

  • FWD has a native library which is used on the FWD Client to access APIs in the operating system to do things that cannot be done directly in pure Java.
  • The FWD Spawner (which is called by the FWD Application Server to launch FWD Client JVM instances) loads a JVM using JNI.
Linux Windows
Run the following commmands:

sudo rm /usr/lib/libjvm.so
sudo ln -s /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so /usr/lib/libjvm.so

This will ensure that the build process for linking the spawner and the native library (libp2j.so) will properly link to the correct JVM.
Ensure the java link library jvm.lib is copied into the GCC lib directory so that it can be found during compilation:

32-bit: copy %JAVA_HOME%\lib\jvm.lib \mingw32\i686-w64-mingw32\lib\

64-bit: copy %JAVA_HOME%\lib\jvm.lib \mingw64\x86_64-w64-mingw32\lib\

Install libffi

In order to support the calling of native library calls from Java code, FWD depends on Portable Foreign Function Interface Library (also known as libffi). The FWD JNI library is dependent upon libffi.

The following instructions are valid for a system being used for running the FWD Client as well as for building FWD. To support both use cases, you MUST install the development version of libffi. If you do not need to build FWD, you can follow the libffi install instructions for only executing the FWD client.

In all cases, the bitness of this library must match the OS architecture and also the bitness of the JVM used by the FWD clients.

Linux Windows
On Linux, libffi can be installed using sudo apt-get install libffi-dev.
The third party libffi-6.dll is required for use of FWD on Windows. The DLL must be installed into the system directory (usually C:\Windows\system32).

As a convenience, the 32-bit and 64-bit pre-built libffi6.dll archives are available.

The library files libffi*.*a are required to build FWD native modules for Windows. Current project implementation requires the libffi-6 related libs to be installed in (GCC) compiler libraries directory. When using the prebuilt binaries, the directory for lib files is: i686-pc-mingw32\.libs for 32-bit package and x86_64-w64-mingw32\.libs for 64-bit package. The all files except *.dll must be copied into mingw32\i686-w64-mingw32\lib directory for 32-bit MinGW, or into mingw64\x86_64-w64-mingw32\lib for 64-bit MinGW. The include files from i686-pc-mingw32\include or x86_64-w64-mingw32\include must be copied into mingw32\i686-w64-mingw32\include or mingw64\x86_64-w64-mingw32\include for 32-bit and 64-bit respectively.

For details on building libffi from source code, please see Building and Installing libffi on Windows.

Install Apache Ant

The FWD build process (and usually the execution of the various conversion project build tasks) uses Apache Ant (version 1.9 or later).

Linux Windows
Install it using sudo apt-get install ant. Download the ant binaries in the zip archive from official site: http://ant.apache.org/bindownload.cgi. Unpack it to the desired location, set the system variable ANT_HOME to point to the path where ant was unzipped, then include the %ANT_HOME%\bin in the Path.

Use the ant -version command to verify that ant is properly installed. It shows the version of ant.

Install Pluggable Authentication Modules (a.k.a. PAM, Linux only)

Pluggable Authentication Modules (PAM) is a flexible mechanism for authenticating users. It is optional and only available on Linux/UNIX. It is used to perform OS-level authentication when the FWD Server launches a FWD Client, via the spawner tool.

This can be easily installed using:

sudo apt-get install libpam0g-dev

To use it during native task build, add this property to the gradlew build command:

./gradlew ... -Dhave.pam=yes

Install unzip - Linux Only

The commonly available unzip utility can be used to extract files from a ZIP archive. This may be needed to extract the source code from the FWD archive or from some of the other downloaded software (in the case of Windows).

Linux Windows
Install it using sudo apt-get install unzip. Windows does not have the command line unzip.exe tool. To unzip all archive files in ZIP format it is possible to use either Windows GUI zip extractor (zip support is already included in the Windows XP (and above) File Explorer) or if the JDK is already installed the command jar xf some_zipped_file.zip can be used as an equivalent to unzip.

Patch NCURSES - Linux Only

On Linux/UNIX, building the FWD native library libp2j.so requires linking with a customized version of NCURSES. Even if the console ChUI driver is not going to be used on this system, this step is still necessary.

Please see Patching NCURSES for the details.

Patch TERMINFO - Linux Only

This step is only needed if you choose to run the client with the native (console) driver.

Please see Patching TERMINFO for the details.

Install FWD Source Code

The FWD project code is available as a ZIP format archive file in the downloads section of this site. This archive should include all of the source code and supporting files needed to build the FWD project on a given system.

The archive filename is normally in the form fwd_<branch_name>_revision_<revno>_<date>.zip. For this example, assume that the fwd_trunk_revision_11145_20170324.zip is in your home directory and that the FWD code is to be installed in a directory named ~/projects (Linux) or c:\projects (Windows). Given these assumptions, installation is as simple as the following commands to extract the zip file:

Linux Windows
cd ~/projects
unzip ~/fwd_trunk_revision_11145_20170324.zip
First install JDK to use jar tool as zip extractor.
cd c:\projects
jar xf c:\PathToBundle\fwd_trunk_revision_11145_20170324.zip

A directory ~/projects/p2j (Linux) or c:\projects\p2j (Windows) will be created with the source tree of the FWD project. For the remainder of the chapter, the environment variable $FWD (Linux) or %FWD% (Windows) will be a reference to that directory. The "p2j" is a historical name for "Progress 4GL to Java". References to that name are still present in the code. Those references will be removed and/or replaced with references to "fwd", but that work is not yet complete.

This can be "saved" into the environment to simplify subsequent commands:

Linux Windows
export FWD=~/projects/p2j
Please note that this will only set the variable for the current terminal session. You will need to set this each time your start a new terminal session, if it is needed.
Go to the Control Panel/System/Advanced System Settings/Environment Variables and add system environment variable with name FWD and value c:\projects\p2j. This will set it persistently for all future command prompts.

You can create and assign the environment variable for the current command prompt using set FWD=c:\projects\p2j. This will be valid within current session only.

Build FWD

Build FWD using the all target:

Linux Windows
Use the gradlew script to build:

cd $FWD
./gradlew all
Use gradlew.bat to build:

cd %FWD%
gradlew.bat all

This will fetch all the Java dependencies, javac compile all Java source code, create the FWD jar files and compile/link the native code.

By default the build script provides only brief log output. If the build fails for any reason use the --info command line parameter to get more detailed log output of the failure, i.e. ./gradlew --info all.

More details can be found in Building FWD From Source.

Install FWD Collation Services for H2 Database

H2 is an in-memory database used implicitly by the FWD runtime for legacy temp-table support and for other internal purposes. This means that even if the permanent database servers being used are not H2, a Java-based text collator used by H2 must be installed on the machine where the FWD Application Server is run. If a permanent database is H2 and is on a different machine, then it must be installed there, too.

To ensure that each H2 database collates its string data in the same manner as Progress, a custom implementation of java.text.spi.CollatorProvider is used. This resource is installed and made available to the FWD runtime using the Java Extension Mechanism. This is necessary in order for J2SE's Locale and related classes to have access to the custom collation services that are needed by FWD.

Make sure that you have built FWD already, so that the p2jspi.jar is available.

Linux Windows
To enable this support, the p2jspi.jar file must be copied from the $FWD/build/lib directory to the Java Extension directory for the target JVM. For an Oracle JVM, this directory is located at $JAVA_HOME/jre/lib/ext. Note that administrator rights may be required, depending upon where the JVM is installed.

If the $JAVA_HOME is set, assume it corresponds to a (for example) /usr/lib/jvm/default-java/ path; otherwise, this path must be found if the JAVA_HOME environment variable is not set. Do not make the mistake of putting the p2jspi.jar into the /usr/lib/jvm/default-java/ext/ directory - this is an empty directory which is NOT used for extensions. The correct directory is /usr/lib/jvm/default-java/jre/lib/ext/ in this example.

After copying it, make sure it can be accessed by all users:
sudo chmod 0444 /usr/lib/jvm/default-java/jre/lib/ext/p2jspi.jar
To enable this support, the p2jspi.jar file must be copied to the target Java environments. The jar file must be copied to BOTH the private JRE of the installed JDK as well as to the default public JRE.

If the JDK home directory is c:\jdk180\, then the p2jspi.jar file must be copied to c:\jdk180\jre\lib\ext which is the "private JRE" location.

Assuming the home directory of your default public JRE is C:\Program Files\Java\jre1.8.0_(RELEASE), the p2jspi.jar must be copied to C:\Program Files\Java\jre1.8.0_(RELEASE)\lib\ext. (A quick and relatively reliable way to find out the default JRE location is to issue the command java -verbose and see where is the rt.jar file loaded from.)

The reason the copy must occur to multiple locations is because different methods of running Java code or its utility programs, may lookup the Java environment differently. Programs may use hidden pathing information in the Windows registry, or the JAVA_HOME environment variable.

To copy p2jspi.jar, administrator permissions may be required if Java was installed in the default location. After copying it, check if the other users have read access to this file.

For the temp-table and dirty databases, the server will automatically default to use the en_US_P2J collation.

To set this value for permanent databases or to override the default for the temp-table and dirty databases, requires a configuration entry in the FWD directory. Please see the Directory and Directory Configuration Reference chapters for more information on the FWD directory.

The directory node path is /server/standard/database/{database_name}/p2j/embedded-collation. The default setting should be en_US_P2J, as in:

<node class="container" name="p2j">
    ...
  <node class="string" name="embedded-collation">
    <node-attribute name="value" value="en_US_P2J"/>
  </node>
</node>

Currently, this custom collation service provider only exists for the ISO-8859-1 character set. If you need to support Progress-like collation for a different character set, you most likely will need to provide a new, custom implementation of java.text.spi.CollatorProvider, which requires special preparation/packaging. Please refer to the Internationalization chapter for additional information.

When using H2 for a permanent database (embedded or not), if the core structure DDL generated during M0 is not used to create the tables, then the collation must be set manually before any table is created. Just execute the following command with the database:

set collation en_US_P2J;

The default setting for the collation can be found in the p2j.cfg.xml file, under the cfg/schema/namespace/dialect-specific/collation node. This value is used during M0 to generate the set collation DDL for the H2 dialect.

<schema metadata="standard" >
   <namespace name=”...”>
...
      <dialect-specific name="h2">
         <parameter name="collation" value="en_US_P2J" />
      </dialect-specific>
   </namespace>
</schema>

Currently, only the en_US_P2J collation is supported.

If the p2jspi.jar file is installed properly and the FWD directory has the embedded-collation property set correctly, the FWD runtime environment will initialize the embedded H2 databases to use the en_US_P2J collation upon server startup.

If the JVM cannot find the en_US_P2J collation rules upon server startup, the following error message is issued to the server log:

Error setting P2J-specific collation;  please ensure p2jspi.jar is installed in Java extension directory

This is a fatal error, which indicates that the p2jspi.jar file is not properly installed as a Java Extension for the server's JVM. In this case, double check that you have installed the jar file for the JVM actually being used to launch the FWD server (avoiding a common mistake when multiple JVMs are installed on a development system), and ensure that you have copied the jar to the correct ext subdirectory (since the HotSpot JVM misleadingly has two of them, as discussed above).

Database Server

The FWD project can use multiple databases as its backing Relational Database Management System (RDBMS). If you will be working on modifications to the persistence layer of FWD or if you are otherwise going to implement code that is directly using the database, then it may be important to have a local installation of the database in question. Implementing such a scenario is straightforward.

It is important to note that it is easy to use the H2 database (that is built-in to the FWD server) as a database server for your application's imported Progress data. This is less work and has many advantages for a development system. H2 is not recommended for production use, but can be successfully used for testing or development.

The installation and setup of a local database server is not necessary in the following cases:

  • The Progress 4GL code does not access any database features.
  • The Progress 4GL code's use of database features is limited to temp-table (or work-table) usage.
  • The necessary test database(s) are accessible over the network on another system.

If any of the above conditions exist, then this section's instructions can be skipped.

The following chapters can be used to setup a local database installation:

PostgreSQL on Linux
SQLServer on Windows
H2 - any platform

IDE integration

To simplify integrating the project into an IDE the build contains the task fetchLibs, which resolves all the Java dependencies and stores them locally.

Linux Windows
Issue the command ./gradlew fetchLib and point your IDE to $FWD/lib to satisfy all the required Java dependencies for your IDE project. Issue the command gradlew.bat fetchLib and point your IDE to %FWD%\lib to satisfy all the required Java dependencies for your IDE project.

Subsequent Setup

The instructions in this chapter are sufficient for getting a running FWD development environment. It is important to understand the tasks that may be necessary in addition to the steps documented here. This chapter does not document any steps that setup a specific application or database. The following tasks may be necessary to provide an environment that supports a specific application conversion project or imported test database. It is likely this additional work is necessary to constitute a complete test or development environment that allows work on a specific converted application. Please use the referenced documentation to accomplish these tasks.

Task Reference
Convert and execute one of the sample conversion projects to test that the installation is working. Hotel ChUI Demo Application or Hotel GUI Demo Application (both contain the code, project configuration and instructions on how to convert and run them)
Use one of the sample conversion projects as a template to create a conversion project for another application. ChUI application or GUI application.
Copy a working conversion project to a new development system. Backup and Restore of a Conversion Project chapter, FWD Conversion Handbook
Configure or customize the FWD runtime environment. FWD Runtime Installation, Configuration and Administration Guide
Create a new conversion project from scratch. FWD Conversion Handbook
Install and setup a converted application for testing or debugging. Running and Testing Converted Code chapter, FWD Conversion Handbook
Migrate data from a Progress database into a relational database for use with the converted application. Data Migration chapter, FWD Conversion Handbook
Setup FWD dependencies (3rd party software projects that have been customized) for modification, build and debugging. Patching and Building Customized 3rd Party Libraries chapter, FWD Developer Guide.

© 2004-2017 Golden Code Development Corporation. ALL RIGHTS RESERVED.