Chapter 2 How to Build FWD from Source - FWD - Golden Code Redmine

How to Build FWD from Source¶

How to Build FWD from Source

This chapter is out of date and is no longer being maintained. It is only being retained for historical purposes. The replacement chapter can be found in Building FWD From Source.

Introduction¶

FWD is implemented almost completely in Java. This means that the primary dependencies of the build environment are a Java Development Kit (JDK) and the Apache ANT build automation tools.

This Java code does use a range of open source Java modules (packaged as jar files). The corresponding jar files must be present on the build system in order to successfully compile the FWD Java code.

Java programs are normally platform neutral. When applications or tools are written in pure Java, it is possible for such code to be run on any operating system or hardware platform that supports the corresponding required level of J2SE (Java 2 Standard Edition).

While the vast majority of the FWD code is written in pure Java, there is a small amount of platform-specific “native” code which is required for FWD at runtime. It is not used during conversion, but there is native code used for both the server and the client. On the server, native code is used to spawn batch programs, appserver instances and web client processes. On the client, it is used for character mode user interface processing, process launching, file system support, shared library API support and other tasks to directly support the functionality of converted Progress 4GL code.

This platform-specific code is written in C and is implemented using the Java Native Interface (JNI). In order to build a version of FWD that fully functional runtime A customized version of that native code is included in the FWD project. To build it, one must install a series of platform-specific C language build tools and libraries/header files.

This chapter will provide detailed instructions for how to setup this environment. It will also explain how to run the build process once all dependencies are satisfied.

32-bit vs 64-bit

native or not

Software Requirements¶

Prepare a system meeting the following software requirements:

Linux (or another UNIX-compatible operating system)¶

This code is currently compiled as both 32-bit and 64-bit shared libraries for Linux.

FWD has been successfully run on a variety of recent Linux distributions from Ubuntu, SuSE, Red Hat and CentOS. There are no specific dependencies on any particular distribution. It is expected that any UNIX-compatible system would work properly.

Windows¶

At the time of this writing, FWD conversion or runtime is not fully tested on Windows OS, but steps were taken to allow conversion and runtime (both server and client side) to work properly on the Windows OS. As the platform-dependent code was ported for Windows too, FWD build will create a shared library (either 32 or 64 bit, depending on the OS architecture) compatible with Windows.

Java Development Kit (JDK)¶

Version: 7.0 or later

[CA]Java 6 or earlier is no longer supported by FWD. For this reason, a JDK 1.7 or later implementation is required to build FWD.

Either the 32-bit or 64-bit version may be used, depending on the application size. At the time of this writing, converting large projects do not require more than 4GB of memory, but you might require more than 4GB to compile and create jar files, for very large projects.[/CA]

For client usage, the 32-bit JVM is best.

Since the memory requirements of server processes can be large, it is expected that the 64-bit version may be necessary for some installations. The application server process (JVM) is where the majority of working memory exists for each client (accessing the system). The actual memory used will depend on the queries, the block flow, recursion and other features of the original 4GL. The application code is actually running in the common server process, which is different from how Progress 4GL works (where the application runs in the client process unless an “appserver” is in use). However, for a development environment, it is less likely that 64-bit is needed.

In general, try to use the 32-bit version unless there is a specific reason to do otherwise.

The PATH for the user should be set to include the directory containing the command line executables (e.g. java or javac) of this installation. In Linux this would be the bin/ directory. In other words, this should be the default installation for any shell or command prompt from which the FWD technology will be built or used.

It is also recommended to set the user's CLASSPATH(on Unix/Linux) as follows:

export CLASSPATH=.

For the native code to compile, ensure the libvjm.so file (on Linux) and the jvm.lib file (on Windows) are accessible via the user's PATH. This results in either creating a /usr/lib/libjvm.so symlink to the actual file (on Linux/UNIX) or adding the parent folder of this file to the user's PATH (on both Linux and Windows).

Apache Ant¶

Version: 1.6 or later. At the moment of writing this document 1.8.2 was found in the Linux repositories.

Due to the ANTLR dependencies, version 1.5.3 does not work. Please ensure that Ant can be found via the user's PATH.

GNU Compiler Collection (GCC) C compiler and binutils (Linux/UNIX only)¶

Make sure that version 4.7.3 (or later) of gcc is installed. This should also include the standard utilities (binutils) package. The native portions of FWD are built using gcc, make and related tools.

[CA]MinGW compiler and utils (Windows only)[/CA]¶

MinGW is used for Windows development of the FWD native code. This is an open source, Windows port of gcc and the other common build tools that exist on UNIX/Linux (the linker, make...) and is packaged in a manner that allows full WIN32 development. As this is not using the cygwin compatibility environment at runtime, the native FWD code will link to the msvcrt library to get the basic C runtime.

To install:

Take one of the full archive: mingw-w32-bin-i686-2013MMDD.7z or mingw-w64-bin-x86_64-2013MMDD.7z. Unpack to \mingw32 or \mingw64.
Include the appropriate \mingwXX\bin to the PATH variable, i.e. C:\mingw64\bin or C:\mingw32\bin, depending on Windows architecture.
The 64-bit binaries are found at http://www.drangon.org/mingw. The 32-bit binaries and much more info can be found at http://mingw.org.

[CA]libffi[/CA]¶

On both Linux and Windows, for the native code to compile is required for the Portable Foreign Function Interface Library to be installed; this is a library used by FWD to implement the actual dispatch of the native API calls.

On Linux, install it using:

sudo apt-get install libffi-dev

On Windows, the library needs to be built from source code. An installed and working MinGW (32-bit or 64-bit, depending on Windows OS) is required (see the MingW compiler and Utils (Windows only) section for how to install MinGW). Follow these steps to build and install libffi:

1. Download MSYS from http://sourceforge.net/projects/mingw-w64/files/External%20binary%20packages%20%28Win64%20hosted%29/MSYS%20%2832-bit%29/ and unpack it in a desired location. No special installation required.

2. Take the libffi sources from http://sourceforge.net/libffi , unpack to desired location.

3. Run MSYS shell - msys.bat from \msys directory.

4. Go to the libffi sources, for example: cd c:\libffi-3.0.13

5. Prepare target build config:

32 bit: sh ./configure

64 bit: sh ./configure --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32

6. Run make

7. After successful build, these binaries and include files are created:

32 bit: libffi-(root)/i686-pc-mingw32/.libs and libffi-(root)/i686-pc-mingw32/include
64 bit: libffi-(root)/x86_64-w64-mingw32/.libs
and libffi-(root)/x86_64-w64-mingw32/include

8. Copy headers, libraries, and dll to the corresponding MinGW folders.

[GES]: This needs to be more specific, both as to the list of where to find the things to copy and the exact locatons to which they need to be copied. I have tried to interpret this but it is unclear. What link libraries need to be copied? Why does the DLL need to be copied if we are linking to the link libraries? The paths below need help and checking.

32-bit:

copy libffi-(root)\i686-pc-mingw32\include\*.h \mingw32\i686-w64-mingw32\include\

copy libffi-(root)\i686-pc-mingw32\.libs\?????? \mingw32\?????\

64-bit:

copy libffi-(root)\x86_64-w64-mingw32\include\*.h \mingw64\x86_64-w64-mingw32\include\

copy libffi-(root)\x86_64-w64-mingw32\.libs\?????? \mingw64\?????\

Then there was also this note from one of your emails:

B. about libffi: №1634 note 74 contains the archives for 32 and 64 bits. As I understand, these need to be copied into the mingw folder, but I think the 64 bit archive contains a problem: I had to rename the x86_64-w64-mingw32/libs/ folder to x86_64-w64-mingw32/lib/, else mingw could not pick up the libffi.

Is this still an issue?[/GES]

[CA]Pluggable Authentication Modules (Linux/UNIX Only)[/CA]¶

PAM is a flexible mechanism for authenticating users and is optional; it is used to perform OS-level authentication when the FWD Server launches a FWD Client, via the spawner tool. This can be easily installed using:

sudo apt-get install libpam0g-dev

To use it during native task build, add this property to the ant launch command:

ant ... -Dhave.pam=yes

NCURSES Development Headers¶

In prior releases of Debian/Ubuntu distributions, the ncurses.h header file was present in a standard gcc installations, but recent versions (e.g. Ubuntu 12.10 and later) no longer provide this by default. Make sure to install the libncurses5-dev package to get the suitable headers.

unzip¶

Among many options, the commonly available unzip utility can be used to extract files from a ZIP archive. This will need to be used to extract the FWD archive.

Install FWD Project Code¶

The FWD project code should be available as a ZIP format archive file. This archive should include all of the source code and supporting files needed to build the FWD project on a given system.

For the sake of this example, assume that the zip file name is ~/p2j_20100620.zip and that the p2j code is to be installed in a directory named ~/projects. Given these assumptions, installation is as simple as the following commands to extract the zip file:

cd ~/projects
unzip ~/p2j_20100620.zip

A directory ~/projects/p2j will be created with the entire contents of the FWD project. For the remainder of the chapter, the environment variable $P2J will be a reference to ~/projects/p2j.

This can be "saved" into the environment to simplify subsequent commands:

export P2J=~/projects/p2j

Satisfying 3rd Party Software Dependencies¶

Build FWD¶

The FWD project must be built. The commands:

cd $P2J
ant all

will compile all code and then create jar files and native libraries that store the compiled code.

All of the FWD conversion and most of the runtime tools are stored in a file called p2j.jar. That file in turn is dependent upon other jar files, which have been specified in the p2j.jar file's manifest. This means that as long as the p2j.jar can be found in the Java CLASSPATH and those other jar files exist in the same directory, the tools will work properly.

The p2j.jar can be found in the $P2J/build/lib/ directory, along with each of the dependent jar files which are shipped with the FWD project. It is best to use those versions since they are tested and are known to work.

When running FWD commands, it will be necessary to use the -classpath $P2J/build/lib/p2j.jar:. parameter on the java command line.

[CA]Besides p2j.jar the build process also produces:

the shared library libp2j.so(on Linux/UNIX) or p2j.dll (on Windows), which provides native support for i.e. child process creation at runtime via Java Native Interface (JNI), native API calls, etc. It resides in $P2J/build/lib/ directory and is only used at runtime, when the converted application makes use of Progress 4GL commands that deal with process creation (INPUT THROUGH, INPUT-OUTPUT THROUGH, OS-COMMAND, UNIX etc), executes native APIs, uses memptr objects, etc.
the spawn tool, a file named spawn (on Linux/UNIX) or spawn.exe (on Windows). It resides in $P2J/build/native and is used by the FWD Server runtime to automatically launch FWD Clients, in cases when web clients, batch processes or appserver Agents are launched from FWD Server side.

Should (re)building of the FWD native library and spawn tool be necessary for any reason, the following build step can be run:

cd $P2J
ant native

The above compiles libp2j.so/p2j.dll and copies it to $P2J/build/lib/ directory. From there it can be picked up when setting up the runtime environment.

64-bit build and cross-compile builds¶

If you are not sure of what version you have installed or just want to check use the following command:

file build/native/libp2j.so

If you are running on a 32-bit operation system then things are quite simple: you can only have 32-bit JVM and the use the libp2j.so/p2j.dll delivered with FWD archive.

Things are more complicated on a 64 bit installation:

In the case of 32-bit JVM then the “stock” libp2j.so/p2j.dll will be just fine. If you want to recompile it, you must add the -m32 switch to C compiler (gcc) in order to cross-compile and generate the code to fit the JVM bitness as the default target for the compile is the native, 64-bit.
When using a 64-bit JVM it is mandatory to recompile the library. Just use the above commands as the compiler will natively generate 64-bit code.

The table below summarize the build cases:

		JVM	JVM
		32-bit	64-bit
OS	32-bit	1. use stock 2. rebuild normally (optional)	n/a
OS	64-bit	1. use stock 2. rebuild using `-m32` compiler switch (optional)	rebuild normally (mandatory)

Remember that [CA]on Linux/UNIX[/CA] you need to have the ncurses library already patched and recompiled with the same bit-size as the one you want to target.

[CA]Available Ant Tasks[/CA]¶

Following is a graphical representation of the ant tasks defined by the FWD's build file:

from which:

clean will remove the entire build/ and dist/ folders, plus all files generated by the antlr and native tasks.
clean-native will remove only the build/native/ folder plus the header files generated in the src/native/ folder, by the native task.
antlr is responsible for running ANTLR transformer against all grammars used by FWD conversion engine or FWD runtime.
compile will compile the entire java source code and save them in the build/classes/ folder.
native will compile the native shared library (to be called via JNI) and also build the spawn tool. This will create, in the build/native/ folder:
- the libp2j.so or p2j.dll file, holding the FWD's native API.
- the spawner or spawner.exe file, the tool used for spawning FWD clients from server side.
jar will create the FWD jar files and place the in the build/lib/ folder:
- p2jadmin.jar - is the jar downloaded by the applet running the FWD Administration Console.
- p2j.jar - is the main FWD file which contains both the FWD conversion and runtime infrastructure.
- p2jrt.jar - this jar is supposed to hold only the FWD runtime, but the subset of FWD packages contained by this jar are not fully tested. We don't recommend using this file, the p2j.jar file needs to be used for both conversion and runtime.
- p2jpl.jar - this jar is used exclusively on the database server to support custom, server-side functions implemented in Java.
- p2jspi.jar - includes service provider interface (SPI) classes for FWD collator support.
javadoc will create Javadoc API documentation from the FWD source files, and place it under the dist/ folder. It can be accessed using the dist/docs/api/index.html file.

Project

General

Profile

FWD

Wiki