FWD Project Source Code Tour - FWD - Golden Code Redmine

FWD Project Source Code Tour¶

FWD Project Source Code Tour

The FWD Project consists of two primary components: the FWD Conversion Technology and the FWD Runtime Environment. This chapter explores the source code of each of these components to provide a basic lay of the land. Some aspects of the implementation of various areas are discussed, but this chapter is not intended as an in-depth presentation of implementation details. For a more thorough treatment of the implementation of the FWD conversion and runtime technologies, please refer to the book FWD Internals.

Project Structure¶

The FWD Project is organized as follows:

Directory / File	Purpose
`lib`	Contains all third party libraries required to build and/or run the FWD Project. The project has been made to work with these specific versions of these libraries, and some of them have been patched to provide necessary fixes or functionality. The associated patches are included.
`locale`	Contains UNIX locale support files, including the source code version of `en_US@p2j_basic`. This is a customized locale definition for FWD use, based upon standard en_US and POSIX locale definitions, but with custom `LC_CTYPE` and `LC_COLLATE` sections to mimic Progress' basic (ISO-8859-1 character set) database collation behavior as closely as possible.
`manifest`	Contains several files which are used when creating the manifests of the FWD Project `jar` archives.
`pl`	FWD uses the open source PL/Java project to implement Java-based, user-defined functions which run within the PostgreSQL database server process. This directory contains a deployment descriptor, `p2jpl.ddr`, which defines the signatures of all of the user-defined functions defined by the FWD Project. It is used when building the `p2jpl.jar` library, which is the `jar` archive installed in each PostgreSQL database used by a converted application.
`rules`	TRPL programs which implement the bulk of the conversion process. This directory is further organized into subdirectories along functional lines. These are described in further detail in the FWD Conversion Technology section below.
`src`	Java and C source code for the project. The Java classes contained in this directory implement both the conversion process and the runtime environment. The packages are organized largely into functional categories. These are described in further detail in the sections below. The only C code, located in src/native, results in building a native libp2j.so shared library, which is called by the client component at runtime via JNI for launching child processes in a terminal and also for waiting for them to exit.
`testcases`	Test cases written in the Progress 4GL. These are generally short programs which test some aspect of the conversion technology or the runtime environment.
`build.xml`	FWD uses Ant (ant.apache.org) as its build tool. This file defines the build tasks and targets which clean up and prepare the project, generate various lexers and parsers from ANTLR grammars, compile Java source code, generate JavaDoc documentation, archive the compiled classes into jar archives, and generate DDL for a relational database schema.

FWD Conversion Technology¶

The FWD Conversion Technology enables the conversion of Progress 4GL source code to Java source code with equivalent behavior. Once a Progress application is converted to Java, the work of the Conversion Technology essentially is done; it is no longer needed in order for the application to function. Actually, the preceding statement is a bit of an oversimplification, since conversion typically is an iterative process; you likely will perform the conversion process many times in the course of converting any non-trivial application, as you address issues and refine the converted output. As we discuss in the second part of this book, you might even rely on the FWD Conversion Technology for the ongoing maintenance and development of your post-conversion application, if you choose to continue your development in the Progress 4GL. However, for the purpose of running your converted application, the FWD Conversion Technology is not necessary.

Since it is the goal of this chapter to introduce the source code which implements the FWD Conversion Technology, some knowledge of the overall conversion process is assumed as a prerequisite. For a description of this process, please refer to the Conversion_Handbook, specifically the Conversion Technology Architecture chapter. Certain terms referenced below are introduced in that chapter.

One such noteworthy term is TRPL, which is an acronym for Tree Processing Language. TRPL is a programming language with Java-like syntax which runs on the Java platform, and is specialized for the analysis and manipulation of Abstract Syntax Trees (ASTs). It is one of the core technologies upon which the conversion process relies. An in-depth treatment of the syntax and semantics of TRPL is beyond the scope of this document. If you need to implement new TRPL programs or modify existing ones, please refer to the book FWD Internals.

The FWD Conversion Technology is implemented as a combination of Java and TRPL programs, consisting of:

a Conversion Driver to manage the conversion process from end to end;
a Progress 4GL schema export (.df) file lexer and parser;
a Progress 4GL source code preprocessor;
a Progress 4GL source code lexer and parser;
a TRPL technology stack which compiles and runs TRPL programs.

Java Source Code¶

The overall conversion solution is implemented in Java, in that much of the technology is written directly in the Java language, and the TRPL environment both runs on the Java platform and has an engine implemented in Java. All Java source code in the FWD Project resides in the FWD Project's src directory. Within this directory, the source code is arranged into packages. Of those, the following contain conversion related classes:

Package	Purpose
`com.goldencode.ast`	Creating, reading, modifying, annotating, persisting, deleting, and otherwise manipulating general purpose ASTs.
`com.goldencode.expr`	An expression engine which compiles and executes Java classes from TRPL source code statements. The TRPL engine is built atop this package.
`com.goldencode.html`	Classes used to simplify creation of HTML output. Used by the FWD reporting facility.
`com.goldencode.p2j.convert`	Specialized Progress 4GL AST to Java AST conversion services. Implements libraries which are accessible to TRPL programs. Contains the `ConversionDriver` class, which is the entry point to the conversion process.
`com.goldencode.p2j.e4gl`	Classes used to provide a WebSpeed compatible embedded 4GL (E4GL) preprocessor.
`com.goldencode.p2j.pattern`	A framework to post-process ASTs using pattern-matching techniques, for the purpose of examination, analysis, transformation, and ultimately conversion from one programming language's syntax to another's.
`com.goldencode.p2j.preproc`	Implements a full set of classes including lexers, parsers, utility classes and command line driver code that constitute a stand alone Progress 4GL compatible language preprocessor.
`com.goldencode.p2j.schema`	Provides services to extract database schema information from a Progress schema dump file (`.df`) for the purposes of inspection and analysis; combines this data with information from conversion hints in order to create an analogous relational database schema and the DMO interfaces and classes appropriate for the resulting schema.
`com.goldencode.p2j.uast`	Provides a lexer, parser, call tree processing and symbol resolution to create an intermediate form representation (AST) of a Progress 4GL source tree in a manner that allows programmatic tree walking and manipulation for the purposes of inspection, analysis, interpretation, translation and conversion.
`com.goldencode.p2j.util`	A functionally broad collection of classes which perform various runtime and conversion related services. The conversion related services include symbol resolution, specialized data structures, and statistics collection (for reporting purposes). This package may be refactored in the future to organize some of the larger groups of classes along functional lines.
`com.goldencode.p2j.xml`	Classes used to simplify manipulation of XML documents, including parsing, programmatic creation, and serialization. Provides a library of these services to TRPL programs, which are used in the generation of XML configuration files, such as Hibernate ORM (`.hbm.xml`) documents and the `dmo-index.xml` file, which is used by the converted application at runtime.
`com.goldencode.trpl`	Partial, next generation TRPL implementation; not currently in use.
`com.goldencode.util`	Utility classes for string and XML document manipulation. Also includes a handful of data structures.

The TRPL Technology consists of three logical components, as depicted in the image to the right. TRPL programs encode the rules of conversion. Outside of the various lexers and parsers for Progress 4GL source code and schema export files, it is within these programs that the majority of the analysis and conversion intelligence resides.

TRPL programs are compiled and executed in a TRPL Engine, which is implemented in Java. TRPL programs may access services in a library of TRPL-aware Java classes. These libraries provide access to certain internal state information in the conversion engine, as well as other functionality to assist in the analysis and conversion of Progress programs.

Each of these logical components is discussed in more detail below.

TRPL Programs¶

The TRPL programs which govern the conversion process are also known as rule-sets, since the conversion engine is at its heart a rules engine, and each TRPL program decomposes into numerous rules. Rule-sets are implemented as XML files, with one file containing one or more rule-sets. Filenames have an extension of either “.xml” or “.rules”. Supporting files may have different extensions, such as “.tpl” or “.rpt”, but these generally do not contain TRPL logic. By convention, a TRPL program with the extension .xml represents a “master” or “driver” TRPL program, which can invoke more specialized rule-sets (those with an extension of .rules) as necessary. All of the standard conversion rule-sets for the FWD Project are located in the rules subdirectory. They are organized in various subdirectories as follows:

Location	Purpose
`annotations`	Perform the core analysis of Progress source code, in preparation for downstream conversion processing. This includes the identification and removal of unreachable code, variable analysis, control flow analysis, record buffer and frame scoping, where clause analysis and refactoring (including index selection), screen buffer management, validation handling, and much more. The driver program in this collection of rule-sets is `annotations.xml`, which in turn invokes many specialized programs from this subdirectory. The input to the annotations rule-sets is an intermediate form of Progress source code, as ASTs. The results of this analysis generally are stored as annotations in the Progress ASTs, which are accessed later by downstream processing. In some cases, annotations rules also transform Progress ASTs by removing or re-organizing branches or entire trees, so the annotated output of these rule-sets may be structurally different than the input ASTs.
`convert`	Process the annotated Progress ASTs and produce the corresponding set of Java ASTs as an intermediate form, and then the corresponding Java source code. These rules leverage all the information gathered by the annotations phase of the conversion process to ensure the Java ASTs represent Java source code which will behave the same as the original Progress application. It is this phase of the process which implements the linkage between the converted application and the FWD runtime environment, in the form of calls integrated into the converted Java source code, to FWD runtime class libraries. In addition to many rule-sets which perform some specialized aspect of the creation and transformation of Java ASTs, this subdirectory contains the following programs worthy of special mention: • `frame_generator.xml`, which produces Java ASTs for frame definitions re-factored from Progress source code; • `base_structure.xml`, which creates a skeleton Java AST for each Progress program; • `core_conversion.xml`, which implements many details of conversion, fleshing out the Java AST skeletons created by base_structure.xml@; • `java_templates.tpl`, which contains templates of various AST fragments which rule-sets can inject into a Java AST and further manipulate; • `brew.xml`, which finally anti-parses the completed Java ASTs into syntactically correct Java source code.
`fixups`	These programs run early in the processing of Progress 4GL source files, just after the source code is parsed. The ASTs created by the Progress parser are “fixed up”, so as to be easier to process downstream. This includes dropping out some AST nodes that are clearly unnecessary, normalizing and simplifying certain expressions, beginning the process of integrating schema-level validation rules into converted programs, and adding early annotations which are more convenient to deal with in a TRPL program than in the context of the Progress parser itself. The primary program in this group is `post_parse_fixups.xml`, which in turn invokes more specialized TRPL rule-sets.
`include`	Libraries of re-usable TRPL functions which are included in other TRPL programs. The most widely used of these is `common-progress.rules`, which contains many general-purpose functions. Another program, `report.rules`, contains functions used by many of the programs/reports in the `reports` subdirectory.
`reports`	This subdirectory contains rule-sets and report definitions for use with FWD's reporting facility (see the Reporting chapter of the FWD Conversion Handbook). Files ending in `.xml` contain rule-sets which can be run by the user to generate various ad-hoc reports. One rule-set in particular, `simple_search.xml`, implements a grep-like utility which is fully aware of Progress 4GL syntax. Two of the files in this subdirectory, `profile.rpt` and `schema.rpt`, encode report definitions for all of the “canned” reports provided with FWD, for in-depth views of an application's source code and schema information, respectively.
`schema`	Convert all database schema constructs to artifacts which can be used by the new system to manage database migration and runtime access to database data. Inputs to these rule-sets include intermediate, AST output generated by the schema export (`.df`) file parser and by the Progress 4GL source code parser. Outputs generated by these rule-sets include: • a Progress-to-Object (P2O) document for each permanent database and Progress source file containing temp-table/workfile definitions; these are used for downstream schema processing, to generate relational schema DDL, and during the data import process to govern export file (`.d`) parsing and to create table indexes; • Java source code (DMO interface and implementation classes); • Hibernate object-to-relational mapping (ORM) documents (in XML form); • a `dmo_index.xml` file used by the FWD runtime environment.
`unreachable`	Implementation of the rules invoked during the annotations phase of conversion, to determine whether certain Progress source code is unreachable at runtime. This processing forms the core of dead code analysis, and identifies and removes unreachable code which, if left behind and converted, would cause failures during compilation of the converted Java code.

TRPL Engine¶

The TRPL engine compiles and runs TRPL programs and provides a framework upon which service libraries, implemented in Java, can be built. The primary purpose of the TRPL engine is to walk a set of ASTs in a depth-first enumeration from root to leaf, and to execute rules defined by a TRPL program as this walk triggers various events. These events include:

global initialization (before any ASTs are visited);
local initialization (before each individual AST is visited);
visiting each AST node;
descending a branch level (traversing the tree vertically from a parent AST node to its first child);
visiting the next child in a chain of sibling AST nodes;
ascending a branch level (traversing the tree vertically from the last child to its parent AST node);
local termination (after each individual AST has been visited);
global termination (after all ASTs have been visited).

At the core of the TRPL engine is a general-purpose expression engine, implemented in the package com.goldencode.expr. The expression engine compiles individual TRPL statements into Java classes on-the-fly and loads these into the JVM via a custom class loader. The expression engine is general purpose in that it processes arithmetic and logical expressions, and is not aware of the TRPL rules engine built above it.

The package com.goldencode.p2j.pattern contains the core implementation of the TRPL rules engine. The class PatternEngine is the primary coordinator of running a TRPL program. It is responsible for loading and preparing each TRPL program and for driving the enumeration of a specific set of ASTs. The classes Rule, RuleContainer, and RuleSet together implement the control flow within a TRPL program.

The TRPL engine allows Java-implemented service libraries to be made accessible to TRPL programs via the PatternWorker interface, which permits the registration of Java methods as callable functions within TRPL programs.

Service Libraries¶

Service libraries are implementations of the com.goldencode.p2j.pattern.PatternWorker interface. They form the bridge between TRPL programs and functionality implemented in Java. Service libraries are used to expose certain internal state of the TRPL engine to TRPL programs, in the form of functions and variables which can be accessed by TRPL logic. They are also used to provide access to I/O services, specialized data collection and analysis services, and generally to any logic which is better implemented in the Java language than in TRPL.

FWD's standard, TRPL service libraries are spread throughout various packages, organized by the services they provide:

com.goldencode.p2j.convert
com.goldencode.p2j.pattern
com.goldencode.p2j.schema
com.goldencode.p2j.uast
com.goldencode.p2j.util
com.goldencode.p2j.xml

FWD Runtime Environment¶

The FWD Runtime Environment provides a general purpose, secure, distributed platform on which to execute a converted application. Furthermore, it provides a compatibility infrastructure upon which a converted application relies in order to behave like it did before conversion. A more detailed discussion of the runtime environment from a functional perspective is provided in the FWD Runtime Environment Overview chapter of this book.

The runtime environment is mainly implemented in Java source code, located in the p2j/src directory, the only exception being p2j/src/native, which contains a small amount of code written in C. Within the p2j/src directory, the Java source code is arranged into packages. Of those, the following contain runtime related classes:

Package	Purpose
`com.goldencode.expr`	An expression engine which compiles and executes Java classes from logical and arithmetic expressions. Used by the Security Manager to implement certain, configurable security policies.
`com.goldencode.p2j`	Common exception types used at runtime.
`com.goldencode.p2j.admin`	Provides an administration server and a browser-based GUI applet that works as an admin client to offer an easy remote GUI for various FWD administration tasks. Implementation of the Administrative Console server API and classes shared by the server and client Administrative Console infrastructure.
`com.goldencode.p2j.admin.client`	Administrative Console classes used only on the client side; these are stored in a `jar` archive which is loaded to the administrative user's browser as an applet.
`com.goldencode.p2j.cfg`	Provides a way of finding pieces of initial (bootstrap) configuration.
`com.goldencode.p2j.directory`	Implementation of the application server's Directory Service. A set of classes that manage a hierarchy of directory objects and provide retrieval and editing capabilities for the objects and their attributes.
`com.goldencode.p2j.io`	Provides a new useful extension to input streams.
`com.goldencode.p2j.main`	Generalized command line driver classes used to launch specific FWD clients and servers based on information from command line parameters and bootstrap configuration files. Also defines the interface for implementing server and context hooks (see Runtime Hooks chapter). This package contains the `ServerDriver` and `ClientDriver`, the command line entry points to the FWD server and FWD (CHUI) client, respectively.
`com.goldencode.p2j.net`	Implementation of the Distributed Application Protocol (DAP). Provides a multiplexed, symmetric low level network transport, transaction requester/server support and all necessary queuing, addressing, naming, routing and security integration needed to create a logical network of leaf and router nodes that can implement a flexible distributed application system.
`com.goldencode.p2j.persist`	Main persistence framework implementation; runtime services which support persistence of an application's business domain Data Model Objects (DMOs) using the Hibernate Object-to-Relational-Mapping (ORM) framework.
`com.goldencode.p2j.persist.dialect`	Implements additional dialect-specific details required by FWD but not covered by Hibernate's standard dialect support.
`com.goldencode.p2j.persist.dirty`	Classes which implement FWD's dirty sharing engine. This mechanism shares certain, uncommitted database changes across user sessions. This sharing is accomplished via tightly synchronized, multi-session access to an embedded, transient, in-memory database resident in the FWD server's JVM. This is required to mimic Progress' unusual mode of transaction isolation. Please refer to the book FWD Internals for additional details on this facility.
`com.goldencode.p2j.persist.embed`	Contains a helper class which initializes two H2 databases embedded in the application server's JVM. These are used for temporary table support and for the dirty sharing engine (see `com.goldencode.p2j.persist.dirty`). Includes setting an appropriate collation for the server's locale, and installing User Defined Functions (UDFs) in the databases, which are compatible with Progress 4GL behavior, for use in HQL queries.
`com.goldencode.p2j.persist.event`	Persistence events which are used primarily for internal housekeeping, including support for dirty sharing of uncommitted changes across user sessions.
`com.goldencode.p2j.persist.hql`	A Hibernate Query Language (HQL) lexer and parser implementation (and helper classes) used by `com.goldencode.p2j.persist.HQLPreprocessor`. HQL queries in converted business logic are preprocessed and augmented by the persistence framework to implement required runtime behavior, based on information which is not available during conversion. This includes the database dialect in use, the value of query substitution parameters, and numerous other details.
`com.goldencode.p2j.persist.pl`	Classes required by user defined, server-side, database functions implemented in Java, when using the PL/Java framework.
`com.goldencode.p2j.persist.remote`	Classes which support connections to databases for which the current FWD server is not authoritative.
`com.goldencode.p2j.persist.trigger`	Partial implementation of assign and delete database triggers. At this time, trigger logic must be implemented manually.
`com.goldencode.p2j.persist.type`	Hibernate user type implementations which manage the FWD data wrapper classes `character`, `date`, `decimal`, `integer`, `logical`, `raw`, and `rowid`. This provides a mapping of these Java classes to SQL data types which Hibernate can handle.
`com.goldencode.p2j.security`	Implementation of the FWD Security Manager subsystem. Includes a set of classes which control application access to abstract resources and manage security credentials.
`com.goldencode.p2j.spi`	Extends the locales available to the FWD server's JVM. Contains a service provider for specialized collators which are governed by Progress-like collation rules, and all implementations of custom collators which are shipped with the FWD Project. Currently, only one such implementation is included, for use with the ISO-8859-1 character set.
`com.goldencode.p2j.ui`	Classes which make up the user interface available to converted applications. This package includes only those classes which run only on the FWD server, or which are shared across the FWD server and FWD client.
`com.goldencode.p2j.ui.chui`	User interface implementation which supports a character mode user interface (CHUI) compatible with Progress' CHUI environment. These classes run only on the FWD Thin Client. Business logic does not run on these classes; they implement general-purpose, user interface services which the FWD server accesses as primitives.
`com.goldencode.p2j.util`	A functionally broad collection of classes which perform various runtime and conversion related services. The runtime services include Progress-like transaction and block management, converted `ACCUMULATE` statement and `ACCUM` function behavior, all of the FWD data wrapper type implementations (`character`, `date`, `decimal`, `integer`, `logical`, `memptr`, `raw`, `recid`, and `rowid`), Progress-compatible operator functions and many built-in functions, access to environment information and operating system services (including I/O), process launching, several data structure implementations, and various other features. This package may be refactored in the future to organize some of the larger groups of classes along functional lines.
`com.goldencode.util`	A collection of general-purpose of utility classes, including XML and string manipulation helpers, and some data structure implementations.

The C code in directory p2j/src/native contains logic which is concerned with launching child processes, possibly in the same terminal as the parent process, and waiting for their termination status. The code builds as a native shared library (libp2j.so, residing in p2j/build/lib) that is invoked via JNI by class com.goldencode.p2j.util.LaunchManager.

Project

General

Profile

FWD

Wiki