Understanding Conversion Projects¶
- Understanding Conversion Projects
- Phase 1 - Assessment
- Phase 2 - Proof of Concept
- Phase 3 - Technical Enablement
- Phase 3A - 4GL Code Improvements
- Phase 3B - FWD Improvements
- Phase 3C Quality Assurance and Testing
- Phase 3D Deployment Preparation
- Phase 3E Debug/Fix/Tune
- Phase 4 - Deployment
This chapter documents the high level project plan for using FWD to convert a 4GL application to Java. It provides a summary of the important steps of a proven methodology which can be used in conjunction with FWD Transform (automated code conversion) and FWD Enhanced Runtime technology (runtime compatibility execution environment) to convert Progress 4GL applications into a drop-in replacement in Java.
FWD Transform is a fully automated code transformation toolkit. The term fully automated means that the tool that converts Progress 4GL source code into semantically identical Java source code is a batch job that is completely non-interactive AND it processes entire applications in a single run AND the resulting converted Java code needs no manual editing before it can compile and execute.
One might be forgiven for assuming that the extreme level of automation in FWD Transform means that migrating an entire application from the 4GL to Java is as easy as just pushing a button. The use of FWD Transform (the automated conversion tools) is just an important step in a larger process. Each application is uniquely complex, usually having been built over decades of development by many different people. This means there is a range of 4GL language features in use and a variability with how those features are used. There are also some number of platform-specific or environmental dependencies that affect how the conversion is handled. Likewise, organizations have development, deployment and support processes that will need at least some changes to support a migration using FWD. There is more work in a project than just running the conversion tools. Importantly, much of that work must be done in advance of running the tools.
Do not let this fact deter you. Migrating off of OpenEdge using FWD will be 10% or less of the effort (and cost) of any other migration alternative. Even though there is more to a project than just pushing a button, the result achieved and the effort/cost needed makes FWD the only practical approach to switch away from OpenEdge. Some projects can be executed in a month or two. Others might take a year or more to get to the point of deployment. This is an amazing result in comparison to the idea of a manual rewrite which would take 5-10 years for a few million lines of 4GL code. This chapter will provide a better idea of which factors cause a project to take more effort.
This chapter will document all the steps in this process, including the planning, preparation, testing, technical and process enablement that is needed to successfully migrate away from OpenEdge. This methodology has been proven with real applications. It will work for any 4GL application, so long as the approach is followed.
During any project to convert/migrate to Java, the development and testing process tends to be iterative such that the FWD Transform automated conversion tools will be run many times as the conversion is refined and enhanced. Still, most of the project plan documented here is not specifically related to running the conversion tools.
The project plan can be split into the following phases:
The developer seeking to use the FWD technology to convert an application must have access to the following:
- Application source code, database schemas and suitable test data. See the chapter on Gathering Inputs for more details on how to gather up all necessary inputs to the automated conversion process.
- Properly installed and configured OpenEdge development system on which the 4GL application is fully operational. This should be a fully functional test environment which is considered the baseline for comparison with the converted application. The project is complete when the converted application's operation is identical to the application on this system. This is often referred to as the “source” system. The instructions for the setup of this system are dependent upon the Progress 4GL technology in use and that is beyond the scope of this book. It is best if this system is enabled for development mode so that changes to the application code (or standalone testcases) can be tested.
- Properly installed and configured system on which the converted application can run. Often this same system can be used as the system on which the conversion is run, but this is not a requirement. This is often referred to as the “target” system. To understand the system requirements please see the System Requirements chapter. For the purposes of this book, this is a test system which where the converted application will be run to compare against the 4GL version of the application running on the source system. In practice, this test system can also be the production installation for the organization using FWD.
- If the target system is not also the system on which the FWD conversion technology will be used to convert the application, a properly installed and configured system must be available. This is often referred to as the “conversion” system. Please see the Conversion Environment Setup chapter for more details.
- Application development staff that understand the code and functionality of the 4GL application to be converted. Consulting with this staff will be important during the project as questions or decisions must be made.
- Application-specific environment requirements, including:
- Connectivity to unique networks or legacy applications required for the source and target application to execute properly.
- Setup of peripherals including printers, bar code readers, card scanners or other unique equipment or hardware needed for the application to execute properly.
- File system configuration with the sub-directory (i.e. folder) structure and permissions expected by the application.
- Configuration or data files required for the application to process properly.
- Access to all external programs (binary executables), scripts, libraries or other 3rd party non-4GL code that are accessed/executed using 4GL language statements such as shell access (e.g.
OS-COMMAND), stream access (e.g.
INPUT THROUGH), native API calls (e.g.
RUNof procedures defined as
PROCEDURE EXTERNAL), COM Automation or .NET classes.
Phase 1 - Assessment¶
Converting to Java is a big change. It comes with many benefits, but will take some work. Even though the core conversion work is fully automated, there is work involved in making the switch. It is important to properly plan the conversion project. The Assessment phase is designed to make a plan that can succeed.
Even if you are just trying to get achieve a Proof of Concept (POC), it is very important to execute the Assessment first. A POC is most easily done when the developer understands which parts of the 4GL code are already supported by FWD. Finding this out by running the automated conversion process or by trying to execute the converted code will be a tedious and time wasting process. This is true because the automated conversion process has not been designed to detect downstream failures. Since it can take hours to run the automated conversion process before hitting a problem, trying to get everything converted without planning ahead will be sure to waste the developer's time. Cutting out the Assessment will not save you time. You've been warned.
If the 4GL application was trivial, using FWD to convert it would not be needed. Any non-trivial 4GL application will have language features in use that will cover a large range of capabilities. Most 4GL developers are very surprised by what actually exists inside the code that has been developed for decades. As a general rule, the larger the code base, the broader the range of 4GL feature usage.
It is common for the 4GL developers to believe that the feature usage is smaller than it is. When dealing with hundreds of thousands of lines of code (LOC) or millions of LOC, it is guaranteed that there will be many features used that would not be remembered or thought of by the 4GL developers. Considering most such applications are built by teams of people that have changed over time, it is highly likely that there is a great deal of code in the application which is completely unfamiliar to most (sometimes all) of the current developers.
The effort to convert using FWD is directly proportional to the 4GL features in use. If there are features used which are not yet supported by FWD, then those features will have to be dropped, replaced or implemented in FWD. There may also be features in use that don't make sense in the converted system. For example, in a GUI application that is being converted to a web UI, the use of OCX or .NET controls is not going to work. There are approaches to deal with this, but it must be planned for.
It is also possible that all 4GL features in use are fully supported by FWD. In that case, the project effort will mostly be related to the organizational aspects of making the shift and the actual technical work will be modest.
The starting point for all assessments is the code and schemata of the application. FWD Analytics is used to look deeply inside the entire application and determine the technical scope of the project.
Parse the 4GL Code and Schemata¶
To look deeply into the code and schemata that comprise a 4GL application, one must use automation. Any non-trivial amount of code cannot be effectively reviewed using only manual efforts. Many applications have millions of lines of code, manually reviewing such projects is simply not possible. FWD provides extensive Code Analytics tooling that leverages automation to allow a deep, accurate and comprehensive review of exactly what is inside the 4GL application.
In order to use automation to analyze code, the code must be transformed into a structure that is easier to process programmatically. The "front end" of the FWD conversion tools is used to handle this transformation. This has nothing to do with converting to Java. Rather, each program and schema in the 4GL application is parsed and turned into a Abstract Syntax Tree (AST). It is only the "middle" and "code back end" portions of the conversion process which handle the actual conversion to Java. The front end is simply responsible for putting the code and schemata into a form that is optimal for programattic analysis.
The FWD tools treats the set of all 4GL procedures and include files that comprise a complete application as one project. Each project is processed by the FWD tools in a single batch run. That process requires access to the application source code, the database schemata and a range of configuration data.
The schemata (in
.df format) are processed and loaded into a schema dictionary. The schema dictionary is used to lookup, search or otherwise reference the detailed structure of each database used by the application. Virtually all Progress 4GL source code has some form of database usage. Since 4GL database code can only be parsed and properly interpreted when the database schema is known, the schema dictionary is an essential component before any meaningful processing of 4GL source code can be done.
Next the 4GL source code is processed. Each external procedure or OO 4GL class is run through the FWD preprocessor, then this fully expanded "cache" file is read as a character stream. This stream is converted into tokens or words (“lexing”) and is output as a stream of tokens. The parser reads the stream of tokens for each source file and structures it into “sentences” with related meaning (“parsing”). The sentences created by the parser are organized into a tree data structure called an Abstract Syntax Tree (AST). This data structure is much better designed for automated processing than the original, more verbose syntax. A great deal of ambiguous and non-regular text is resolved and syntactic sugar in the source code is removed. The ASTs output from the front end are the input upon which the rest of the conversion process operates AND they are the input for FWD Analytics.
The process for parsing these files:
- Gathering Inputs - Pull together all the code and schemata of the application.
- Project Setup - A directory tree must be created and populated with the application source code and schema files. Configuration files must be created and placed in the proper directories.
- Running the Front End Conversion - Run the front end which loads the schema and then preprocesses/parses the 4GL source code. An AST will be created for every source file and schema file.
The execution of the the conversion front-end for the first time will usually entail resolving a range of problems related to the application's source code, schema files, other files that are available or missing and other details the project configuration which need to be defined. In many ways, the FWD tools are more stringent than the Progress 4GL runtime. Many syntax errors will be reported during the front-end run that may not be reported in the 4GL until the runtime attempts to execute the invalid code. These syntax errors are an example of the things that may need to be resolved before the front-end can complete successfully.
It is also possible that the 4GL source code cannot parse due to latent problems in the FWD parser. It is possible that some issues can be bypassed with minor 4GL code changes but the typical approach is to fix the parser when these issues are found.
The effort for this stage of the Assessment can range from hours to weeks depending on how carefully the inputs were gathered, how careful the project setup instructions were followed and if there are any latent FWD parser bugs encountered. For someone with expertise in parsing, even the largest and most complex projects can be parsed with 2-3 weeks of effort. This is the most time consuming part of the Assessment.
It is very strongly advised to carefully follow the guidelines in Avoiding Common Mistakes. This WILL SAVE many hours or days of unnecessary work.
Run Reports using FWD Analytics¶
This cannot occur until after the entire application has successfully parsed. At that point the rest of the Assessment process depends upon the availability of FWD Analytics. To enable FWD Analytics, there is a process for Calculating Report Results. This "slices and dices" the code and schemata up and uses this to pre-calculate a wide range of reports about the application.
For the subsequent steps the FWD Analytics web UI is used to access those reports. Please see Code Analytics for details on how to start the web server and use the reports.
The successful run of the conversion front-end does not imply that the project will fully convert without improvements or additions to FWD.
The breadth and depth of legacy features and core functionality already supported by FWD is very large. Even trivial application code would not work at all if core functionality such as database access, transaction management, user interface, file-system I/O, process launching, etc. were not working extremely well.
FWD supports a wide array of complex features, such as GUI (including ADM/ADM2), web deployment, ChUI mode, Appserver mode, batch processes, Windows native API support, transactions, sub-transactions, UNDO, block level retry, infinite loop protection, dynamic database, etc. Large, complex ERP systems have been converted and work correctly with FWD.
The approach to development of the language compatibility of FWD is to implement those features needed by applications which are using FWD. This is a statement of popularity and common usage. Anything used by many applications will likely already be supported. Things not common in applications are less likely to be supported. The approach tends to bias the work toward features that have long been available in OpenEdge (and the Progress 4GL before it was OpenEdge). Long availability means it is more likely to have been used widely. That doesn't mean that features OpenEdge v10.x and v11.x are missing. For example, there is a robust support for all of the v10 data types. But it does mean that applications with code from Progress 4GL v9.x will be very well supported indeed. Some v10 and v11 applications will be fully supported and some may require missing features.
At this time, a handful of larger features (mostly OO 4GL and ProDataSets) and many smaller features (e.g., various language statements, built-in functions, methods and attributes, certain metadata tables, etc.) are not yet implemented. A summary of these features can be seen in Not Yet Supported ABL Features.
Far more is already done than needs to still be finished, but for a large application (in the millions of lines of code) it is still likely that some changes to FWD may be needed before the application can be converted and executed.
The gap analysis process is crucial for planning the conversion project. You must know in advance if you can run the conversion tools and if you can run the resulting converted code. Trying to convert an application that is not yet supported will just be a frustrating process because it will almost certainly abend somewhere in the conversion tool run, but it may take some hours to get there. Then you might think that you can make some code edits (to bypass the problem) and re-run conversion. This is true, but again it is likely to hit an abend in the conversion tools. This iterative process is very slow because each problem (of which there may by hundreds, thousands or many more) will cause a failure that takes hours to figure out.
Instead, it is imperative to identify the gaps first. Then make a plan to resolve each one. Then the code can be converted automatically.
In addition to the functional gaps, there are some features of the 4GL which are not supported in the FWD environment. In particular, consider that the current FWD implementation yields a statically converted Java application. This means that features such as dynamic generation and execution of 4GL code at runtime will not work. Another example is runtime preprocessor usage. Such usage must be identified and resolved before a full conversion is run.
FWD provides a set of Code Analytics tools for deeply inspecting large applications, even applications in the tens of millions of lines of code. By using these tools one can identify all 4GL language features that are used in an application. There is a set of canned reports which are of great value in identifying all “gaps” between the current 4GL source code and schema files and the features which are supported in FWD. These reports must be run and reviewed, to produce a gap analysis between the source and target environments.
See the Gap Analysis chapter for details on how to do this.
The Gap Analysis may take a few days to prepare for a large application.
It is important to review the long term objectives for the application.
This analysis should include:
- Objectives (e.g. Does the application need to provide a first class web and mobile experience?)
- UI Paradigm (GUI, Web, Mobile...)
- OS and Hardware Platform
- ABL Inhibitors and Limitations (e.g. What technical objectives have been difficult or impossible to achieve?)
- Market Trends
- Objectives (e.g. Does the application need to provide a first class web and mobile experience?)
- Unmet Customer Needs
- Growth Opportunities, New Markets
- Competitive Threats
- Development Team
- Retaining and Hiring Talent
- Tools and Enviroment
- Support Team and Processes
- Development Team
- Specifics of Approach
- Schedule (e.g. Must consider the customer base's ability to accept change.)
Mapping these requirements into the possible approaches is important to find the right plan. For example, some parts of the existing application may not have a long term future and the plan could be adjusted accordingly.
After this review, some work may be added to the project and it will certainly guide the prioritization of the work that is done.
It is beyond the scope of this book to explore this topic more deeply.
4GL Improvements Analysis¶
The conversion project is a good opportunity to make some 4GL changes that can lead to a better long term result. For example, there may be some level of cleanup of the code or elimination of technical debt, that is worth doing as part of the conversion.
However, the core design of FWD supports an iterative process where the automated conversion is run many times. One primary reason for running the conversion is to pick up the latest 4GL code changes and include them in the result. This means that 4GL changes can be made before, during and after the conversion project. Since the conversion project itself is already a big change, it is usually a good idea to limit the underlying cleanup/improvements to those that are:
- Absolutely necessary to make the converted application a success; OR
- Important enough to warrant the relative risk of the change.
Generally, it is best to do less rather than more. Future changes can be done in the 4GL and then the application can be reconverted at any time. Additionally, if some or all of the development is converted to Java, those portions can be reworked using the wide range of Java tools for refactoring and improvement.
The automated conversion process has extensive tools for customizing the converted output. See Customizing Conversion Rules for details on these tools. The section entitled "Techniques for Integrating Java Code into Reconverted 4GL Code" is especially useful.
The following areas of the application are commonly considered for improvements. A review of these areas may also identify some custom or non-standard conversion requirements. Use FWD Analytics to identify the locations in the code that are associated with each area. This review will usually take a few days for a large application.
Platform Specific Dependencies
Startup and Navigation and Configuration
4GL Idioms That Do Not Convert Or Which Convert Sub-Optimally
The standard conversion rules included in FWD are designed to be used for all applications. Nevertheless, there may be aspects of the current 4GL code which are not desirable to bring forward, and which were not easy to remove by hand in Phase 3A, perhaps due to their pervasive presence throughout the code base.
It may also be preferable to refactor the code, for instance to separate the static definitions of the user interface frames and temp-/work-tables from business logic.
The flexible nature of FWD Transform affords ample opportunity for both standardized refactoring, as well as customized transformations. The extensible TRPL rules engine which drives the conversion process allows patterns to be easily identified in the existing system and can transform these corresponding features to use different backing implementations in the next generation system.
In some cases, such refactoring is necessary simply to make the new system function, for example if the hardware and OS platform of the converted system is different than that on which the current code runs. In other cases, refactoring can involve the identification and removal of dead or unreachable code from the project.
In addition and perhaps most interestingly, this extensible refactoring capability can be leveraged to perform conversion improvements which are tailored to the 4GL application being converted. By writing custom conversion rules which recognize coding patterns/idioms specific to the application, these idioms can be replaced with a preferred implementation in the converted system. For example, all call-outs to a particular, platform-specific library or to an outdated or sub-optimal Progress procedure could be identified and replaced with calls to a more recent or portable API, or to an optimized, hand-coded library.
If customized conversion is needed, the requirements must be identified here. The custom conversion rules would be written in Phase 3B.
Detailed Project Planning¶
Based on the Gap Analysis, Requirements Analysis and 4GL Improvements Analysis, specific lists of tasks must be prepared. These tasks are used to plan the schedule, size the effort and determine the required skills/resources. From this finished plan, costs can be determined.
A specific guide on estimating the size of the project is outside the scope of this book. However, the most time consuming portions of a FWD conversion project are summarized here.
Estimating Phase 3A¶
The 4GL Improvements phase of the project is designed to include high priority work that supports the overall objective of the conversion.
Another reason to remove or replace code is if a 3 rd party (such as Progress Software Corporation) owns the rights to some of the source code and that code cannot be legally converted due to lack of a license to make a derivative work.
Either way, this is traditional 4GL development work resulting in modifications to the 4GL source code of the application. This work can be planned using existing 4GL development processes for estimation. The existing 4GL development team would be the most obvious resources to work these tasks.
Estimating Phase 3B¶
Although the FWD technology supports a large proportion of the Progress 4GL, since it is not 100% complete, in any given application there may be some gaps that must be filled before the application can be converted and executed. These features must be coded in Java (for the runtime) or ANTLR (for the preprocessors, the lexer or the parser) or TRPL (if the change is in the conversion rules).
The effort to add support for missing language features to the FWD environment (conversion and runtime) can often be one of the largest work efforts of the project. Some language features are trivial and could be added in minutes. Other features are substantial enough to take weeks or months to implement. The more complicated the feature, the more time consuming the development work will be to replace it. The largest features still unsupported in FWD will still only take a matter of months, not years.
The real effort in this development work is usually in fully understanding the compatibility specification of the language feature thoroughly enough to write an implementation from scratch to replace it. If the feature is complex, debugging and fixing that feature may take a substantial amount of time. The time necessary to debug and fix new features is significantly less if the up-front work on understanding the feature was suitably accurate and comprehensive. Since so much of the 4GL is undocumented, hidden or unexpected behavior, this up-front work is usually difficult and slow. DO NOT bypass this work. Any "short cut" will end up taking much longer to complete fully than if the proper investigation was done first.
Plan for testcases to be written and for there to be a significant effort to look past the obvious functionality in order to find all of the hidden, quirky, undocumented, boundary conditions and error handling that is actually possible. From there, writing the implementation is usually the smaller part of the work.
For this phase, it is critical to engage Java developers that have experience in supporting/enhancing FWD.
Estimating Phase 3C¶
The Quality Assurance and Testing phase exists to certify that the resulting application works as a drop-in replacement.
To deal with the complexity of shifting an entire application to Java, deep, up front analysis of an application’s code base is used to determine in advance which features a code base uses and to find potential problem areas. Nevertheless, some issues don’t turn up until a converted application is being tested. These generally are categorized as:
- Application Defects (an application bug that exists before and after conversion)
- Conversion Defects (some edge case is found that has not been converted correctly)
- Runtime Defects (cases where the FWD runtime is not behaving correctly)
- Performance Hot Spots (cases where performance in FWD is not as fast as the OpenEdge)
The larger a code base, the more variety of use cases exists, and the more likely it is that issues will be encountered. Time is needed after the gaps are filled in FWD to test the converted application and resolve these issues. As such, the actual testing done in this phase is usually done later in Phase 3 since it is dependent upon the application being ready for execution. This means that many or all of the results from Phases 3A and 3B will need to be complete first.
For estimation purposes, the following guidelines will help size the testing effort:
- Expect that the testing effort will be at least as large as that undertaken for a major version change in the application. It is assumed that such a change would require testing of all major work flows that are critical to the application.
- Plan for multiple testing iterations. The number of iterations will usually depend on the scope of the changes needed to the 4GL code and to FWD. When these changes are small in scope, the number of iterations might just be 2 or 3. If the scope is extensive, then up to 5 iterations might be reasonable.
- To the degree that the "base" testing does not cover the changes being made in the 4GL Improvements (Phase 3A) or in FWD Improvements (Phase 3B), add extra test effort that will be executed in the early test iteration(s).
- Make sure that testing is planned for:
- Reporting, printing and other forms of output.
- Use of peripherals or other specialized hardware that is needed for the application.
- Integration with external systems.
To the degree that the above test plans are not already created, effort may need to be made in this phase to develop adequate test scenarios.
Where possible, this effort can be greatly reduced using automated testing. This is optimal if an automated test harness already exists in which a suitable baseline of tests for the application can be run on a non-interactive basis. This would make the certification testing easy.
In most cases, no such baseline and testing environment exists. It is possible to create such a baseline as part of the project, however it should not be assumed that this is required.
For Character UI (ChUI) applications the Automated Testing Harness for TTY Applications can be used to automate the testing.
For GUI applications, many of the existing Windows-based automated test tooling (e.g. QTP) will not work in the converted environment. DO NOT spend time implementing GUI test automation using tooling before you know that it will work for FWD. The FWD UI implementation does not have the low level Windows implementation that is deeply encoded into OpenEdge. For this reason, the Windows tools will usually not work. Contact Golden Code for assistance.
Estimating Phase 3D¶
Each team involved in the deployment planning will have some changes needed for their processes. This may involve training on new tools or technology. It will also involve a survey of the existing processes and selective replacement/improvement as meets the needs of the organization.
Expect that each such team will need at least one person to spend 2-3 months on these activities. Often this person can then cross-train the rest of the team, however some training may be useful to have the entire team go through.
Phase 2 - Proof of Concept¶
A Proof of Concept (POC) is not a required phase of a conversion project. However, it can be useful to remove any doubt about whether FWD can suitably support the application being converted.
The core idea is that a portion of the application code and schemata will be converted to Java, a test database will be migrated into an RDBMS (PostgreSQL, SQLServer...) and the resulting application will be made available for testing. Successfully converting and running the POC code proves the suitability of FWD for use as the foundational technology for the migration of the application.
It is assumed that FWD and its dependencies are installed sufficiently for both conversion and runtime usage. Please see the Installation Configuration and Administration Guide for details.
For comparison purposes, it is important to have access to a test/development system. This would be a system on which the original Progress 4GL application can be run. This includes test database access and access to the development environment (including the procedure editor and data dictionary). These accounts must be available with full developer privileges. This means that the developers must have the ability to modify the application code or write standalone test cases that allow specific/arbitrary ABL code to be executed.
The following describes the process and guidelines for successfully completing a POC:
Runtime Configuration and Setup
A POC is really just a miniature conversion project. All of the same steps are done, but the POC is done on a cut down subset of the full application. This makes the POC less work than the full conversion project. Do not be discouraged by the steps above. Golden Code makes it easy by providing all the services needed for a turnkey POC.
Need help? Contact Golden Code for assistance.
Phase 3 - Technical Enablement¶
In this phase, there are 3 sub-phases that can be executed in parallel.
Phase 3A deals with the transformation of the 4GL application code. By improving the 4GL inputs to the conversion process, a more optimal Java output can be achieved.
Phase 3B is the work needed to the FWD technology (both conversion and runtime) that are necessary to meet the requirements of application being converted. This includes the incremental addition of any missing features used by the 4GL application code which are not supported today, plus any enhancements or optimizations found to be necessary along the way.
Phase 3C includes the planning and execution of a thorough Quality Assurance (QA) process to certify that the converted application is a replacement for the original application.
Phase 3A - 4GL Code Improvements¶
During the Assessment phase, a deep analysis is undertaken of the application code and the organization's requirements. Based on this, it is very likely that some number of improvements have been identified for the 4GL code. This phase is designed to do some manual preparation of the 4GL code base and schemata to ensure the best result from the automated conversion process.
As experts in the design and implementation of the current application, current development personnel would lead this part of the effort. The people working on the FWD Improvements (Phase 3B) will assist in the analysis of the code base and schema(s) with reporting tools, helping identify areas of the application which may need special attention and preparation for conversion. The outcome of this effort is a "cleaned-up" and "slimmed down" Progress 4GL code base and schema, in the sense that many of the areas of the application which are best not brought forward, or which are likely to cause problems in conversion, will have been addressed in some way.
Obsolete modules or features are best removed from the project at this point. Use the Call Graph Analyzer to build the call graph and determine the exact list of dead code that can be dropped.
Changing OS/hardware platforms will require platform-specific dependencies to be addressed.
Certain features of the 4GL which are sub-optimal to convert - in that they add no particular value or could be replaced with other/simpler implementations - will be removed or replaced at this point of the process. In general, any application complexity which is best addressed by modifications to the existing code base or schema(s) will be addressed in this step.
Because the project entails a conversion rather than a rewrite, the essential business logic of the new system is derived from the current system's code base. This ensures the new system is a drop-in replacement for the current system.
Some gaps identified in the Assessment are best resolved by making code changes to the 4GL source code or the schema files. There are cases in which source code changes are the only way to resolve a gap. These 4GL code changes are made like any other Progress 4GL development activity.
For the list of common reasons to make code changes and for details on potential solutions, please see the chapter entitled Code Preparation.
Phase 3B - FWD Improvements¶
There are three kinds of improvements made to FWD in this phase:
- Missing Features. If the original application (4GL code or schemata) uses features of the Progress 4GL environment which are not yet supported by FWD, then some improvements may be required in the FWD compatibility runtime environment. These should have been identified during the gap analysis.
- Latent Issues. It is also possible that the 4GL code uses supported language features in ways not previously anticipated, which will necessitate changes to the FWD runtime. This may be identified in advance through the gap analysis or it may manifest as latent defects in the conversion or runtime.
- 4GL Language or Runtime Enhancements. FWD can and does implement language enhancements and runtime features which provide capabilities that do not exist in OpenEdge. It is common for customers (both 4GL developers and business people) to request improvements and features to provide a better platform for the strategic development of the application.
The purpose of phase 3B is to address these improvements.
This effort will usually be undertaken primarily by Java developers that are experienced in enhancing and supporting the FWD project. Many improvements can be simply defined as matching the 4GL language feature in a compatible manner. This effort, while iterative, can begin immediately, since there are well known functional gaps in the runtime which need to be filled to support the application being converted.
Some changes will be discovered through testing cycles of the converted system. As testable iterations of the converted system become available, it is possible to discover additional functionality which needs to be enhanced or optimized, as well as for latent defects to emerge which need to be addressed. This part of the work tends to be "back-end loaded" in the project (it comes nearer to the end of the phase).
The outcome of this phase is an enhanced version of FWD which meets the functional and performance needs of the application being converted.
Fill Gaps in FWD Conversion and/or FWD Runtime¶
The gap analysis may have identified changes required to the FWD conversion process and/or to the FWD runtime which are required to allow the application to be converted and properly executed. For those gaps which are simple to remove or replace by making low-risk changes to the 4GL source code, it is often the best approach to do so. However, for any features that are intrinsic or which cannot be easily removed or replaced, FWD modifications may be necessary.
Often, this may some support for some 4GL language statement, function, method, attribute or option which FWD does not yet support or which is not yet fully supported.
But there are also requirements for use of specific non-supported platforms that may highlight gaps that need to be filled. Certain features (e.g. character user interfaces which provide TTY/terminal support) may have portions of the FWD conversion or runtime which need to be ported to the target platform (hardware and operating system) in order to operate properly.
Likewise, the application itself may be dependent upon platform-specific functionality such as programs that are run from language statements such as OS-COMMAND. If the target platform does not have a compatible replacement for such programs, a replacement must be created.
It is beyond the scope of this book to detail how this can be done. For more details on how to modify the FWD conversion process, please see the FWD Internals and the FWD Conversion Reference books. For details on how to modify the FWD runtime, please see the FWD Developer Guide and the FWD Internals books.
Design and Implement Custom Conversions¶
This is an optional step. By default, the conversion and runtime will provide an ABL-compatible approach to all supported ABL features. If a custom conversion is desired, then the automation tools can be exploited for that purpose.
Some areas that often can be enhanced or improved are security, startup, navigation and/or configuration. If these or other custom conversion approaches are required, the details of those requirements must be fully defined and the approach determined.
These aspects of the source application tend to be highly customized/unique but having a highly standardized result in the target has great value. For example, the target runtime has a well defined, standardized security infrastructure for managing user identities but the source application often has a custom approach to user identities. For this reason custom conversion processing (and even runtime support) is often desirable.
In other cases, the organization may have objectives for how certain code is to be converted, that may optimize the result. Regardless of the reason, the FWD tools provide a powerful and flexible mechanism to implement changes across massive amounts of source code, on a fully automated basis.
Designing and implementing such changes is beyond the scope of this book. Please see the FWD Developer Guide for more detail. It must be noted that depending on the requirements, the work to design or implement custom conversion processing can be non-trivial. For this reason it is important to handle this early in the project since it can have a significant impact on schedules, effort and resource allocation. This will involve writing custom conversion rules and possibly making modifications to FWD (the conversion or runtime).
At this point the project and any FWD modifications or customizations should be ready for conversion. The automated conversion process can be executed and the results can be built. For details on how to run the automated conversion process, please see the chapter on Running Conversion.
Upon successful completion, the converted Java application code will need to be compiled/built. Please see the chapters on Building Converted Code and Getting Your Project to Compile for details.
Problems may become visible the first time the conversion process is run or when getting the converted application to build. These problems will have to be resolved and then the process retried. This process is usually iterative when modifications have been made to FWD or when custom features have been implemented. In addition, any latent defects in the conversion processing that are triggered by something unexpected in the application's source code or schema will have to be fixed.
Once the application converts and builds successfully, this step is complete.
Phase 3C Quality Assurance and Testing¶
The objective of Phase 3C is to identify any unexpected deviations of the converted application from the original 4GL version.
Establish a Certification Baseline¶
The converted Java application is meant to be a drop-in replacement for the original. For this reason, the original application is used as a benchmark upon which the success of the converted application can be measured. A reliable way to do this is by testing the converted application and comparing the results to the same tests run with the same data on the original application.
There will be some existing testing process used for QA on major version upgrades of the application. This set of testcases is the default certification baseline for the converted system.
These tests can be run manually, but it is significantly better to automate the testing. Regardless, the project will use this baseline to define the user acceptance of the target system.
Each test must be created by users or business analysts which are expert in the application. Generally, each test definition should include the following:
- Description of the function or test.
- Required preconditions.
- Inputs including descriptions of any variation of data.
- Basic logical flow.
- Alternative logical flows, including error handling.
- Outputs including descriptions of any variation of data.
- Conditions for determining success.
While this book does not provide much further assistance in planning for testing or for the design of a testing baseline, the following guidelines have been found to be useful:
- Maximize data stability.
- Avoid certain bad practices.
- When using automation, frequently synchronize execution of the test with the current state of the application to properly simulate what a real interactive user does.
Data stability is the concept that each time a test runs, the data is the same (or as close as possible) to the state of the application at the time the tests were originally encoded. This maximizes the amount of input data that can be hard coded into test definitions and greatly simplifies the coding of tests. An example of how to ensure data stability is to restore the application's database to a specific/known backup before running testing.
Bad practices to avoid include the following:
- It is easy to encode a test which is dependent upon the state of the file system, security permissions/rights, the specific user's configured environment or other application state that is variable over time, user or other criteria. If an application changes its behavior based on state that is variable, this state must be controlled carefully before testing is run to avoid failures. For example, a reporting application (which writes a report in a text file in response to some user input) may work one way the first time the report is generated and may add additional prompts in subsequent runs. In this example, if the report file already exists (due to a prior run) the application may prompt the user to confirm that it should overwrite the report file. This variablity of the applications behavior must be made deterministic, through careful control over the environment and state of the system and application.
- Many applications will have different behavior (and appearance) on the first time a particular screen or function is accessed than in subsequent times.
Test step synchronization is used in automated testing toolkits. Test scenario execution is not allowed to continue until it is confirmed that the application has fully and successfully responded to all prior input. For example, many applications provide a capability to buffer keystrokes (for the user to type-ahead). However, some applications don't provide type-ahead and even some applications that do support type-ahead, have limits on how type-ahead can be used. This inherently limits the speed with which key input can be sent to the application when automating such interactive cases. Since automation tools can easily overrun the application (sending keys that will be discarded) and since the application's response time is variable (and under load or other conditions it may vary widely), most applications have significant limitations on how much can be typed ahead before the user must wait for the application to catch up.
In interactive usage, this is part of the end-user's job. They naturally have to wait at specific points for the application to catch up before proceeding. This is absolutely essential to do with great frequency in automating testing. If this is not done, the test execution will appear unreliable or will fail based on timing of responses from the application. Make sure any automation tools include a variety of synchronization mechanisms that can be used to fully avoid failures by frequent synchronization.
It is highly recommended to ensure that all critical work flows are covered in the certification baseline. Time should be spent in this phase filling in any important gaps in the coverage. This means that new scenarios must be designed and if automation is being used, those tests should be encoded and tested on the OpenEdge side to ensure they work.
Make sure that features such as the following are well covered:
- Reports, printing and other forms of output.
- Use of peripherals or other specialized hardware that is needed for the application.
- Integration with external systems.
It is not required to use automated testing, but it is highly desirable. Investment in test automation has an extremely high rate of return. This is particularly true for a conversion project, because of its highly iterative nature. As runtime features are added or fixed, or conversion rules are enhanced or added to meet the particular needs of the application being converted, it is critical to be able to perform consistent regression testing as quickly as possible. This ensures the integrity of the application is maintained while enabling quick turnaround over many, short development cycles.
More importantly, an investment in automated testing will have long term benefits for all future development.
Automated testing can double as a certification mechanism if all of the critical work flows are encoded as tests. By carefully defining a broad suite of test cases in the original, Progress environment, a baseline of test results is established, which must be matched by the converted application. When all tests complete identically in both environments (excepting behavior which was intentionally changed), given identical data/input sets, the converted application is considered a certified replacement for the original.
Automated testing requires that a comprehensive set of test cases be scripted, which covers as much of the application's functionality as possible. An automation harness is used to run the test scripts in multiple, parallel threads of execution, as if live users were using the system concurrently. This allows the test environment to be used both for functional and for scalability testing.
The harness must run against both the original environment and the new environment, so that baseline results captured against the current system can be used to compare against those produced with the new system. The baseline captures both screen information and non-screen information (e.g., report output, exported data), and the harness performs comparisons against this baseline at well-defined points during test execution. Failures are noted and reported at the end of the test run.
Since the harness will be run many times against the new system, a baseline must be captured using a specific data/input set on the original system. This same data set must be restored on the new system prior to each test cycle. The non-data inputs (key and mouse input) are recorded in the encoded tests which the harness will execute. This ensures that the logical state of the system is the same at the start of testing and that the same inputs will be provided during testing. If the systems are equivalent, then the system outputs will be equivalent.
Because the tests can run concurrently in multiple, parallel streams, and the timing of those streams may vary across harness runs, careful planning is required to account for any dependencies in the data or in the environment. For example, test cases in one thread of execution must not depend on data or file system changes made asynchronously by another, unless the tests are also designed to wait for those changes to be made. The test harness provides features to enable this behavior. This helps in simulating realistic work flows.
Since the definition of an appropriate set of test cases is a planning-intensive, largely manual process, it is critical to begin this effort as early as possible, to ensure the broadest possible test coverage. This will ensure issues are caught and addressed quickly. The test suite will prove a valuable resource for future development as well, since it can grow over time to encompass new features, even after conversion.
The current 4GL developers and/or QA staff, as experts in the application domain and the software's practical use, will have primary responsibility for the definition of a test suite and companion data set(s).
For automated testing batch, server or ChUI code, the Automated Test Harness is a capable toolset.
For the testing of the GUI code, a different testing toolkit will be needed.
TODO: link to the FWD plans for automated GUI testing.
Early Functional Testing¶
Each module or major feature of the source application must be tested in isolation for both functional and performance criteria. external resource access, database support, control flow and user interface support are examples of core functions to be tested. Any defects found must be fixed and the test cycle repeated.
Areas of the FWD runtime or conversion process that were modified or customized should receive extra testing. Any areas of the application which had 4GL source modifications should also receive extra attention.
The idea here is to use a combination of the certification baseline and additional testing which is designed to hit parts of the system that are more likely to expose problems. Early testing phases are intended to be a shake down of the system, flushing out the most serious issues, especially in parts of FWD that were changed in Phase 3B.
Integrated System Testing¶
Once the core work flows are working well, it is important to test the system in a more real-world environment.
Modules or major features of the source application which by nature integrate with other parts of the application or with external resources must must be tested for both functional and performance criteria. Multi-user processing, resource sharing/locking and other more complex features must be stress tested at this time. Any defects found must be fixed and the test cycle repeated starting with unit testing.
Performance or capacity testing should also be done at this time.
It is beyond the scope of this book to provide further details on integrated system testing.
Certification/User Acceptance Testing¶
At the end of Phase 3C, the official certification baseline previously established is now used to officially certify that the target application is functionally identical to the source application. This is optimally done via an automated test harness.
It is very important to ensure that "power users" are unleashed on the system with the objective to try to break it. This can usually be done simulaneously with Integrated System Testing. Any defects found must be fixed and the test cycle repeated.
It is beyond the scope of this book to provide further details on certification/user acceptance testing.
Phase 3D Deployment Preparation¶
Although FWD is designed to be a drop-in replacement for end-users, it does represent a change in the infrastructure for the application. This means that existing deployment processes must be changed. The processes needed for the converted application are typically simpler and naturally map to modern/cloud requirements. For example, web applications do not need to be installed on every user's desktop.
Even though the new processes are not hard and are often less work than the processes they replace, one still must spend the time to properly plan for the changes.
The existing database administration team should be trained on the new DBMS. This is usually quite easy and options like PostgreSQL have been found to be just as easy (or easier) to deploy and maintain as the OpenEdge database.
The FWD technology and the converted application are both based on Java and a range of modern, open source technologies. There are many standard deployment and management technologies that can be used and the stack itself is very much cloud and container "friendly". DevOps tooling (e.g. Ansible, Puppet, SaltStack, Chef...) is easy to implement to automate CI/CD.
The application's support team (e.g. help desk) needs to be learn about the differences in the implementations and their support processes need to be reworked to match the new software stack.
Whether the deployment is in the cloud, on-premise or both, the team doing the deployments must have a repeatable process for implementing:
- The cutover/migration of existing OpenEdge installations.
- The implementation of fresh ("greenfield") installations which never had OpenEdge.
The system administrators need to understand the new software stack and their support processes will need modifications to map into the new environment.
The deployment processes is typically refined by using them on testing/staging envronments. After the process is validated in testing/staging sytems, it would be expected to be validated in the pilot deployments (Phase 4).
Phase 3E Debug/Fix/Tune¶
To deal with the complexity of shifting an entire application to Java, deep, up front analysis of an application’s code base is used to determine in advance which features a code base uses and to find potential problem areas.
Nevertheless, some issues don’t turn up until a converted application is being tested. These generally are categorized as:
- Application Defects (an application bug that exists before and after conversion)
- Conversion Defects (some edge case is found that has not been converted correctly)
- Runtime Defects (cases where the FWD runtime is not behaving correctly)
- Performance Hot Spots (cases where performance in FWD is not as fast as the OpenEdge)
The larger a code base, the more variety of use cases exists, and the more likely it is that issues will be encountered. Time is needed after the Phases 3A and 3B to test the converted application and resolve these issues.
Phase 4 - Deployment¶
A common approach is to select a small set of customers/sites that are representative of the full install base. Pilot the new version in those customers, which will give an opportunity to shake out any issues with the deployment processes before full scale deployment.
The point of this process is to result in an identical target application, so there is no end user training needed. However the operations personnel and software developers must be properly trained to support, modify and maintain the new environment.
Please contact Golden Code Development (www.goldencode.com) for a list of training offerings which can be used to handle this requirement.
The runtime environment must be properly installed on the target platform/hardware. Change windows must be scheduled and software/configuration change packages must be built if any software distribution process is required for deployment. Freeze windows for application usage must be established. During that window, the latest data must be migrated/imported into the target database. All application-specific environment features must be properly installed and configured. Final, non-destructive tests should be executed on this system to confirm that it is operating the same as the test systems.
Please see the FWD Runtime Installation, Configuration and Administration Guide for details on the installation, setup and operation of the FWD environment.
Create the Database and Import Data¶
Unless the application uses no database support or only temp-table support, there will be a database that must be created. In all likelihood, any database will also need to have some data migrated from a currently running system. In some cases, the application may have an initial data set that must be imported to initialize the database with some application-specific minimum data.
The chapter on Data Migration for details on how to use the FWD tools to create the database(s) and optionally to import data to populate any database(s).
Define the Runtime Configuration¶
At a minimum, the configuration details for connecting to the database(s) will have to be provided to the FWD runtime. Once accomplished, most applications can be run at this point. See the chapter on Running and Testing Converted Code for details.
There are cases where the FWD runtime will need more extensive configuration. This will be driven by needs for non-default 4GL runtime features or when customization is needed for things such as networking addresses, digital certificates for encryption and resource/user/group definitions for security. These must be defined as appropriate for the target environment and the organization's requirements. Once defined, this must be encoded in the proper configuration format, most of which resides in an application “directory” that is read and managed by the FWD application server (part of the FWD runtime). Please see the FWD Runtime Installation, Configuration and Administration Guide for complete details.
At this point the converted application can executed and can be tested in comparison with the source application.
The old system is shut down, turned off or otherwise disabled and the users are silently directed to the new target application. Larger than usual support staff must be available to ensure that any unexpected issues can be quickly handled. A contingency plan to switch back must be ready beforehand to ensure that the risk from unexpected "showstoppers" is reduced.
It is beyond the scope of this book to provide further details on cutover.
All defects must be tracked, prioritized and fixed. Test cycles must be repeated to ensure that no regressions occur.
It is beyond the scope of this book to provide further details on defect support.
© 2004-2022 Golden Code Development Corporation. ALL RIGHTS RESERVED.