Project

General

Profile

Wiki »

Using Ansible for Management of a FWD Ubuntu System

Introduction

The VM management bzr repository at https://proj.goldencode.com/downloads/automation/fwd_mgmt/ provides scripts to automate the configuration of an existing Ubuntu 16.04 machine, to be used as a FWD conversion and/or application server machine.

The scripts are written for an open source systems management tool called Ansible.

To use these scripts, there will be two systems:

  1. the TARGET system, which will be configured by the Ansible scripts
  2. the CONTROLLER system, on which the Ansible scripts will be executed

For these instructions, substitute the following placeholders in each command:

  • <server_hostname> This is the hostname of the target system (VM) on which you are installing.
  • <admin_username> This is the admin user account on the target system (VM) on which you are installing.

This process is used to create systems that can be used to deploy build, test or production systems. This can be used on physical systems or on virtual machines. The result is designed for and tested on a single server system. There is no requirement for a Graphical Desktop environment (e.g. Gnome or KDE) to be installed.

The following components are installed:

  • PostgreSQL Database Cluster
  • Imported database instance for the application database (only a single DB instance is supported at this time). This is optional.
  • FWD Application Server
  • Converted Application (converted binaries and the runtime configuration). Optionally the conversion environment is fully setup and executed.
  • FWD Web Client and the Spawner
  • Standard Embedded Mode Web Application

This is sufficient to run remote FWD clients (ChUI, Swing GUI, locally installed web clients), ChUI clients and both kinds of GUI web client. If the system does have a graphical desktop environment, then it would be possible to run any of the interactive clients (e.g. Swing clients) there as well.

Prerequisites

Ubuntu 16.04 System

This process does not install the Ubuntu operating system itself. It is assumed that there are two existing systems (or virtual machines) running Ubuntu 16.04. While it is possible to run the automated installation on a TARGET Ubuntu 16.04 system that has had other software installed and configured, starting with a fresh install will ensure that the process will work. The controller is not sensitive to this same issue.

Project Configuration

The application must have an established project configuration. This includes the conversion tool configuration and scripts for conversion as well as the configuration needed for executing the converted result.

The project configuration DOES NOT include the application's source code.

To create a project configuration for an application, start with ChUI Hotel Project as a Template or GUI Hotel Project as a Template. These provide a project configuration that can be customized for your project.

If a system is needed for exploration and prototyping purposes, the two sample projects can be used as is (ChUI Project Configuration, see Hotel ChUI Demo Application or GUI Project Configuration, see Hotel GUI Demo Application).

In short, to create a runtime system you must have an existing project configuration or one of the FWD sample projects. The project configuration must be zipped up without any top level directory included in the zip. This must have the same folder structure as the Hotel demo projects. The zip will be extracted to /opt/<appname>/ folder in the TARGET system.

In the instructions below, the project configuration zip will be referred to as <fwd_app_cfg_zip>.

FWD

By default the process will download the latest FWD code. You may override this to use a specific version. The public versions available can be found in Downloads. In the instructions below, the FWD zip will be referred to as <fwd_zip>.

In the case where you are making edits to the FWD project and need to build this zip for your customized FWD distribution, the zip file must have a root p2j/ folder. It will be extracted to /opt/ and a symbolic link /opt/<appname>/p2j/ will be created, targeting the /opt/p2j/ folder.

Converted Application

There are two ways to deploy an application:

  1. Put the 4GL source code on the TARGET system and run conversion. This is called convert_in_place mode. This is useful for development and test systems.
  2. Run conversion on some other system and gather the converted results (no source code) for deployment to the TARGET system. This is called deploy_preconverted_result mode. This is most common for production systems.

These two modes are mutually exclusive: you can either deploy for conversion or deploy a preconverted application.

Convert From 4GL Source on the TARGET System

The following inputs are needed when running convert_in_place mode.

4GL Source Code

Create a zip file that includes all necessary 4GL source code and resources (e.g. icons, bitmaps). In the instructions below, the source code zip will be referred to as <fwd_app_code_zip>. The zip will be extracted to the /opt/<appname>/abl/ folder.

The source must be zipped up without any top level directory included in the zip. The structure of the source tree inside should contain all the directories that would normally be present when running or compiling in OpenEdge. The project configuration will have a PROPATH defined that references these locations.

Schemata

Create a zip file that includes all necessary 4GL schema files (in .df form). In the instructions below, the schemata zip will be referred to as <fwd_app_schemata_zip>. The zip will be extracted to the /opt/<appname>/data/ folder.

The source must be zipped up without any top level directory included in the zip.

4GL Source Code Patches (Optional)

This is only needed if you are converting from 4GL source code on the TARGET system.

This is an optional zip file which contains any patches to apply to the 4GL source code. It would also include a patch script named patches.sh. Without the patches.sh, nothing will be done. Normally, this would be coded to use the patch utility to edit source files based on patch files created with diff.

In the instructions below, the patches zip will be referred to as <fwd_app_patches_zip>. The patches must be zipped up without any top level directory included in the zip. If this exists, it will be extracted to the /opt/<appname>/abl/ folder and Ansible will execute the patches.sh script. The patches.sh script must reside in the top level of the zip, such that it will be found as /opt/<appname>/abl/patches.sh when it is extracted.

Possenet Code (Optional)

This is only needed if you are converting from 4GL source code on the TARGET system.

If your project uses ADM or ADM2, then it will need to convert with the Possenet version of the ADM/ADM2 framework instead of the version that is included with OpenEdge. There are other use cases where this code may be needed. Please see Possenet for details.

Download the latest Possenet code or use a specific Possenet zip. In the instructions below, the Possenet zip will be referred to as <possenet_zip>. If specified, the Possenet sources will be extracted to /opt/<appname>/abl/ folder.

To enable Possenet sources, add the fwd_possenet=true Ansible option when invoking the installation script (more details below). This is enough to let the Ansible scripts download the latest Possenet code . If you want to use a specific Possenet zip, set the fwd_possenet_url="file:///opt/<possenet_zip>" option.

Deploy Pre-Converted Results

For deploy_preconverted_result mode, all inputs will be provided in a single zip file.

In this mode you must create a zip file with the jars and conversion artifacts of a previously converted application. The easiest way to do this is to run ant deploy.dist AFTER you successfully convert (and compile). It will generate the necessary zip file in distribution/<app_name>_jars_deploy_<YYYYMMDD>.zip.

In the instructions below, the binary deployment zip will be referred to as <fwd_app_deploy_zip>. The zip will be extracted to the /opt/<appname>/ folder.

If you manually create the zip file, the source must be zipped up without any top level directory included in the zip. It should include these contents:

build/lib/*
adm_windows.json
name_map.cache
ddl/*
data/namespace/*
cfg/registry.xml

This zip is used to avoid conversion on the TARGET system. The results are run on the TARGET system using the FWD runtime environment.

Exported Database .d Files

A zip file containing the dump (OpenEdge database export) of all ABL tables as exported in .d files. The data dump files must be zipped up without any top level directory included in the zip. In the instructions below, the dump zip will be referred to as <fwd_app_data_dump_zip>. The content of this zip will be extracted to /opt/appname/data/dump/<dbname>/ folder.

If this file is missing, the data import will be bypassed.

Configuring the TARGET System

First, on the TARGET system, do the following:

1. Login to the fresh Ubuntu Server 16.04 64-bit system.

Make sure you have admin rights on the account you are using (<admin_username>). You need to be at a terminal (command prompt). If you are accessing this remotely, it will look something like this:

ssh <server_hostname> -l <admin_username>

2. Install some prequisite software.

sudo apt-get update && sudo apt-get install python

If you are prompted for your "sudo password", just use the same password as you use for login. If you are using an AWS VM, it will probably already be setup for key-based authentication and passwordless sudo so it may not be an issue.

3. Copy the following zip files (as needed and described above) to your home folder on the TARGET system.

  • All Scenarios
    • fwd_app_cfg_zip
    • data_dump_zip
    • fwd_zip
  • Conversion on the TARGET System
    • fwd_app_code_zip
    • fwd_schema_zip
    • fwd_app_patches_zip (optional)
    • possenet_zip (optional)
  • Deployment of Pre-Converted Code
    • fwd_app_deploy_zip

4. Move all these files into /opt/ and set the proper permissions.

sudo mv *.zip /opt/ && cd /opt/ && sudo chmod 0755 *.zip && cd

Configuring the CONTROLLER system

Next, on the CONTROLLER do the following:

1. Setup key-based SSH access.

If the target system is not already setup for key-based SSH access, you must enable it. If the CONTROLLER hasn't any SSH keys built yet, use ssh-keygen -t rsa to generate them before running this command:

scp .ssh/id_rsa.pub <target_system_admin_username>@<target_system_hostname>:.ssh/authorized_keys2 && ssh <target_system_admin_username>@<target_system_hostname> chmod 0600 .ssh/authorized_keys2

2. Install some pre-requisite software.

sudo apt-get update && sudo apt-get install python ansible bzr bzrtools python-paramiko

3. Check out the FWD-specific Ansible-driven automation project.

bzr co https://proj.goldencode.com/downloads/automation/fwd_mgmt && cd fwd_mgmt

4. Run the install.sh command.

Use the existing install.sh script to either deploy (via the deploy_preconverted_result mode) or install for conversion (via convert_in_place mode) your application.

The syntax for this script is:

./install.sh -srvhost=<host> -adminuser=<username> -mode=[convert_in_place|deploy_preconverted_result] <ansible_opts>...

where:
  1. <host> must be replaced with the target system's hostname
  2. <username> must be replaced with the target Ubuntu system's admin user account name
  3. use -mode=convert_in_place to install for conversion or -mode=deploy_preconverted_result to install an already converted application
  4. <ansible_opts> are specified as opt1=val1 opt2=val2 ... optn=valn - they will be passed directly to the ansible playbook.

Depending on the mode used, only certain files or Ansible options are required.

Depending on how the TARGET machine is configured, it might take ~20 minutes for ansible to complete the scripts.

It is recommended to do a reboot to pick up any updated kernels that occur during the installation process. The FWD application servers should autostart.

Common Ansible options and associated files

TODO: enhance to work with multiple DBs
TODO: enhance to work with multiple user accounts

Each option must be specified via ansible_opt=value. If the value contains spaces, enclose it in quotes. Unless specified, all options are mandatory.

  1. fwd_install_root - it represents the location where the FWD project and the Application will be installed. If not specified, defaults to /opt/.
  2. user_acct_name - an user account which will be created and configured on the target system, under which the FWD client will be executed
  3. user_acct_pw - the password for the user account above
  4. fwd_app_name - the application name
  5. fwd_app_cfg_zip - the converted project configuration files, will be unzipped in i.e. /opt/<fwd_app_name>.
  6. fwd_db_name - the database name.
  7. fwd_app_data_dump_zip - optional, required only if fwd_import_db=true. Contains a zip (with no folder structure) with the data dump files for the ABL exported database
  8. fwd_distro_url - optional. If you are using a custom FWD installation, the value will look like file:///opt/<fwd_distro_zip>. Otherwise, the latest FWD will be downloaded.
  9. pg_mode - optional, defaults to true. This allows to enable import to Postgresql (when true) or to H2 (when false).
  10. db_port - optional, defaults to 5432. Specify the PostgreSQL port.
  11. fwd_import_db - optional, defaults to false. When set to true, the import will be performed. If this is not specified or if it is explicitly set to false, then you will need to run import separately (through the normal process for running a data import for your application).

Ansible options and files for convert_in_place mode

  1. fwd_possenet - optional, defaults to false. When true, it specifies that the applications requires the Possenet sources.
  2. fwd_possenet_url - optional. If using a custom Possenet zip, specify its value like file:///opt/possenet.zip. If fwd_possenet=true, then the Possenet sources will be extracted to the application's abl/ folder.
  3. fwd_app_code_zip - the application's legacy code and resources. Will be extracted to the application's abl/ folder.
  4. fwd_app_patches_zip - optional. Contains a script named patches.sh and/or other patch files to be applied to ./abl/ folder
  5. fwd_app_schemata_zip - a zip containing a single file, with the application's exported schema. This file will be extracted as data/<appname>.df.
  6. fwd_convert_app - optional, defaults to false. It indicates the application will be converted by the Ansible scripts. If this is not specified or if it is explicitly set to false, then you will need to run conversion separately (through the normal process for running a FWD conversion for your application). WARNING: for large applications, this may take hours to complete.

Ansible options and files for deploy_preconverted_result mode

  1. fwd_app_deploy_zip - specify a zip file name which contains the application's converted Jars and other conversion-generated files.

How the Target System is Configured

For a detailed look, you can check the roles targeted by the 'conversion_reference_system.yml' and 'deploy_reference_system.yml' files from the 'fwd_mgmt' project. Following are some high-level details on what the Ansible scripts will perform on the target machine:

  • Installs the necessary packages (FWD dependencies) to run and compile FWD.
  • Ensure the Ubuntu installation is up-to-date.
  • Adds a 'fwd' account under which the FWD server will run (you can modify it via the fwd_acct_name=<name> Ansible option for install.sh).
  • Installs and patches ncurses, installs the JDK and will build the FWD project in /opt/p2j/.
  • Install a postgresql database, version 9.5, and creates a cluster named fwd on port 5432 (if not otherwise overridden).
  • Creates a user account using the user_acct_name and user_acct_pw Ansible options.
  • Installs the application in /opt/<app_name> (for convert or deploy, depending on the mode).
  • Will configure the /opt/<app_name>/deploy/server/directory.xml to work on the target system.
  • If gen_certs=true was passed to install.sh, it will re-generate the certificates (without the root CA, this can be re-used).
  • Configure the target system to automatically launch the FWD application server and the embedded server. These can be controlled via sudo service fwd start or sudo service fwd_embedded_web start from an admin OS account.
  • Configure the firewall to allow connection to the following ports, from outside the target machine:
    • 443
    • 8443 - to access the embedded application clients
    • 7443 - for the AW application's virtual desktop clients
    • 7449:7459 - to allow the AW web clients to be accessed from another machine. These port must match the directory.xml configuration in webclient/portsrange.
  • the /etc/hosts will be changed, so that all lines which are ending with the <server_host>, will be set as <target-machine-ip> <server_host>. This is required so that the target machine will be able to server the web clients properly.

Future Improvements

The following items are WIP for the Ansible configuration:

  • Enable support of ACME/Let's Encrypt real certificates, to avoid self-signed certificates.
  • Add reverse proxy implementation to allow a single port to be used and multiplexed for multiple users. This will allow access to the machine running the application server from machines outside its network (i.e. internet), by having only one port exposed to the internet.
  • Firewall configuration for production usage.
  • Custom configuration of the FWD clients and users.

© 2004-2017 Golden Code Development Corporation. ALL RIGHTS RESERVED.