Internationalization¶

Internationalization (or I18N) is a very complex subject. This chapter is not designed to provide a comprehensive treatment on the subject. Rather, the purpose is to document the minimum necessary steps to handle internationalization issues for getting an application successfully converted and running.

The most common I18N issue is to properly process character data (strings) for an application. This encompasses a range of issues including mapping the proper glyphs (visual representations of a character) to each character, sorting and the translation of character data from input sources (e.g. files or child processes) and to output sources (e.g. files, child processes or the terminal). There are many character sets in existence and Unicode is the standard that can handle all of the world's languages with a single definition. The problem is that most of the world's data, databases and source code are not using Unicode yet, especially in the Progress 4GL. The Java language handles character data internally as Unicode data, but inputs and outputs are not necessarily in Unicode. Java can process internally in a different character set than Unicode, but that requires much more intensive coding to provide that facility. The approach that FWD takes is to follow the path of least resistance in Java (to allow everything to work in Unicode internally), but to ensure that all input data and output data is translated from and to the correct encodings as needed. This means that the proper encoding of every input and output must be known and honored.

At this time, only a single encoding (character set and language combination) is honored in an application and in the conversion. This means that the same encoding must be used for all inputs and outputs. Inputs include the source code, schema files, exported data dump files for database import and any other data files or child processes that may be used at runtime. Outputs include the populated database, the terminal user interface and any files or child processes (e.g. printers) that may be written to. The way to make this work is to ensure that everything uses the same encoding (which is defined by a specific locale).

Besides character data, there are also other I18N issues such as date formats and number separators. These must be specified application-wide as well, to ensure that everything is consistent.

The character encoding and these other configuration data (e.g. date formats) are all combined into a definition of an environment that is specific to a language, country and character set. This definition is called a locale. The locale is something that is shared across multiple applications on the same system and it is usually a shared facility provided by or in the operating system.

All the examples in this chapter are for a Linux system, but other systems such as UNIX or Windows will have similar facilities. The specific commands and techniques will vary but the concepts should be the same.

Conversion_Locale

Input File Encoding¶

All input files should be encoded with the same character set. Find out the character encoding for the source code. Make sure that all source code files use that same encoding. When the schema files (.df files) are exported, make sure to use that same encoding in the export. When the data is dumped/exported from the database (.d files), make sure to use that same encoding. If there are any other files or inputs for the application, ensure they use that encoding. Anything that is not encoded properly should be converted.

This encoding must be the same character set that will be used when running the FWD commands. If it is not the default character set, then the default locale must be overridden when the FWD commands are executed. See Setting the Locale for FWD for details.

Operating System Locale Support¶

The locale is usually a set of data files that are read and used during processing of the C runtime library and/or the operating system APIs. Subsystems such as Java will depend upon the locale definitions to properly handle I18N issues.

If the specific locale needed is not yet defined in the operating system's locale definitions, it must be added. There may be an installation program or other utilities to handle this. On Linux, character sets, locales and the tools to compile new locale definitions are included with the C library. The most important tool is named localedef.

1. To examine the currently installed locales and character sets, identify the paths used on the system. Run this:

localedef --help

Near the end of the output, there will be a display like this:

System's directory for character maps : /usr/share/i18n/charmaps
                       repertoire maps: /usr/share/i18n/repertoiremaps
                       locale path    : /usr/lib/locale:/usr/share/i18n

Look inside the /usr/lib/locale/ (in this case) to find the locales that are already installed. If the required locale is not present, it will need to be compiled.

2. To compile the proper locale definition for Linux, use a command similar to this:

sudo localedef --no-archive -f IBM866 -i ru_RU ru_RU.IBM866

--no-archive causes the compiled locale definition to be created with a name the same as the last command line parameter (ru_RU.IBM866). The definition will be created in a new directory of the same name which is in the main locale path (usually /usr/lib/locale/). In this case the compiled locale will be named ru_RU.IBM866 and it will be located in /usr/lib/locale/ru_RU.IBM866/.

The -f parameter specifies the character map name to be used. There must be a <character_map_name>.gz in the directory where character maps are stored (usually /usr/share/i18n/charmaps/). In this case the character map is IBM866 and there should be a file named /usr/share/i18n/charmaps/IBM866.gz.

The -i parameter specifies the language and country definitions to use. There must be a <lang>_<country> file of the same name in the directory for the input definitions (usually /usr/share/i18n/locales/). In this case the language is ru, the country code is RU so the parameter is ru_RU and there must be a file named /usr/share/i18n/locales/ru_RU.

3. Confirm the new locale is visible in the locale list using locale -a. It is important to check this in a regular user account (not root), because the permissions of the locale directory can hide a locale from normal users. The <lang>_<country>.<charset> locale should appear in the list. If it does not, ensure the new locale's file system permissions are correct. They should look like this:

/usr/lib/locale:
drwxr-xr-x 3 root root pl_PL.IBM852

/usr/lib/locale/pl_PL.IBM852/:
-rw-r--r-- 1 root root LC_ADDRESS
-rw-r--r-- 1 root root LC_COLLATE
-rw-r--r-- 1 root root LC_CTYPE
-rw-r--r-- 1 root root LC_IDENTIFICATION
-rw-r--r-- 1 root root LC_MEASUREMENT
drwxr-xr-x 2 root root LC_MESSAGES
-rw-r--r-- 1 root root LC_MONETARY
-rw-r--r-- 1 root root LC_NAME
-rw-r--r-- 1 root root LC_NUMERIC
-rw-r--r-- 1 root root LC_PAPER
-rw-r--r-- 1 root root LC_TELEPHONE
-rw-r--r-- 1 root root LC_TIME

This can be achieved with the following commands (the order is important):

sudo chmod 0755 /usr/lib/locale/pl_PL.IBM852/
sudo chmod 0644 /usr/lib/locale/pl_PL.IBM852/*
sudo chmod 0755 /usr/lib/locale/pl_PL.IBM852/LC_MESSAGES/
sudo chmod 0644 /usr/lib/locale/pl_PL.IBM852/LC_MESSAGES/*

Note that some Linux system updates may modify permissions on these file system resources. If you find error messages to this effect, reissue the chmod commands above to restore permissions to their proper settings.

Setting the Locale for FWD¶

On Linux, the locale of the current process is set using the LANG environment variable. On most Linux systems, by default the LANG is set to en_US.UTF-8 (English language, US country with the UTF-8 character set). This will be picked up by all FWD tools (conversion or runtime) since the Java Virtual Machine (JVM) will naturally honor the default locale (LANG setting). The JVM is the infrastructure that allows Java programs to execute.

To force all input/output processing for the JVM to a specific locale, use the following syntax:

LANG=<locale> <command>

This overrides the LANG environment variable for the lifetime of that specific command. Alternatively, this can be set as the system default or as the default for the user's shell based on startup script entries (e.g. ~/.bashrc).

This must be used with all FWD conversion tools (e.g. ReportDriver, ConversionDriver and PatternEngine). Likewise, it must be used to start FWD servers, FWD clients and any FWD batch processes (e.g. ServerDriver or ClientDriver). This example runs the bogus Whatever program with a Russian locale:

LANG=ru_RU.ibm866 java -classpath $P2J_HOME/p2j/lib/p2j.jar com.goldencode.p2j.Whatever

Whenever possible it is best to run the Java process using UTF-8 as the default character set. If the only requirement for an override is for the conversion process to read source code files (external procedures, classes and include files) that have been encoded in a specific charset, then the preferred method to handle this is to set a global hint (e.g. put it in the abl/directory.hints file) using the source-charset hint. See Conversion Hints and look in the preprocessor section.

PostgreSQL Locale¶

The PostgreSQL database server on Linux relies upon the locale support of the operating system to encode and collate string data. A PostgreSQL database cluster is initialized using a particular locale, which permanently determines the collation and default character encoding of all databases created in that cluster.

As discussed in the Database Setup chapter, the FWD project provides a custom Linux locale (en_US@p2j_basic) for Progress-like, basic collation using the ISO-8859-1 character set. At the time of this writing, no custom locales have been created for other character sets. Creating a new, custom locale from scratch is a topic beyond the scope of this document, which is described in the book FWD Developer Guide.

Assuming you have created or obtained a custom locale you wish to install in Linux and with which you want to initialize a PostgreSQL cluster (for example, ru_RU@ p2j_basic), you would follow the instructions pertaining to the en_US@p2j_basic locale provided in the Database Server Setup for PostgreSQL on Linux chapter, replacing all references in the instructions to en_US@p2j_basic with ru_RU@p2j_basic (or with the name of your particular, custom locale).

By default, all database instances created in a cluster will use the default character set of the cluster. So, a database created in a cluster initialized with the en_US@p2j_basic locale will by default encode character data using the ISO-8859-1 character set (identified as LATIN1 by PostgreSQL). Assuming a custom locale named ru_RU@p2j_basic would use the IBM866 character set, a database created in such a cluster would default to the IBM866 character set. This default can be overridden using the -E option to the createdb utility when creating individual databases, but we recommend omitting this option and allowing the default behavior. Please consult the PostgreSQL user documentation for a list of available character sets.

Once you have initialized your cluster and created a database with the appropriate character encoding, you are ready to import your data. When importing data exported from a Progress database, special care needs to be taken to ensure the proper encoding is maintained. The import process uses the JVM to decode character data from your export (.d) files, to handle that data as Unicode characters while in the JVM, and finally to encode that data to the encoding expected by your database server. If the the default character set of the JVM does not match the character set of the data export files you have dumped from your Progress database, you will need to set the LANG environment variable to specify the encoding of the export files when running the import command.

If the data export files are encoded in the IBM866 character set, for instance, the following command should be run from the $P2J_HOME directory to launch the import program:

LANG=ru.RU.ibm866 java
   -server
   -classpath p2j/build/lib/p2j.jar:build/lib/{my_app}.jar:cfg:
   -Djava.util.logging.config.file=cfg/logging.properties
   -DP2J_HOME=.
   com.goldencode.p2j.pattern.PatternEngine
      -d 2
      "dbName=\"{export_file_path}\"" 
      "maxThreads={num_threads}" 
      schema/import
      data/namespace
      "{schema_name}.p2o"

Please refer to the discussion of importing data in the Data Migration chapter for the meanings of the substitution parameters {my_app}, {export_file_path}, {num_threads}, and {schema_name} in the command above.

H2 Database Collation¶

The FWD runtime environment uses the H2 open source database (www.h2database.com) to manage all temporary tables and for certain internal processing; also, H2 can be used to manage the permanent database for testing and development purposes. When used internally (for the temporary tables or housekeeping), this database is embedded in the JVM process which runs the FWD server and as such, it does not use the same locale services as the PostgreSQL server. Generally, if the localization settings of the FWD server JVM are appropriate to your locale, the embedded H2 databases will also behave properly with respect to locale.

However, special care needs to be taken to ensure character data in database records managed by H2 are collated in the same way as they would be in a Progress database, which is different in some respects than Java's default text collation behavior. To address this mismatch, the FWD project provides a custom implementation of the java.text.spi.CollationProvider interface, for use with the FWD server JVM. Installation and configuration of this service provider is discussed in the H2 String Collation Service Provider Installation (Deprecated version is kept as ~~String Collation Service Provider Installation~~) chapter.

To ensure the H2 database collates its records properly for a locale other than en_US, a new implementation of the java.text.spi.CollationProvider interface will have to be developed, packaged, and installed.

if compatibility with Java8 and previous version is required, the new library is installed using using the Java Extension Mechanism. This would produce a jar file similar to p2jspi.jar, which would be installed in the public EXT directory of the installed Java, being publicly available.
starting with Java8, the search order of locale sensitive services can be configured by using a system property named java.locale.providers. We will use SPI,JRE in order to inform JRE that our implementation of the CollationProvider is to be processed first (found in a local folder), and defaulting to the system JRE standard locations. Regardless of the order in which the search will be performed, the list of directories containing the extension is set in the system property java.ext.dirs, which should defined at the same time.

To enable a specific collation for the database the following configurations should be done:

static configuration (without server running). In cfg/p2j.cfg.xml, for each namespace which is designed to use H2 dialect, the following entry will be added:

      <namespace name="testDb" [...]>
         <dialect-specific name="h2">
            <parameter name="collation" value="en_US@cp1252_fwd_basic" />
         </dialect-specific>
      <metadata name="standard" />

This is used mainly when the database is imported.

server runtime configuration. In directory.xml, for each database, at /server/standard/database/<db-name>/p2j/, add the following:

              <node class="string" name="embedded-collation">
                <node-attribute name="value" value="en_US@cp1252_fwd_basic"/>
              </node>

Notes:

the same value for the collation must be set for a specific database for both static and runtime configuration.
the _temp database should also have the same collation set as expected in 4GL. The location in directory.xml is /server/standard/database/_temp/p2j/embedded-collation. This database does not have a static configuration.
for any database, if no collation is set, the default en_US@iso88591_fwd_basic is used.

The topic of developing and packaging such a service is beyond the scope of this document. It is discussed in the FWD Developer Guide book. TODO: is it really?

When H2 is used to manage the permanent database, the collation needs to be set using the SET COLLATION statement before any table is created, using H2's web console or its script running tool - see the Data Migration chapter on how this scripting tool is used to install the schema in the database.

Date Formatting¶

The order of date components (day, month and year fields) can vary from country to country or place to place. When date values are converted to and from strings, the internal date fields must be parsed or rendered in the right order. By default the order will be month - day - year (MDY). If the default is incorrect for this application, the FWD conversion and runtime tools provide a means to override the date order.

For the conversion, this can be set as the date-order global parameter (see the Project Setup chapter). In particular, this heavily affects how the data import process works, since all date fields in the data dump files (.d files) can only be properly parsed with prior knowledge of the order of the date fields.

To specify order of the 3 date sub-components, use a 3 character string using the letters "M", "D" and "Y" once each. The order from index 0 to index 2 represents the left to right ordering of the components. The leftmost date component is defined by the character at index position 0, the middle component is defined by the character at index 1 and the rightmost date component is defined by the character at index 2.

At runtime, this same knowledge must be provided to the FWD server to allow date processing to work as expected. The chapter on Running Converted Code provides guidance on getting a test server running with the converted application. Before the FWD server is started, the dateFormat value must be specified in the directory. Find the section of the directory under the path /server/default/runtime/default/ and add a section similar to this:

<node class="string" name="dateFormat">
   <node-attribute name="value" value="YMD"/>
</node>

Only specify this if you need to override the default of “MDY”.

Number Formatting¶

The characters used to parse number input and and format number output can vary from country to country or place to place. When numbers are converted to and from strings, the character that separates the integer and fractional portions of a decimal number must be known to parse or render the decimal “point”. For example , the period “.” character is the decimal separator in 139.78. Likewise, the integer portion of a number (decimal or not) is often separated into groups by a specific character. For example , the comma “,” character is the group separator in 31,888. If the appropriate number separators can be read from the locale of the JVM, then no extra effort is needed. If the JVM cannot obtain the locale-specific separators, then a comma will be used for the group separator and a period will be used as the decimal separator. If for whatever reason, the default values are incorrect, the FWD conversion and runtime tools provide a means to override the separators.

For the conversion, these can be set as the number-group-sep and number-decimal-sep global parameters (see the Project Setup chapter). In particular, this heavily affects how the data import process works, since all number fields in the data dump files (.d files) can only be properly parsed with prior knowledge of the separator characters.

At runtime, this same knowledge must be provided to the FWD server to allow number formatting to work as expected. The chapter on Running Converted Code provides guidance on getting a test server running with the converted application. Before the FWD server is started, the numberGroupSep and numberDecimalSep values must be specified in the directory. Find the section of the directory under the path /server/default/runtime/default/ and add a section similar to this:

<node class="string" name="numberGroupSep">
   <node-attribute name="value" value="."/>
</node>
<node class="string" name="numberDecimalSep">
   <node-attribute name="value" value=","/>
</node>

By default the numberGroupSep is a comma “,” and the numberDecimalSep is a period “.”. Only specify these entries in the directory if the defaults are not acceptable.

Translation Support¶

Introduction¶

FWD provides translation support for the 4GL legacy application.

Applications translated with OpenEdge Translation Manager can be converted and executed using all the translation data implemented using the OE Translation Manager. This process is fully automated if the translation database is kept up to date with the legacy sources.

The use of Translation Manager is not a requirement, legacy applications can be trnaslated after they are converted.

FWD implements gettext, the proven and widely used internationalization and localization (I18N and L10N) system. The biggest advantage of gettext is its ability to separate writing code from the work of text translation. The typical workflow of translations in gettext applications is as follows:

write code, enclose translatable literals in translation methods (see Converted Code for more details)
create a set of .po files from the sources, a grep-like tool is used to extract the translatable literals
add translations to the *.po entries, as needed
generate resource bundles from the translated *.po files
deploy the resource bundles, which hold the translations for runtime usage, in the application

The FWD I18N workflow utilizes some of the tools from the GNU gettext package. GNU gettext is the most commonly used implementation of gettext. It is available for all the most commonly used OS platforms, including Windows and Linux.

Requirements¶

FWD v4¶

This feature was added in #3817 and the changes are complete as of branch 3821c revision 13125. This is first available to the public as FWD v4.

GNU gettext¶

Installation is as follows:

Ubuntu	Windows
sudo apt-get update -y sudo apt-get install -y gettext	Download the correct binary package from https://mlocati.github.io/articles/gettext-iconv-windows.html and run the downloaded executable. Make sure the installed executables are on the system PATH.

Standard Project Layout¶

The details below may reference the standard project layout as can be seen in Hotel_GUI_Demo_Application.

Converting a Translated Project¶

To convert an existing 4GL application translated with OpenEdge Translation Manager perform the following steps:

export the translations from Translation Manager (TM)
configure and convert the legacy application
run the legacy application

Exporting Translations from TM¶

TM has the capability to export the translation data. However the current release of TM, which is hosted on Github (https://github.com/gquerret/adetran), has some limitations and doesn't export all the required information. We extended the Github hosted TM and patched this limitation.

To export the translation data download the improved TM from Translation Manager Extended Export Version. This extended version must be compiled/built in the 4GL. Run the Translation Manager in OpenEdge and then:

Switch to the Data tab.
File -> Export...
In the Export dialog select the UTF-8 code page and export.
Repeat the export for all the target languages.

Configuring and Converting the Legacy Application¶

During conversion the translatable literals are identified and matched with the translation data exported from TM in the previous step. The result will be a set of *.po files (one for each external procedure - *.w and *.p files, or one for each class file). These files will be converted to resource bundles and deployed with the converted legacy application.

Proceed as follows:

1. Add the following parameters in <project_root>/cfg/p2j.cfg.xml: i18n-enable, tm-translations, text-expansion.

Example:

       <parameter name="i18n-enable"      value="true" />
       <parameter name="tm-translations"  value="./translations_fr.csv,./translations_de.csv,./translations_cs.csv" />
       <parameter name="text-expansion"   value="100" />

The parameter tm-translations holds the translation data exported from TM in the previous step.

text-expansion is the percentage value of how much the translatable literals will be expanded to accommodate longer length target languages. This value has the same meaning as the "Growth Table" percentage field in the TM "Compile" dialog (used for building translated procedures). You may omit this if you don't use text expansion.

2. Convert the application as usual.

3. After conversion the directory <project root>/translations will contain *.po files with all the translatable strings encountered in the converted sources and the matched translations.

4. Inspect the conversion log. Look for the messages "Failed to resolve TM translation for", these indicate a match in the exported TM data was not found for a translatable literal in the legacy sources.

5. Translate all the unmatched literals by editing the corresponding *.po files.

6. Generate resource bundles. Execute <project_root>/p2j/tools/i18n/gen-bundles.sh. This will generate Java resource bundles from the *.po files and output them in <project root>/build/classes.

7. Execute ant jar to pack the resource bundles in the project jar file.

8. Execute ant deploy.prepare to deploy the project jar.

Running the Converted Application¶

With all the resource bundles deployed in the previous step, it is time to fire them up. There are two ways to let the runtime know what the effective target language should be. The 4GL code can assign the CURRENT-LANGUAGE session attribute or by adding the configuration parameter currentLanguage in directory.xml. The runtime inspects the following directory paths when searching for currentLanguage: /server/<serverID>/runtime/<account_or_group>/<id>, /server/<serverID>/runtime/default/<id>, /server/default/runtime/<account_or_group>/<id>, /server/default/runtime/default/<id>.

The values assigned to CURRENT-LANGUAGE or currentLanguage directory parameter are those originally used in the legacy application and as such exported in the translation data from Translation Manager.

If you assigned text-expansion in p2j.cfg.xml before conversion, also assign the same value in text-expansion directory attribute. Put it in one of the locations stated above for currentLanguage.

Converting an Untranslated 4GL Project¶

To convert a 4GL application that has no existing translations, perform the following steps:

configure and convert the legacy application
translate the texts identified during conversion
deploy the translated data and run the legacy application

Configuring and Converting the Legacy Application¶

During conversion the translatable literals are identified and output in a set of *.po files (one for each external procedure - *.w and *.p files, or one for each class file). These files will be converted to resource bundles and deployed with the converted legacy application.

Proceed as follows:

Add the following parameters in <project_root>/p2j/cfg/p2j.cfg.xml: i18n-enable, text-expansion.
Example:

       <parameter name="i18n-enable"      value="true" />
       <parameter name="text-expansion"   value="100" />

text-expansion is the percentage value of how much the translatable literals will be expanded to accommodate more talkative target languages. This value has the same meaning as the "Growth Table" percentage field in the TM "Compile" dialog (used for building translated procedures). You may omit this if you don't use text expansion.

Convert the application as usual.

Translating the Texts Identified During Conversion¶

After conversion the directory <project root>/translations will contain *.po files with all the translatable strings encountered in the converted sources. There will be one *.po file per external procedure. Create n sets of the *.po files (where n is the number of target languages) by copying the original *.po files. The naming of the resulting translated *.po files is important. The name of the file must include the target language name. It is in the form <procedure_base_file_name>_<target_language>.po. For example, if you end up with the .po file myprocedure.po after conversion, you will need to create two copies in the same dir myprocedure_Czech.po and myprocedure_German.po if your target languages are Czech and German. Note that Czech and German will also be the values your application will assign to CURRENT-LANGUAGE and currentLanguage directory parameter when setting the effective target language.

Translate the texts in the *.po copies made above.

Deploying the Translations and Running the Converted Application¶

Generate resource bundles. Execute <project_root>/p2j/tools/i18n/gen-bundles.sh. This will generate Java resource bundles from the *.po files in <project_root>/translations and output them in <project root>/build/classes.
Execute ant jar to pack the resource bundles in the project jar file.
Execute ant deploy.prepare to deploy the project jar.
Follow the section Running the Converted Application.

Reference¶

Converted Code¶

During conversion of legacy application FWD identifies translatable string literals and generates a translatable Java representation. In the generated Java code all the literals are wrapped in translation methods, see the example below.

The original legacy code in the procedure file test.p:

def var fi as char init "Hello world!" view-as fill-in label "Message".
message "Hello world!".

The converted code (business logic):

/**
 * Business logic (converted to Java from the 4GL source code
 * in test.p).
 */
public class Test
{
   public static final long __trid = I18nOps.initReferent(com.goldencode.testcases.Test.class);

   public static final TranslationManager __tm = TranslationManager.getInstance();

   public static final String FMT_STR_1 = __tr("x(20)");

   TestF fFrame = GenericFrame.createFrame(TestF.class, "f");

   @LegacySignature(type = Type.VARIABLE, name = "fi")
   character fi = UndoableFactory.character(__tr("Hello world!"));

   public static String __tr(String msg)
   {
      return __tm.translate(__trid, msg);
   }

   public static String __tr(long ctxt, String msg)
   {
      return __tm.translate(__trid, ctxt, msg);
   }

   public static I18nString __tre(String msg)
   {
      return __tm.translateExpand(__trid, msg);
   }

   public static I18nString __tre(long ctxt, String msg)
   {
      return __tm.translateExpand(__trid, ctxt, msg);
   }

   /**
    * External procedure (converted to Java from the 4GL source code
    * in test.p).
    */
   @LegacySignature(type = Type.MAIN, name = "test.p")
   public void execute()
   {
      externalProcedure(Test.this, new Block((Body) () ->
      {
         fFrame.openScope();

         fFrame.enable();

         message(__tr(2, "Hello world!"));
      }));
   }
}

The converted code (UI):

public interface TestF
extends CommonFrame
{
   public static final long __trid = I18nOps.initReferent(com.goldencode.testcases.Test.class);

   public static final TranslationManager __tm = TranslationManager.getInstance();

   public static final Class configClass = TestFDef.class;

   public static String __tr(String msg)
   {
      return __tm.translate(__trid, msg);
   }

   public static String __tr(long ctxt, String msg)
   {
      return __tm.translate(__trid, ctxt, msg);
   }

   public static I18nString __tre(String msg)
   {
      return __tm.translateExpand(__trid, msg);
   }

   public static I18nString __tre(long ctxt, String msg)
   {
      return __tm.translateExpand(__trid, ctxt, msg);
   }

   public character getFi();

   public void setFi(character parm);

   public void setFi(String parm);

   public void setFi(BaseDataType parm);

   public FillInWidget widgetFi();

   public static class TestFDef
   extends WidgetList
   {
      FillInWidget fi = new FillInWidget();

      public void setup(CommonFrame frame)
      {
         frame.setDown(1);
         fi.setDataType("character");
         fi.setFormat(__tr("x(20)"));
         fi.setLabel(__tre(1, "Hello world!"));
      }

      {
         addWidget("fi", "fi", fi);
      }
   }
}

In the converted Java code, beside the actual translatable literals, there are other code elements created to aid the translation. These are __trid and __tm fields, and the methods __tr, __tre.

__tm is the API facade for all the translation needs.
__trid is a handle for the resource bundle in use for the current external procedure (see the section TODO). Note that both the generated Java files (business logic and ui) reference the same resource bundle.
__tr and __tre methods are the translation methods, translatable literals are passed to them and the translations are returned back. They are always generated locally to hide the details of passing __trid in TranslationManager.

OpenEdge Translation Manager and 4GL treat the same string in different places in the code as different entities. To disambiguate the same string literals FWD uses a context identifier, a number identifying a string literal in the code. In the example above "Hello world!" string is converted as __tr("Hello world!"), __tr(1, "Hello world!") and __tr(2, "Hello world!"). The first occurrence of a string in the legacy 4GL code is always converted without the context identifier. The subsequent occurrences receive a context identifier with 1 increments, second occurrence 1, third occurrence 2, etc. Context identifiers are included together with the literals in *.po files, and so the same source text literal may get different translations for the different occurrences in the code.

Resource Bundles¶

The converted application needs a storage of the translation data. This is where resource bundles come in play. Resource bundles are binary deployable pieces of the converted application in the form of simple Java classes implementing the Java ResourceBundle interface. There is one resource bundle class per a 4GL external procedure or a legacy class per one target language. If the application is translated in n target languages there will be n resource bundle classes per a procedure file or a legacy class.

Resource bundles are created from the translated *.po files with the msgfmt tool, which comes in the GNU gettext bundle.

See the resource bundle class below for the following *.po file.

msgid "" 
msgstr "" 
"Language: Czech\n" 
"Content-Type: text/plain; charset=UTF-8\n" 

#: ./test.p:1:25
msgid "Hello world!" 
msgstr "Ahoj světe!" 

#: ./test.p:1:62
msgctxt "1" 
msgid "Hello world!" 
msgstr "Ahoj světe!" 

#: ./test.p:1:84
msgid "x(20)" 
msgstr "x(20)" 

#: ./test.p:4:9
msgctxt "2" 
msgid "Hello world!" 
msgstr "Ahoj světe!"

package com.goldencode.testcases;

import java.util.Enumeration;
import java.util.MissingResourceException;
import java.util.ResourceBundle;

public class Test_Czech extends ResourceBundle {
   private static final String[] table;

   public Test_Czech() {
   }

   public Object handleGetObject(String var1) throws MissingResourceException {
      int var2 = var1.hashCode() & 2147483647;
      int var3 = var2 % 7 << 1;
      String var4 = table[var3];
      return var4 != null && var1.equals(var4) ? table[var3 + 1] : null;
   }

   public Enumeration getKeys() {
      return new Enumeration() {
         private int idx;

         {
            for(this.idx = 0; this.idx < 14 && Test_Czech.table[this.idx] == null; this.idx += 2) {
            }

         }

         public boolean hasMoreElements() {
            return this.idx < 14;
         }

         public Object nextElement() {
            String var1 = Test_Czech.table[this.idx];

            do {
               this.idx += 2;
            } while(this.idx < 14 && Test_Czech.table[this.idx] == null);

            return var1;
         }
      };
   }

   public ResourceBundle getParent() {
      return this.parent;
   }

   static {
      String[] var0 = new String[]{"", "Language: Czech\nContent-Type: text/plain; charset=UTF-8\n", "Hello world!", "Ahoj světe!", null, null, "1\u0004Hello world!", "Ahoj světe!", "x(20)", "x(20)", "2\u0004Hello world!", "Ahoj světe!", null, null};
      table = var0;
   }
}

As you can see the resource bundle maps keys, the source untranslated texts, to the translated values. The actual implementation depends on the number of translation pairs. In the example above msgfmt transforms a hash of an untranslated text to create an index in an array of all the translations. But msgfmt chooses a different strategy if it detects collisions in the hashes.

The class name of the resource bundle holds the target language of the legacy application. If the target language is "Czech" the resource bundle name must be decorated with "Czech", like in the example above "Test_Czech". This is important for FWD runtime to resolve the appropriate resource bundle. In the call I18nOps.initReferent a referent class is given, this is usually the converted external procedure Java class. The referent id returned from initReferent is passed to TranslationManager when translating strings. Thanks to this id and the current language FWD has all the needed information to resolve the correct resource bundle.

GNU gettext¶

The translation process relies on GNU gettext, a popular and commonly used implementation of gettext. It is available for all major operating systems. See the project main page https://www.gnu.org/software/gettext/.

GNU gettext provides various utility programs for handling translation files. FWD translation process relies on one of them: msgfmt.

msgfmt generates a database of all translations from po files the running program queries to provide the target translations. See https://www.gnu.org/software/gettext/manual/html_node/msgfmt-Invocation.html for more details.

For complete GNU gettext documentation see https://www.gnu.org/software/gettext/manual/html_node/index.html.

Project

General

Profile

FWD

Wiki

Internationalization¶

Input File Encoding¶

Operating System Locale Support¶

Setting the Locale for FWD¶

PostgreSQL Locale¶

H2 Database Collation¶

Date Formatting¶

Number Formatting¶

Translation Support¶

Introduction¶

Requirements¶

FWD v4¶

GNU gettext¶

Standard Project Layout¶

Converting a Translated Project¶

Exporting Translations from TM¶

Configuring and Converting the Legacy Application¶

Running the Converted Application¶

Converting an Untranslated 4GL Project¶

Configuring and Converting the Legacy Application¶

Translating the Texts Identified During Conversion¶

Deploying the Translations and Running the Converted Application¶

Reference¶

Converted Code¶

Resource Bundles¶

GNU gettext¶