FWD » Core Development » Conversion Tools

We have multiple customers that split up a large 4GL application into multiple independent modules, each of which can stand on its own or run in combination with some other modules. This gives flexibility at development time (different teams can work on each module) and at deployment time (customers can choose which modules they use).

Evidently, OE allows splitting up the schema for a logical database into multuple standalone .df files (which we are calling "shards"). These can be somehow loaded together as a single database but they are maintained as separate files. Today, FWD can only load a single .df for a logical database. This task is intended to support a list of shards which will be loaded together.

Eric made some related comments in #6256-28.

One customer that uses this feature extensively organizes all activity in apps (each one like an independent module). This means that all development and compilation is done with code that only ever uses a specific sub-set of the overall schema shards. The customer reports that the same 4GL code can fail to compile with a merged schemata (that has all shards).

They found this during a test where they provided a merged schemata and all apps as a single code base. When they tried to compile it in the 4GL, they had to resolve a large number of ambiguous field names as a result. In the 4GL, each app can be compiled separately and then run in the same system with a single database that has all shards present. As long as the compiled code is unambiguous at the time it is compiled, runtime ambiguity will not exist.

FWD needs to support this capability. This means that apps/modules need to be able to be converted in pieces and executed at runtime as a larger system without issues.

This means that apps/modules need to be able to be converted in pieces and executed at runtime as a larger system without issues.

I believe the key here is to put the DMOs from a given profile into a package that is not just specific to the logical name of the database but also include a path segment for the profile. This would create a unique namespace per-profile per logical schema.

+Danut/Alex

I did some testing in an OE environment and want to see if I got it right regarding how this feature works.
I set up a new progress project with the following architecture:

/BigApplication
  /app1
      /data
          common_shard.df      <-- Defines Table1, Table2
      /src
          code_app1.p

  /app2
      /data
          common_shard.df      <-- Defines Table3, Table4
      /src
          code_app2.p
  main.p

Here is app1/data/common_shard.df:

ADD TABLE "Table1" 
  AREA "Schema Area" 
  DUMP-NAME "table1" 

ADD FIELD "CalcValue" OF "Table1" AS decimal 
  FORMAT "->>>,>>9.99" 
  INITIAL "0" 
  MAX-WIDTH 11

ADD FIELD "Amount" OF "Table1" AS decimal 
  FORMAT "->>>,>>9.99" 
  INITIAL "0" 
  MAX-WIDTH 11

ADD TABLE "Table2" 
  AREA "Schema Area" 
  DUMP-NAME "table2" 

ADD FIELD "CalcResult" OF "Table2" AS decimal 
  FORMAT "->>>,>>9.99" 
  INITIAL "0" 
  MAX-WIDTH 11

ADD TABLE "Table3" 
  AREA "Schema Area" 
  DUMP-NAME "table3" 

ADD FIELD "LogData" OF "Table3" AS character 
  FORMAT "X(40)" 
  INITIAL "" 
  MAX-WIDTH 80

ADD FIELD "Amount" OF "Table3" AS decimal 
  FORMAT "->>>,>>9.99" 
  INITIAL "0" 
  MAX-WIDTH 11

ADD TABLE "Table4" 
  AREA "Schema Area" 
  DUMP-NAME "table4" 

ADD FIELD "BusinessStatus" OF "Table4" AS character 
  FORMAT "X(20)" 
  INITIAL "" 
  MAX-WIDTH 40

Let's assume we need both data definitions in the same database to run the entire application. However, the issue is that both Table1 and Table3 contain a field named Amount.

This is app1/src/code_app1.p:

DEFINE FRAME myFrame
    CalcValue LABEL "Valoare" 
    Amount    LABEL "Total" 
    WITH 1 COLUMN SIDE-LABELS.

FIND FIRST Table1 NO-LOCK NO-ERROR.
IF AVAILABLE Table1 THEN DO:
    DISPLAY CalcValue Amount WITH FRAME myFrame.
END.  

So, in code_app1.p, we want to use Amount for a frame definition, but before we open a buffer scope for Table1.

Let's say main.p is responsible for compiling code_app1.p:

COMPILE "app1/src/code_app1.p".

If we use a dedicated database for app1 and load its data definition files, main.p runs without errors. However, if we use a shared database where both app1/data/common_shard.df and app2/data/common_shard.df are loaded, compiling code_app1.p produces the following output:

---------------------------
Error (Press HELP to view stack trace)
---------------------------
** Amount is ambiguous with Table1.Amount and Table3.Amount (72)
** C:\Users\Administrator\Progress\Developer Studio 4.3.1\workspace\DfShards\app1\src\code_app1.p Could not understand line 3. (196)

---------------------------
OK   Help   
---------------------------

To resolve this, we must compile code_app1.p in an isolated environment and deploy the resulting code_app1.r binar when connected to the main database. This allows main.p to run without errors, even though the main database contains both Table1.Amount and Table3.Amount.

Does this experiment clearly demonstrate how this feature works, or are there other aspects that haven't been captured yet?

At production, the customer will expect a merged database not to run against a shared. This means there is an assumption in this sharded mode is that any tables that get used across applications (across shards) meet one of these conditions:

• No ambiguity exists across the entire merged database. OR
• The 4GL code is written to be explicit enough to avoid ambiguity. An example would be for field references to be explicitly qualified.

A customer that uses this sharded concept must enforce one of these, usually through some level of code review or other "discipline" at development time. Leaving this condition in the codebase would be invalid.

Understood, that makes perfect sense for production discipline. But does this setup still fundamentally confirm the OE behavior we need to target for this task? Specifically, I'm referring to how the r-code uses its own schema definitions when running against a merged database.

Octavian Adrian Gavril wrote:

Understood, that makes perfect sense for production discipline. But does this setup still fundamentally confirm the OE behavior we need to target for this task? Specifically, I'm referring to how the r-code uses its own schema definitions when running against a merged database.

Yes, it correctly describes how the setup would work in OE.

Constantin/Eric: Comments?

Greg Shah wrote:

Octavian Adrian Gavril wrote:

Understood, that makes perfect sense for production discipline. But does this setup still fundamentally confirm the OE behavior we need to target for this task? Specifically, I'm referring to how the r-code uses its own schema definitions when running against a merged database.

Yes, it correctly describes how the setup would work in OE.

Constantin/Eric: Comments?

The description is accurate. The compiled r-code disambiguates all field/buffer references; if you run that r-code against a database which has the shard loaded, it will work in OpenEdge.

On our side, there is one more complexity: for runtime conversion, if we keep the .p2o artifacts one per shard, then we will need to know which one to load. But, considering that a runtime query requires the buffers to be specified before parsing it, we can infer from that. Otherwise, we will need a .p2o for the merged schema for runtime.

If the details are in annotations in the DMOs, do we need the P2O documents? Annotations seems cleaner.

Greg Shah wrote:

If the details are in annotations in the DMOs, do we need the P2O documents? Annotations seems cleaner.

Is not just .p2o, .dict are loaded by SchemaDictionary also, at runtime. We may be able to rebuild this from the DMO annotations if all info is there. We can do it lazily or at once at server startup, but this will add some delay for the FWD server startup or initial query conversion.

I think it is best to remove the .dict files too.

Greg Shah wrote:

If the details are in annotations in the DMOs, do we need the P2O documents? Annotations seems cleaner.

The p2o documents are also used for import.

Since they are structured in a tree form, it makes TRPL navigation trivial. In case of import, the p2o is a kind of source of truth (since the .df which was transformed into .p2o should be naturally paired with data stored in the .p files). Rewriting the import to use the annotations is possible, but that direct link is lost. However, since the annotation were basically created from .p2o, and are also stored in application jar, an indirect link still exists.

At any rate, I think moving the import away from TRPL is better. It will couple more the DMOs with the data and will avoid the extra cost of compiling the TRPL source(s). The performance gain would be minimal and only observable in really small databases. The heavy work is (and will be) performed by the import worker.

Using TRPL for import is overkill since the tree structure is so trivial. So, yes, I agree it should move away.

Greg Shah wrote:

This means that apps/modules need to be able to be converted in pieces and executed at runtime as a larger system without issues.

I believe the key here is to put the DMOs from a given profile into a package that is not just specific to the logical name of the database but also include a path segment for the profile. This would create a unique namespace per-profile per logical schema.

Isn't this already handled by the fact that each module has its own package root?

For example, I've converted app1 and I'm using its JAR file in another empty main app that has a merged .df. When app1 is run in this environment, it will use the com.goldencode.app1.dmo.fwd package.

I understand that we don't support this, but I'm trying to set up an environment that simulates the current problem in FWD.

Yes, using a per-module namespace would be a way to solve the issue I noted in #6320-5.