Project

General

Profile

Bug #9950

« Previous | Next »

Upgrade the metadata schema to a newer version

Added by Ovidiu Maxiniuc about 1 year ago. Updated 5 months ago.

Status:
Test
Priority:
High
Assignee:
Ovidiu Maxiniuc
Target version:
-
Start date:
Due date:
% Done:

80%

billable:
No
vendor_id:
GCD
case_num:
version_reported:
version_resolved:
reviewer:
Constantin Asofiei
production:
No
env_name:
topics:

select_standard.sh Magnifier (809 Bytes) Roger Borrello, 02/11/2026 11:51 AM

Related issues

Related to Database - Feature #9549: implement missing metadata features New
Related to Database - Feature #4155: remove conversion dependency on metadata .df file WIP

History

#1 Updated by Ovidiu Maxiniuc about 1 year ago

One of the latest customer projects uses a newer schema (standard.df) of the _meta database than the one we have. I was not yet able to identify the exact revision where the new fields were added in 4GL. The conversion seems to be fine, but there are some issues at runtime. We wrote some runtime support for managing the VST tables (_Lock, _Connect). These 'updaters' work in background and keep the VST up to date with the system changes.

The problem is that some tables in the new _meta schema have additional fields (they are always incrementally additive) which should also be maintain/set/updated. The reason we noticed this was because at least one of the fields was mandatory and the updater, unaware of its existence, left the column unassigned, therefore NULL. This causes an exception at database level since the mandatory fields are not null columns.

The goal of this task is to upgrade the _meta schema (standard.df) to the latest available (an up-to-date dump will be necessary) and upgrade the 'updaters' so that the behaviour like the above are avoided. Beside these classes, the MetadataManager will most likely require upgrade when the _meta database is initially populated.

#3 Updated by Constantin Asofiei about 1 year ago

  • Assignee set to Ovidiu Maxiniuc
  • Priority changed from Normal to High

Ovidiu, please get some preliminary changes in a branch, so we can patch 9457c and test FWD server startup.

#4 Updated by Ovidiu Maxiniuc about 1 year ago

  • Status changed from New to WIP
  • reviewer Constantin Asofiei added
I noticed the following changes in the currently supported meta files (from DDL PoV):
  • 10 (mandatory) columns added to meta_connect;
  • 1 column added to meta_myconnection_;
  • 5 (mandatory) columns added to meta_trans;
  • 2 columns added to meta_userio;

In the latest schema I could see additional 5 meta tables: _DbServiceManager, _DbServiceManagerObjects, _Repl-AgentActivity, _Repl-AgentControlActivity, _Repl-InterAgentActivity.

Interesting enough, this standard.df lacks the following tables: _aud-audit-data, _aud-audit-data-value, _aud-audit-policy, _aud-event, _aud-event-policy, _aud-field-policy, _aud-file-policy. I think they were intentionally left out to simplify the schema (so that it is a bit smaller than the one we currently use).

I created 9950a and committed the changes. My test starts successful and allows the clients to connect (that was also a problem unidentified with customer PoC).
Committed revision 15895.

Should I patch 9457c as well?

#5 Updated by Constantin Asofiei about 1 year ago

  • Related to Feature #9549: implement missing metadata features added

#6 Updated by Ovidiu Maxiniuc 9 months ago

I added backward compatibility with projects that have not upgraded from OE 11.6. By default 11.6 is the default meta schema level. To benefit from the new fields in the connection-related VST tables one should set the metadataLevel in his directory.xml to 117 (OE 11.7). From my investigations, this is the OE version where the new fields were added. This approach allows for later upgrades to newer OE versions.

I must admit, the code is not as extensible as I would have liked. Initially, I started with a different approach, but it expanded to a lot of interfaces and classes (that provided better isolation of OE versions) but when I put in balance I realised the the complexity is not worth it. So here is the more pragmatic solution.

I also rebased 9950a to latest trunk so now the current revision is 16263. I will start the tests with projects using different metadata levels.

#7 Updated by Constantin Asofiei 9 months ago

  • % Done changed from 0 to 80
  • Status changed from WIP to Review

#8 Updated by Constantin Asofiei 6 months ago

Greg/Eric, this task is about standard.df for OE12.x which has additional fields for tables we manage, and these do not exist in our current standard.df meta schema.

The approach currently is to make OE12.x and OE17.x standard.df version configurable via directory.xml, and change FWD so that it uses the different internal structure for these tables, depending on the version.

#9 Updated by Greg Shah 6 months ago

In the past we have always found that we can use the latest standard.df even with older code. The newer versions have been backwards compatible, so all that is needed is just a reconversion with the later .df. Is that compatibility broken in 12.8?

FYI: my long term intention is to implement our own manually written Java classes for metadata and to have those be always present in FWD, removing the need for any standard.df and eliminating any knowledge of this from the conversion. The way we do this now make no sense and it just adds friction and makes things more fragile.

#10 Updated by Greg Shah 6 months ago

  • Related to Feature #4155: remove conversion dependency on metadata .df file added

#11 Updated by Constantin Asofiei 6 months ago

Greg Shah wrote:

In the past we have always found that we can use the latest standard.df even with older code. The newer versions have been backwards compatible, so all that is needed is just a reconversion with the later .df. Is that compatibility broken in 12.8?

There are new fields added in 12.8 - and depending on what an app does with these tables, it may break them.

#12 Updated by Constantin Asofiei 5 months ago

Ovidiu: please prepare tasks in each customer project to upgrade to the new standard.df file. We will do this depending on how the customer switches to trunk.

Otherwise, is 9950b safe to merge at this point?

#17 Updated by Ovidiu Maxiniuc 5 months ago

Constantin Asofiei wrote:

Ovidiu: please prepare tasks in each customer project to upgrade to the new standard.df file. We will do this depending on how the customer switches to trunk.

Done. Please let me know (in private) if there are any customers I forgot about / unaware of.

Otherwise, is 9950b safe to merge at this point?

I am not aware of any reason not to.

#18 Updated by Constantin Asofiei 5 months ago

Ovidiu, please remind me - are there meta tables stored in the physical DB which are affected? This is required to know for existing productions installations, how to upgrade.

#19 Updated by Ovidiu Maxiniuc 5 months ago

Constantin Asofiei wrote:

[...] are there meta tables stored in the physical DB which are affected? This is required to know for existing productions installations, how to upgrade.

No. The affected runtime is only for VST tables related to client connections, whose data is persisted fully in temporary _meta database. Due to the nature of VST, this kind of information is not meaningful from one execution of FWD server to another, therefore not persisted.
Currently, the only _meta tables stored in/moved to permanent database are: _user, _sec-authentication-domain, and _sec-authentication-system. None of these are updated.

#21 Updated by Constantin Asofiei 5 months ago

  • Status changed from Review to Merge Pending

Please merge after 10894a.

#23 Updated by Roger Borrello 5 months ago

Are there any immediate changes to customer applications post merge?

#24 Updated by Ovidiu Maxiniuc 5 months ago

Roger,
The application's behaviour shouldn't be affected. The upgrade should consist of:
  • upgrade FWD to r16415
  • convert the application using the updated .df (it's about 350KiB size, just let me know how we should proceed here, I can commit the file directly to project repository).

Executing only one of these will prevent the server from starting. There are no other operations to be performed for build, deployment or runtime.
I am not sure I answered your question. If that's the case, please rephrase.

#25 Updated by Roger Borrello 5 months ago

Ovidiu Maxiniuc wrote:

Roger,
The application's behaviour shouldn't be affected. The upgrade should consist of:
  • upgrade FWD to r16415
  • convert the application using the updated .df (it's about 350KiB size, just let me know how we should proceed here, I can commit the file directly to project repository).

Executing only one of these will prevent the server from starting. There are no other operations to be performed for build, deployment or runtime.
I am not sure I answered your question. If that's the case, please rephrase.

I think that answered the question... If we upgrade the runtime of FWD to trunk_16415, but have not reconverted (both with the updated standard.df and trunk_16415) there will be trouble.

#26 Updated by Ovidiu Maxiniuc 5 months ago

  • Status changed from Merge Pending to Test

Roger Borrello wrote:

[...] If we upgrade the runtime of FWD to trunk_16415, but have not reconverted (both with the updated standard.df and trunk_16415) there will be trouble.

Yes, that is pretty much correct.
  • trunk_16415 itself does not contain changes in how the .df is converted, but expects the _meta schema to be defined as in the new standard.df;
  • OTOH, standard.df adds some mandatory fields which the old FWD are not (yet) aware and therefore will remain null. The SQL will reject the records causing FWD to crash.

BTW,
Branch 9950b was merged into trunk as rev. 16415 and archived.

#27 Updated by Constantin Asofiei 5 months ago

Ovidiu, I did another review of the new standard.df and the _trans meta table has new mandatory fields - did you test this part?

#28 Updated by Ovidiu Maxiniuc 5 months ago

Not explicitly.
But this should not be a problem. When the DMO is initialised, the mandatory fields are automatically populated, as described in DMO annotations. We do not process these fields the same way for _connect tables.

At any rate, I am going to redo the tests, and put the accent on this specific table.

#29 Updated by Roger Borrello 5 months ago

I just wanted to post a method by which an application can be backward compatible for building. Some projects have multiple custom branches, and it could get messy trying to maintain build environments.

Create a data/select_standard.sh in the configuration project. Sample attached. It will have policies governing which data/standard.df database to use. The default is trunk.

Dependency is that there are 2 versions of data/standard.df in place:
  • data/standard_pre9950.df
  • data/standard_post9950.df

The script will determine via FWD version and comparing branch/revision which to use.

Usage in the build.xml is the addition of a target which will copy the correct one in place (falling back to leaving the data/standard.df in place). Call this target from a pre-build target:

   <target name="init-standard-df" depends="init-ant-contrib" 
           description="Determine correct standard.df based on p2j build">

      <!-- Call helper script -->
      <exec executable="${data.rel}/select_standard.sh" 
            outputproperty="standard.selector.result" 
            failonerror="true">
          <arg value="${data.rel}"/>
          <arg value="${fwd.lib.home}/p2j.jar"/>
      </exec>
      <echo message="Standard selection result: ${standard.selector.result}"/>

      <!-- Determine source file -->
      <condition property="standard.source" 
                 value="${data.rel}/standard_post9950.df" 
                 else="${data.rel}/standard_pre9950.df">
          <equals arg1="${standard.selector.result}" arg2="post9950"/>
      </condition>

      <!-- Copy only if not skip -->
      <if>
         <not>
            <equals arg1="${standard.selector.result}" arg2="skip"/>
         </not>
         <then>
            <echo message="Copying ${standard.source} to ${data.rel}/standard.df"/>
            <copy file="${standard.source}" 
                  tofile="${data.rel}/standard.df" 
                  overwrite="true"/>
         </then>
         <else>
            <echo message="Skipping standard.df update (fallback condition met)"/>
         </else>
      </if>
   </target>

Also available in: Atom PDF