FWD » Core Development » Database

• DEFINE TEMP-TABLE
  • table options NAMESPACE-URI and NAMESPACE-PREFIX are marked runtime partial
  • field option COLUMN-CODEPAGE
• TEMP-TABLE handle usage
  • PRIMARY attribute (sets or gets the primary index)
  • SCHEMA-MARSHAL attribute

I remember putting some effort in NAMESPACE-URI and NAMESPACE-PREFIX implementation. They were working for my testcases. Of course, they were not extensively tested and customer code can reveal some aspects which were not foreseen. So I would like to investigate those testcases.

Some support for COLUMN-CODEPAGE exists. At least they are collected at TemporaryBuffer level (registeredCodePages). But the rest of runtime support is missing.

OTOH, the PRIMARY attribute lacks both both conversion and runtime support. The good news is that it seems easy to implement it.

The SCHEMA-MARSHAL should be fully functional. As with NAMESPACE-URI and NAMESPACE-PREFIX it is probably not tested enough.

Igor, I think PRIMARY has already been implemented in branch 6129a. I think that is your target. Anyway that branch will be merged into 3821c soon.

To avoid influencing you, I think you should try to write some very simple testcases with the above attribute/options. Be sure you use them in both read and write mode, where possible.

Ovidiu Maxiniuc wrote:

Igor, I think PRIMARY has already been implemented in branch 6129a. I think that is your target. Anyway that branch will be merged into 3821c soon.

To avoid influencing you, I think you should try to write some very simple testcases with the above attribute/options. Be sure you use them in both read and write mode, where possible.

Thank you, Ovidiu.
I will start working on this on Monday.

Ovidiu,
Which branch should I use for this task?
Thank you.

6129a

As I've mentioned in #6453-10 the issues mentioned above formally do not related to the scope of this task. Should I try to fix them?

Eric has the final say, but I think these should be cleared now.

Agreed, but adding missing functionality is highest priority; if fixing any of these items looks to be particularly time consuming, please note the reason and move it to the back of the priority queue.

Eric Faulhaber wrote:

Agreed, but adding missing functionality is highest priority; if fixing any of these items looks to be particularly time consuming, please note the reason and move it to the back of the priority queue.

Got it.
Thank you.
I will work on TEMP-TABLE handle first.
I do not expect that fixing issues from #6453-11 will be time consuming except the one with dynamic table - I have no experience with such tables and cannot say how difficultit it will be to fix the issue.

It looks that the value of the SCHEMA-MARSHAL attribute has no effect on the WRITE-XMLSCHEMA output (both for 4GL and FWD).
What it should affect?
Thank you.

Igor Skornyakov wrote:

It looks that the value of the SCHEMA-MARSHAL attribute has no effect on the WRITE-XMLSCHEMA output (both for 4GL and FWD).
What it should affect?
Thank you.

It looks that this attribute affects temp-table serialization when it is used as a parameter in a call. However, I have no idea how to test this.
Ovidiu,
can you please advise and/or provide at least a simplest code sample when it is used?
Thank you.

Ovidiu Maxiniuc wrote:

OTOH, the PRIMARY attribute lacks both both conversion and runtime support. The good news is that it seems easy to implement it.

As far as I can see, the PRIMARY attribute is fully supported and this support passes simple compatibility tests (assignment on the PREPARED state fails properly, and works fine when table is not prepared).

Igor Skornyakov wrote:

Ovidiu Maxiniuc wrote:

OTOH, the PRIMARY attribute lacks both both conversion and runtime support. The good news is that it seems easy to implement it.

As far as I can see, the PRIMARY attribute is fully supported and this support passes simple compatibility tests (assignment in the PREPARED state fails properly, and works fine when table is not prepared).

1. The initial value of th PRIMARY attribute is the first defined index in 4GL but UNKNOWN in FWD.
2. The is a typo in the 3131 message (assignment in the PREPARED state).

Igor Skornyakov wrote:

Actually there are minor incompatibilities:

1. The initial value of th PRIMARY attribute is the first defined index in 4GL but UNKNOWN in FWD.
2. The is a typo in the 3131 message (assignment in the PREPARED state).

Fixed in 6129a/14418.

Igor Skornyakov wrote:

It looks that the value of the SCHEMA-MARSHAL attribute has no effect on the WRITE-XMLSCHEMA output (both for 4GL and FWD).
What it should affect?

This attribute is used internally, when a table is passed as a parameter, as opposed to writing to an external destination (like a file). So unrelated to any JSON/XML serialization.

It looks that this attribute affects temp-table serialization when it is used as a parameter in a call. However, I have no idea how to test this.

Exactly.

can you please advise and/or provide at least a simplest code sample when it is used?

I implemented this using the reference manual and tested on real customer application. The source/target should be in different processes, as I understand. Constantin is best suited to provide you this code which is related to appservers and REST.

Ovidiu Maxiniuc wrote:

I implemented this using the reference manual and tested on real customer application. The source/target should be in different processes, as I understand. Constantin is best suited to provide you this code which is related to appservers and REST.

Thank you, Ovidiu.
Indeed as I can see from the code it affects only remote calls.

Constantin - can you please provide a sample code or at least a clue how it should look like?
Thank you.

It is easy to fix FWD to add index data to the WRITE-XMLSCHEMA output. All we need is no comment !this.isSchemaOnly check in the XmlExport.writeTableSchema(BufferImpl, TempTableSchema, boolean, boolean) method (line 875). In fact I do not understand the reason this check was added.

However it looks that that the logic in the XmlImport.readTableSchema(String xmlNodeName, String nsPrefix, String nsUri) method expects the XML schema structure which is very different from what is created by XmlExport. In particular it just fails if a schema contains index data.
At this moment I'm implementing another version of this method alongside with the existing one.
Don't you think that instead of manual implementtation it makes sense to use some open-source code, such as https://github.com/xmlet/XsdParser (MIT license)?
Thank you.

Igor Skornyakov wrote:

Don't you think that instead of manual implementtation it makes sense to use some open-source code, such as https://github.com/xmlet/XsdParser (MIT license)?

Another (better?) alternative is Apache XmlSchema (Apache License 2.0). It looks actively maintained,

We are very careful about introducing new dependencies into the project and in fact we are striving to simplify by reducing non-critical dependencies. The key requirements for any functionality in FWD are:

• correct/compatible behavior;
• performance;
• simplicity;
• minimum development effort (including long term maintainability).

Do either of the projects suggested offer improvements in those areas compared to something we implement (or already have implemented) ourselves?

Eric Faulhaber wrote:

We are very careful about introducing new dependencies into the project and in fact we are striving to simplify by reducing non-critical dependencies. The key requirements for any functionality in FWD are:

• correct/compatible behavior;
• performance;
• simplicity;
• minimum development effort (including long term maintainability).

Do either of the projects suggested offer improvements in those areas compared to something we implement (or already have implemented) ourselves?

• for this particular piece of functionality performance is not an issue at all since I do not think that this functionality is heavily used in any real life application
• XML Schema is a very standard thing but it is pretty complicated. The hand-made processing will either need to implement multiple corner cases or use "shortcuts" (as we do in XmlImport ignoring e.g. complexType, sequence, or annotation ) which makes the logic fragile.
• using standard libraries for standard things obviously makes the code more simple since it has to implement only application-specific logic and don't care about complexities of the underlying standard layer.
• the development efforts will also be less. Most Apache projects are very reliable in terms of maintainability. For (READ|WRITE)-XML(Schema)? methods which use a limited subset of XML Schema standard the maintanability is very unliky will ever become a problem.

Igor Skornyakov wrote:

I undestand this. My points is

• for this particular piece of functionality performance is not an issue at all since I do not think that this functionality is heavily used in any real life application
• XML Schema is a very standard thing but it is pretty complicated. The hand-made processing will either need to implement multiple corner cases or use "shortcuts" (as we do in XmlImport ignoring e.g. complexType, sequence, or annotation ) which makes the logic fragile.
• using standard libraries for standard things obviously makes the code more simple since it has to implement only application-specific logic and don't care about complexities of the underlying standard layer.
• the development efforts will also be less. Most Apache projects are very reliable in terms of maintainability. For (READ|WRITE)-XML(Schema)? methods which use a limited subset of XML Schema standard the maintanability is very unliky will ever become a problem.

I agree with most of your points, except the performance assumption in your first point (and implicit in your third point). We can never make this assumption. We are implementing this feature set precisely because we found it in use in a real application, which is extremely performance sensitive. Whether this specific feature is in particularly heavy use, I don't know for sure, but we cannot afford to be casual about performance decisions.

That being said, I don't know if performance of these libraries is better or worse than what we would create. However, it is almost certain that a third party solution will be designed to provide a broader set of features than we typically need for a particular FWD feature. When we implement it ourselves, we can focus on just what we need (and expect to need) and nothing more. This lends itself to a simpler implementation, though I do take your point about reliability of standard solutions.

Ovidiu, as the person who has done most of the work in this area so far, please provide your thoughts. I have not looked at either library, so other than a general desire to avoid adding dependencies to the project unless they are well worth it, I honestly don't have a strong preference either way.

Eric Faulhaber wrote:

I agree with most of your points, except the performance assumption in your first point (and implicit in your third point). We can never make this assumption. We are implementing this feature set precisely because we found it in use in a real application, which is extremely performance sensitive. Whether this specific feature is in particularly heavy use, I don't know for sure, but we cannot afford to be casual about performance decisions.

My note about performance is based on the assumption that it doesn't make sense to created a lot of dynamic temp-tables using READ_XMLSCHEMA. This assumption can be wrong of course. In any case it is difficult to reason about performance without real perfomance tests. The assumptions based on common sense can be wrong.

Ovidiu - what do you think?
Thank you.

Also note that sometimes it is harder to meet specific 4GL compatibility requirements using 3rd party libraries that were written to be general purpose but which have their own implementation choices. The 4GL can require weird behavior that was never anticipated in a general purpose library.

Greg Shah wrote:

Also note that sometimes it is harder to meet specific 4GL compatibility requirements using 3rd party libraries that were written to be general purpose but which have their own implementation choices. The 4GL can require weird behavior that was never anticipated in a general purpose library.

I understand this. But in case of XML schema this not likely. Moreoever right now FWD cannot read a valid schema created by FWD.

As a first observation, I agree with Eric's note #6453-24: FWD is quite a large project, it has 200+ dependencies (jar files). Remember that they need to be loaded in memory, even if only a fraction of each one's code is used.

I do not expect the performance to bottle-neck when reading/writing XML/JSON. Usually there is one or maybe several files read/exported with user interaction. Unless the application is built with this exact goal in mind. We can do some tests and time them to get a feeling of the expected overhead compared to direct-generation we use at this moment. Also there will surely be an additional memory cost - especially for large table/datasets, but this will be released once the operation ends.

The big problem I see is binary compatibility of the output files. I ran tens may hundreds of tests with various parameters passed to serialization methods to get the closer output to that of 4GL. I am positive that there are cases which I could not cover or I did not thought of so the output might not be exactly the same. Using a 3rd party library to generate to files for us will only make this worse. Although they may grant the correct structure of the XML (which we must handle in the background), the other aspects might not be easily configurable.

Ovidiu Maxiniuc wrote:

As a first observation, I agree with Eric's note #6453-24: FWD is quite a large project, it has 200+ dependencies (jar files). Remember that they need to be loaded in memory, even if only a fraction of each one's code is used.

I do not expect the performance to bottle-neck when reading/writing XML/JSON. Usually there is one or maybe several files read/exported with user interaction. Unless the application is built with this exact goal in mind. We can do some tests and time them to get a feeling of the expected overhead compared to direct-generation we use at this moment. Also there will surely be an additional memory cost - especially for large table/datasets, but this will be released once the operation ends.

The big problem I see is binary compatibility of the output files. I ran tens may hundreds of tests with various parameters passed to serialization methods to get the closer output to that of 4GL. I am positive that there are cases which I could not cover or I did not thought of so the output might not be exactly the same. Using a 3rd party library to generate to files for us will only make this worse. Although they may grant the correct structure of the XML (which we must handle in the background), the other aspects might not be easily configurable.

Thank you, Ovidiu.
I will try to fix the existing code.

Igor/Eric/Ovidiu: we use the WRITE-XML and READ-XML behavior to serialize the payload both for read and write, for XML table or dataset arguments. As SOAP can be considered 'micro-services', performance for both of these is important.

Constantin Asofiei wrote:

Igor/Eric/Ovidiu: we use the WRITE-XML and READ-XML behavior to serialize the payload both for read and write, for XML table or dataset arguments. As SOAP can be considered 'micro-services', performance for both of these is important.

Thank you Constantin,
So far I see no problems with WRITE-XML/READ-XML. I see a number of issues with WRITE-XMLSCHEMA/READ-XMLSCHEMA. Are they used even if not called directly by an application?
Thank you.

I see multiple incompatibilities between FWD and 4GL in the WRITE-XMLSCHEMA output.
In particular some element attributes are not privided in FWD. It looks that these happens because XmlExport.isSchemaOnly is set to true in XmlExport.writeTableSchema.
The logic behind this described in the isSchemaOnly field Javadoc doesn't look correct.
Have I missed something?
Thank you.

Igor, regarding the change in 6129a/14418 - are you sure PRIMARY defaults to the first defined index, and not the first defined UNIQUE index (if it exists) and only after that the first non-unique index?

Constantin Asofiei wrote:

Igor, regarding the change in 6129a/14418 - are you sure PRIMARY defaults to the first defined index, and not the first defined UNIQUE index (if it exists) and only after that the first non-unique index?

This is a good question, thank you. I was thinking about this but forgot to check.
In my test I've not defined any UNIQUE index. I will double check

Igor,
Please do the test in all cases: the table was prepared or not, the index was set as primary or not. I do not expect that the uniqueness will make any difference, but you can test that, too.

IIRC, when the table is not prepared, the reference manual was wrong, in the sense that the attribute returns ? only if no index was defined as primary. When there is such index, it is correctly returned.

The implicitly primary index is computed only after the table was prepared. This is decided by the first if conditional in the attribute implementation. If the old implementation was wrong was because it did not stop after the first declared primary index was found. You can define multiple primary indexes in a non-prepared temp-table, but this attribute will return the first. You can always overwrite the primary attribute if the table is not prepared, but this is no more possible after that.

I've noticed a strange thing.
For some field data types (e.g. datetime, recid, handle and com-hangle) in the static temp table the Property annotation of the corresponding field in the generated DMO interface has defined format property even if the field format was not specified. For other types this is not the fact.
Was it done on purpose?
Thank you.

Ovidiu Maxiniuc wrote:

Igor,
Please do the test in all cases: the table was prepared or not, the index was set as primary or not. I do not expect that the uniqueness will make any difference, but you can test that, too.

IIRC, when the table is not prepared, the reference manual was wrong, in the sense that the attribute returns ? only if no index was defined as primary. When there is such index, it is correctly returned.

The implicitly primary index is computed only after the table was prepared. This is decided by the first if conditional in the attribute implementation. If the old implementation was wrong was because it did not stop after the first declared primary index was found. You can define multiple primary indexes in a non-prepared temp-table, but this attribute will return the first. You can always overwrite the primary attribute if the table is not prepared, but this is no more possible after that.

Thank you, Ovidiu.
I will try to perform extensive testing.

The codePage attribute of the Property annotation is not populated in the generated DMO interface for the static temporary table even if specified.

Just noticed an issue in ErrorManager 6129a/14418.
You fixed error message for error 3131 wrongly. There are tens of the places where the 3131 is provided with parameters in the old order. Please change it back then put the parameters in the expected order in AbstractTempTable.primaryIndex(String primary). It should be:

instead of

Thank you!

Ovidiu Maxiniuc wrote:

Just noticed an issue in ErrorManager 6129a/14418.
You fixed error message for error 3131 wrongly. There are tens of the places where the 3131 is provided with parameters in the old order. Please change it back then put the parameters in the expected order in AbstractTempTable.primaryIndex(String primary). It should be:

ErrorManager.recordOrThrowError(3131, "PRIMARY", "", LegacyResource.TEMP_TABLE + " widget");

instead of

ErrorManager.recordOrThrowError(3131, "PRIMARY", LegacyResource.TEMP_TABLE + " widget", "");

Thank you!

I see. Sorry. Will change back.
Please note however, that in my test the previous version generated message different from one that I see with 4GL.

Igor Skornyakov wrote:

I see. Sorry. Will change back.
Please note however, that in my test the previous version generated message different from one that I see with 4GL.

How was your message different? Do you have a testcase?

Ovidiu Maxiniuc wrote:

Igor Skornyakov wrote:

I see. Sorry. Will change back.
Please note however, that in my test the previous version generated message different from one that I see with 4GL.

How was your message different? Do you have a testcase?

Of course I have a test. Otherwise I had no reasons to make changes. At this moment I do not remember exactly what was the difference. I will provide this information after the changes will be reverted. I will do it later today. I want to finish with WRITE-XMLSCHEMA first.

Constantin Asofiei wrote:

Igor, regarding the change in 6129a/14418 - are you sure PRIMARY defaults to the first defined index, and not the first defined UNIQUE index (if it exists) and only after that the first non-unique index?

Constantin.
Accoring to my test 4GL does not make difference between UNIQUE and 'normal' indexes when selecting default value of the PRIMARY attribute. Only the index order is considered when no index is defined as primary one.

• Fixed issues with WRITE-XMLSCHEMA/READ-XMLSCHEMA described in #6453-11 except the issue with WRITE-XMLSCHEMA for dynamic temp-tables.
• Reverted the change in error 3131 message template in ErrorManager. The only change is a single call in the AbstractTempTable.

Committed to 6129a/14473.

Please note that at this moment the FWD output of WRITE-XMLSCHEMA is not exactly the same as with 4GL. This is because of issue with static TEMP-TABLE conversion described in #6453-38 and #6453-40.
Should I try to fix them now?
Thank you.

• AbstractTempTable: shouldn't the last parameter of the 3131 error message be LegacyResource.TEMP_TABLE + " widget"? Just asking because of the similar statement several lines above. I might be wrong, I did not do any tests;
• TableMapper: I have not studied the new signatures (which use the new TempTable parameter) for getters and setters of the field attributes, but I think the newly added ones should have a similar signature. The getXmlDataType() in particular: this value could be obtained using getSerializeOptions(tt).getXmlDataType();
• XmlExport: missing javadoc of the new method writeColumnAttrs().

I think you should add the missing attributes described in #6453-38 and #6453-40.

Ovidiu Maxiniuc wrote:

Review of 6129a/14473.

• AbstractTempTable: shouldn't the last parameter of the 3131 error message be LegacyResource.TEMP_TABLE + " widget"? Just asking because of the similar statement several lines above. I might be wrong, I did not do any tests;

I think you're wrong. I have the test and the previous version produced a messsage different from one in 4GL. I haven't tested other use cases.

• TableMapper: I have not studied the new signatures (which use the new TempTable parameter) for getters and setters of the field attributes, but I think the newly added ones should have a similar signature. The getXmlDataType() in particular: this value could be obtained using getSerializeOptions(tt).getXmlDataType();

I was thinking about this, but at the poisnnt were I need to XML-DATA_TYPE value I have to access to the corresponding TEMP-TABLE.

• XmlExport: missing javadoc of the new method writeColumnAttrs().

Thank you. Will be fixed.

I think you should add the missing attributes described in #6453-38 and #6453-40.

Thank you. Working on this.
Please note however that #6453-38 is not about missing parameter but about an extra one. My question was about the reason for adding default format value for some data types.
Thank you.

Ovidiu Maxiniuc wrote:

• TableMapper: I have not studied the new signatures (which use the new TempTable parameter) for getters and setters of the field attributes, but I think the newly added ones should have a similar signature. The getXmlDataType() in particular: this value could be obtained using getSerializeOptions(tt).getXmlDataType();
• XmlExport: missing javadoc of the new method writeColumnAttrs().

Fixed in 6129a/14474.

Igor Skornyakov wrote:

Ovidiu Maxiniuc wrote:

• TableMapper: I have not studied the new signatures (which use the new TempTable parameter) for getters and setters of the field attributes, but I think the newly added ones should have a similar signature. The getXmlDataType() in particular: this value could be obtained using getSerializeOptions(tt).getXmlDataType();
• XmlExport: missing javadoc of the new method writeColumnAttrs().

Fixed in 6129a/14474.

Igor, you still have lfi.getSerializeOptions(null).getXmlDataType(); in getXmlDataType(). Any mutable attribute must pass the TempTable instance.

Constantin Asofiei wrote:

Fixed in 6129a/14474.

Igor, you still have lfi.getSerializeOptions(null).getXmlDataType(); in getXmlDataType(). Any mutable attribute must pass the TempTable instance.

Fixed in 6129a/14475.

I see that KW_COL_CP AST (COLUMN-CODEPAGE) exists in .schema file but is missed in .p2o one.
What is the right place to fix this?
Thank you.

I think rules/schema/fixups.xml. See the other attributes are processed around lines 690.

Ovidiu Maxiniuc wrote:

I think rules/schema/fixups.xml. See the other attributes are processed around lines 690.

Thank you!

Added COLUMN-CODEPAGE field attribute support (conversion and WRITE-XMLSCHEMA)

Committed to 6129a/14482.

Igor Skornyakov wrote:

I've noticed a strange thing.
For some field data types (e.g. datetime, recid, handle and com-hangle) in the static temp table the Property annotation of the corresponding field in the generated DMO interface has defined format property even if the field format was not specified. For other types this is not the fact.
Was it done on purpose?
Thank you.

The root cause of this is that default formats specified for the datetime, recid, handle, and com-hangle datatapes in the defFormats map initiated in java_dmo.xml and used in dmo_common.rules are different from ones returned by the defaultFormatString() method for the corresponding BDTs.
It is possible of course to make them identical but in any case with cannot reliably check if the FORMAT attribute was explicitly specified for the TEMP-TABLE field if this format is the same as a default one for this data type.
Since it seems very unlikely that the default format added to schema (even if it was not explicitely specified for the table) can cause any problems, I suggest to leave it as-is now.
What do you think?
Thank you.

Ovidiu Maxiniuc wrote:

Igor Skornyakov wrote:

It looks that the value of the SCHEMA-MARSHAL attribute has no effect on the WRITE-XMLSCHEMA output (both for 4GL and FWD).
What it should affect?

This attribute is used internally, when a table is passed as a parameter, as opposed to writing to an external destination (like a file). So unrelated to any JSON/XML serialization.

It looks that this attribute affects temp-table serialization when it is used as a parameter in a call. However, I have no idea how to test this.

Exactly.

can you please advise and/or provide at least a simplest code sample when it is used?

I implemented this using the reference manual and tested on real customer application. The source/target should be in different processes, as I understand. Constantin is best suited to provide you this code which is related to appservers and REST.

Constantin/Marian: Can you please advise Igor on how he can test the behavior of SCHEMA-MARSHAL?

Igor Skornyakov wrote:

Igor Skornyakov wrote:

I've noticed a strange thing.
For some field data types (e.g. datetime, recid, handle and com-hangle) in the static temp table the Property annotation of the corresponding field in the generated DMO interface has defined format property even if the field format was not specified. For other types this is not the fact.
Was it done on purpose?
Thank you.

The root cause of this is that default formats specified for the datetime, recid, handle, and com-hangle datatapes in the defFormats map initiated in java_dmo.xml and used in dmo_common.rules are different from ones returned by the defaultFormatString() method for the corresponding BDTs.
It is possible of course to make them identical but in any case with cannot reliably check if the FORMAT attribute was explicitly specified for the TEMP-TABLE field if this format is the same as a default one for this data type.
Since it seems very unlikely that the default format added to schema (even if it was not explicitely specified for the table) can cause any problems, I suggest to leave it as-is now.
What do you think?
Thank you.

In fact 4GL ignores FORMAT value in the WRITE-XMLSCHEMA even if it is explicitely specified but is the same as a default one. So, if we want to have exactly the same output in FWD we need to make defaultFormatString() method result for BDTs the same as specified in defFormats.
I assume the correct values are in the java_dmo.xml. At lest they are the same I can see in the 4GL documentation.
Should I fix defaultFormatString() methods to match defFormats?
Thank you.

It looks that the problem with WRITE-XMLSCHEMA for dynamic TEMP-TABLE in FWD is that for all fields of such table serializeHidden is true.
Is this intentional?
Thank you,

Igor Skornyakov wrote:

It looks that the problem with WRITE-XMLSCHEMA for dynamic TEMP-TABLE in FWD is that for all fields of such table serializeHidden is true.
Is this intentional?

I don't see why it would be, definitively not like that in 4GL.

Marian Edu wrote:

Igor Skornyakov wrote:

It looks that the problem with WRITE-XMLSCHEMA for dynamic TEMP-TABLE in FWD is that for all fields of such table serializeHidden is true.
Is this intentional?

I don't see why it would be, definitively not like that in 4GL.

Got it, thank you.
Working on the fix.

Greg Shah wrote:

Constantin/Marian: Can you please advise Igor on how he can test the behavior of SCHEMA-MARSHAL?

This is basically used when passing temp-table (separate or part of a dataset) from one side (usually the client) where the other side receive it as handle (table or dataset) so it doesn't know it's schema. By default SCHEMA-MARSHAL is set to FULL so when passed along the schema is also included next to the data, for MIN some lees important table/field/index properties are not serialised but the data structure is there so the other end can always work with it... the only real difference is when set to NONE so no schema information is being send. This is fine if the other end has the temp-table/dataset definition - aka parameter is defined as table/dataset for - but it will throw if schema is unknown there - aka parameter is table/dataset handle, the error being thrown there is 12323.

This can be tested using a `client` connection to the appssrv (using `connect`) and set the SCHEMA-MARSHAL property of temp-table before the call, or using REST send only the data payload to an end-point that expects a table handle. For that the following procedure from testcases project can be used: appsrv/api/table/pipe_table_handle.p. For REST you will see in SoapUI project appsrv/test/web/pasoe-fwd-soapui-project.xml when the pipe_table_handle endpoint is being called the TempTable also has the xsd:schema attached.

Marian Edu wrote:

Greg Shah wrote:

Constantin/Marian: Can you please advise Igor on how he can test the behavior of SCHEMA-MARSHAL?

This is basically used when passing temp-table (separate or part of a dataset) from one side (usually the client) where the other side receive it as handle (table or dataset) so it doesn't know it's schema. By default SCHEMA-MARSHAL is set to FULL so when passed along the schema is also included next to the data, for MIN some lees important table/field/index properties are not serialised but the data structure is there so the other end can always work with it... the only real difference is when set to NONE so no schema information is being send. This is fine if the other end has the temp-table/dataset definition - aka parameter is defined as table/dataset for - but it will throw if schema is unknown there - aka parameter is table/dataset handle, the error being thrown there is 12323.

This can be tested using a `client` connection to the appssrv (using `connect`) and set the SCHEMA-MARSHAL property of temp-table before the call, or using REST send only the data payload to an end-point that expects a table handle. For that the following procedure from testcases project can be used: appsrv/api/table/pipe_table_handle.p. For REST you will see in SoapUI project appsrv/test/web/pasoe-fwd-soapui-project.xml when the pipe_table_handle endpoint is being called the TempTable also has the xsd:schema attached.

I see. Thanks a lot!

Should I fix defaultFormatString() methods to match defFormats?

Probably not. I think that may cause regressions.

For handle, comhandle, I understand that the deviations are related to the fact that we store a 64-bit (long) resource ID. I think we require the difference to avoid string conversion failures.

Hynek: I see this change in the AGPL commit of trunk rev 11146, however I cannot explain where it came from. I did the check in, but I know you prepared this change (at least the scripts used). Perhaps I included some pending changes by mistake. Do you have any idea where the changes came from?

For recid, the recide.defFmt does match the one used in java_dmo.xml. Likewise datetime.defaultFormatString() seems to have the correct value. Did I misread something?

Greg Shah wrote:

Should I fix defaultFormatString() methods to match defFormats?

Probably not. I think that may cause regressions.

For handle, comhandle, I understand that the deviations are related to the fact that we store a 64-bit (long) resource ID. I think we require the difference to avoid string conversion failures.

Hynek: I see this change in the AGPL commit of trunk rev 11146, however I cannot explain where it came from. I did the check in, but I know you prepared this change (at least the scripts used). Perhaps I included some pending changes by mistake. Do you have any idea where the changes came from?

For recid, the recide.defFmt does match the one used in java_dmo.xml. Likewise datetime.defaultFormatString() seems to have the correct value. Did I misread something?

For datetime the difference is only in case.

For datetime the difference is only in case.

Please analyze if this case difference matters in our runtime. If not, then it can be changed.

Igor Skornyakov wrote:

It looks that the problem with WRITE-XMLSCHEMA for dynamic TEMP-TABLE in FWD is that for all fields of such table serializeHidden is true.

It looks that the problem is in DMO generation for dynamic temp-table in DynamicTablesHelper.createDynamicDMO.
The root = (Aast) results.getStoredObject("p2o.prog") is

_temp [DATA_MODEL]:4294967297 @0:0
      (peerid=1, package=com.goldencode.p2j.persist.dynamic, path=com/goldencode/p2j/persist/dynamic/, permanent=false, srcfile=null, historical=_temp)
   DynamicRecord1 [CLASS]:4294967298 @0:0
         (peerid=2, historical=dtt, table=dtt1, hasdenormalizedfields=false)
      recid [PRIMARY]:4294967299 @0:0
            (column=recid, datatype=Long)
      field1 [PROPERTY]:4294967300 @0:0
            (peerid=3, column=field1, historical=achar, datatype=character, label=char-attr, format=XXXX, serialize-hidden=true, serialize-name=char, xml-node-name=pk, xml-node-type=attribute, order=10, fieldid=1, propertyid=1)
         initial [INITIAL]:4294967301 @0:0
            99aa [STRING]:4294967302 @0:0
      field2 [PROPERTY]:4294967303 @0:0
            (peerid=6, column=field2, historical=aint, datatype=integer, label=int-attr, format=99999, serialize-hidden=true, xml-data-type=long, xml-node-type=attribute, order=20, fieldid=2, propertyid=2)
         initial [INITIAL]:4294967304 @0:0
            1 [NUM_LITERAL]:4294967305 @0:0
      field3 [PROPERTY]:4294967306 @0:0
            (peerid=9, column=field3, historical=fchar, datatype=character, label=char, format=XXXX, serialize-hidden=true, serialize-name=char-field, xml-node-name=char-node, xml-node-type=ELEMENT, order=30, fieldid=3, propertyid=3)
         initial [INITIAL]:4294967307 @0:0
            99aa [STRING]:4294967308 @0:0
      field4 [PROPERTY]:4294967309 @0:0
            (peerid=12, column=field4, historical=fcharcs, datatype=character, label=char-cs, format=XXXX, case-sensitive=true, serialize-hidden=true, xml-node-type=ELEMENT, order=40, fieldid=4, propertyid=4)
         initial [INITIAL]:4294967310 @0:0
            aa88 [STRING]:4294967311 @0:0
      field5 [PROPERTY]:4294967312 @0:0
            (peerid=15, column=field5, historical=fcharext, datatype=character, label=char-ext, format=x(8), extent=8, case-sensitive=true, serialize-hidden=true, xml-node-name=extent-node, xml-node-type=ELEMENT, order=50, fieldid=5, propertyid=5, composite=4294967360)
         initial [INITIAL]:4294967313 @0:0
            99ee [STRING]:4294967314 @0:0
      field6 [PROPERTY]:4294967315 @0:0
            (peerid=18, column=field6, historical=fdecimal, datatype=decimal, label=decimal, help=help, format=->>,>>9.99, col_lab=decimal-column, scale=2, serialize-hidden=true, xml-node-type=ELEMENT, order=60, fieldid=6, propertyid=6)
         initial [INITIAL]:4294967316 @0:0
            0 [DEC_LITERAL]:4294967317 @0:0
      field7 [PROPERTY]:4294967318 @0:0
            (peerid=21, column=field7, historical=fint, datatype=integer, label=int, format=99999, serialize-hidden=true, xml-node-type=ELEMENT, order=70, fieldid=7, propertyid=7)
         initial [INITIAL]:4294967319 @0:0
            1 [NUM_LITERAL]:4294967320 @0:0
      field8 [PROPERTY]:4294967321 @0:0
            (peerid=24, column=field8, historical=fint64, datatype=int64, label=int64, format=99999, serialize-hidden=true, serialize-name=int64-field, xml-node-type=ELEMENT, order=80, fieldid=8, propertyid=8)
         initial [INITIAL]:4294967322 @0:0
            4 [NUM_LITERAL]:4294967323 @0:0
      field9 [PROPERTY]:4294967324 @0:0
            (peerid=27, column=field9, historical=fbool, datatype=logical, label=bool, format=yes/no, serialize-hidden=true, serialize-name=logical, xml-node-type=ELEMENT, order=90, fieldid=9, propertyid=9)
         initial [INITIAL]:4294967325 @0:0
            true [BOOL_TRUE]:4294967326 @0:0
      field10 [PROPERTY]:4294967327 @0:0
            (peerid=30, column=field10, historical=fdate, datatype=date, label=date, format=99/99/99, serialize-hidden=true, xml-node-type=ELEMENT, order=100, fieldid=10, propertyid=10)
         initial [INITIAL]:4294967328 @0:0
            10/12/22 [DATE_LITERAL]:4294967329 @0:0
      field11 [PROPERTY]:4294967330 @0:0
            (peerid=33, column=field11, historical=fdatetime, datatype=datetime, label=datetime, format=99/99/9999 HH:MM:SS.SSS, serialize-hidden=true, xml-node-type=ELEMENT, order=110, fieldid=11, propertyid=11)
         initial [INITIAL]:4294967331 @0:0
            10/12/2022 18:32:41.447 [DATETIME_LITERAL]:4294967332 @0:0
      field12 [PROPERTY]:4294967333 @0:0
            (peerid=36, column=field12, historical=fdatetime-tz, datatype=datetimetz, label=datetime-tz, format=99/99/9999 HH:MM:SS.SSS+HH:MM, serialize-hidden=true, xml-node-type=ELEMENT, order=120, fieldid=12, propertyid=12)
         initial [INITIAL]:4294967334 @0:0
            10/12/2022 18:32:41.447+03:00 [DATETIME_TZ_LITERAL]:4294967335 @0:0
      field13 [PROPERTY]:4294967336 @0:0
            (peerid=39, column=field13, historical=fblob, datatype=blob, label=blob, format=, col_lab=blob-column, serialize-hidden=true, xml-node-type=ELEMENT, order=130, fieldid=13, propertyid=13)
         initial [INITIAL]:4294967337 @0:0
            ? [UNKNOWN]:4294967338 @0:0
      field14 [PROPERTY]:4294967339 @0:0
            (peerid=42, column=field14, historical=fclob1, datatype=clob, label=clob1, format=, serialize-hidden=true, xml-node-type=hidden, order=140, fieldid=14, propertyid=14)
         initial [INITIAL]:4294967340 @0:0
            ? [UNKNOWN]:4294967341 @0:0
      field15 [PROPERTY]:4294967342 @0:0
            (peerid=45, column=field15, historical=fclob2, datatype=clob, label=clob2, format=, serialize-hidden=true, xml-node-type=ELEMENT, order=150, fieldid=15, propertyid=15)
         initial [INITIAL]:4294967343 @0:0
            ? [UNKNOWN]:4294967344 @0:0
      field16 [PROPERTY]:4294967345 @0:0
            (peerid=48, column=field16, historical=frecid, datatype=integer, subtype=recid, label=recid, format=->,>>>,>>9, serialize-hidden=true, xml-node-type=ELEMENT, order=160, fieldid=16, propertyid=16)
         initial [INITIAL]:4294967346 @0:0
            ? [UNKNOWN]:4294967347 @0:0
      field17 [PROPERTY]:4294967348 @0:0
            (peerid=51, column=field17, historical=fhandle, datatype=handle, label=handle, format=>>>>>>9, serialize-hidden=true, xml-node-type=ELEMENT, order=170, fieldid=17, propertyid=17)
         initial [INITIAL]:4294967349 @0:0
            ? [UNKNOWN]:4294967350 @0:0
      field18 [PROPERTY]:4294967351 @0:0
            (peerid=54, column=field18, historical=fcom-handle, datatype=comhandle, label=com-handle, format=>>>>>>9, serialize-hidden=true, xml-node-type=ELEMENT, order=180, fieldid=18, propertyid=18)
         initial [INITIAL]:4294967352 @0:0
            ? [UNKNOWN]:4294967353 @0:0
      field19 [PROPERTY]:4294967354 @0:0
            (peerid=57, column=field19, historical=fraw, datatype=raw, format=, serialize-hidden=true, xml-node-type=ELEMENT, order=190, fieldid=19, propertyid=19)
         initial [INITIAL]:4294967355 @0:0
             [NULL_LITERAL]:4294967356 @0:0
      field20 [PROPERTY]:4294967357 @0:0
            (peerid=58, column=field20, historical=frowid, datatype=rowid, subtype=rowid, format=, serialize-hidden=true, xml-node-type=ELEMENT, order=200, fieldid=20, propertyid=20)
         initial [INITIAL]:4294967358 @0:0
            ? [UNKNOWN]:4294967359 @0:0
      composite8 [COMPOSITE]:4294967360 @0:0
            (extent=8, table=dtt1__8, class=Composite8, property=[4294967312])
       [UNIQUE]:4294967361 @0:0
         field7 [CONSTR_COL]:4294967362 @0:0
      dtt1_idx1 [INDEX]:4294967363 @0:0
            (peerid=61, historical=idx1)
         field3 [INDEX_COL]:4294967364 @0:0
               (peerid=62, refid=4294967306, datatype=character, ignore-case=true)
      dtt1_idx2 [INDEX]:4294967365 @0:0
            (peerid=63, historical=idx2, primary=true)
         field1 [INDEX_COL]:4294967366 @0:0
               (peerid=64, refid=4294967300, datatype=character, ignore-case=true)
      dtt1_idx3 [INDEX]:4294967367 @0:0
            (peerid=65, historical=idx3, unique=true)
         field7 [INDEX_COL]:4294967368 @0:0
               (peerid=66, refid=4294967318, datatype=integer)

However Aast iface = (Aast) results.getStoredObject(DMO_IFACE_CATEGORY, ifaceName) is:

compilation unit [COMPILE_UNIT]:8589934593 @0:0
   com.goldencode.p2j.persist.dynamic._temp [KW_PACKAGE]:8589934594 @0:0
   com.goldencode.p2j.persist.annotation.* [KW_IMPORT]:8589934595 @0:0
   Table [ANNOTATION]:8589934596 @0:0
      = [ASSIGN]:8589934597 @0:0
         name [REFERENCE]:8589934598 @0:0
         dtt1 [STRING]:8589934599 @0:0
      = [ASSIGN]:8589934600 @0:0
         legacy [REFERENCE]:8589934601 @0:0
         dtt [STRING]:8589934602 @0:0
   Indices [ANNOTATION]:8589934603 @0:0
       [INITIALIZER]:8589934604 @0:0
         Index [ANNOTATION]:8589934605 @0:0
            = [ASSIGN]:8589934606 @0:0
               name [REFERENCE]:8589934607 @0:0
               dtt1_idx1 [STRING]:8589934608 @0:0
            = [ASSIGN]:8589934609 @0:0
               legacy [REFERENCE]:8589934610 @0:0
               idx1 [STRING]:8589934611 @0:0
            = [ASSIGN]:8589934612 @0:0
               components [REFERENCE]:8589934613 @0:0
                [INITIALIZER]:8589934614 @0:0
                  IndexComponent [ANNOTATION]:8589934615 @0:0
                     = [ASSIGN]:8589934616 @0:0
                        name [REFERENCE]:8589934617 @0:0
                        field3 [STRING]:8589934618 @0:0
                     = [ASSIGN]:8589934619 @0:0
                        legacy [REFERENCE]:8589934620 @0:0
                        fchar [STRING]:8589934621 @0:0
         Index [ANNOTATION]:8589934622 @0:0
            = [ASSIGN]:8589934623 @0:0
               name [REFERENCE]:8589934624 @0:0
               dtt1_idx2 [STRING]:8589934625 @0:0
            = [ASSIGN]:8589934626 @0:0
               legacy [REFERENCE]:8589934627 @0:0
               idx2 [STRING]:8589934628 @0:0
            = [ASSIGN]:8589934629 @0:0
               primary [REFERENCE]:8589934630 @0:0
               true [BOOL_TRUE]:8589934631 @0:0
            = [ASSIGN]:8589934632 @0:0
               components [REFERENCE]:8589934633 @0:0
                [INITIALIZER]:8589934634 @0:0
                  IndexComponent [ANNOTATION]:8589934635 @0:0
                     = [ASSIGN]:8589934636 @0:0
                        name [REFERENCE]:8589934637 @0:0
                        field1 [STRING]:8589934638 @0:0
                     = [ASSIGN]:8589934639 @0:0
                        legacy [REFERENCE]:8589934640 @0:0
                        achar [STRING]:8589934641 @0:0
         Index [ANNOTATION]:8589934642 @0:0
            = [ASSIGN]:8589934643 @0:0
               name [REFERENCE]:8589934644 @0:0
               dtt1_idx3 [STRING]:8589934645 @0:0
            = [ASSIGN]:8589934646 @0:0
               legacy [REFERENCE]:8589934647 @0:0
               idx3 [STRING]:8589934648 @0:0
            = [ASSIGN]:8589934649 @0:0
               unique [REFERENCE]:8589934650 @0:0
               true [BOOL_TRUE]:8589934651 @0:0
            = [ASSIGN]:8589934652 @0:0
               components [REFERENCE]:8589934653 @0:0
                [INITIALIZER]:8589934654 @0:0
                  IndexComponent [ANNOTATION]:8589934655 @0:0
                     = [ASSIGN]:8589934656 @0:0
                        name [REFERENCE]:8589934657 @0:0
                        field7 [STRING]:8589934658 @0:0
                     = [ASSIGN]:8589934659 @0:0
                        legacy [REFERENCE]:8589934660 @0:0
                        fint [STRING]:8589934661 @0:0
   DynamicRecord1 [KW_INTERFACE]:8589934662 @0:0
         (access=public, pkgname=com.goldencode.p2j.persist.dynamic._temp, extends=Temporary)
       [CS_INSTANCE_METHODS]:8589934663 @0:0
         getField1 [METHOD_DECL]:8589934667 @0:0
               (access=public, rettype=com.goldencode.p2j.util.character)
         Property [ANNOTATION]:8589934668 @0:0
               (getter-id=8589934667)
            = [ASSIGN]:8589934669 @0:0
               id [REFERENCE]:8589934670 @0:0
               1 [NUM_LITERAL]:8589934671 @0:0
            = [ASSIGN]:8589934672 @0:0
               name [REFERENCE]:8589934673 @0:0
               field1 [STRING]:8589934674 @0:0
            = [ASSIGN]:8589934675 @0:0
               column [REFERENCE]:8589934676 @0:0
               field1 [STRING]:8589934677 @0:0
            = [ASSIGN]:8589934678 @0:0
               legacy [REFERENCE]:8589934679 @0:0
               achar [STRING]:8589934680 @0:0
            = [ASSIGN]:8589934681 @0:0
               format [REFERENCE]:8589934682 @0:0
               XXXX [STRING]:8589934683 @0:0
            = [ASSIGN]:8589934684 @0:0
               initial [REFERENCE]:8589934685 @0:0
               99aa [STRING]:8589934686 @0:0
            = [ASSIGN]:8589934687 @0:0
               label [REFERENCE]:8589934688 @0:0
               char-attr [STRING]:8589934689 @0:0
            = [ASSIGN]:8589934690 @0:0
               order [REFERENCE]:8589934691 @0:0
               10 [NUM_LITERAL]:8589934692 @0:0
            = [ASSIGN]:8589934693 @0:0
               serializeHidden [REFERENCE]:8589934694 @0:0
               true [BOOL_TRUE]:8589934695 @0:0
            = [ASSIGN]:8589934696 @0:0
               serializeName [REFERENCE]:8589934697 @0:0
               char [STRING]:8589934698 @0:0
            = [ASSIGN]:8589934699 @0:0
               xmlNodeName [REFERENCE]:8589934700 @0:0
               pk [STRING]:8589934701 @0:0
            = [ASSIGN]:8589934702 @0:0
               xmlNodeType [REFERENCE]:8589934703 @0:0
               attribute [STRING]:8589934704 @0:0
         setField1 [METHOD_DECL]:8589934705 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589934706 @0:0
               com.goldencode.p2j.util.Text [REFERENCE_DEF]:8589934707 @0:0
                     (name=field1)
         getField2 [METHOD_DECL]:8589934708 @0:0
               (access=public, rettype=com.goldencode.p2j.util.integer)
         Property [ANNOTATION]:8589934709 @0:0
               (getter-id=8589934708)
            = [ASSIGN]:8589934710 @0:0
               id [REFERENCE]:8589934711 @0:0
               2 [NUM_LITERAL]:8589934712 @0:0
            = [ASSIGN]:8589934713 @0:0
               name [REFERENCE]:8589934714 @0:0
               field2 [STRING]:8589934715 @0:0
            = [ASSIGN]:8589934716 @0:0
               column [REFERENCE]:8589934717 @0:0
               field2 [STRING]:8589934718 @0:0
            = [ASSIGN]:8589934719 @0:0
               legacy [REFERENCE]:8589934720 @0:0
               aint [STRING]:8589934721 @0:0
            = [ASSIGN]:8589934722 @0:0
               format [REFERENCE]:8589934723 @0:0
               99999 [STRING]:8589934724 @0:0
            = [ASSIGN]:8589934725 @0:0
               initial [REFERENCE]:8589934726 @0:0
               1 [STRING]:8589934727 @0:0
            = [ASSIGN]:8589934728 @0:0
               label [REFERENCE]:8589934729 @0:0
               int-attr [STRING]:8589934730 @0:0
            = [ASSIGN]:8589934731 @0:0
               order [REFERENCE]:8589934732 @0:0
               20 [NUM_LITERAL]:8589934733 @0:0
            = [ASSIGN]:8589934734 @0:0
               serializeHidden [REFERENCE]:8589934735 @0:0
               true [BOOL_TRUE]:8589934736 @0:0
            = [ASSIGN]:8589934737 @0:0
               xmlDataType [REFERENCE]:8589934738 @0:0
               long [STRING]:8589934739 @0:0
            = [ASSIGN]:8589934740 @0:0
               xmlNodeType [REFERENCE]:8589934741 @0:0
               attribute [STRING]:8589934742 @0:0
         setField2 [METHOD_DECL]:8589934743 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589934744 @0:0
               com.goldencode.p2j.util.NumberType [REFERENCE_DEF]:8589934745 @0:0
                     (name=field2)
         getField3 [METHOD_DECL]:8589934746 @0:0
               (access=public, rettype=com.goldencode.p2j.util.character)
         Property [ANNOTATION]:8589934747 @0:0
               (getter-id=8589934746)
            = [ASSIGN]:8589934748 @0:0
               id [REFERENCE]:8589934749 @0:0
               3 [NUM_LITERAL]:8589934750 @0:0
            = [ASSIGN]:8589934751 @0:0
               name [REFERENCE]:8589934752 @0:0
               field3 [STRING]:8589934753 @0:0
            = [ASSIGN]:8589934754 @0:0
               column [REFERENCE]:8589934755 @0:0
               field3 [STRING]:8589934756 @0:0
            = [ASSIGN]:8589934757 @0:0
               legacy [REFERENCE]:8589934758 @0:0
               fchar [STRING]:8589934759 @0:0
            = [ASSIGN]:8589934760 @0:0
               format [REFERENCE]:8589934761 @0:0
               XXXX [STRING]:8589934762 @0:0
            = [ASSIGN]:8589934763 @0:0
               initial [REFERENCE]:8589934764 @0:0
               99aa [STRING]:8589934765 @0:0
            = [ASSIGN]:8589934766 @0:0
               label [REFERENCE]:8589934767 @0:0
               char [STRING]:8589934768 @0:0
            = [ASSIGN]:8589934769 @0:0
               order [REFERENCE]:8589934770 @0:0
               30 [NUM_LITERAL]:8589934771 @0:0
            = [ASSIGN]:8589934772 @0:0
               serializeHidden [REFERENCE]:8589934773 @0:0
               true [BOOL_TRUE]:8589934774 @0:0
            = [ASSIGN]:8589934775 @0:0
               serializeName [REFERENCE]:8589934776 @0:0
               char-field [STRING]:8589934777 @0:0
            = [ASSIGN]:8589934778 @0:0
               xmlNodeName [REFERENCE]:8589934779 @0:0
               char-node [STRING]:8589934780 @0:0
            = [ASSIGN]:8589934781 @0:0
               xmlNodeType [REFERENCE]:8589934782 @0:0
               ELEMENT [STRING]:8589934783 @0:0
         setField3 [METHOD_DECL]:8589934784 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589934785 @0:0
               com.goldencode.p2j.util.Text [REFERENCE_DEF]:8589934786 @0:0
                     (name=field3)
         getField4 [METHOD_DECL]:8589934787 @0:0
               (access=public, rettype=com.goldencode.p2j.util.character)
         Property [ANNOTATION]:8589934788 @0:0
               (getter-id=8589934787)
            = [ASSIGN]:8589934789 @0:0
               id [REFERENCE]:8589934790 @0:0
               4 [NUM_LITERAL]:8589934791 @0:0
            = [ASSIGN]:8589934792 @0:0
               name [REFERENCE]:8589934793 @0:0
               field4 [STRING]:8589934794 @0:0
            = [ASSIGN]:8589934795 @0:0
               column [REFERENCE]:8589934796 @0:0
               field4 [STRING]:8589934797 @0:0
            = [ASSIGN]:8589934798 @0:0
               legacy [REFERENCE]:8589934799 @0:0
               fcharcs [STRING]:8589934800 @0:0
            = [ASSIGN]:8589934801 @0:0
               format [REFERENCE]:8589934802 @0:0
               XXXX [STRING]:8589934803 @0:0
            = [ASSIGN]:8589934804 @0:0
               initial [REFERENCE]:8589934805 @0:0
               aa88 [STRING]:8589934806 @0:0
            = [ASSIGN]:8589934807 @0:0
               label [REFERENCE]:8589934808 @0:0
               char-cs [STRING]:8589934809 @0:0
            = [ASSIGN]:8589934810 @0:0
               order [REFERENCE]:8589934811 @0:0
               40 [NUM_LITERAL]:8589934812 @0:0
            = [ASSIGN]:8589934813 @0:0
               caseSensitive [REFERENCE]:8589934814 @0:0
               true [BOOL_TRUE]:8589934815 @0:0
            = [ASSIGN]:8589934816 @0:0
               serializeHidden [REFERENCE]:8589934817 @0:0
               true [BOOL_TRUE]:8589934818 @0:0
            = [ASSIGN]:8589934819 @0:0
               xmlNodeType [REFERENCE]:8589934820 @0:0
               ELEMENT [STRING]:8589934821 @0:0
         setField4 [METHOD_DECL]:8589934822 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589934823 @0:0
               com.goldencode.p2j.util.Text [REFERENCE_DEF]:8589934824 @0:0
                     (name=field4)
         getField5 [METHOD_DECL]:8589934825 @0:0
               (access=public, rettype=com.goldencode.p2j.util.character[])
         getField5 [METHOD_DECL]:8589934826 @0:0
               (access=public, rettype=com.goldencode.p2j.util.character)
             [LPARENS]:8589934827 @0:0
               int [REFERENCE_DEF]:8589934828 @0:0
                     (name=index)
         Property [ANNOTATION]:8589934829 @0:0
               (getter-id=8589934826)
            = [ASSIGN]:8589934830 @0:0
               id [REFERENCE]:8589934831 @0:0
               5 [NUM_LITERAL]:8589934832 @0:0
            = [ASSIGN]:8589934833 @0:0
               name [REFERENCE]:8589934834 @0:0
               field5 [STRING]:8589934835 @0:0
            = [ASSIGN]:8589934836 @0:0
               column [REFERENCE]:8589934837 @0:0
               field5 [STRING]:8589934838 @0:0
            = [ASSIGN]:8589934839 @0:0
               legacy [REFERENCE]:8589934840 @0:0
               fcharext [STRING]:8589934841 @0:0
            = [ASSIGN]:8589934842 @0:0
               format [REFERENCE]:8589934843 @0:0
               x(8) [STRING]:8589934844 @0:0
            = [ASSIGN]:8589934845 @0:0
               initial [REFERENCE]:8589934846 @0:0
               99ee [STRING]:8589934847 @0:0
            = [ASSIGN]:8589934848 @0:0
               label [REFERENCE]:8589934849 @0:0
               char-ext [STRING]:8589934850 @0:0
            = [ASSIGN]:8589934851 @0:0
               order [REFERENCE]:8589934852 @0:0
               50 [NUM_LITERAL]:8589934853 @0:0
            = [ASSIGN]:8589934854 @0:0
               caseSensitive [REFERENCE]:8589934855 @0:0
               true [BOOL_TRUE]:8589934856 @0:0
            = [ASSIGN]:8589934857 @0:0
               serializeHidden [REFERENCE]:8589934858 @0:0
               true [BOOL_TRUE]:8589934859 @0:0
            = [ASSIGN]:8589934860 @0:0
               xmlNodeName [REFERENCE]:8589934861 @0:0
               extent-node [STRING]:8589934862 @0:0
            = [ASSIGN]:8589934863 @0:0
               xmlNodeType [REFERENCE]:8589934864 @0:0
               ELEMENT [STRING]:8589934865 @0:0
            = [ASSIGN]:8589934866 @0:0
               extent [REFERENCE]:8589934867 @0:0
               8 [NUM_LITERAL]:8589934868 @0:0
         getField5 [METHOD_DECL]:8589934869 @0:0
               (access=public, rettype=com.goldencode.p2j.util.character)
             [LPARENS]:8589934870 @0:0
               com.goldencode.p2j.util.NumberType [REFERENCE_DEF]:8589934871 @0:0
                     (name=index)
         setField5 [METHOD_DECL]:8589934872 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589934873 @0:0
               int [REFERENCE_DEF]:8589934874 @0:0
                     (name=index)
               com.goldencode.p2j.util.Text [REFERENCE_DEF]:8589934875 @0:0
                     (name=element)
         setField5 [METHOD_DECL]:8589934876 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589934877 @0:0
               com.goldencode.p2j.util.NumberType [REFERENCE_DEF]:8589934878 @0:0
                     (name=index)
               com.goldencode.p2j.util.Text [REFERENCE_DEF]:8589934879 @0:0
                     (name=element)
         setField5 [METHOD_DECL]:8589934880 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589934881 @0:0
               com.goldencode.p2j.util.Text [REFERENCE_DEF]:8589934882 @0:0
                     (name=element)
         setField5 [METHOD_DECL]:8589934883 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589934884 @0:0
               com.goldencode.p2j.util.Text[] [REFERENCE_DEF]:8589934885 @0:0
                     (name=elements)
         getField6 [METHOD_DECL]:8589934886 @0:0
               (access=public, rettype=com.goldencode.p2j.util.decimal)
         Property [ANNOTATION]:8589934887 @0:0
               (getter-id=8589934886)
            = [ASSIGN]:8589934888 @0:0
               id [REFERENCE]:8589934889 @0:0
               6 [NUM_LITERAL]:8589934890 @0:0
            = [ASSIGN]:8589934891 @0:0
               name [REFERENCE]:8589934892 @0:0
               field6 [STRING]:8589934893 @0:0
            = [ASSIGN]:8589934894 @0:0
               column [REFERENCE]:8589934895 @0:0
               field6 [STRING]:8589934896 @0:0
            = [ASSIGN]:8589934897 @0:0
               legacy [REFERENCE]:8589934898 @0:0
               fdecimal [STRING]:8589934899 @0:0
            = [ASSIGN]:8589934900 @0:0
               format [REFERENCE]:8589934901 @0:0
               ->>,>>9.99 [STRING]:8589934902 @0:0
            = [ASSIGN]:8589934903 @0:0
               initial [REFERENCE]:8589934904 @0:0
               0 [STRING]:8589934905 @0:0
            = [ASSIGN]:8589934906 @0:0
               columnLabel [REFERENCE]:8589934907 @0:0
               decimal-column [STRING]:8589934908 @0:0
            = [ASSIGN]:8589934909 @0:0
               label [REFERENCE]:8589934910 @0:0
               decimal [STRING]:8589934911 @0:0
            = [ASSIGN]:8589934912 @0:0
               help [REFERENCE]:8589934913 @0:0
               help [STRING]:8589934914 @0:0
            = [ASSIGN]:8589934915 @0:0
               order [REFERENCE]:8589934916 @0:0
               60 [NUM_LITERAL]:8589934917 @0:0
            = [ASSIGN]:8589934918 @0:0
               serializeHidden [REFERENCE]:8589934919 @0:0
               true [BOOL_TRUE]:8589934920 @0:0
            = [ASSIGN]:8589934921 @0:0
               xmlNodeType [REFERENCE]:8589934922 @0:0
               ELEMENT [STRING]:8589934923 @0:0
            = [ASSIGN]:8589934924 @0:0
               scale [REFERENCE]:8589934925 @0:0
               2 [NUM_LITERAL]:8589934926 @0:0
         setField6 [METHOD_DECL]:8589934927 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589934928 @0:0
               com.goldencode.p2j.util.NumberType [REFERENCE_DEF]:8589934929 @0:0
                     (name=field6)
         getField7 [METHOD_DECL]:8589934930 @0:0
               (access=public, rettype=com.goldencode.p2j.util.integer)
         Property [ANNOTATION]:8589934931 @0:0
               (getter-id=8589934930)
            = [ASSIGN]:8589934932 @0:0
               id [REFERENCE]:8589934933 @0:0
               7 [NUM_LITERAL]:8589934934 @0:0
            = [ASSIGN]:8589934935 @0:0
               name [REFERENCE]:8589934936 @0:0
               field7 [STRING]:8589934937 @0:0
            = [ASSIGN]:8589934938 @0:0
               column [REFERENCE]:8589934939 @0:0
               field7 [STRING]:8589934940 @0:0
            = [ASSIGN]:8589934941 @0:0
               legacy [REFERENCE]:8589934942 @0:0
               fint [STRING]:8589934943 @0:0
            = [ASSIGN]:8589934944 @0:0
               format [REFERENCE]:8589934945 @0:0
               99999 [STRING]:8589934946 @0:0
            = [ASSIGN]:8589934947 @0:0
               initial [REFERENCE]:8589934948 @0:0
               1 [STRING]:8589934949 @0:0
            = [ASSIGN]:8589934950 @0:0
               label [REFERENCE]:8589934951 @0:0
               int [STRING]:8589934952 @0:0
            = [ASSIGN]:8589934953 @0:0
               order [REFERENCE]:8589934954 @0:0
               70 [NUM_LITERAL]:8589934955 @0:0
            = [ASSIGN]:8589934956 @0:0
               serializeHidden [REFERENCE]:8589934957 @0:0
               true [BOOL_TRUE]:8589934958 @0:0
            = [ASSIGN]:8589934959 @0:0
               xmlNodeType [REFERENCE]:8589934960 @0:0
               ELEMENT [STRING]:8589934961 @0:0
         setField7 [METHOD_DECL]:8589934962 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589934963 @0:0
               com.goldencode.p2j.util.NumberType [REFERENCE_DEF]:8589934964 @0:0
                     (name=field7)
         getField8 [METHOD_DECL]:8589934965 @0:0
               (access=public, rettype=com.goldencode.p2j.util.int64)
         Property [ANNOTATION]:8589934966 @0:0
               (getter-id=8589934965)
            = [ASSIGN]:8589934967 @0:0
               id [REFERENCE]:8589934968 @0:0
               8 [NUM_LITERAL]:8589934969 @0:0
            = [ASSIGN]:8589934970 @0:0
               name [REFERENCE]:8589934971 @0:0
               field8 [STRING]:8589934972 @0:0
            = [ASSIGN]:8589934973 @0:0
               column [REFERENCE]:8589934974 @0:0
               field8 [STRING]:8589934975 @0:0
            = [ASSIGN]:8589934976 @0:0
               legacy [REFERENCE]:8589934977 @0:0
               fint64 [STRING]:8589934978 @0:0
            = [ASSIGN]:8589934979 @0:0
               format [REFERENCE]:8589934980 @0:0
               99999 [STRING]:8589934981 @0:0
            = [ASSIGN]:8589934982 @0:0
               initial [REFERENCE]:8589934983 @0:0
               4 [STRING]:8589934984 @0:0
            = [ASSIGN]:8589934985 @0:0
               label [REFERENCE]:8589934986 @0:0
               int64 [STRING]:8589934987 @0:0
            = [ASSIGN]:8589934988 @0:0
               order [REFERENCE]:8589934989 @0:0
               80 [NUM_LITERAL]:8589934990 @0:0
            = [ASSIGN]:8589934991 @0:0
               serializeHidden [REFERENCE]:8589934992 @0:0
               true [BOOL_TRUE]:8589934993 @0:0
            = [ASSIGN]:8589934994 @0:0
               serializeName [REFERENCE]:8589934995 @0:0
               int64-field [STRING]:8589934996 @0:0
            = [ASSIGN]:8589934997 @0:0
               xmlNodeType [REFERENCE]:8589934998 @0:0
               ELEMENT [STRING]:8589934999 @0:0
         setField8 [METHOD_DECL]:8589935000 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589935001 @0:0
               com.goldencode.p2j.util.NumberType [REFERENCE_DEF]:8589935002 @0:0
                     (name=field8)
         isField9 [METHOD_DECL]:8589935003 @0:0
               (access=public, rettype=com.goldencode.p2j.util.logical)
         Property [ANNOTATION]:8589935004 @0:0
               (getter-id=8589935003)
            = [ASSIGN]:8589935005 @0:0
               id [REFERENCE]:8589935006 @0:0
               9 [NUM_LITERAL]:8589935007 @0:0
            = [ASSIGN]:8589935008 @0:0
               name [REFERENCE]:8589935009 @0:0
               field9 [STRING]:8589935010 @0:0
            = [ASSIGN]:8589935011 @0:0
               column [REFERENCE]:8589935012 @0:0
               field9 [STRING]:8589935013 @0:0
            = [ASSIGN]:8589935014 @0:0
               legacy [REFERENCE]:8589935015 @0:0
               fbool [STRING]:8589935016 @0:0
            = [ASSIGN]:8589935017 @0:0
               format [REFERENCE]:8589935018 @0:0
               yes/no [STRING]:8589935019 @0:0
            = [ASSIGN]:8589935020 @0:0
               initial [REFERENCE]:8589935021 @0:0
               true [STRING]:8589935022 @0:0
            = [ASSIGN]:8589935023 @0:0
               label [REFERENCE]:8589935024 @0:0
               bool [STRING]:8589935025 @0:0
            = [ASSIGN]:8589935026 @0:0
               order [REFERENCE]:8589935027 @0:0
               90 [NUM_LITERAL]:8589935028 @0:0
            = [ASSIGN]:8589935029 @0:0
               serializeHidden [REFERENCE]:8589935030 @0:0
               true [BOOL_TRUE]:8589935031 @0:0
            = [ASSIGN]:8589935032 @0:0
               serializeName [REFERENCE]:8589935033 @0:0
               logical [STRING]:8589935034 @0:0
            = [ASSIGN]:8589935035 @0:0
               xmlNodeType [REFERENCE]:8589935036 @0:0
               ELEMENT [STRING]:8589935037 @0:0
         setField9 [METHOD_DECL]:8589935038 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589935039 @0:0
               com.goldencode.p2j.util.logical [REFERENCE_DEF]:8589935040 @0:0
                     (name=field9)
         getField10 [METHOD_DECL]:8589935041 @0:0
               (access=public, rettype=com.goldencode.p2j.util.date)
         Property [ANNOTATION]:8589935042 @0:0
               (getter-id=8589935041)
            = [ASSIGN]:8589935043 @0:0
               id [REFERENCE]:8589935044 @0:0
               10 [NUM_LITERAL]:8589935045 @0:0
            = [ASSIGN]:8589935046 @0:0
               name [REFERENCE]:8589935047 @0:0
               field10 [STRING]:8589935048 @0:0
            = [ASSIGN]:8589935049 @0:0
               column [REFERENCE]:8589935050 @0:0
               field10 [STRING]:8589935051 @0:0
            = [ASSIGN]:8589935052 @0:0
               legacy [REFERENCE]:8589935053 @0:0
               fdate [STRING]:8589935054 @0:0
            = [ASSIGN]:8589935055 @0:0
               format [REFERENCE]:8589935056 @0:0
               99/99/99 [STRING]:8589935057 @0:0
            = [ASSIGN]:8589935058 @0:0
               initial [REFERENCE]:8589935059 @0:0
               10/12/22 [STRING]:8589935060 @0:0
            = [ASSIGN]:8589935061 @0:0
               label [REFERENCE]:8589935062 @0:0
               date [STRING]:8589935063 @0:0
            = [ASSIGN]:8589935064 @0:0
               order [REFERENCE]:8589935065 @0:0
               100 [NUM_LITERAL]:8589935066 @0:0
            = [ASSIGN]:8589935067 @0:0
               serializeHidden [REFERENCE]:8589935068 @0:0
               true [BOOL_TRUE]:8589935069 @0:0
            = [ASSIGN]:8589935070 @0:0
               xmlNodeType [REFERENCE]:8589935071 @0:0
               ELEMENT [STRING]:8589935072 @0:0
         setField10 [METHOD_DECL]:8589935073 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589935074 @0:0
               com.goldencode.p2j.util.date [REFERENCE_DEF]:8589935075 @0:0
                     (name=field10)
         getField11 [METHOD_DECL]:8589935076 @0:0
               (access=public, rettype=com.goldencode.p2j.util.datetime)
         Property [ANNOTATION]:8589935077 @0:0
               (getter-id=8589935076)
            = [ASSIGN]:8589935078 @0:0
               id [REFERENCE]:8589935079 @0:0
               11 [NUM_LITERAL]:8589935080 @0:0
            = [ASSIGN]:8589935081 @0:0
               name [REFERENCE]:8589935082 @0:0
               field11 [STRING]:8589935083 @0:0
            = [ASSIGN]:8589935084 @0:0
               column [REFERENCE]:8589935085 @0:0
               field11 [STRING]:8589935086 @0:0
            = [ASSIGN]:8589935087 @0:0
               legacy [REFERENCE]:8589935088 @0:0
               fdatetime [STRING]:8589935089 @0:0
            = [ASSIGN]:8589935090 @0:0
               format [REFERENCE]:8589935091 @0:0
               99/99/9999 HH:MM:SS.SSS [STRING]:8589935092 @0:0
            = [ASSIGN]:8589935093 @0:0
               initial [REFERENCE]:8589935094 @0:0
               10/12/2022 18:32:41.447 [STRING]:8589935095 @0:0
            = [ASSIGN]:8589935096 @0:0
               label [REFERENCE]:8589935097 @0:0
               datetime [STRING]:8589935098 @0:0
            = [ASSIGN]:8589935099 @0:0
               order [REFERENCE]:8589935100 @0:0
               110 [NUM_LITERAL]:8589935101 @0:0
            = [ASSIGN]:8589935102 @0:0
               serializeHidden [REFERENCE]:8589935103 @0:0
               true [BOOL_TRUE]:8589935104 @0:0
            = [ASSIGN]:8589935105 @0:0
               xmlNodeType [REFERENCE]:8589935106 @0:0
               ELEMENT [STRING]:8589935107 @0:0
         setField11 [METHOD_DECL]:8589935108 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589935109 @0:0
               com.goldencode.p2j.util.date [REFERENCE_DEF]:8589935110 @0:0
                     (name=field11)
         getField12 [METHOD_DECL]:8589935111 @0:0
               (access=public, rettype=com.goldencode.p2j.util.datetimetz)
         Property [ANNOTATION]:8589935112 @0:0
               (getter-id=8589935111)
            = [ASSIGN]:8589935113 @0:0
               id [REFERENCE]:8589935114 @0:0
               12 [NUM_LITERAL]:8589935115 @0:0
            = [ASSIGN]:8589935116 @0:0
               name [REFERENCE]:8589935117 @0:0
               field12 [STRING]:8589935118 @0:0
            = [ASSIGN]:8589935119 @0:0
               column [REFERENCE]:8589935120 @0:0
               field12 [STRING]:8589935121 @0:0
            = [ASSIGN]:8589935122 @0:0
               legacy [REFERENCE]:8589935123 @0:0
               fdatetime-tz [STRING]:8589935124 @0:0
            = [ASSIGN]:8589935125 @0:0
               format [REFERENCE]:8589935126 @0:0
               99/99/9999 HH:MM:SS.SSS+HH:MM [STRING]:8589935127 @0:0
            = [ASSIGN]:8589935128 @0:0
               initial [REFERENCE]:8589935129 @0:0
               10/12/2022 18:32:41.447+03:00 [STRING]:8589935130 @0:0
            = [ASSIGN]:8589935131 @0:0
               label [REFERENCE]:8589935132 @0:0
               datetime-tz [STRING]:8589935133 @0:0
            = [ASSIGN]:8589935134 @0:0
               order [REFERENCE]:8589935135 @0:0
               120 [NUM_LITERAL]:8589935136 @0:0
            = [ASSIGN]:8589935137 @0:0
               serializeHidden [REFERENCE]:8589935138 @0:0
               true [BOOL_TRUE]:8589935139 @0:0
            = [ASSIGN]:8589935140 @0:0
               xmlNodeType [REFERENCE]:8589935141 @0:0
               ELEMENT [STRING]:8589935142 @0:0
         setField12 [METHOD_DECL]:8589935143 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589935144 @0:0
               com.goldencode.p2j.util.date [REFERENCE_DEF]:8589935145 @0:0
                     (name=field12)
         getField13 [METHOD_DECL]:8589935146 @0:0
               (access=public, rettype=com.goldencode.p2j.util.blob)
         Property [ANNOTATION]:8589935147 @0:0
               (getter-id=8589935146)
            = [ASSIGN]:8589935148 @0:0
               id [REFERENCE]:8589935149 @0:0
               13 [NUM_LITERAL]:8589935150 @0:0
            = [ASSIGN]:8589935151 @0:0
               name [REFERENCE]:8589935152 @0:0
               field13 [STRING]:8589935153 @0:0
            = [ASSIGN]:8589935154 @0:0
               column [REFERENCE]:8589935155 @0:0
               field13 [STRING]:8589935156 @0:0
            = [ASSIGN]:8589935157 @0:0
               legacy [REFERENCE]:8589935158 @0:0
               fblob [STRING]:8589935159 @0:0
            = [ASSIGN]:8589935160 @0:0
               format [REFERENCE]:8589935161 @0:0
                [STRING]:8589935162 @0:0
            = [ASSIGN]:8589935163 @0:0
               initialNull [REFERENCE]:8589935164 @0:0
               true [BOOL_TRUE]:8589935165 @0:0
            = [ASSIGN]:8589935166 @0:0
               columnLabel [REFERENCE]:8589935167 @0:0
               blob-column [STRING]:8589935168 @0:0
            = [ASSIGN]:8589935169 @0:0
               label [REFERENCE]:8589935170 @0:0
               blob [STRING]:8589935171 @0:0
            = [ASSIGN]:8589935172 @0:0
               order [REFERENCE]:8589935173 @0:0
               130 [NUM_LITERAL]:8589935174 @0:0
            = [ASSIGN]:8589935175 @0:0
               serializeHidden [REFERENCE]:8589935176 @0:0
               true [BOOL_TRUE]:8589935177 @0:0
            = [ASSIGN]:8589935178 @0:0
               xmlNodeType [REFERENCE]:8589935179 @0:0
               ELEMENT [STRING]:8589935180 @0:0
         setField13 [METHOD_DECL]:8589935181 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589935182 @0:0
               com.goldencode.p2j.util.blob [REFERENCE_DEF]:8589935183 @0:0
                     (name=field13)
         getField14 [METHOD_DECL]:8589935184 @0:0
               (access=public, rettype=com.goldencode.p2j.util.clob)
         Property [ANNOTATION]:8589935185 @0:0
               (getter-id=8589935184)
            = [ASSIGN]:8589935186 @0:0
               id [REFERENCE]:8589935187 @0:0
               14 [NUM_LITERAL]:8589935188 @0:0
            = [ASSIGN]:8589935189 @0:0
               name [REFERENCE]:8589935190 @0:0
               field14 [STRING]:8589935191 @0:0
            = [ASSIGN]:8589935192 @0:0
               column [REFERENCE]:8589935193 @0:0
               field14 [STRING]:8589935194 @0:0
            = [ASSIGN]:8589935195 @0:0
               legacy [REFERENCE]:8589935196 @0:0
               fclob1 [STRING]:8589935197 @0:0
            = [ASSIGN]:8589935198 @0:0
               format [REFERENCE]:8589935199 @0:0
                [STRING]:8589935200 @0:0
            = [ASSIGN]:8589935201 @0:0
               initialNull [REFERENCE]:8589935202 @0:0
               true [BOOL_TRUE]:8589935203 @0:0
            = [ASSIGN]:8589935204 @0:0
               label [REFERENCE]:8589935205 @0:0
               clob1 [STRING]:8589935206 @0:0
            = [ASSIGN]:8589935207 @0:0
               order [REFERENCE]:8589935208 @0:0
               140 [NUM_LITERAL]:8589935209 @0:0
            = [ASSIGN]:8589935210 @0:0
               serializeHidden [REFERENCE]:8589935211 @0:0
               true [BOOL_TRUE]:8589935212 @0:0
            = [ASSIGN]:8589935213 @0:0
               xmlNodeType [REFERENCE]:8589935214 @0:0
               hidden [STRING]:8589935215 @0:0
         setField14 [METHOD_DECL]:8589935216 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589935217 @0:0
               com.goldencode.p2j.util.clob [REFERENCE_DEF]:8589935218 @0:0
                     (name=field14)
         getField15 [METHOD_DECL]:8589935219 @0:0
               (access=public, rettype=com.goldencode.p2j.util.clob)
         Property [ANNOTATION]:8589935220 @0:0
               (getter-id=8589935219)
            = [ASSIGN]:8589935221 @0:0
               id [REFERENCE]:8589935222 @0:0
               15 [NUM_LITERAL]:8589935223 @0:0
            = [ASSIGN]:8589935224 @0:0
               name [REFERENCE]:8589935225 @0:0
               field15 [STRING]:8589935226 @0:0
            = [ASSIGN]:8589935227 @0:0
               column [REFERENCE]:8589935228 @0:0
               field15 [STRING]:8589935229 @0:0
            = [ASSIGN]:8589935230 @0:0
               legacy [REFERENCE]:8589935231 @0:0
               fclob2 [STRING]:8589935232 @0:0
            = [ASSIGN]:8589935233 @0:0
               format [REFERENCE]:8589935234 @0:0
                [STRING]:8589935235 @0:0
            = [ASSIGN]:8589935236 @0:0
               initialNull [REFERENCE]:8589935237 @0:0
               true [BOOL_TRUE]:8589935238 @0:0
            = [ASSIGN]:8589935239 @0:0
               label [REFERENCE]:8589935240 @0:0
               clob2 [STRING]:8589935241 @0:0
            = [ASSIGN]:8589935242 @0:0
               order [REFERENCE]:8589935243 @0:0
               150 [NUM_LITERAL]:8589935244 @0:0
            = [ASSIGN]:8589935245 @0:0
               serializeHidden [REFERENCE]:8589935246 @0:0
               true [BOOL_TRUE]:8589935247 @0:0
            = [ASSIGN]:8589935248 @0:0
               xmlNodeType [REFERENCE]:8589935249 @0:0
               ELEMENT [STRING]:8589935250 @0:0
         setField15 [METHOD_DECL]:8589935251 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589935252 @0:0
               com.goldencode.p2j.util.clob [REFERENCE_DEF]:8589935253 @0:0
                     (name=field15)
         getField16 [METHOD_DECL]:8589935254 @0:0
               (access=public, rettype=com.goldencode.p2j.util.recid)
         Property [ANNOTATION]:8589935255 @0:0
               (getter-id=8589935254)
            = [ASSIGN]:8589935256 @0:0
               id [REFERENCE]:8589935257 @0:0
               16 [NUM_LITERAL]:8589935258 @0:0
            = [ASSIGN]:8589935259 @0:0
               name [REFERENCE]:8589935260 @0:0
               field16 [STRING]:8589935261 @0:0
            = [ASSIGN]:8589935262 @0:0
               column [REFERENCE]:8589935263 @0:0
               field16 [STRING]:8589935264 @0:0
            = [ASSIGN]:8589935265 @0:0
               legacy [REFERENCE]:8589935266 @0:0
               frecid [STRING]:8589935267 @0:0
            = [ASSIGN]:8589935268 @0:0
               format [REFERENCE]:8589935269 @0:0
               ->,>>>,>>9 [STRING]:8589935270 @0:0
            = [ASSIGN]:8589935271 @0:0
               initialNull [REFERENCE]:8589935272 @0:0
               true [BOOL_TRUE]:8589935273 @0:0
            = [ASSIGN]:8589935274 @0:0
               label [REFERENCE]:8589935275 @0:0
               recid [STRING]:8589935276 @0:0
            = [ASSIGN]:8589935277 @0:0
               order [REFERENCE]:8589935278 @0:0
               160 [NUM_LITERAL]:8589935279 @0:0
            = [ASSIGN]:8589935280 @0:0
               serializeHidden [REFERENCE]:8589935281 @0:0
               true [BOOL_TRUE]:8589935282 @0:0
            = [ASSIGN]:8589935283 @0:0
               xmlNodeType [REFERENCE]:8589935284 @0:0
               ELEMENT [STRING]:8589935285 @0:0
         setField16 [METHOD_DECL]:8589935286 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589935287 @0:0
               com.goldencode.p2j.util.NumberType [REFERENCE_DEF]:8589935288 @0:0
                     (name=field16)
         getField17 [METHOD_DECL]:8589935289 @0:0
               (access=public, rettype=com.goldencode.p2j.util.handle)
         Property [ANNOTATION]:8589935290 @0:0
               (getter-id=8589935289)
            = [ASSIGN]:8589935291 @0:0
               id [REFERENCE]:8589935292 @0:0
               17 [NUM_LITERAL]:8589935293 @0:0
            = [ASSIGN]:8589935294 @0:0
               name [REFERENCE]:8589935295 @0:0
               field17 [STRING]:8589935296 @0:0
            = [ASSIGN]:8589935297 @0:0
               column [REFERENCE]:8589935298 @0:0
               field17 [STRING]:8589935299 @0:0
            = [ASSIGN]:8589935300 @0:0
               legacy [REFERENCE]:8589935301 @0:0
               fhandle [STRING]:8589935302 @0:0
            = [ASSIGN]:8589935303 @0:0
               format [REFERENCE]:8589935304 @0:0
               >>>>>>9 [STRING]:8589935305 @0:0
            = [ASSIGN]:8589935306 @0:0
               initialNull [REFERENCE]:8589935307 @0:0
               true [BOOL_TRUE]:8589935308 @0:0
            = [ASSIGN]:8589935309 @0:0
               label [REFERENCE]:8589935310 @0:0
               handle [STRING]:8589935311 @0:0
            = [ASSIGN]:8589935312 @0:0
               order [REFERENCE]:8589935313 @0:0
               170 [NUM_LITERAL]:8589935314 @0:0
            = [ASSIGN]:8589935315 @0:0
               serializeHidden [REFERENCE]:8589935316 @0:0
               true [BOOL_TRUE]:8589935317 @0:0
            = [ASSIGN]:8589935318 @0:0
               xmlNodeType [REFERENCE]:8589935319 @0:0
               ELEMENT [STRING]:8589935320 @0:0
         setField17 [METHOD_DECL]:8589935321 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589935322 @0:0
               com.goldencode.p2j.util.handle [REFERENCE_DEF]:8589935323 @0:0
                     (name=field17)
         getField18 [METHOD_DECL]:8589935324 @0:0
               (access=public, rettype=com.goldencode.p2j.util.comhandle)
         Property [ANNOTATION]:8589935325 @0:0
               (getter-id=8589935324)
            = [ASSIGN]:8589935326 @0:0
               id [REFERENCE]:8589935327 @0:0
               18 [NUM_LITERAL]:8589935328 @0:0
            = [ASSIGN]:8589935329 @0:0
               name [REFERENCE]:8589935330 @0:0
               field18 [STRING]:8589935331 @0:0
            = [ASSIGN]:8589935332 @0:0
               column [REFERENCE]:8589935333 @0:0
               field18 [STRING]:8589935334 @0:0
            = [ASSIGN]:8589935335 @0:0
               legacy [REFERENCE]:8589935336 @0:0
               fcom-handle [STRING]:8589935337 @0:0
            = [ASSIGN]:8589935338 @0:0
               format [REFERENCE]:8589935339 @0:0
               >>>>>>9 [STRING]:8589935340 @0:0
            = [ASSIGN]:8589935341 @0:0
               initialNull [REFERENCE]:8589935342 @0:0
               true [BOOL_TRUE]:8589935343 @0:0
            = [ASSIGN]:8589935344 @0:0
               label [REFERENCE]:8589935345 @0:0
               com-handle [STRING]:8589935346 @0:0
            = [ASSIGN]:8589935347 @0:0
               order [REFERENCE]:8589935348 @0:0
               180 [NUM_LITERAL]:8589935349 @0:0
            = [ASSIGN]:8589935350 @0:0
               serializeHidden [REFERENCE]:8589935351 @0:0
               true [BOOL_TRUE]:8589935352 @0:0
            = [ASSIGN]:8589935353 @0:0
               xmlNodeType [REFERENCE]:8589935354 @0:0
               ELEMENT [STRING]:8589935355 @0:0
         setField18 [METHOD_DECL]:8589935356 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589935357 @0:0
               com.goldencode.p2j.util.comhandle [REFERENCE_DEF]:8589935358 @0:0
                     (name=field18)
         getField19 [METHOD_DECL]:8589935359 @0:0
               (access=public, rettype=com.goldencode.p2j.util.raw)
         Property [ANNOTATION]:8589935360 @0:0
               (getter-id=8589935359)
            = [ASSIGN]:8589935361 @0:0
               id [REFERENCE]:8589935362 @0:0
               19 [NUM_LITERAL]:8589935363 @0:0
            = [ASSIGN]:8589935364 @0:0
               name [REFERENCE]:8589935365 @0:0
               field19 [STRING]:8589935366 @0:0
            = [ASSIGN]:8589935367 @0:0
               column [REFERENCE]:8589935368 @0:0
               field19 [STRING]:8589935369 @0:0
            = [ASSIGN]:8589935370 @0:0
               legacy [REFERENCE]:8589935371 @0:0
               fraw [STRING]:8589935372 @0:0
            = [ASSIGN]:8589935373 @0:0
               format [REFERENCE]:8589935374 @0:0
                [STRING]:8589935375 @0:0
            = [ASSIGN]:8589935376 @0:0
               order [REFERENCE]:8589935377 @0:0
               190 [NUM_LITERAL]:8589935378 @0:0
            = [ASSIGN]:8589935379 @0:0
               serializeHidden [REFERENCE]:8589935380 @0:0
               true [BOOL_TRUE]:8589935381 @0:0
            = [ASSIGN]:8589935382 @0:0
               xmlNodeType [REFERENCE]:8589935383 @0:0
               ELEMENT [STRING]:8589935384 @0:0
         setField19 [METHOD_DECL]:8589935385 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589935386 @0:0
               com.goldencode.p2j.util.BinaryData [REFERENCE_DEF]:8589935387 @0:0
                     (name=field19)
         getField20 [METHOD_DECL]:8589935388 @0:0
               (access=public, rettype=com.goldencode.p2j.util.rowid)
         Property [ANNOTATION]:8589935389 @0:0
               (getter-id=8589935388)
            = [ASSIGN]:8589935390 @0:0
               id [REFERENCE]:8589935391 @0:0
               20 [NUM_LITERAL]:8589935392 @0:0
            = [ASSIGN]:8589935393 @0:0
               name [REFERENCE]:8589935394 @0:0
               field20 [STRING]:8589935395 @0:0
            = [ASSIGN]:8589935396 @0:0
               column [REFERENCE]:8589935397 @0:0
               field20 [STRING]:8589935398 @0:0
            = [ASSIGN]:8589935399 @0:0
               legacy [REFERENCE]:8589935400 @0:0
               frowid [STRING]:8589935401 @0:0
            = [ASSIGN]:8589935402 @0:0
               format [REFERENCE]:8589935403 @0:0
                [STRING]:8589935404 @0:0
            = [ASSIGN]:8589935405 @0:0
               initialNull [REFERENCE]:8589935406 @0:0
               true [BOOL_TRUE]:8589935407 @0:0
            = [ASSIGN]:8589935408 @0:0
               order [REFERENCE]:8589935409 @0:0
               200 [NUM_LITERAL]:8589935410 @0:0
            = [ASSIGN]:8589935411 @0:0
               serializeHidden [REFERENCE]:8589935412 @0:0
               true [BOOL_TRUE]:8589935413 @0:0
            = [ASSIGN]:8589935414 @0:0
               xmlNodeType [REFERENCE]:8589935415 @0:0
               ELEMENT [STRING]:8589935416 @0:0
         setField20 [METHOD_DECL]:8589935417 @0:0
               (access=public, rettype=void)
             [LPARENS]:8589935418 @0:0
               com.goldencode.p2j.util.rowid [REFERENCE_DEF]:8589935419 @0:0
                     (name=field20)
       [CS_INNER_CLASSES]:8589934664 @0:0
         Buf [KW_INTERFACE]:8589934665 @0:0
               (extends=DynamicRecord1, TempTableBuffer)
             [CS_INSTANCE_METHODS]:8589934666 @0:0

If I undestand correctly here serialize-hidden value is wrong.

I have absolutely no experience with this.
What should I start looking at?
Thank you.

The serialize-hidden is a known issue which was fixed in a different branch. To fix it, open DynamicTablesHelper.java and replace

tableField.putAnnotation("serialize-name", field.getSerializeName());

         if (field.getSerializeName() != null)
         {
            tableField.putAnnotation("serialize-name", field.getSerializeName());
         }

Ovidiu Maxiniuc wrote:

The serialize-hidden is a known issue which was fixed in a different branch. To fix it, open DynamicTablesHelper.java and replace
[...]with[...]
I cannot tell the exact line since they are changed now. Also there are more attributes which suffer from same defect.

Thank you, Ovidiu.
Unfortunatelly this doesn't help. It seems that serialize-hidden gets the incorrect value earlier (see #6453-67).
Thank you.

Greg Shah wrote:

For datetime the difference is only in case.

Please analyze if this case difference matters in our runtime. If not, then it can be changed.

I've resolved the issue with datatime by replacing equals with equalIgnoreCase when comparing formats in WRITE-XMLSCHEMA.
I think it is safer.
Committed to 6129a/14484.

Igor Skornyakov wrote:

Ovidiu Maxiniuc wrote:

The serialize-hidden is a known issue which was fixed in a different branch. To fix it, open DynamicTablesHelper.java and replace
[...]with[...]
I cannot tell the exact line since they are changed now. Also there are more attributes which suffer from same defect.

Thank you, Ovidiu.
Unfortunatelly this doesn't help. It seems that serialize-hidden gets the incorrect value earlier (see #6453-67).

Replacing

         tableField.putAnnotation("serialize-hidden", field.isSerializeHidden());

         if (field.isSerializeHidden())
         {
            tableField.putAnnotation("serialize-hidden", field.isSerializeHidden());
         }

resolves the main problem with serialize-hidden. Now (READ|WRITE)-XML(SCHEMA)? produce reasonable results for dynamic temporary tables.
However there are still some incompatibilities.
Working on them.

In TableMapper.loadFields for if INITIAL is NOW or TODAY for date/datetime field we calculate the value of correspinding function and use it as INITIAL value in the LegacyFieldInfo.
This results in the incorrect initail value after PREPARE for dynamic TEMP-TABLE. Moreover, I think that is is never correct. It seems that it is better to have Supplier<BaseDataType> instead of BaseDataType for the initialValue field in the LegacyFieldInfo with a corresponding change in c'tor.
Have I missed something?
Thank you.

In TableMapper.loadFields for if INITIAL is NOW or TODAY for date/datetime field we calculate the value of correspinding function and use it as INITIAL value in the LegacyFieldInfo.

See #6301.

Greg Shah wrote:

In TableMapper.loadFields for if INITIAL is NOW or TODAY for date/datetime field we calculate the value of correspinding function and use it as INITIAL value in the LegacyFieldInfo.

See #6301.

I see. It looks indeed related to what I've described, but exectly the same. Should I try to fix it now?
Thank you.

Are you saying that Eric's solution in 3821c revs 14279 and 14280 is not sufficient to resolve the issue you have found?

Greg Shah wrote:

Are you saying that Eric's solution in 3821c revs 14279 and 14280 is not sufficient to resolve the issue you have found?

Looking at 3821c I'm not sure that it contains the fix for the problem. In any case I need at least to have these changes in 6129a.

Fixed a number of issues with WRITE-XML and READ-XMLSCHEMA.

Committed to 6129a/14488.

At this moment I see only one issue with (READ|WRITE)-XML(SCHEMA)? in addition to #6453-72. The The COLUMN-CODEPAGE attribute is lost on dynamic TEMP-TABLE PREPARE.
Working on this.

Igor Skornyakov wrote:

Looking at 3821c I'm not sure that it contains the fix for the problem.

We have some technical debt, in that TableMapper redundantly implements some features that are handled by RecordMeta, PropertyMeta, DmoMeta. I would rather not add to this redundancy by re-inventing the wheel for this feature/fix. We will need to figure out how to leverage the existing solution for dynamic temp-tables.

In any case I need at least to have these changes in 6129a.

A rebase will be done in the next few days.

Eric Faulhaber wrote:

Igor Skornyakov wrote:

Looking at 3821c I'm not sure that it contains the fix for the problem.

We have some technical debt, in that TableMapper redundantly implements some features that are handled by RecordMeta, PropertyMeta, DmoMeta. I would rather not add to this redundancy by re-inventing the wheel for this feature/fix. We will need to figure out how to leverage the existing solution for dynamic temp-tables.

In any case I need at least to have these changes in 6129a.

A rebase will be done in the next few days.

I see. Thank you.

I've noticed the following conversion problem:
The field definition

    FIELD fdatetime-tz1 as datetime-tz initial "10/17/2022 13:47:48.426+03:00" label "datetime-tz1" 

   @Property(id = 14, name = "fdatetimeTz1", column = "fdatetime_tz1", legacy = "fdatetime-tz1", format = "99/99/9999 HH:MM:SS.SSS+HH:MM", initial = "10/17/2022", label = "datetime-tz1", order = 130)
   public datetimetz getFdatetimeTz1();

As we can see the time part is lost.

Igor Skornyakov wrote:

I've noticed the following conversion problem:
The field definition
[...]
is converted to
[...]

As we can see the time part is lost.

It looks that this happens at a very early conversion step. Thie corresponding AST node is:

      <ast col="25" id="34359738519" line="17" text="fdatetime-tz1" type="FIELD_DATETIME_TZ">
        <annotation datatype="java.lang.String" key="historical" value="fdatetime-tz1"/>
        <annotation datatype="java.lang.Long" key="fdefid-low" value="-6034864375743484910"/>
        <annotation datatype="java.lang.Long" key="fdefid-high" value="-1525000044317686479"/>
        <annotation datatype="java.lang.Long" key="order" value="130"/>
        <annotation datatype="java.lang.String" key="label" value="datetime-tz1"/>
        <annotation datatype="java.lang.Long" key="peerid" value="47244640299"/>
        <ast col="40" id="34359738522" line="17" text="initial" type="KW_INIT">
          <ast col="48" id="34359738523" line="17" text="10/17/2022" type="DATE_LITERAL">
            <annotation datatype="java.lang.Boolean" key="is-literal" value="true"/>
            <annotation datatype="java.lang.String" key="original-text" value="&quot;10/17/2022 13:47:48.426+03:00&quot;"/>
            <annotation datatype="java.lang.Long" key="oldtype" value="2937"/>
          </ast>
        </ast>
      </ast>

Igor Skornyakov wrote:

Igor Skornyakov wrote:

I've noticed the following conversion problem:
The field definition
[...]
is converted to
[...]

As we can see the time part is lost.

It looks that this happens at a very early conversion step. Thie corresponding AST node is:
[...]
We see that the type of the KW_INIT is DATE_LITERAL instead of DATETIME_TZ_LITERAL.

In the .parser file I see:

            define field [DEFINE_FIELD]:17179869551 @0:0
               fdatetime-tz1 [SYMBOL]:17179869552 @17:11
               as [KW_AS]:17179869554 @17:25
                  datetime-tz [KW_DATE_TZ]:17179869556 @17:28
               initial [KW_INIT]:17179869558 @17:40
                  10/17/2022 [DATE_LITERAL]:17179869560 @17:48
               label [KW_LABEL]:17179869562 @17:80
                  "datetime-tz1" [STRING]:17179869564 @17:86

Look at initializer_constant in progress.g. There is a relex() step that occurs when the literal is quoted text. Something probably goes wrong there.

Greg Shah wrote:

Look at initializer_constant in progress.g. There is a relex() step that occurs when the literal is quoted text. Something probably goes wrong there.

Thank you!

As far as I understand relex() cannot properly and reliably detect the literal type since it does not consider vtype at all. For example 4GL accepts "10/17/2022 13:47:48.426" as INITIAL value for datetime-tz with session timezone but how relex() can decide if it is DATETIME_TZ_LITERAL or DATETIME_LITERAL? Right now it treats it as DATE_LITERAL.

Unfortunately, I'm not familiar with the ANTLR-based parsing good enough to fix this problem quickly. And it looks not directly related to the scope of this task.
Should I create a separate issue?
Thank you.

Igor Skornyakov wrote:

As far as I understand relex() cannot properly and reliably detect the literal type since it does not consider vtype at all. For example 4GL accepts "10/17/2022 13:47:48.426" as INITIAL value for datetime-tz with session timezone but how relex() can decide if it is DATETIME_TZ_LITERAL or DATETIME_LITERAL? Right now it treats it as DATE_LITERAL.

Unfortunately, I'm not familiar with the ANTLR-based parsing good enough to fix this problem quickly. And it looks not directly related to the scope of this task.
Should I create a separate issue?

Yes, please do, and let's focus on the remaining core items of this task. What is left?

Eric Faulhaber wrote:

Yes, please do, and let's focus on the remaining core items of this task. What is left?

At this moment I'm trying to create a test for SCHEMA-MARSHAL. Hope to finish tomorrow. And this is the last thing (if #6453-83 changes are OK).

Marian Edu wrote:

Greg Shah wrote:

Constantin/Marian: Can you please advise Igor on how he can test the behavior of SCHEMA-MARSHAL?

This is basically used when passing temp-table (separate or part of a dataset) from one side (usually the client) where the other side receive it as handle (table or dataset) so it doesn't know it's schema. By default SCHEMA-MARSHAL is set to FULL so when passed along the schema is also included next to the data, for MIN some lees important table/field/index properties are not serialised but the data structure is there so the other end can always work with it... the only real difference is when set to NONE so no schema information is being send. This is fine if the other end has the temp-table/dataset definition - aka parameter is defined as table/dataset for - but it will throw if schema is unknown there - aka parameter is table/dataset handle, the error being thrown there is 12323.

This can be tested using a `client` connection to the appssrv (using `connect`) and set the SCHEMA-MARSHAL property of temp-table before the call, or using REST send only the data payload to an end-point that expects a table handle. For that the following procedure from testcases project can be used: appsrv/api/table/pipe_table_handle.p. For REST you will see in SoapUI project appsrv/test/web/pasoe-fwd-soapui-project.xml when the pipe_table_handle endpoint is being called the TempTable also has the xsd:schema attached.

Marian,
I've finished (hopefully) with other substasks and started working on testing SCHEMA-MARSHAL support.
At this moment I have a problem with understanding how to run tests you've mentioned with 4GL. The OE documentation looks huge and not clear enough (for me). Can you please recommend a reasonably short and concise description of this functionality with examples?
Thank you.

Igor Skornyakov wrote:

Marian,
I've finished (hopefully) with other substasks and started working on testing SCHEMA-MARSHAL support.
At this moment I have a problem with understanding how to run tests you've mentioned with 4GL. The OE documentation looks huge and not clear enough (for me). Can you please recommend a reasonably short and concise description of this functionality with examples?

Igor, there is one example you might use/extend maybe in testcases project - appsrv/test/table/test_table_pipe_table.p and make a new one that calls table/pipe_table_handle.p instead (it's appsrv/api/table/pipe_table_handle.p but the appsrv propath should include appsrv/api).

I can try to write something like that is needed, I don't think we still have the appsrv setup and for FWD I think Constantin is the best person to advise on how you can run something on the appsrv or start an appsrv in FWD.

Marian Edu wrote:

Igor Skornyakov wrote:

Marian,
I've finished (hopefully) with other substasks and started working on testing SCHEMA-MARSHAL support.
At this moment I have a problem with understanding how to run tests you've mentioned with 4GL. The OE documentation looks huge and not clear enough (for me). Can you please recommend a reasonably short and concise description of this functionality with examples?

Igor, there is one example you might use/extend maybe in testcases project - appsrv/test/table/test_table_pipe_table.p and make a new one that calls table/pipe_table_handle.p instead (it's appsrv/api/table/pipe_table_handle.p but the appsrv propath should include appsrv/api).

I can try to write something like that is needed, I don't think we still have the appsrv setup and for FWD I think Constantin is the best person to advise on how you can run something on the appsrv or start an appsrv in FWD.

Thank you, Marian.
At this moment I need to understand how to setup and run app. server with 4GL. As I wrote before, OE documentation is too wordy and the information is spread across multiple places.
If I will be able to run at least one test from testcases/appsrv with 4GL I believe I will be able to move forward pretty fast.
For FWD I can at least analyze/debug our code, so it seems to be easier, but I understand that I have to make sure that our support for SCHEMA-MARSHAL is compatible with 4GL.
In fact I understand that the only place in FWD where SCHEMA-MARSHAL value affects the functionality is the PropertyDefinition.writeExternal() method which was AFAIK implemented based on OE specs. The method is short and simple so in fact I need to make sure that 4GL behavior is indeed the same as described in the documentation. This is why I need to be able to run 4GL tests.

Igor Skornyakov wrote:

If I will be able to run at least one test from testcases/appsrv with 4GL I believe I will be able to move forward pretty fast.

For that you need to setup an appsrv locally, or maybe there is one available on <customer> remote dev that you can use. You need to be able to deploy the code from testcases repo there and made that available in the appsrv PROPATH.

For FWD I can at least analyze/debug our code, so it seems to be easier, but I understand that I have to make sure that our support for SCHEMA-MARSHAL is compatible with 4GL.

If it's easier for you we can write the 4GL tests on our side and then you run those on FWD, either way it will take some time to test exactly which attributes are ignored for min-schema but when no-schema is used it should be easier to see the other side will fail if it doesn't receive a valid table schema.

In fact I understand that the only place in FWD where SCHEMA-MARSHAL value affects the functionality is the PropertyDefinition.writeExternal() method which was AFAIK implemented based on OE specs.

That I can't comment upon I'm afraid as I don't know anything about it.

Marian Edu wrote:

If it's easier for you we can write the 4GL tests on our side and then you run those on FWD, either way it will take some time to test exactly which attributes are ignored for min-schema but when no-schema is used it should be easier to see the other side will fail if it doesn't receive a valid table schema.

BTW: what can go wrong if we ignore SCHEMA-MARSHAL and always send all schema attributes? I understand that it can affect performance but I do not think that the impact will be noticeable (except maybe some exotic use cases when tables with many fields with multiple attributes but just a couple of records are very frequently transmitted). Can it affect application logic?
Thank you.

Igor Skornyakov wrote:

BTW: what can go wrong if we ignore SCHEMA-MARSHAL and always send all schema attributes? I understand that it can affect performance but I do not think that the impact will be noticeable (except maybe some exotic use cases when tables with many fields with multiple attributes but just a couple of records are very frequently transmitted).

The impact might be noticeable if they are using a dataset intensive framework like that of Consultingwerk, the API always use the same dataset definition but then at times it only needs one or two tables from a slightly larger number in the dataset and the definition can also be quite large with all those fancy 'display only' properties one can find in a 4GL data structure - it can easily be way larger than the actual data sent :(

Can it affect application logic?

Nope, receiving a no-schema table when one doesn't have thee schema already is simply an error and no application is built to generate errors (on purpose at least).

We need to match the results exactly. No short cuts.

Greg Shah wrote:

We need to match the results exactly. No short cuts.

I see. So I definitely need to be able to run corresponding tests in 4GL environment

Igor Skornyakov wrote:

Marian,
Sorry for disturbing you, but I've stuck with the problem of running my test with the OE App. Server.
The test is pretty simple (see attached).
I understand how to modify tt-marshal.p to work with app. server. My problem is how to deploy tt-marshal-accept.p to this server.

Hi Igor, 'deployment' for AppSrv is not what you expect from a tomcat based server - I mean yeah, for rest adapter and web handler mapping there is a war to be deployed but otherwise deploying 4GL code mean just that you need to put the files somewhere in the AppSrv's PROPATH and that's all, no need to do anything else, not even restart the AppSrv if the files are new :)

Marian Edu wrote:

Igor Skornyakov wrote:

Marian,
Sorry for disturbing you, but I've stuck with the problem of running my test with the OE App. Server.
The test is pretty simple (see attached).
I understand how to modify tt-marshal.p to work with app. server. My problem is how to deploy tt-marshal-accept.p to this server.

Hi Igor, 'deployment' for AppSrv is not what you expect from a tomcat based server - I mean yeah, for rest adapter and web handler mapping there is a war to be deployed but otherwise deploying 4GL code mean just that you need to put the files somewhere in the AppSrv's PROPATH and that's all, no need to do anything else, not even restart the AppSrv if the files are new :)

Oh, I see. Thanks a lot!

Igor Skornyakov wrote:

When running attached test with an application server, I see the following in the server log:
[...]
The handle in question is the input paramater of the procedure. Why it is not initialized?

Because you can't pass handles over the wire (appsrv) - those are only valid in the same AVM instance.

You need to use table-handle parameters instead, check the samples I've mentioned in testcases repo.

Marian Edu wrote:

Igor Skornyakov wrote:

When running attached test with an application server, I see the following in the server log:
[...]
The handle in question is the input paramater of the procedure. Why it is not initialized?

Because you can't pass handles over the wire (appsrv) - those are only valid in the same AVM instance.

You need to use table-handle parameters instead, check the samples I've mentioned in testcases repo.

Thanks a lot! I was finally able to run my test!

A bit strange. The OE versions looks identical in both FULL and NONE case.
https://docs.progress.com/ru-RU/bundle/openedge-abl-reference-117/page/NO-SCHEMA-MARSHAL-attribute.html and link to MIN-SCHEMA-MARSHAL at the bottom of the page. The SCHEMA-MARSHAL page is a bit ...not so explicit.

For FWD implementation, see DsTableDefinition. There is is switch in c'tor which decides which attributes to serialize beased on SCHEMA-MARSHAL attribute. They were picked from reference manual.

Ovidiu Maxiniuc wrote:

A bit strange. The OE versions looks identical in both FULL and NONE case.
https://docs.progress.com/ru-RU/bundle/openedge-abl-reference-117/page/NO-SCHEMA-MARSHAL-attribute.html and link to MIN-SCHEMA-MARSHAL at the bottom of the page. The SCHEMA-MARSHAL page is a bit ...not so explicit.

For FWD implementation, see DsTableDefinition. There is is switch in c'tor which decides which attributes to serialize beased on SCHEMA-MARSHAL attribute. They were picked from reference manual.

Unfortunatelly, I was unable to find an explicit list of serialized attributes in the OE docs. If you can provide a reference it will be great.
The page you're referencing to contains the following:

This attribute suppresses index descriptions and some field information (such as label, help, field validation expression, and so on) when marshaling data. It does marshal field names, data types, and extents.

On the other side I understand that the OE documentation does not always provide accurate information. This is why I was asked to write tests.
If you see why the test can provide misleading results, please let me know.
Thank you.

When I initially added this attribute I used a possible older manuals. It seems it is unchanged, though.
I think your testes are excellent. I salute the idea of serializing the result instead of just checking individual attributes which were documented in pdf.

My surprise was to see the last htt-MIN.xsd and htt-FULL.xsd are pretty identical. Initially, I thought they were the output from OE, but I understand now that they are from FWD, instead. I see that the indices are also missing from the schema.

Ovidiu Maxiniuc wrote:

My surprise was to see the last htt-MIN.xsd and htt-FULL.xsd are pretty identical. Initially, I thought they were the output from OE, but I understand now that they are from FWD, instead. I see that the indices are also missing from the schema.

Yes, as I've mentioned before, the differences between OE and FWD are significant.
I hope however, that I will be able to fix it in a couple of days.

Igor Skornyakov wrote:

Ovidiu Maxiniuc wrote:

My surprise was to see the last htt-MIN.xsd and htt-FULL.xsd are pretty identical. Initially, I thought they were the output from OE, but I understand now that they are from FWD, instead. I see that the indices are also missing from the schema.

Yes, as I've mentioned before, the differences between OE and FWD are significant.
I hope however, that I will be able to fix it in a couple of days.

I wouldn't bother too much with MIN/FULL differences... the real use case here was the 'NONE' variant used by some heavy-duty OERA followers with huge datasets passed around all the time when they just need to pass one row in a single table at times so the schema part greatly outweigh the data payload :)

It looks that when SCHEMA-MARSHAL is FULL, 4GL serializes all field attributes of the table parameter of the remote call that WRITE-XMLSCHEMA does except XML-DATA-TYPE and XML-NODE-NAME.

If the SCHEMA-MARSHAL is MIN the FORMAT, LABEL, COLUMN-LABEL, HELP and DECIMALS attributes are not serialized.

Marian Edu wrote:

I wouldn't bother too much with MIN/FULL differences... the real use case here was the 'NONE' variant used by some heavy-duty OERA followers with huge datasets passed around all the time when they just need to pass one row in a single table at times so the schema part greatly outweigh the data payload :)

I understand that our policy is to be as close to 4GL logic as possible.

Now is the time to get all details exactly correct. I'm glad it is less likely to be an issue, but please do implement full compatibility.

Greg Shah wrote:

Now is the time to get all details exactly correct. I'm glad it is less likely to be an issue, but please do implement full compatibility.

Sure. This is exactly what I'm working on.

Marian Edu wrote:

I wouldn't bother too much with MIN/FULL differences... the real use case here was the 'NONE' variant used by some heavy-duty OERA followers with huge datasets passed around all the time when they just need to pass one row in a single table at times so the schema part greatly outweigh the data payload :)

1. As I can see from the code at this moment we treat NONE as MIN
2. I do not see any logic supporting NONE with a pre-defined schema or even a message A NO-SCHEMA-MARSHAL table cannot be used as a parameter where the receiving side does not have a pre-prepared schema. (12323) generated by 4GL.

Marian,
I do not see tests for the NONE value of the SCHEMA_MARSHAL attribute in yout appsrv test suite. Have I missed something?
Thank you.

Igor Skornyakov wrote:

The support for SCHEMA-MARSHAL NONE value is still missing (see #6453-116). In particular I have no idea in what situations it should be usefull (what "receiving side does have a pre-prepared schema" exactly means).

Igor, sorry for being late on this one... it just slipped my mind I guess. The use for that error message is when the receiving side has the temp-table/dataset parameter defined as handle (table-handle, dataset-handle) so it doesn't know it's structure - if it receives a table with no schema (shema-marshal was set to none on the sending side) that can't be of no use, you need to have the schema to work with the table even dynamically :(

Marian Edu wrote:

Igor, sorry for being late on this one... it just slipped my mind I guess. The use for that error message is when the receiving side has the temp-table/dataset parameter defined as handle (table-handle, dataset-handle) so it doesn't know it's structure - if it receives a table with no schema (shema-marshal was set to none on the sending side) that can't be of no use, you need to have the schema to work with the table even dynamically :(

Thank you Marian. But my question was about the situations when NONE value of the SCHEMA_MARSHAL is acceptable. Is it sufficient that at the receiving side exists a temporary table with the specified name?
Thank you.

1. TableMapper.getLegacyIndexInfo ignores ASC/DESC options for index fields.
2. it is not very convenient to parse this string at the receiving side. In fact it looks that we just ignore index data on un-marshalling.

Do we really need to keep index data in the TableParameter as a string? May be it is better to keep index data in a structured form (like we do with table fields and PropertyDefinition) and use readExternal/writeExternal for (un)mashalling?
What do you think?
Thank you.

• DynamicTablesHelper - please do not use StringUtils.isEmpty in apache commons, add a helper in com.goldencopde.util.StringHelper if you want to avoid doing s != null && !s.isEmpty().
• PropertyDefinition
  • please describe in what situations the new fields are used. I ask because PropertyDefinition is used by the FWD OpenClient support, and AFAIK this does not support XML-NODE-TYPE and others. Maybe we need a sub-class of this for these cases, otherwise we will keep transferring these, without any benefit at all.
  • please compact all the boolean fields into a single bit-representation and use that for read/write.
• TableWrapper.schemaMarshalLevel is not serialized in read/writeExternal
• AppServerHelper - if (tempDMO != null && test is redundant, tempDMO is referenced a couple of lines before this.
• some files need history entries
• did you test with DATASET parameters? There is DsTableDefinition which duplicates to some extent the TableWrapper code.

Constantin Asofiei wrote:

Igor, review for the #6453-117 patch:

• DynamicTablesHelper - please do not use StringUtils.isEmpty in apache commons, add a helper in com.goldencopde.util.StringHelper if you want to avoid doing s != null && !s.isEmpty().

Of course, I can do it. But we have an explicit dependency on commons-lang. What not to use it?

• PropertyDefinition
  • please describe in what situations the new fields are used. I ask because PropertyDefinition is used by the FWD OpenClient support, and AFAIK this does not support XML-NODE-TYPE and others. Maybe we need a sub-class of this for these cases, otherwise we will keep transferring these, without any benefit at all.

As I can see from my test the additional field attributes are transmitted by 4GL with TableParameter in a remote call. In fact it seems that PropertyDefinition is heavily overused in our code. Nine c'tors is a good illustration of this. Maybe it makes sense to use a completely different class for TableParameter (de)serialization?

• please compact all the boolean fields into a single bit-representation and use that for read/write.

I'm not sure that it is a good idea. We will reduce the payload for one byte but at a cost of more complicated code. However, of you insist, I will do this.

• TableWrapper.schemaMarshalLevel is not serialized in read/writeExternal

It was not serialized before, but I plan to add it when I will implement support for the NONE value of the SCHEMA-MARSHAL attr.

• AppServerHelper - if (tempDMO != null && test is redundant, tempDMO is referenced a couple of lines before this.

Thank you, will be fixed.

• some files need history entries

OK, I will double check.

• did you test with DATASET parameters? There is DsTableDefinition which duplicates to some extent the TableWrapper code.

Not yet. I have a lot of issues with TableWrapper so far.

List of Supported 3rd Party Libraries

Greg Shah wrote:

List of Supported 3rd Party Libraries

Sorry, I do not understand. I see both commons-lang and commons-lang3.
Can I use them directly?
Thank you.

Igor Skornyakov wrote:

Greg Shah wrote:

List of Supported 3rd Party Libraries

Sorry, I do not understand. I see both commons-lang and commons-lang3.
Can I use them directly?
Thank you.

No.

Greg Shah wrote:

Sorry, I do not understand. I see both commons-lang and commons-lang3.
Can I use them directly?
Thank you.

No.

OK. Thank you.

Igor Skornyakov wrote:

At this moment we keep index data in the TableParameter as a string created by TableMapper.getLegacyIndexInfo.
I see two problems with this approach:

1. TableMapper.getLegacyIndexInfo ignores ASC/DESC options for index fields.
2. it is not very convenient to parse this string at the receiving side. In fact it looks that we just ignore index data on un-marshalling.

• the caller send a TABLE parameter and the callee (on appserver side) has a TABLE-HANDLE
• the caller's TABLE has unique, non-unique and primary indexes, asc or desc
• check at the callee the schema of the received table: dump the index information and see how it looks like. My point is that we've seen cases where only unique/primary indexes were being marshalled.

Constantin Asofiei wrote:

Igor Skornyakov wrote:

At this moment we keep index data in the TableParameter as a string created by TableMapper.getLegacyIndexInfo.
I see two problems with this approach:

1. TableMapper.getLegacyIndexInfo ignores ASC/DESC options for index fields.
2. it is not very convenient to parse this string at the receiving side. In fact it looks that we just ignore index data on un-marshalling.

Please test on the appserver with this case:

• the caller send a TABLE parameter and the callee (on appserver side) has a TABLE-HANDLE
• the caller's TABLE has unique, non-unique and primary indexes, asc or desc
• check at the callee the schema of the received table: dump the index information and see how it looks like. My point is that we've seen cases where only unique/primary indexes were being marshalled.

Constantin.
In my test the table at the receving side has no indexes at all. See #6453-106.

Igor Skornyakov wrote:

Thank you Marian. But my question was about the situations when NONE value of the SCHEMA_MARSHAL is acceptable. Is it sufficient that at the receiving side exists a temporary table with the specified name?

Yes, the receiving side must have the schema available so the input parameter must be of type table/dataset so static not handle. I suspect the 4GL will use the local schema to read the data payload - since there is no schema received I don't see how the schema check can be performed so there will probably be some other run-time errors if the data payload doesn't match the schema - wrong data type for column, less columns than defined in schema, less tables in dataset than what is in schema, etc...

Marian Edu wrote:

Igor Skornyakov wrote:

Thank you Marian. But my question was about the situations when NONE value of the SCHEMA_MARSHAL is acceptable. Is it sufficient that at the receiving side exists a temporary table with the specified name?

Yes, the receiving side must have the schema available so the input parameter must be of type table/dataset so static not handle. I suspect the 4GL will use the local schema to read the data payload - since there is no schema received I don't see how the schema check can be performed so there will probably be some other run-time errors if the data payload doesn't match the schema - wrong data type for column, less columns than defined in schema, less tables in dataset than what is in schema, etc...

Sorry for my ignorance. How the declaration of such parameter should look like?
Thank you.

Igor Skornyakov wrote:

Sorry for my ignorance. How the declaration of such parameter should look like?

[[https://docs.progress.com/bundle/openedge-develop-abl-applications-122/page/Using-a-temp-table-as-a-parameter.html]]

DEFINE [ INPUT | INPUT-OUTPUT] PARAMETER TABLE FOR temp-table-name

Marian Edu wrote:

[[https://docs.progress.com/bundle/openedge-develop-abl-applications-122/page/Using-a-temp-table-as-a-parameter.html]]

[...]

Thank you!

One more problem with table parameter in remote call.
If the parameter is declared like this:

DEF INPUT PARAM table FOR ttx BIND.

I'm working now on table parameter issues described in #6453-133 and #6453-134.
After that I plan to return to work on transmission of indexes' definitions.
Please let me know what you think about my suggestion #6453-120 and related idea about the PropertyDefinition from #6453-122.
Thank you.

Igor Skornyakov wrote:

Please let me know what you think about my suggestion #6453-120 and related idea about the PropertyDefinition from #6453-122.

Constantin, Ovidiu, (maybe Stanislav?): this is not my area of expertise. Can one of you unblock Igor with an answer to this question? Thanks.

I've added reporting 12323 error reporting ('A NO-SCHEMA-MARSHAL table cannot be used as a parameter where the receiving side does not have a pre-prepared schema') using ErrorManager.throw(12323).
However, I see this error message only in the app. server log, because the error is not propagated to the caller were my test intercepts the error and prints the message to its output.
With 4GL I see the error both in the app. server logs and in the application output file.
What I'm doing wrong?
Thank you.

Igor, you need to use ErrorManager.recordOrThrow most likely.

I'm still thinking about the index issue.

Eric Faulhaber wrote:

Please let me know what you think about my suggestion #6453-120 and related idea about the PropertyDefinition from #6453-122.

Constantin, Ovidiu, (maybe Stanislav?): this is not my area of expertise. Can one of you unblock Igor with an answer to this question? Thanks.

The initial implementation was done using only the manual reference. Only a bit later, I had encounters of this 4GL feature in the customer's code. As you observed, the manuals are very brief for some methods, attributes and other artefacts. The current String solution was added as a compromise unknowing how much of the data was necessary: it offered a good compromise between the data size and the actual information payload. It was also in use by 4GL in other places.

It seems now that the full set of knowledge about the indexes is needed. So I am definitely for a structured serialization, like for other elements of dataset/tables. The protocol is less important as FWD is at the both side of the transport. We just need to be sure the de-serialization works correctly in all cases and to choose the optimum binary data format for it.

Ovidiu Maxiniuc wrote:

It seems now that the full set of knowledge about the indexes is needed. So I am definitely for a structured serialization, like for other elements of dataset/tables. The protocol is less important as FWD is at the both side of the transport. We just need to be sure the de-serialization works correctly in all cases and to choose the optimum binary data format for it.

Thank you, Ovidiu. If there are no objections, I will use a structured serialization fo indexes.

Constantin Asofiei wrote:

Igor, you need to use ErrorManager.recordOrThrow most likely.

Yes, this helps. Thank you, Constantin!

In 4GL when SCHEMA-MARCHAL is NONE and the input parameter of the called procedure is defined as table FOR <local temp-table> BIND the actual parameter value is the temp table which is the same as the local one, but with the name taken from the callee.
How can this be done in FWD in the TemporaryBuffer.associate call?
Thank you.

Igor Skornyakov wrote:

In 4GL when SCHEMA-MARCHAL is NONE and the input parameter of the called procedure is defined as table FOR <local temp-table> BIND the actual parameter value is the temp table which is the same as the local one, but with the name taken from the callee.
How can this be done in FWD in the TemporaryBuffer.associate call?
Thank you.

Sorry, I was looking at the wrong place in a log. Please disregard.

Constantin Asofiei wrote:

Igor, you need to use ErrorManager.recordOrThrow most likely.

I'm still thinking about the index issue.

Constantin,
I'm working on the indexes (un)marshaling now. What is your decision? I see that encoding indexes data as a string in the TableWrapper is used in the LegacyJavaAppserver.
Can I at least change this string structure by adding ASC/DESC option?
Thank you.

1. FWD does not create a separate schema file on WRITE-XMLSCHEMA if a NAMESPACE-URI for a table in the DATASET is different from one of the DATASET.
2. FWD prodata:prefix is not added to the root element on WRITE-XMLSCHEMA.
3. The NAMESPACE-PREFIX is not added to the field name in the xpatch for the xsd:unique element in the DATASET SCHEMA.
4. The NAMESPACE-PREFIX is not added to the elements on WRITE-XML at all.
5. FWD incorrectly specifies schema location on WRITE-XML.

Should I fix these issues in the scope of this task?
Thank you.

Optimally, yes. But I know Eric has other tasks for you as well. How long do you estimate the fixes to take?

Greg Shah wrote:

Optimally, yes. But I know Eric has other tasks for you as well. How long do you estimate the fixes to take?

I think it will take a day or two.
Please note, however, that I'm not familiar with the DATASET support in FWD so this estimation can be not very accurate.

For some reason FWD processes TEMP-TABLE which is part of the DATASET input parameter differently than when it is an input parameter per se.

If a table is an input parameter is it created only at the final destination (AppServerHelper.preProcessParameters) while for a DATASET it happens in the Agent.preProcessArguments.
This causes a problem when SCHEMA-MARSHAL is NONE because at this moment we do not know if the corresponding parameter will be binded or not.
Do we really need to create tables in a DATASET parameter that early?

In addition to the previous note.
In fact, there are many places when we work with a standalone temp table in a different way than when it is a part of a dataset. This includes (de)serialization of remote call arguments and (READ|WRITE)-XML(SCHEMA)? calls.
This results in code duplication and makes the modification of the corresponding logic much more tricky.
Do we have really good reasons for this?
Thank you.

I think it will take a day or two.

Please do resolve these issues now. If it is going to take much longer than 2 days, let us know.

there are many places when we work with a standalone temp table in a different way than when it is a part of a dataset. This includes (de)serialization of remote call arguments and (READ|WRITE)-XML(SCHEMA)? calls.
This results in code duplication and makes the modification of the corresponding logic much more tricky.

Hmm. I suspect this was due to how we evolved the various parts and had tight schedules. This has happened many times, even just in the persistence layer. If we don't have good compatibility and regression tests then we can't know the implications of a change and often might implement in parallel rather than make common code.

Ovidiu will have to give the real answer, but we would prefer to have this factored better.

Yes, this is true. Another reason (actually more important, at least at the beginning) was that, in the initial simple testcases the output seemed quite different.

However, if you can refactor the serializer it would be a good thing. But, IIRC, the reader (deserializer) suffer more of this flaw.

To get a bit of help, there are some testcases in xfer test suites. Look for subdrectories like dataset/(dynamic|mixed|static)/methods/(read|write)_xml[_schema]/. They should cover at least some of your needs. I am not aware of similar tests for temp-tables.

Ovidiu Maxiniuc wrote:

Yes, this is true. Another reason (actually more important, at least at the beginning) was that, in the initial simple testcases the output seemed quite different.

However, if you can refactor the serializer it would be a good thing. But, IIRC, the reader (deserializer) suffer more of this flaw.

To get a bit of help, there are some testcases in xfer test suites. Look for subdrectories like dataset/(dynamic|mixed|static)/methods/(read|write)_xml[_schema]/. They should cover at least some of your needs. I am not aware of similar tests for temp-tables.

Thank you, Ovidiu. However, in my tests, there is no difference between a standalone TEMP-TABLE and one which is a part of a DATASET if we consider (un)marshaling of the argument of a remote call or (READ|WRITE)-XML(SCHEMA)?.

What about #6453-150?
Thank you.

Thank you, Ovidiu. However, in my tests, there is no difference between a standalone TEMP-TABLE and one which is a part of a DATASET if we consider (un)marshaling of the argument of a remote call or (READ|WRITE)-XML(SCHEMA)?.

Very well. This is s strong reason to refactor the inter-process serialization (the Externalizable TableWrapper and DatasetWrapper). Note that this is different than invoking (READ|WRITE)-XML(SCHEMA) methods. I do not know if the data structures will make it easy for you.

What about #6453-150?

I am not familiar with those classes. Probably Constantin is the best suited person to answer such questions.

Ovidiu Maxiniuc wrote:

Very well. This is s strong reason to refactor the inter-process serialization (the Externalizable TableWrapper and DatasetWrapper). Note that this is different than invoking (READ|WRITE)-XML(SCHEMA) methods. I do not know if the data structures will make it easy for you.

Thank you, Ovidiu. I will think about how to do it in a minimally intrusive way.

What about #6453-150?

I am not familiar with those classes. Probably Constantin is the best suited person to answer such questions.

Constantin,
Can you please clarify the situation?
Thank you.

Igor Skornyakov wrote:

Constantin Asofiei wrote:

Igor, you need to use ErrorManager.recordOrThrow most likely.

Yes, this helps. Thank you, Constantin!

Sorry. I've just noticed that this does not resolve the issue completely. With ErrorManager.recordOrThrow I see the error message in the server.log but the error is not propagated to the caller.

Igor Skornyakov wrote:

Fixed DATASET (un)marshaling. See the attached .diff file.
Please review. There are many pended changes. I would prefer to start working on the unification of the standalone and dataset temp-tables support after the changes will be approved and committed.
Thank you.

I tried to review your changes. Indeed, at 2k lines, this a large update. My problem is that I could not find the right revision to apply the patch. It failed for 14354 for a lot of files, so I went back to 14343 (the last big commit). The result was similar.

I also tried inspecting the .diff file directly but the changes are out of context so not very useful. It seems to me that the changes are correct, but there are multiple tasks (schema-marshal, bound table parameters, initial attribute value - to name a few) merged in this single file.

Please let me know the revision you used as a base for marshal3.diff, or - even better - rebase it to latest revision (you need to do this anyway) and repost the new .diff.

1. NAMESPACE-URI and NAMESPACE-PREFIX DATASET attributes were not (de)serialized at all in the remote calls (fixed)
2. The problem with external XML schema which is created when the table namespace is different from the dataset one is more difficult to resolve than I initially expected (was not implemented, now mostly done for WRITE-XMLSCHEMA).

This means that my initial estimation (1-2 days) was not accurate. Now I think that I need at least two days more.

Completely (as far as my tests demonstrate) fixed WRITE-XMLSCHEMA for standalone tables and datasets, including external schema creation and min-xmlschema/omit-initial-values flags and processing of the SCHEMA-MARSHAL for datasets (this was tricky since it was lost in several points on its way from the caller to the callee).

What is still to be done is adding prefixes to WRITE-XML output for the datasets (should be easy) and implementing support for external schemas in READ-XMLSCHEMA.

Igor Skornyakov wrote:

I work with branch 6129b. Please find the .diff for rev.14320 attached.

• general code aspect. My OCD kicks in: I am aware that this is not affect performance but please let an empty line between field declarations and between method definitions. And no empty lines between import statements.
• TempTableResultSet:387: you are creating an intermediary mprops tree for collecting PropertyDefinition s. Why not collecting then directly into the lprops list? Because we try to maximize performance, a good old while instead of the forEachRemaining might be a better choice;
• PropertyDefinition externalization: the serializeHidden, caseSensitive, and full flags can be grouped as bits in a single byte;
• DsTableDefinition
  • line 526: I think the line can be better rewritten as:

    return properties != null ? new ArrayList<>(properties) : Collections.emptyList();

  • writeExternal() and readExternal() methods. The code is badly indented caused by melding sources maybe? The logic for objectProps seem unfinished;
• TemporaryBuffer:2616 and DataSet:869: Throwing error 12323 is really dry for the code reader. Please add a comment with the actual text message of the error;
• TempTableBuilder: I preferred the previous approach of static factory methods. Probably it's just a matter of taste. Please update the missing information in javadoc of the new addNewField() method.
• TableWrapper:601, see note about DsTableDefinition:526;
• DmoMeta: the idea for delaying evaluation for date-related literals is interesting, but adds a bit of insecurity: if some code will read the property type and try to cast the initial value to expected class will result in a CCE. At least this must be documented in P2JField.

Ovidiu Maxiniuc wrote:

Igor Skornyakov wrote:

I work with branch 6129b. Please find the .diff for rev.14320 attached.

Review of marshal4.diff
Overall: a big update, handling multiple issues. Nice work. Here are my comments:

• general code aspect. My OCD kicks in: I am aware that this is not affect performance but please let an empty line between field declarations and between method definitions. And no empty lines between import statements.
• TempTableResultSet:387: you are creating an intermediary mprops tree for collecting PropertyDefinition s. Why not collecting then directly into the lprops list? Because we try to maximize performance, a good old while instead of the forEachRemaining might be a better choice;
• PropertyDefinition externalization: the serializeHidden, caseSensitive, and full flags can be grouped as bits in a single byte;
• DsTableDefinition
  • line 526: I think the line can be better rewritten as:[...]
  • writeExternal() and readExternal() methods. The code is badly indented caused by melding sources maybe? The logic for objectProps seem unfinished;
• TemporaryBuffer:2616 and DataSet:869: Throwing error 12323 is really dry for the code reader. Please add a comment with the actual text message of the error;
• TempTableBuilder: I preferred the previous approach of static factory methods. Probably it's just a matter of taste. Please update the missing information in javadoc of the new addNewField() method.
• TableWrapper:601, see note about DsTableDefinition:526;
• DmoMeta: the idea for delaying evaluation for date-related literals is interesting, but adds a bit of insecurity: if some code will read the property type and try to cast the initial value to expected class will result in a CCE. At least this must be documented in P2JField.

Thank you, Ovidiu.
I will address this issue together with new changes since most of the mentioned classes have been already changed.

Fixed WRITE-XML support for DATASET

Igor Skornyakov wrote:

• TempTableResultSet:387: you are creating an intermediary mprops tree for collecting PropertyDefinition s. Why not collecting then directly into the lprops list? Because we try to maximize performance, a good old while instead of the forEachRemaining might be a better choice;

This is done to ensure the correct ordering of lprops, which was initially implemented differently and (as far as understand) less efficiently.

I looked now at neighbour code from the same method. It seems to be that the build of lprops is a bit overly complicated. The TempTableBuilder.getOrderedPropertyNames() should return the properties in the correct order. Or we could use DmoMeta.getFields(false), but we need to be careful for the extent fields.

• writeExternal() and readExternal() methods. The code is badly indented caused by melding sources maybe? The logic for objectProps seem unfinished;

The situation with objectProps looks strange. I could not add this myself, but I do not see this in the repo version of the code anymore.Should I remove it as well?

I see similar code in TableWrapper.writeExternal(). Maybe that's the source?

• DmoMeta: the idea for delaying evaluation for date-related literals is interesting, but adds a bit of insecurity: if some code will read the property type and try to cast the initial value to expected class will result in a CCE. At least this must be documented in P2JField.

Sorry, I do not understand your point about security. Anyway, without delayed evaluation, the assigned default values will be simply incorrect.

What I meant was that a ClassCastException could be thrown if code inside FWD will cast the BaseDataType initial value to date instead of using assign() method. I agree that the old code behaves incorrectly in that the dynamic values was evaluated at the wrong time.

Igor Skornyakov wrote:

See a new .diff file for 6129b.14320 attached. It also contains changes described in #6453-162.

• DsTableDefinition:199: It seems strange that SM_FULL and SM_MIN will retain the same amount of information. And SM_DEFAULT less than SM_MIN. But if this is what the tests revealed we have to live with this.
• DataSet:2369: I think this is logically incorrect and I wonder if compilable in Java. I guess you meant || instead of &; OTOH, the parameter is not expected to be null here;
• DataSet:3921: in what conditions can buf be null here? It seems to me like a flaw at the caller instead;
• XmlExport:
  • line 675: this is a bit strange construct. I would expect to be written much simpler as:

    ErrorManager.recordOrShowError(13095, false)

    but I see this pattern in other places, too. If not modal is the problem please add an overloaded method in ErrorManager like this:

    public static void recordOrShowError(int id, boolean isError, boolean isModal, String... params)

    and call it instead;
  • line 1103 & other: adding this conditional might introduce imbalance between the element tags (and indentation). Intentionally, this method (and other similar) are using simple code blocks to mimic the tags in output stream and make it easier to debug. The opening and closing tags must go in pairs, according to same conditions. Opening a tag/indentation in other method breaks the structured code;
  • generally, there are a lot of changes here. I cannot check if all are correct without proper testing (do not have the time now), so I assume you made the changes based on your set of testcases.

Ovidiu Maxiniuc wrote:

Igor Skornyakov wrote:

• TempTableResultSet:387: you are creating an intermediary mprops tree for collecting PropertyDefinition s. Why not collecting then directly into the lprops list? Because we try to maximize performance, a good old while instead of the forEachRemaining might be a better choice;

This is done to ensure the correct ordering of lprops, which was initially implemented differently and (as far as understand) less efficiently.

I looked now at neighbour code from the same method. It seems to be that the build of lprops is a bit overly complicated. The TempTableBuilder.getOrderedPropertyNames() should return the properties in the correct order. Or we could use DmoMeta.getFields(false), but we need to be careful for the extent fields.

• writeExternal() and readExternal() methods. The code is badly indented caused by melding sources maybe? The logic for objectProps seem unfinished;

The situation with objectProps looks strange. I could not add this myself, but I do not see this in the repo version of the code anymore.Should I remove it as well?

I see similar code in TableWrapper.writeExternal(). Maybe that's the source?

• DmoMeta: the idea for delaying evaluation for date-related literals is interesting, but adds a bit of insecurity: if some code will read the property type and try to cast the initial value to expected class will result in a CCE. At least this must be documented in P2JField.

Sorry, I do not understand your point about security. Anyway, without delayed evaluation, the assigned default values will be simply incorrect.

What I meant was that a ClassCastException could be thrown if code inside FWD will cast the BaseDataType initial value to date instead of using assign() method. I agree that the old code behaves incorrectly in that the dynamic values was evaluated at the wrong time.

Igor Skornyakov wrote:

See a new .diff file for 6129b.14320 attached. It also contains changes described in #6453-162.

• DsTableDefinition:199: It seems strange that SM_FULL and SM_MIN will retain the same amount of information. And SM_DEFAULT less than SM_MIN. But if this is what the tests revealed we have to live with this.
• DataSet:2369: I think this is logically incorrect and I wonder if compilable in Java. I guess you meant || instead of &; OTOH, the parameter is not expected to be null here;
• DataSet:3921: in what conditions can buf be null here? It seems to me like a flaw at the caller instead;
• XmlExport:
  • line 675: this is a bit strange construct. I would expect to be written much simpler as:[...]but I see this pattern in other places, too. If not modal is the problem please add an overloaded method in ErrorManager like this:[...]and call it instead;
  • line 1103 & other: adding this conditional might introduce imbalance between the element tags (and indentation). Intentionally, this method (and other similar) are using simple code blocks to mimic the tags in output stream and make it easier to debug. The opening and closing tags must go in pairs, according to same conditions. Opening a tag/indentation in other method breaks the structured code;
  • generally, there are a lot of changes here. I cannot check if all are correct without proper testing (do not have the time now), so I assume you made the changes based on your set of testcases.

Thank you, Ovidiu.
I will review your comments at the weekend.

Ovidiu Maxiniuc wrote:

• TempTableResultSet:387: you are creating an intermediary mprops tree for collecting PropertyDefinition s. Why not collecting then directly into the lprops list? Because we try to maximize performance, a good old while instead of the forEachRemaining might be a better choice;

This is done to ensure the correct ordering of lprops, which was initially implemented differently and (as far as understand) less efficiently.

I looked now at neighbour code from the same method. It seems to be that the build of lprops is a bit overly complicated. The TempTableBuilder.getOrderedPropertyNames() should return the properties in the correct order. Or we could use DmoMeta.getFields(false), but we need to be careful for the extent fields.

Well, I think that you're right. However, I suggest postponing the optimization. Let's do it when the changes will be committed and no regression will be revealed. We already have a lot of pended changes.

• writeExternal() and readExternal() methods. The code is badly indented caused by melding sources maybe? The logic for objectProps seem unfinished;

The situation with objectProps looks strange. I could not add this myself, but I do not see this in the repo version of the code anymore.Should I remove it as well?

I see similar code in TableWrapper.writeExternal(). Maybe that's the source?

Indeed, I do not see objectProps processing even in 60129a, so it looks like you're right. The question is should I remove it? After all, I understand that DsTableDefinition and TableWrapper are essentially about the same. My guess is that objectProps processing was just missed in the first one.

• DmoMeta: the idea for delaying evaluation for date-related literals is interesting, but adds a bit of insecurity: if some code will read the property type and try to cast the initial value to expected class will result in a CCE. At least this must be documented in P2JField.

Sorry, I do not understand your point about security. Anyway, without delayed evaluation, the assigned default values will be simply incorrect.

What I meant was that a ClassCastException could be thrown if code inside FWD will cast the BaseDataType initial value to date instead of using assign() method. I agree that the old code behaves incorrectly in that the dynamic values was evaluated at the wrong time.

I understand this. However, we can do nothing about this. A good thing is that these kinds of bugs manifest themselves in a very clear way so can be easily detected a fixed.

Igor Skornyakov wrote:

See a new .diff file for 6129b.14320 attached. It also contains changes described in #6453-162.

• DsTableDefinition:199: It seems strange that SM_FULL and SM_MIN will retain the same amount of information. And SM_DEFAULT less than SM_MIN. But if this is what the tests revealed we have to live with this.

According to my tests, SM_MIN only reduces the number of field attributes and information about indexes. The information about the table itself remains the same

• DataSet:2369: I think this is logically incorrect and I wonder if compilable in Java. I guess you meant || instead of &; OTOH, the parameter is not expected to be null here;

Indeed, it was a typo. Thank you. Strangely enough, it compiles. Fixed now.

• DataSet:3921: in what conditions can buf be null here? It seems to me like a flaw at the caller instead;

This happens when the corresponding table has NONE value of the #6453-150

• XmlExport:
  • line 675: this is a bit strange construct. I would expect to be written much simpler as:[...]but I see this pattern in other places, too. If not modal is the problem please add an overloaded method in ErrorManager like this:[...]and call it instead;

Done.

• line 1103 & other: adding this conditional might introduce imbalance between the element tags (and indentation). Intentionally, this method (and other similar) are using simple code blocks to mimic the tags in output stream and make it easier to debug. The opening and closing tags must go in pairs, according to same conditions. Opening a tag/indentation in other method breaks the structured code;

Indeed, manual formatting is difficult to maintain and debug. But I had to add conditionals to achieve compatibility with 4GL.
I believe that it was not a good idea to perform formatting manually, especially for such a complicated document as XML Schema. Maybe it is better to use Transformer like described here: https://mkyong.com/java/how-to-write-xml-file-in-java-stax-writer/#write-and-pretty-print-xml-content-transformer? This will greatly simplify out code. If we will use this approach for schemas the additional overhead should not be that big since table/dataset schema typically is a small document. Of course, this is a simple but massive change and it is better to do it in a separate patch after the current changes will be committed.

• generally, there are a lot of changes here. I cannot check if all are correct without proper testing (do not have the time now), so I assume you made the changes based on your set of testcases.

Of course, I did. All my changes are done based on my tests and tested with these tests. However, I understand that it does not guarantee that the new version of the code is 100% correct, but the same can be said about the current version )).

Do you think that I can commit the changes now?
Thank you.

In addition to the previous note. I
Why not extend XMLStreamWriter so that it will care about indentation itself? This will reduce the number of indent() calls and ndentLevel adjustments dramatically. After all, these actions are required only on writeStartElement/writeEndElement calls.
To ensure that elements are properly nested/closed we can add a single method
public void writeElement(String name, ElementBuilder elementBuilder) where ElementBuilder is a functional interface with a single method (public void build()).
After that, we can refactor the code moving the logic between writeStartElement and writeEndElement calls into a lambda expression and passing it to the writeElement.
I believe that it will be much less error-prone than the current approach base just on the code formatting because the compiler will validate the document structure.
What do you think?
Thank you.

Igor Skornyakov wrote:

Why not extend XMLStreamWriter so that it will care about indentation itself? [...] What do you think?

Yes, I also thought of some similar manner of fixing the issue. The current blocks will be kept and bound the lambdas. In fact that's one of the reasons I added them. IIRC, the problem was at that time we were not sure of the how the usage of lambdas affect the performance so I did not make the final step.

I think we should handle this in a separate task. Please add one in the Database project. We probably need a similar class (if not the same) for the JSON serialization where the data is pretty much the same, but simpler (no schema, different, simpler syntax).

Ovidiu Maxiniuc wrote:

Igor Skornyakov wrote:

Why not extend XMLStreamWriter so that it will care about indentation itself? [...] What do you think?

Yes, I also thought of some similar manner of fixing the issue. The current blocks will be kept and bound the lambdas. In fact that's one of the reasons I added them. IIRC, the problem was at that time we were not sure of the how the usage of lambdas affect the performance so I did not make the final step.

I think we should handle this in a separate task. Please add one in the Database project. We probably need a similar class (if not the same) for the JSON serialization where the data is pretty much the same, but simpler (no schema, different, simpler syntax).

Thank you, Ovidiu.
I will create the task later.

Igor Skornyakov wrote:

What I meant was that a ClassCastException could be thrown if code inside FWD will cast the BaseDataType initial value to date instead of using assign() method. I agree that the old code behaves incorrectly in that the dynamic values was evaluated at the wrong time.

I understand this. However, we can do nothing about this. A good thing is that these kinds of bugs manifest themselves in a very clear way so can be easily detected a fixed.

Indeed.

Of course, I did. All my changes are done based on my tests and tested with these tests. However, I understand that it does not guarantee that the new version of the code is 100% correct, but the same can be said about the current version )).
Do you think that I can commit the changes now?

Certainly, the patch is clearly an improvement over the current revision on bzr. Please commit it.

Ovidiu Maxiniuc wrote:

Certainly, the patch is clearly an improvement over the current revision on bzr. Please commit it.

Thank you.
Committed to 6129b/14322.

It appears that READ-XMLSCHEMA for the DATASET is broken even in the situation of a single namespace (w/o external schemas).
Working on the fix.

• AbstractTempTable: the new method lacks the javadoc;
• XmlImport:
  • method removeNsPrefix, addChildField(): please separate the @param tags by an empty line from the main description and the @return tag (if present), inside the method's javadoc;
  • method processDsBuffers: I think ds.setBuffers(dsBuffers); line should be moved after the for block. OTOH, looking at the code, I understand that it tries to add each first buffer of the tables to the dataset. Most likely, these are the default buffers and they can be accessed directly, using AbstractTempTable.defaultBuffer().

Tests used for this task committed to xfer.goldencode.com/opt/testcases/temp-table-marshal. rev. 1300.

Only one change before the commit, please: in XmlImport.java, instead of

Buffer tbuffer = ttb.defaultBufferHandle().unwrapBuffer();

Buffer tbuffer = (Buffer) ttb.defaultBufferHandle().getResource();

We know the type of the result, so we can avoid the costly ContextLocal.get().

Ovidiu Maxiniuc wrote:

Only one change before the commit, please: in XmlImport.java, instead of [...]use[...]

We know the type of the result, so we can avoid the costly ContextLocal.get().

Fixed.

Committed to 6129b/14326.

• how safe is to abend here?
  

                            ttb = extenalSchemas.get(tableRef);
                            if (ttb == null)
                            {
                               throw new IllegalStateException("Unknown table reference: [" 
                                                                + tableRef + "]");
                            }
  

• how safe is to use server-side check if a file exists instead of going through the normal lookup process? Here I mean the schema can be a file in the jar, on client-side system, etc.
  

  String extPath = !Serializator.TYPE_FILE.equals(
                                   source.getType().toUpperCase()) ? null : 
                                   Paths.get(source.getFileName()).getParent().toString() + 
                                                                    "/" + schemaLocation;
  

• this is leaking TempTableBuilder instances:
  

                          TempTableBuilder ttb = new TempTableBuilder();
                          if (!ext.importTempTableSchema(ttb)) 
                          {
                             return false;  // delete ttb...
                          }
  

how safe is to abend here?

I think that it is safe because actually, it should never happen if the schema is correct and the reference matches already processed import..

We can't know what future changes may expose this as a valid code path. For example, in #6371, we are allowing SQL DDL changes at the database to be read back in and used to rewrite DMOs in memory.

As a rule, please only ever use legacy compatible exceptions.

Greg Shah wrote:

how safe is to abend here?

I think that it is safe because actually, it should never happen if the schema is correct and the reference matches already processed import..

We can't know what future changes may expose this as a valid code path. For example, in #6371, we are allowing SQL DDL changes at the database to be read back in and used to rewrite DMOs in memory.

As a rule, please only ever use legacy compatible exceptions.

I've simulated incorrect schema with my 4GL test and replaced throw new IllegalStateException with the correct 4GL warning.
Committed to 6129b/14329.

Igor,

Related to 6129b/14329: after ErrorManager.recordOrShowError do not throw a new exception. After the 4GL condition is recorded the exception, just return false to signal the method failure. The event was already processed. Throwing a new exception will cause the calling method to 'recordOrShow' with a new 4GL condition. Alternatively, you can rely on the caller to raise a 4GL condition, but I see 3 of them already added.

• how safe is to use server-side check if a file exists instead of going through the normal lookup process? Here I mean the schema can be a file in the jar, on client-side system, etc.

I do not check that file exists here. The external schema should be located alongside the 'root' one and I just calculate its location. How this code should be modified to work with schemas in a jar file?

What Constantin means here is that your processing expects the file to be on server-side file system. In the majority of cases, customer's environment this will not be true. The file is usually on client side, or - if it is a stable file and expected to be there - added to build project and deployed inside the application jar. This means the file should be either loaded as a com.goldencode.p2j.util.Stream (see the SourceData for FILE case)or using or ClassLoader.getResourceAsStream. This just crossed my mind: is it possible that this resource to be an URL? Sorry for missing this aspect in my review.

• this is leaking TempTableBuilder instances:
  [...]

Sorry, what do you mean? I think that I saw a lot of similar places in the XmlImport. What should be done here to avoid leaking?

The problem is that TempTableBuilder -s are HandleChain objects which are kept in a linked list of objects iterable from 4GL (see FIRST-*** of SESSION handle). This is a latent bug in the code. To drop the tables which should not survive after a failed deserialization, please iterate this.tables and call (HandleChain.) delete() on them in that event. And this particular case when the object is not added to the list before returning.

Constantin, thank you for your keen eyes and valuable input!

Ovidiu Maxiniuc wrote:

Related to 6129b/14329: after ErrorManager.recordOrShowError do not throw a new exception. After the 4GL condition is recorded the exception, just return false to signal the method failure. The event was already processed. Throwing a new exception will cause the calling method to 'recordOrShow' with a new 4GL condition. Alternatively, you can rely on the caller to raise a 4GL condition, but I see 3 of them already added.

In fact, the caller of this method just returns false on XmlStreamException. Since this exception can be thrown in multiple places I decided to use it in case we decide to log such exceptions in the future. But of course, it can be replaced with just return false. I've tested that no additional error messages are added.

• how safe is to use server-side check if a file exists instead of going through the normal lookup process? Here I mean the schema can be a file in the jar, on client-side system, etc.

I do not check that file exists here. The external schema should be located alongside the 'root' one and I just calculate its location. How this code should be modified to work with schemas in a jar file?

What Constantin means here is that your processing expects the file to be on server-side file system. In the majority of cases, customer's environment this will not be true. The file is usually on client side, or - if it is a stable file and expected to be there - added to build project and deployed inside the application jar. This means the file should be either loaded as a com.goldencode.p2j.util.Stream (see the SourceData for FILE case)or using or ClassLoader.getResourceAsStream. This just crossed my mind: is it possible that this resource to be an URL? Sorry for missing this aspect in my review.

Sorry, I do not understand. I've tested that XML-WRITESCHEMA creates a file in the deploy/client and XML-READSCHEMA reads data from this folder as well. What is the 4GL counterpart for reading data from a .jar file? And how can I figure out the location of the external schema in such a situation?

• this is leaking TempTableBuilder instances:
  [...]

Sorry, what do you mean? I think that I saw a lot of similar places in the XmlImport. What should be done here to avoid leaking?

The problem is that TempTableBuilder -s are HandleChain objects which are kept in a linked list of objects iterable from 4GL (see FIRST-*** of SESSION handle). This is a latent bug in the code. To drop the tables which should not survive after a failed deserialization, please iterate this.tables and call (HandleChain.) delete() on them in that event. And this particular case when the object is not added to the list before returning.

OK, I will double-check. But I've not noticed this logic in other parts of the code where instances of the TempTableBuilder are created. Maybe I've just overlooked this.

Ovidiu Maxiniuc wrote:

The problem is that TempTableBuilder -s are HandleChain objects which are kept in a linked list of objects iterable from 4GL (see FIRST-*** of SESSION handle). This is a latent bug in the code. To drop the tables which should not survive after a failed deserialization, please iterate this.tables and call (HandleChain.) delete() on them in that event. And this particular case when the object is not added to the list before returning.

Sorry, but I'm confused. We create a new TempTableBuilder instance e.g. in the readTableSchema(String tableName, String nsPrefix) which can fail, but I do not see any cleanup. Why values of the XmlImport.tables map are so special compared to the local TempTableBuilder instance in this method?
In fact, I've not found any calls to the TempTableBuilder.delete() except resourceDelete(). Why do we need special actions in this only situation?
Please clarify.
Thank you.

Igor Skornyakov wrote:

Sorry, but I'm confused. We create a new TempTableBuilder instance e.g. in the readTableSchema(String tableName, String nsPrefix) which can fail, but I do not see any cleanup.

Please see my note: This is a latent bug in the code. It have passed unnoticed until now and this is the chance to fix it by adding the missing clean-up code in case of error.

Why values of the XmlImport.tables map are so special compared to the local TempTableBuilder instance in this method?

It is not. Just that there is an additional if which can cause to return to caller without adding the newly created object to tables collection and missed to be cleaned together with the other temp-table objects, so it might leak.

In fact, I've not found any calls to the TempTableBuilder.delete() except resourceDelete(). Why do we need special actions in this only situation?

This class is unique in the fact that we have control of the short lifetime of the TempTableBuilder in case of an error. Normally, these objects are created as result of CREATE TEMP-TABLE statement, and it's the responsibility of the ABL programmer to delete the objects when not needed any more. They will do this by using DELETE OBJECT statement.

Note: I think there is a secondary place where the lifetime of these objects may need improvements: when the tables are deserialized from a remote temp-table parameter. Constantin, please confirm this.

Ovidiu Maxiniuc wrote:

This class is unique in the fact that we have control of the short lifetime of the TempTableBuilder in case of an error. Normally, these objects are created as result of CREATE TEMP-TABLE statement, and it's the responsibility of the ABL programmer to delete the objects when not needed any more. They will do this by using DELETE OBJECT statement.

In fact, I do not understand why we add TempTableBuilder to the HandleChain. I have not found the SESSION FIRST-... method for accessing temporary tables either in 4GL documentation or in SessionUtils.

The corresponding buffers are created in the XmlImport.processDsBuffers() method which is called only after successful schema processing.

I've added iteration over the buffers with logging to my tests and the results are the same in 4GL and FWD both for successful and failed schema read.
However, I've noticed some minor issues and fixed them.

Committed to 6129b/14330.

BTW: It looks the current implementation of the HandleChain logic is pretty fragile because it is difficult to guarantee a complete cleanup in case of exception, especially if one is not caught in the method where the chained entity was created.
Don't you think that it makes sense to add a kind of 'transactional' logic to enable a 'batch' cleanup in a single call from the catch or finally block or even as a part of stack unwinding?
Thank you.

Igor Skornyakov wrote:

In fact, the caller of this method just returns false on XmlStreamException. Since this exception can be thrown in multiple places I decided to use it in case we decide to log such exceptions in the future. But of course, it can be replaced with just return false. I've tested that no additional error messages are added.

Actually, if we replace throwing XmlStreamException with return false this will result in incompatible behavior because the caller treats it as a validation error and adds an additional message.

Igor Skornyakov wrote:

In fact, I do not understand why we add TempTableBuilder to the HandleChain. I have not found the SESSION FIRST-... method for accessing temporary tables either in 4GL documentation or in SessionUtils.

You are right. I cannot give you a straight answer: the logical things to do with dynamic objects is to add them in the list/chain. I assumed, based on existing code that such list exists in 4GL.
Eric, what should we do: remove this list or expose this as a enhanced feature with a method in SESSION handle?
Constantin, I think you added the TEMP-TABLE list as a new entry in HandleChain.WorkArea.firstResource about 10 years ago. Is there a reason to keep it?

Igor Skornyakov wrote:

Actually, if we replace throwing XmlStreamException with return false this will result in incompatible behavior because the caller treats it as a validation error and adds an additional message.

Is the last error message doubled?
I've seen similar behaviour in 4GL: the error messages being added while execution thread backs to calling internal methods. So it is not necessary to emit all 3 errors in same place, just the most appropriate and signal the caller method to add its own errors.

Ovidiu Maxiniuc wrote:

Actually, if we replace throwing XmlStreamException with return false this will result in incompatible behavior because the caller treats it as a validation error and adds an additional message.

Is the last error message doubled?

No, an additional message (which I do not see with 4GL) is added.

Eric, what should we do: remove this list or expose this as a enhanced feature with a method in SESSION handle?

Let's remove it. It is extra work, extra memory that is not needed.

Merged to the trunk rev. 14526.