FWD » Core Development » Base Language

During testing a customer's application, it was found that this method does not behave the same as in Progress.

DEFINE TEMP-TABLE ttMain NO-UNDO SERIALIZE-NAME "Main" 
   FIELD field1 AS CHARACTER SERIALIZE-NAME "field1" 
   FIELD field2 AS CHARACTER
   FIELD parentId AS RECID SERIALIZE-HIDDEN.

DEFINE TEMP-TABLE ttSecondary NO-UNDO SERIALIZE-NAME "Secondary" 
   FIELD field1 AS CHARACTER SERIALIZE-NAME "field1" 
   FIELD field2 AS CHARACTER SERIALIZE-NAME "field2" 
   FIELD anotherField3 AS CHARACTER SERIALIZE-NAME "field3" 
   FIELD parentId AS RECID SERIALIZE-HIDDEN.

DEFINE DATASET ds SERIALIZE-HIDDEN FOR ttMain, ttSecondary
   PARENT-ID-RELATION FOR ttMain, ttSecondary PARENT-ID-FIELD parentId.

DEF VAR lcRes AS LONGCHAR VIEW-AS EDITOR SIZE 75 BY 15 LARGE.
DEF VAR content AS CHARACTER NO-UNDO.
lcRes = '~{ "Main" : [ ~{ "Secondary" : [ ~{ "field1" : "value1", "field2" : "value2" ~}, ~{ "field1" : "value3", "field2" : "value4" ~} ] ~} ] ~}'.

//CURRENT-WINDOW:WIDTH-CHARS = 128.
//CURRENT-WINDOW:HEIGHT-CHARS = 32.
//DISPLAY lcRes.

DATASET ds:READ-JSON(STRING(Progress.Reflect.DataType:LONGCHAR), lcRes, "EMPTY").

FOR FIRST ttMain:
   FOR EACH ttSecondary:
      DISPLAY ttSecondary. //Should print something
   END.
END.

I was investigating the reason for which the parentId field of the ttSecondary temp-table record was not updating with the id of the ttMain temp-table record when loading the dataset. For now, the scope for this task is to fix this synchronization issue. But in order to do this, some tweaking needs to be done for the READ-JSON method because for the program above it cannot even load the contents of the lcRes variable into the dataset. This works perfectly fine in progress.

FOREIGN-KEY-HIDDEN means that fk is calculated automatically (i.e. it knows that the parent's field is used as key in the child record, and sets it automatically in the child record's fk field). If this is missing, then that FK must exist in the .xml, right (for read or write)? Otherwise the child would not know how to map the record with its parent.

To avoid confusion, please also do not use the same field names in the parent and child tables. In your example, ttmain.parentid is not even required, as ttsecondary.parentid will match with the PK of the ttmain record.

Otherwise, for PARENT-ID-RELATION, we have a field in the child table which is automatically set to the parent's recid. And it looks that this is done regardless if serialize-hidden is set at this child's field - even if it appears in the json, it is ignored, as it needs to be an actual PK.

It just the fact that the correspondent of the foreign keys for temp-tables (I don't know if temp-tables have foreign keys), the fields declared using the PARENT-ID-RELATION are not satisfied.

What do you mean by this?

Constantin Asofiei wrote:

If this is missing, then that FK must exist in the .xml, right (for read or write)? Otherwise the child would not know how to map the record with its parent.

I have talked to Teodor about this and when parsing an .xml Progress uses its structure to understand how records should be tied together. This means that the FK doesn't need to exist at all. Teodor please correct me if I'm wrong but I think that while reading an .xml file this field is also ignored as it is for .json .

To avoid confusion, please also do not use the same field names in the parent and child tables. In your example, ttmain.parentid is not even required, as ttsecondary.parentid will match with the PK of the ttmain record.

Yes, sorry it was left by mistake.

It just the fact that the correspondent of the foreign keys for temp-tables (I don't know if temp-tables have foreign keys), the fields declared using the PARENT-ID-RELATION are not satisfied.

What do you mean by this?

There are 2 types of relations that can be declared in a dataset, PARENT-ID-RELATION and DATA-RELATION. I found the code above which looks like it updates some fields in the child record based on the parent record but it only does this if the relation type is DATA-RELATION and is declared with FOREIGN-KEY-HIDDEN. I thought that this is meant for id's like in my case and I was wondering why is this done only for the FOREIGN-KEY-HIDDEN ones.

I've analyzed the method a little better now and it looks like it's not about ids or recids but fields in general.

Constantin Asofiei wrote:

Paul, the problems are multiple:

• childFieldsByChild has as key the temp-table name, and not the serialization name of the table/buffer. In setParentValues, I think this needs to receive as parameter the buffer, and not the serialized name (i.e. resolve the buffer before setParentValues call)

There is a slight problem with this approach. If we have a situation like this:

DEFINE TEMP-TABLE ttMain1 NO-UNDO SERIALIZE-NAME "Main" 
   FIELD field1 AS CHARACTER SERIALIZE-NAME "field1" 
   FIELD field2 AS CHARACTER.
DEFINE TEMP-TABLE ttMain2 NO-UNDO SERIALIZE-NAME "Main" 
   FIELD field1 AS CHARACTER SERIALIZE-NAME "field1" 
   FIELD field2 AS CHARACTER.
DEFINE DATASET ds SERIALIZE-HIDDEN FOR ttMain1, ttMain2, ttSecondary
   PARENT-ID-RELATION FOR ttMain1, ttSecondary PARENT-ID-FIELD parent1Id
   PARENT-ID-RELATION FOR ttMain2, ttSecondary PARENT-ID-FIELD parent2Id.

Then the entry in the childFieldsByChild will get overriden each time and only the last one will remain because the key is the same buffer.

We could go around this by checking if the key exist first and if it is then append to the list but we need to change the structure of the childFieldsByChild's value.

I'm also having trouble understanding the existing implementation. For example, fields that are supposed to be populated from the parent are not initialized in the setParentValues method like the documentation says, but just registered along with their default value. The fields are then set to this value in the fillFromParent method.

Should I replace the implementation and make it more easier to understand?

Committed rev. 16130 on branch 10352a that allows the solution to be compatible with java 8.

Committed rev. 16132 on branch 10352a that introduces a check to avoid NPE for cases when the structure of the payload being read is not the one expected.

There is another problem; note that the following is happening on an READ-XML - ctx.proxy has no record loaded:

                  try
                  {
                     ctx.parentValues.put(childFields.get(i), column.getGetter().invoke(ctx.proxy));
                  }
                  catch (IllegalAccessException | InvocationTargetException e)
                  {
                     LOG.severe("", e);
                     return false;
                  }

Comitted rev. 16133 on branch 10352a that addresses this issue.

Hi Paul, I took a look at 10352a by chance.
Please keep Java 8 compatibility in mind for now:
(If you use Intellij you can go to File - Project Structure - Project - Under Project language level, select 8 – Lambdas, type annotations...)

[ant:javac] /home/sp/gcd/branches/10352a/src/com/goldencode/p2j/persist/DataRelation.java:1089: error: cannot find symbol
[ant:javac]       return Arrays.stream(ret).toList();

Stefanel Pezamosca wrote:

Hi Paul, I took a look at 10352a by chance.
Please keep Java 8 compatibility in mind for now:
(If you use Intellij you can go to File - Project Structure - Project - Under Project language level, select 8 – Lambdas, type annotations...)

Thank you, I committed revision 16135 with the fix.

Also, was the change from String[] to ArrayList necessary?

I wouldn't say necessary but it really simplifies the code a lot and it's more convenient especially for the processRelationFlds method of the Importer class. We could go around it but it would introduce unnecessary complexity. Aside from this, the result was also transformed to a List in JsonExport and XmlExport.

Harness, smoketest and reports passed with 10352a.

Smoke-test for 10352a on another customer app showed no regressions.

There is a unit test from customer application, which fails because of a RECID field.
Please see this test case, which reproduces the issue: ABL CodeABL Code

BLOCK-LEVEL ON ERROR UNDO, THROW.

USING OpenEdge.Core.Assert FROM PROPATH.

CLASS tests.TestReadXmlRecid:
    DEFINE TEMP-TABLE tt1 NO-UNDO
        FIELD name AS CHARACTER.

    DEFINE TEMP-TABLE tt2 NO-UNDO
        FIELD name AS CHARACTER
        FIELD parentId AS RECID.

    DEFINE DATASET ds FOR tt1, tt2
        PARENT-ID-RELATION REL1 FOR tt1, tt2 PARENT-ID-FIELD parentId.

    @Setup.
    METHOD PUBLIC VOID PopulateTempTable():
        DEFINE VARIABLE recordId AS RECID NO-UNDO.
        DEFINE VARIABLE goodRecordId AS RECID NO-UNDO.

        CREATE tt1.
        tt1.name = "Not in dataset".
        recordId = RECID(tt1).

        CREATE tt1.
        tt1.name = "Into the dataset".
        goodRecordId = RECID (tt1).

        FIND FIRST tt1 WHERE RECID (tt1) = recordId.
        DELETE tt1.

        CREATE tt2.
        tt2.name = "Into the dataset tt2.".
        tt2.parentId = goodRecordId.
    END METHOD.

    @Test.
    METHOD PUBLIC VOID TestWriteReadXml():
        DEFINE VARIABLE cLongChar AS LONGCHAR NO-UNDO.
        DATASET ds:WRITE-XML ("longchar", cLongChar).

        DATASET ds:READ-XML ("longchar", cLongChar, "EMPTY", ?, ?).

        FIND FIRST tt1.
        Assert:Equals("Into the dataset", tt1.name).

        FIND FIRST tt2 WHERE tt2.parentId = RECID(tt1).
        Assert:Equals("Into the dataset tt2.", tt2.name).
    END METHOD.

    @Teardown.
    METHOD PUBLIC VOID TearDown():
        DATASET ds:EMPTY-DATASET ().
    END METHOD.
END CLASS.

• before calling WRITE-XML, in the temp-tables, there are 2 records: tt1 "Into the dataset" with RECID 2305 and tt2 "Into the dataset tt2." with RECID 2304, parentID 2305 (I am not using the H2 changes from #10519, but the issue still persists with it).
• the longchar contents is right: XML contentsXML contents

  <?xml version="1.0" encoding="UTF-8" standalone="no"?>
  <ds xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
      <tt1>
          <name>Into the dataset</name>
          <tt2>
              <name>Into the dataset tt2.</name>
              <parentId>2305</parentId>
          </tt2>
      </tt1>
  </ds>
  

• after calling the READ-XML, the temp-table contents are the following: tt1 "Into the dataset" with RECID 2304, tt2 "Into the dataset tt2." with RECID 2304, parentID 2305
• as you may notice, the parent table RECID is not set.

• skipping the parent RECID field on child records.
• actually setting the RECID field on the parent table.

Teodor, please explain more: usually PARENT-ID-RELATION are hidden (thus computed automatically), as the recid is not considered deterministic between multiple imports of the same .xml file. So, what exactly is the problem after calling READ-XML? What does 4GL do? You can try to 'pollute' the temp-table PKs in OpenEdge by creating some records before READ-XML, and see what happens, so the '2305' value from .xml file does not match any real PK in the parent.

Paul, one more fix: Importer.processRelationFlds has the same dependency on serializeName; please fix this.

Constantin Asofiei wrote:

Teodor, please explain more: usually PARENT-ID-RELATION are hidden (thus computed automatically), as the recid is not considered deterministic between multiple imports of the same .xml file. So, what exactly is the problem after calling READ-XML? What does 4GL do? You can try to 'pollute' the temp-table PKs in OpenEdge by creating some records before READ-XML, and see what happens, so the '2305' value from .xml file does not match any real PK in the parent.

OE just ignores this field at parsing and populates this field with a new RECID value, as you tell, but FWD parses this field and sets this wrong value into the parentId field.

Teodor Gorghe wrote:

Constantin Asofiei wrote:

Teodor, please explain more: usually PARENT-ID-RELATION are hidden (thus computed automatically), as the recid is not considered deterministic between multiple imports of the same .xml file. So, what exactly is the problem after calling READ-XML? What does 4GL do? You can try to 'pollute' the temp-table PKs in OpenEdge by creating some records before READ-XML, and see what happens, so the '2305' value from .xml file does not match any real PK in the parent.

OE just ignores this field at parsing and populates this field with a new RECID value, as you tell, but FWD parses this field and sets this wrong value into the parentId field.

This means that the the child part of a PARENT-ID-RELATION is always assumed as 'hidden' and FWD needs to do the same - do not import, calculate automatically.

Exactly.
What should I do for this issue? Create a new task/commit here?

Commit to 10352a.

Constantin Asofiei wrote:

Paul, one more fix: Importer.processRelationFlds has the same dependency on serializeName; please fix this.

Actually, processRelationFlds is needed only for xml parsing now. I committed rev. 16137 on branch 10352a in which I removed the dependency on serializeName and also moved the processRelationFlds method from Importer to XmlImport as it doesn't make sense for a JsonImport instance to have access it.

Teodor please update the branch before committing your change.

Committed revision 16138 on task branch 10352a:
- Skipping relation fields when the parser reaches it. The changes affects READ-XML and READ-JSON.

Constantin, my changes are done, they are ready for review.

Committed revision 16139 on task branch 10352a:
- Changed column.getXmlNodeName() with column.getFieldName()

While testing this, I have encountered another issue:
- the Importer.fillFromParent implementation is not right. The getColumn() expects to receive the XML-NODE-NAME or SERIALIZE-NAME, but the field name is used. Having these attributes different than the field name makes the READ-XML and READ-JSON to fail.

java.lang.invoke.WrongMethodTypeException: cannot convert MethodHandle(String)Column to ()Object

Teodor, regarding your last note - is that fixed in 10352a?

No, it's not.

You can reproduce with any prodataset, with an XML-NODE-NAME on a parent id field different than the legacy name.

I don't quite understand. Beside the issue that I have found, there is another issue. I will document it soon, but why SERIALIZE-NAME or XML-NODE-NAME is used on sorting criteria?
I understand that the actual value is statically, provided from the conversion on DATASET definition, and there are some preprocessing (removing dots), but the value for this field is actually the field legacy name and not the XML node name/SERIALIZE-NAME.

Later update:
- I found the reason for this: the name parameter is the localname from XML/JSON.

Committed revision 16140 on task branch 10352a:
- Fixed the issue reported on #10352-35.

I will document the issue reported on #10352-39 and come with a fix soon.

Committed revision 16141 on task branch 10352a:
- Fixed processNestedRecord when the child record has XML prefix into serialization name.

This commit fixes DATASET:READ-XML when the child temp-table has a namespace prefix into serialization name: CodeCode

BLOCK-LEVEL ON ERROR UNDO, THROW.

USING OpenEdge.Core.Assert FROM PROPATH.

CLASS tests.TestReadXmlRecid:
    DEFINE TEMP-TABLE tt1 NO-UNDO
        FIELD name AS CHARACTER.

    DEFINE TEMP-TABLE tt2 NO-UNDO NAMESPACE-URI "https://goldencode.com/ns-uri" XML-NODE-NAME "prefix:tt2" 
        FIELD name AS CHARACTER
        FIELD parentId AS RECID.

    DEFINE DATASET ds FOR tt1, tt2
        PARENT-ID-RELATION REL1 FOR tt1, tt2 PARENT-ID-FIELD parentId.

    @Setup.
    METHOD PUBLIC VOID PopulateTempTable():
        DEFINE VARIABLE recordId AS RECID NO-UNDO.
        DEFINE VARIABLE goodRecordId AS RECID NO-UNDO.

        CREATE tt1.
        tt1.name = "Not in dataset".
        recordId = RECID(tt1).

        CREATE tt1.
        tt1.name = "Into the dataset".
        goodRecordId = RECID (tt1).

        FIND FIRST tt1 WHERE RECID (tt1) = recordId.
        DELETE tt1.

        CREATE tt2.
        tt2.name = "Into the dataset tt2.".
        tt2.parentId = goodRecordId.
    END METHOD.

    @Test.
    METHOD PUBLIC VOID TestWriteReadXml():
        DEFINE VARIABLE cLongChar AS LONGCHAR NO-UNDO.
        DATASET ds:WRITE-XML ("longchar", cLongChar).
        DATASET ds:WRITE-XML ("file", "output.xml", TRUE).

        DATASET ds:READ-XML ("longchar", cLongChar, "EMPTY", ?, ?).

        FIND FIRST tt1.
        Assert:Equals("Into the dataset", tt1.name).

        FIND FIRST tt2 WHERE tt2.parentId = RECID(tt1).
        Assert:Equals("Into the dataset tt2.", tt2.name).
    END METHOD.

    @Test.
    METHOD PUBLIC VOID TestWriteReadJson():
        DEFINE VARIABLE cLongChar AS LONGCHAR NO-UNDO.
        DATASET ds:WRITE-JSON ("longchar", cLongChar).

        DATASET ds:READ-JSON ("longchar", cLongChar, "EMPTY").

        FIND FIRST tt1.
        Assert:Equals("Into the dataset", tt1.name).

        FIND FIRST tt2 WHERE tt2.parentId = RECID(tt1).
        Assert:Equals("Into the dataset tt2.", tt2.name).
    END METHOD.

    @Teardown.
    METHOD PUBLIC VOID TearDown():
        DATASET ds:EMPTY-DATASET ().
    END METHOD.
END CLASS.

Teodor: please rebase 10352a.

Rebased 10352a to trunk revision 16250.

Rebased branch 10352a with trunk rev. 16359

Rebased branch 10352a with trunk rev. 16377