Project

General

Profile

Bug #10519

ProDataset WRITE-XML issue on nested PARENT-ID-RELATION

Added by Teodor Gorghe 11 months ago. Updated 9 months ago.

Status:
Test
Priority:
Normal
Assignee:
Target version:
-
Start date:
Due date:
% Done:

100%

billable:
No
vendor_id:
GCD
case_num:
version_reported:
version_resolved:
production:
No
env_name:
topics:

Related issues

Related to Base Language - Bug #10352: Fix READ-JSON dataset handle method's behaviour Closed

History

#2 Updated by Teodor Gorghe 11 months ago

Suppose we have a ProDataSet structure like this:

RootNode
|
|- node1
|  |
|  |- node11
|  |  |
|  |  |- commonTable
|  |  |
|  |  |- commonTable
|  |
|  |- node11
|     |
|     |- commonTable
|
|- node2
   |
   |- commonTable

There are multiple issues:
  • WRITE-XML: When SERIALIZE-HIDDEN is set to true, the second node11 is not created, but the contents get inserted into XML. You will end up with a commonTable node into node1. When closing the XML element (the supposed not to be closed is the second node11, but node1 is closed instead. I goes into the root node and WRITE-XML fails. From investigation, I have found that there is an issue with hasChild flag.
  • WRITE-XML, but WRITE-JSON also suffers from this issue: When inserting child records into the element node, all records from the child temp-table gets inserted. There should be inserted only the records matching the parent id with the parent recid.

#3 Updated by Teodor Gorghe 11 months ago

Created task branch 10519.
I have a fix for the first issue, which will come very soon.
For the second issue, there are no issues with WRITE-XML and WRITE-JSON, but rather, node11 and node2 have the same recid (4GL generates unique recid from the entire session, two temp-table cannot have the same recid).

#4 Updated by Teodor Gorghe 11 months ago

  • Status changed from New to WIP

#5 Updated by Teodor Gorghe 11 months ago

Committed revision 16146 on task branch 10519a:
- Reset hasChild flag to initial state while iterating through childs.

#6 Updated by Teodor Gorghe 11 months ago

There is a big problem on how we generate the recid field from a record:

// Procedure entry point
// Temp table definitions

CREATE node11.
CREATE node2.

// In 4GL, the RECID is different between two temp-table, but on FWD, is the same.

The testcase #10519-2 shows why this is not right.

Please also check this scenario:
  • We use READ-XML to import the dataset from XML, with this payload:
  • On WRITE-XML we get back this payload (FWD):

As you can see, this is completely wrong, c4 shouldn't be in node11 and c1/c2 in node2.

I think this requires H2 modifications.

#7 Updated by Constantin Asofiei 11 months ago

Are you saying that if you call CREATE twice one after the other, for different temp-tables, both records will have the same PK in FWD?

#8 Updated by Alexandru Lungu 11 months ago

There are limitations on how we can assign recid to _temp in FWD. There are changes to properly reclaim ids so that it avoids infinite loops, but the exact numbers for the ids is highly dependent upon the memory schema of the 4GL process. I hard-coded some numbers in FWD-H2 (like the size of the pages), but these are purely empirical.

From you comment, I understand that ids should be different across different temporary tables. Please check MultiplexedScanIndex.nextOffset - this is a counter to provide the next batch id. Maybe this should be moved at h2.Session level, instead of "per-table".

#9 Updated by Teodor Gorghe 11 months ago

Constantin Asofiei wrote:

Are you saying that if you call CREATE twice one after the other, for different temp-tables, both records will have the same PK in FWD?

Yes.

#10 Updated by Ovidiu Maxiniuc 11 months ago

I am sure this is a regression.

When I implemented the dataset support I would not have been able to identify the images from 4 or more related tables containing the information on same record of the original temp-table. I used the PKs to properly identify which is which and I was melding the outputs from 4GL and FWD which contained the result of ROWID for the aforementioned tables.

In 4GL, each temp-table pre-allocates a series of 0x800 PKs (not remember if this is the correct number) for each temp-table at first use (or maybe at creation/initialisation time). The old implementation was pretty accurate since FWD PKs were really matching those from 4GL (with some rare exceptions, indeed, but they were not a problem). The only adjustment I had to do is set the base record ID since in 4GL it is incremented for each application executed, but in FWD will reset with each server restart. And I had a lot of those.

At any rate, although this is not mandatory a problem as long as the PKs are different 'inside' each table, but some rowid-based queries might fail (return unexpected results) if the 4GL programmer expects the PK to be unique across all temp-tables.

#11 Updated by Teodor Gorghe 11 months ago

  • Related to Bug #10352: Fix READ-JSON dataset handle method's behaviour added

#12 Updated by Teodor Gorghe 11 months ago

Alexandru Lungu wrote:

There are limitations on how we can assign recid to _temp in FWD. There are changes to properly reclaim ids so that it avoids infinite loops, but the exact numbers for the ids is highly dependent upon the memory schema of the 4GL process. I hard-coded some numbers in FWD-H2 (like the size of the pages), but these are purely empirical.

From you comment, I understand that ids should be different across different temporary tables. Please check MultiplexedScanIndex.nextOffset - this is a counter to provide the next batch id. Maybe this should be moved at h2.Session level, instead of "per-table".

I have some changes for fwd-h2, but I am not really sure if this is right, but the problem is fixed:

I have some questions:
- the current patch might regress something. This code section is multi-threaded? If yes, I should implement just one synchronized function which takes the variable from h2.Session and increments it
- the MultiplexedScanIndex.truncate sets the nextOffset to 0. I think now is no longer required, but I need to think when is called and how often.

#13 Updated by Constantin Asofiei 11 months ago

Lets take a step back. Why do we need in FWD for the recids to be different? What's the minimal standalone test (in a functionally/business logic POV) which requires this?

#14 Updated by Teodor Gorghe 11 months ago

Constantin Asofiei wrote:

Lets take a step back. Why do we need in FWD for the recids to be different? What's the minimal standalone test (in a functionally/business logic POV) which requires this?

Please take a look on #10519-2 PopulateDataset, the expected (4GL) and actual (FWD) WRITE-XML output, described on #10519-6.

#15 Updated by Constantin Asofiei 11 months ago

Teodor Gorghe wrote:

Constantin Asofiei wrote:

Lets take a step back. Why do we need in FWD for the recids to be different? What's the minimal standalone test (in a functionally/business logic POV) which requires this?

Please take a look on #10519-2 PopulateDataset, the expected (4GL) and actual (FWD) WRITE-XML output, described on #10519-6.

Is the problem in READ-XML or WRITE-XML? What is the actual content of the temp-tables (use a FOR EACh) after READ-XML?

#16 Updated by Ovidiu Maxiniuc 11 months ago

The thing that puzzles me is this:

        PARENT-ID-RELATION REL4 FOR node2, commonTable PARENT-ID-FIELD parent_id
        PARENT-ID-RELATION REL5 FOR node11, commonTable PARENT-ID-FIELD parent_id.
In other words, the commonTable table has two parents. This is NOT logical and at the time I implemented the dataset module, I worked with single parent relation in my mind. The only case when these were allowed is when all-but-one of them are disabled. Maybe I got it wrong or maybe this is a special 'feature' of 4GL. Also this might be one of the subtle differences between normal relations (defines with DATA-RELATION) and PARENT-ID-RELATION.

At any rate, if we want this to work as expected, the PKs MUST be unique in FWD as they are in 4GL (this testcase is guaranteed to work correctly only in that case).

#17 Updated by Teodor Gorghe 11 months ago

Constantin Asofiei wrote:

Teodor Gorghe wrote:

Constantin Asofiei wrote:

Lets take a step back. Why do we need in FWD for the recids to be different? What's the minimal standalone test (in a functionally/business logic POV) which requires this?

Please take a look on #10519-2 PopulateDataset, the expected (4GL) and actual (FWD) WRITE-XML output, described on #10519-6.

Is the problem in READ-XML or WRITE-XML? What is the actual content of the temp-tables (use a FOR EACh) after READ-XML?

There was a WRITE-XML problem, which is partially fixed with the current commit on this task branch. The only READ-XML problem (as you know), is the issue addressed by #10352. In order to test with READ-XML, I have took the #10352 fixes.
But the cause of the remaining issue is not by the READ-XML or WRITE-XML. The table contents are attached

As you may notice from the testcase, commonTable has 2 parents: node11 and node2, linked by the commonTable.parent_id. When there are two identical RECIDs from two separate tables then the serialized XML or JSON becomes exactly the one described on #10519-6.
I don't think there is a READ-XML or WRITE-XML issue, since the output is different both when the records are populated by "hand" or using READ-XML.
The H2 changes fixes this issue, but I think there is more work to do, to ensure that is thread-safe and there is no regression on NO-UNDO temp-tables.

Ovidiu Maxiniuc wrote:

The thing that puzzles me is this:[...]In other words, the commonTable table has two parents. This is NOT logical and at the time I implemented the dataset module, I worked with single parent relation in my mind. The only case when these were allowed is when all-but-one of them are disabled. Maybe I got it wrong or maybe this is a special 'feature' of 4GL. Also this might be one of the subtle differences between normal relations (defines with DATA-RELATION) and PARENT-ID-RELATION.

At any rate, if we want this to work as expected, the PKs MUST be unique in FWD as they are in 4GL (this testcase is guaranteed to work correctly only in that case).

Exactly

#18 Updated by Constantin Asofiei 11 months ago

commonTable has 2 parents: node11 and node2, linked by the commonTable.parent_id. When there are two identical RECIDs from two separate tables then the serialized XML or JSON becomes exactly the one described on #10519-6.

What is the code which links the child with the parent via recid? Is the parent table there beside the recid?

Otherwise, is the FWD temp-table content the same as 4GL (excluding the recid) after READ-XML?

#19 Updated by Teodor Gorghe 11 months ago

Constantin Asofiei wrote:

What is the code which links the child with the parent via recid? Is the parent table there beside the recid?

Otherwise, is the FWD temp-table content the same as 4GL (excluding the recid) after READ-XML?

For READ-XML, I don't know just yet the exact code location where the record is created and then linked by RECID. I think there is a simple CREATE and fill with data from XML parsing.
For WRITE-XML, the code location is on XmlExport:2791, which just creates a query and executes it. It fetches the parents with the corresponding RECID and there are no issues there.

Constantin Asofiei wrote:

Is the parent table there beside the recid?

Yes, is there after READ-XML.

Constantin Asofiei wrote:

Otherwise, is the FWD temp-table content the same as 4GL (excluding the recid) after READ-XML?

Yes, is the same.

#20 Updated by Teodor Gorghe 11 months ago

Created task branch 10519a_h2 on p2j_repo/h2.

Committed revision 62 on h2 task branch 10519a_h2:
- Moved MultiplexedScanIndex.nextOffset variable to H2.Session.

The H2 changes can be reviewed.
I will change the task status after I fix the indentation issues on WRITE-XML.

#21 Updated by Teodor Gorghe 11 months ago

  • Status changed from WIP to Review
  • reviewer Constantin Asofiei added

Committed revision 16147 on task branch 10519a:
- Reverted last commit. Removed hasChild = true on writeChildRecords.

#22 Updated by Teodor Gorghe 11 months ago

  • % Done changed from 0 to 100

#23 Updated by Constantin Asofiei 10 months ago

  • Assignee set to Teodor Gorghe
  • reviewer Alexandru Lungu added

Alexandru - please review 10519a_h2 - it looks good to me.

#24 Updated by Alexandru Lungu 10 months ago

  • % Done changed from 100 to 90
  • Status changed from Review to WIP

Review of 10519a_h2:

  • I don't think while (batchId >= (nextOffset = getAndIncNextOffset(session))) block is required anymore. This was created because you could create a batch, create a row, truncate (so all batches are deleted) and then do a rollback. In the case of rollback, the batch should exist so that the created row is inserted back. We created it incrementally so that nextOffset was iterated from 0 to batchId. If you move nextOffset as a session-level counter, then you can simply disregard this use-case.
    • Also, on a long running session, this while loop may iterate a lot, so it is no feasible to still have it in place.
    • Still, we need to ensure that batches.get(batchId) doesn't return null. Maybe you can simply create the batch if it doesn't exist:
      if (batch == null) {
               long multiplex = row.getValue(1).getInt();
               batch = new Batch(FIRST_KEY + batchId * BATCH_SIZE, this.noUndo, multiplex);
               notifyNewBatch(batch, multiplex);
               batches.put(batchId, batch);
               lastBatches.put((long) multiplex, batch);
      }
  • No need to synchronize getAndIncNextOffset. H2 sessions are single-threaded by convention.

#25 Updated by Teodor Gorghe 10 months ago

  • % Done changed from 90 to 100
  • Status changed from WIP to Review

Committed revision 63 on h2 task branch 10519a_h2:
- Addressed code review #10519-24.

#26 Updated by Constantin Asofiei 10 months ago

Alexandru - do you think is safe to commit 10519a_h2 to fwd-h2 trunk? I need to add this to 9457c.

#27 Updated by Alexandru Lungu 10 months ago

I am OK with 10519a_h2. This can be merged to fwd-h2 trunk.
But I want to stress out this is a high risk change and we need all the testing possible to ensure we did not introduce memory leaks or issues with rollbacks.

#28 Updated by Alexandru Lungu 10 months ago

  • Status changed from Review to Merge Pending

#29 Updated by Constantin Asofiei 10 months ago

Alexandru Lungu wrote:

I am OK with 10519a_h2. This can be merged to fwd-h2 trunk.
But I want to stress out this is a high risk change and we need all the testing possible to ensure we did not introduce memory leaks or issues with rollbacks.

Understood, I'll merge to fwd-h2 trunk and 9457c.

#30 Updated by Constantin Asofiei 10 months ago

  • Status changed from Merge Pending to Review

Merged 10519a_h2 to fwd-h2 rev 62 and pushed to projsrv01.

#31 Updated by Constantin Asofiei 10 months ago

Teodor, please fwd-h2 rev 62 in 10519a.

#32 Updated by Teodor Gorghe 10 months ago

Committed revision 16148 on task branch 10519a:
- Upgraded to the latest FWD-H2-1.62.

#33 Updated by Constantin Asofiei 10 months ago

Teodor Gorghe wrote:

Committed revision 16148 on task branch 10519a:
- Upgraded to the latest FWD-H2-1.62.

I've changed to 1.62 in 16149, it was 1.61 in 16148.

#34 Updated by Alexandru Lungu 9 months ago

Anything left to be reviewed?

#35 Updated by Constantin Asofiei 9 months ago

  • Status changed from Review to Merge Pending

This can be merged after 10509a.

#36 Updated by Teodor Gorghe 9 months ago

  • Status changed from Merge Pending to Test

Branch 10519a was merged to trunk rev. 16233 and archived.

Also available in: Atom PDF