Bug #10519
ProDataset WRITE-XML issue on nested PARENT-ID-RELATION
100%
Related issues
History
#2 Updated by Teodor Gorghe 11 months ago
Suppose we have a ProDataSet structure like this:
RootNode | |- node1 | | | |- node11 | | | | | |- commonTable | | | | | |- commonTable | | | |- node11 | | | |- commonTable | |- node2 | |- commonTableThere are multiple issues:
- WRITE-XML: When
SERIALIZE-HIDDENis set to true, the secondnode11is not created, but the contents get inserted into XML. You will end up with acommonTablenode intonode1. When closing the XML element (the supposed not to be closed is the secondnode11, butnode1is closed instead. I goes into the root node andWRITE-XMLfails. From investigation, I have found that there is an issue with hasChild flag. - WRITE-XML, but WRITE-JSON also suffers from this issue: When inserting child records into the element node, all records from the child temp-table gets inserted. There should be inserted only the records matching the parent id with the parent recid.
#3 Updated by Teodor Gorghe 11 months ago
Created task branch 10519.
I have a fix for the first issue, which will come very soon.
For the second issue, there are no issues with WRITE-XML and WRITE-JSON, but rather, node11 and node2 have the same recid (4GL generates unique recid from the entire session, two temp-table cannot have the same recid).
#4 Updated by Teodor Gorghe 11 months ago
- Status changed from New to WIP
#5 Updated by Teodor Gorghe 11 months ago
Committed revision 16146 on task branch 10519a:
- Reset hasChild flag to initial state while iterating through childs.
#6 Updated by Teodor Gorghe 11 months ago
There is a big problem on how we generate the recid field from a record:
// Procedure entry point // Temp table definitions CREATE node11. CREATE node2. // In 4GL, the RECID is different between two temp-table, but on FWD, is the same.
The testcase #10519-2 shows why this is not right.
Please also check this scenario:- We use
READ-XMLto import the dataset from XML, with this payload: Payload - On
WRITE-XMLwe get back this payload (FWD): Payload
As you can see, this is completely wrong, c4 shouldn't be in node11 and c1/c2 in node2.
I think this requires H2 modifications.
#7 Updated by Constantin Asofiei 11 months ago
Are you saying that if you call CREATE twice one after the other, for different temp-tables, both records will have the same PK in FWD?
#8 Updated by Alexandru Lungu 11 months ago
There are limitations on how we can assign recid to _temp in FWD. There are changes to properly reclaim ids so that it avoids infinite loops, but the exact numbers for the ids is highly dependent upon the memory schema of the 4GL process. I hard-coded some numbers in FWD-H2 (like the size of the pages), but these are purely empirical.
From you comment, I understand that ids should be different across different temporary tables. Please check MultiplexedScanIndex.nextOffset - this is a counter to provide the next batch id. Maybe this should be moved at h2.Session level, instead of "per-table".
#9 Updated by Teodor Gorghe 11 months ago
Constantin Asofiei wrote:
Are you saying that if you call
CREATEtwice one after the other, for different temp-tables, both records will have the same PK in FWD?
Yes.
#10 Updated by Ovidiu Maxiniuc 11 months ago
I am sure this is a regression.
When I implemented the dataset support I would not have been able to identify the images from 4 or more related tables containing the information on same record of the original temp-table. I used the PKs to properly identify which is which and I was melding the outputs from 4GL and FWD which contained the result of ROWID for the aforementioned tables.
In 4GL, each temp-table pre-allocates a series of 0x800 PKs (not remember if this is the correct number) for each temp-table at first use (or maybe at creation/initialisation time). The old implementation was pretty accurate since FWD PKs were really matching those from 4GL (with some rare exceptions, indeed, but they were not a problem). The only adjustment I had to do is set the base record ID since in 4GL it is incremented for each application executed, but in FWD will reset with each server restart. And I had a lot of those.
At any rate, although this is not mandatory a problem as long as the PKs are different 'inside' each table, but some rowid-based queries might fail (return unexpected results) if the 4GL programmer expects the PK to be unique across all temp-tables.
#11 Updated by Teodor Gorghe 11 months ago
- Related to Bug #10352: Fix READ-JSON dataset handle method's behaviour added
#12 Updated by Teodor Gorghe 11 months ago
Alexandru Lungu wrote:
There are limitations on how we can assign recid to _temp in FWD. There are changes to properly reclaim ids so that it avoids infinite loops, but the exact numbers for the ids is highly dependent upon the memory schema of the 4GL process. I hard-coded some numbers in FWD-H2 (like the size of the pages), but these are purely empirical.
From you comment, I understand that ids should be different across different temporary tables. Please check
MultiplexedScanIndex.nextOffset- this is a counter to provide the next batch id. Maybe this should be moved ath2.Sessionlevel, instead of "per-table".
I have some changes for fwd-h2, but I am not really sure if this is right, but the problem is fixed:
FWD-H2 patch
I have some questions:
- the current patch might regress something. This code section is multi-threaded? If yes, I should implement just one synchronized function which takes the variable from h2.Session and increments it
- the MultiplexedScanIndex.truncate sets the nextOffset to 0. I think now is no longer required, but I need to think when is called and how often.
#13 Updated by Constantin Asofiei 11 months ago
Lets take a step back. Why do we need in FWD for the recids to be different? What's the minimal standalone test (in a functionally/business logic POV) which requires this?
#14 Updated by Teodor Gorghe 11 months ago
Constantin Asofiei wrote:
Lets take a step back. Why do we need in FWD for the recids to be different? What's the minimal standalone test (in a functionally/business logic POV) which requires this?
Please take a look on #10519-2 PopulateDataset, the expected (4GL) and actual (FWD) WRITE-XML output, described on #10519-6.
#15 Updated by Constantin Asofiei 11 months ago
Teodor Gorghe wrote:
Constantin Asofiei wrote:
Lets take a step back. Why do we need in FWD for the recids to be different? What's the minimal standalone test (in a functionally/business logic POV) which requires this?
Please take a look on #10519-2
PopulateDataset, the expected (4GL) and actual (FWD)WRITE-XMLoutput, described on #10519-6.
Is the problem in READ-XML or WRITE-XML? What is the actual content of the temp-tables (use a FOR EACh) after READ-XML?
#16 Updated by Ovidiu Maxiniuc 11 months ago
The thing that puzzles me is this:
PARENT-ID-RELATION REL4 FOR node2, commonTable PARENT-ID-FIELD parent_id
PARENT-ID-RELATION REL5 FOR node11, commonTable PARENT-ID-FIELD parent_id.In other words, the commonTable table has two parents. This is NOT logical and at the time I implemented the dataset module, I worked with single parent relation in my mind. The only case when these were allowed is when all-but-one of them are disabled. Maybe I got it wrong or maybe this is a special 'feature' of 4GL. Also this might be one of the subtle differences between normal relations (defines with DATA-RELATION) and PARENT-ID-RELATION.
At any rate, if we want this to work as expected, the PKs MUST be unique in FWD as they are in 4GL (this testcase is guaranteed to work correctly only in that case).
#17 Updated by Teodor Gorghe 11 months ago
Constantin Asofiei wrote:
Teodor Gorghe wrote:
Constantin Asofiei wrote:
Lets take a step back. Why do we need in FWD for the recids to be different? What's the minimal standalone test (in a functionally/business logic POV) which requires this?
Please take a look on #10519-2
PopulateDataset, the expected (4GL) and actual (FWD)WRITE-XMLoutput, described on #10519-6.Is the problem in READ-XML or WRITE-XML? What is the actual content of the temp-tables (use a
FOR EACh) after READ-XML?
There was a WRITE-XML problem, which is partially fixed with the current commit on this task branch. The only READ-XML problem (as you know), is the issue addressed by #10352. In order to test with READ-XML, I have took the #10352 fixes.
But the cause of the remaining issue is not by the READ-XML or WRITE-XML. The table contents are attached
here
As you may notice from the testcase, commonTable has 2 parents: node11 and node2, linked by the commonTable.parent_id. When there are two identical RECIDs from two separate tables then the serialized XML or JSON becomes exactly the one described on #10519-6.
I don't think there is a READ-XML or WRITE-XML issue, since the output is different both when the records are populated by "hand" or using READ-XML.
The H2 changes fixes this issue, but I think there is more work to do, to ensure that is thread-safe and there is no regression on NO-UNDO temp-tables.
Ovidiu Maxiniuc wrote:
The thing that puzzles me is this:[...]In other words, the
commonTabletable has two parents. This is NOT logical and at the time I implemented the dataset module, I worked with single parent relation in my mind. The only case when these were allowed is when all-but-one of them are disabled. Maybe I got it wrong or maybe this is a special 'feature' of 4GL. Also this might be one of the subtle differences between normal relations (defines withDATA-RELATION) andPARENT-ID-RELATION.At any rate, if we want this to work as expected, the PKs MUST be unique in FWD as they are in 4GL (this testcase is guaranteed to work correctly only in that case).
Exactly
#18 Updated by Constantin Asofiei 11 months ago
commonTable has 2 parents: node11 and node2, linked by the commonTable.parent_id. When there are two identical RECIDs from two separate tables then the serialized XML or JSON becomes exactly the one described on #10519-6.
What is the code which links the child with the parent via recid? Is the parent table there beside the recid?
Otherwise, is the FWD temp-table content the same as 4GL (excluding the recid) after READ-XML?
#19 Updated by Teodor Gorghe 11 months ago
Constantin Asofiei wrote:
What is the code which links the child with the parent via recid? Is the parent table there beside the recid?
Otherwise, is the FWD temp-table content the same as 4GL (excluding the recid) after READ-XML?
For READ-XML, I don't know just yet the exact code location where the record is created and then linked by RECID. I think there is a simple CREATE and fill with data from XML parsing.
For WRITE-XML, the code location is on XmlExport:2791, which just creates a query and executes it. It fetches the parents with the corresponding RECID and there are no issues there.
Constantin Asofiei wrote:
Is the parent table there beside the recid?
Yes, is there after READ-XML.
Constantin Asofiei wrote:
Otherwise, is the FWD temp-table content the same as 4GL (excluding the recid) after READ-XML?
Yes, is the same.
#20 Updated by Teodor Gorghe 11 months ago
Created task branch 10519a_h2 on p2j_repo/h2.
Committed revision 62 on h2 task branch 10519a_h2:
- Moved MultiplexedScanIndex.nextOffset variable to H2.Session.
The H2 changes can be reviewed.
I will change the task status after I fix the indentation issues on WRITE-XML.
#21 Updated by Teodor Gorghe 11 months ago
- Status changed from WIP to Review
- reviewer Constantin Asofiei added
Committed revision 16147 on task branch 10519a:
- Reverted last commit. Removed hasChild = true on writeChildRecords.
#22 Updated by Teodor Gorghe 11 months ago
- % Done changed from 0 to 100
#23 Updated by Constantin Asofiei 10 months ago
- Assignee set to Teodor Gorghe
- reviewer Alexandru Lungu added
Alexandru - please review 10519a_h2 - it looks good to me.
#24 Updated by Alexandru Lungu 10 months ago
- % Done changed from 100 to 90
- Status changed from Review to WIP
Review of 10519a_h2:
- I don't think
while (batchId >= (nextOffset = getAndIncNextOffset(session)))block is required anymore. This was created because you could create a batch, create a row, truncate (so all batches are deleted) and then do a rollback. In the case of rollback, the batch should exist so that the created row is inserted back. We created it incrementally so thatnextOffsetwas iterated from 0 tobatchId. If you movenextOffsetas a session-level counter, then you can simply disregard this use-case.- Also, on a long running session, this while loop may iterate a lot, so it is no feasible to still have it in place.
- Still, we need to ensure that
batches.get(batchId)doesn't return null. Maybe you can simply create the batch if it doesn't exist:if (batch == null) { long multiplex = row.getValue(1).getInt(); batch = new Batch(FIRST_KEY + batchId * BATCH_SIZE, this.noUndo, multiplex); notifyNewBatch(batch, multiplex); batches.put(batchId, batch); lastBatches.put((long) multiplex, batch); }
- No need to synchronize
getAndIncNextOffset. H2 sessions are single-threaded by convention.
#25 Updated by Teodor Gorghe 10 months ago
- % Done changed from 90 to 100
- Status changed from WIP to Review
Committed revision 63 on h2 task branch 10519a_h2:
- Addressed code review #10519-24.
#26 Updated by Constantin Asofiei 10 months ago
Alexandru - do you think is safe to commit 10519a_h2 to fwd-h2 trunk? I need to add this to 9457c.
#27 Updated by Alexandru Lungu 10 months ago
I am OK with 10519a_h2. This can be merged to fwd-h2 trunk.
But I want to stress out this is a high risk change and we need all the testing possible to ensure we did not introduce memory leaks or issues with rollbacks.
#28 Updated by Alexandru Lungu 10 months ago
- Status changed from Review to Merge Pending
#29 Updated by Constantin Asofiei 10 months ago
Alexandru Lungu wrote:
I am OK with 10519a_h2. This can be merged to fwd-h2 trunk.
But I want to stress out this is a high risk change and we need all the testing possible to ensure we did not introduce memory leaks or issues with rollbacks.
Understood, I'll merge to fwd-h2 trunk and 9457c.
#30 Updated by Constantin Asofiei 10 months ago
- Status changed from Merge Pending to Review
Merged 10519a_h2 to fwd-h2 rev 62 and pushed to projsrv01.
#31 Updated by Constantin Asofiei 10 months ago
Teodor, please fwd-h2 rev 62 in 10519a.
#32 Updated by Teodor Gorghe 10 months ago
Committed revision 16148 on task branch 10519a:
- Upgraded to the latest FWD-H2-1.62.
#33 Updated by Constantin Asofiei 10 months ago
Teodor Gorghe wrote:
Committed revision 16148 on task branch 10519a:
- Upgraded to the latest FWD-H2-1.62.
I've changed to 1.62 in 16149, it was 1.61 in 16148.
#34 Updated by Alexandru Lungu 9 months ago
Anything left to be reviewed?
#35 Updated by Constantin Asofiei 9 months ago
- Status changed from Review to Merge Pending
This can be merged after 10509a.
#36 Updated by Teodor Gorghe 9 months ago
- Status changed from Merge Pending to Test
Branch 10519a was merged to trunk rev. 16233 and archived.