FWD » Core Development » Database

The remaining work:

• Encoding behavior needs better testing (at least #6623 must be fixed).
• BOM handling for file source/target not implemented yet
• Some remaining issues (see #4768-8)

The BOM handling has a testcase already written. See testcases/copy_lob/tests/bom.p.

The BOM handling has a testcase already written. See testcases/copy_lob/tests/bom.p.

OK, the first issue I met in bom.p surprisingly was that the following sequence leads to invalid output file content in 4GL because it runs some conversion due to convert option. FWD doesn't do it and the output file is fine UTF-8.

inputLongchar = ?.
fix-codepage(inputLongchar) = 'UTF-8'.
inputLongchar = "A" + CHR(14844588, "UTF-8", "UTF-8") + "A".
copy-lob from inputLongchar to file 'copy_lob/output/out.tmp' convert target codepage 'UTF-8' no-error.

Stanislav, looking at the 14844588 value, in hex is e282ac. See this: https://community.progress.com/s/article/P181822 Please experiment with -cpstream and see what happens. More, in the docs https://documentation.progress.com/output/ua/OpenEdge_latest/pdsoe/PLUGINS_ROOT/com.openedge.pdt.langref.help/rfi1424920632352.html it states:

If either the source or the target object is a file, the target's code page defaults to -cpstream.

On a side note, test abends in OE on Windows if the target path does not exist on disk.

Stanislav Lomany wrote:

Do you have any idea what and how 4GL produces?

Try launching the 4GL using -cpinternal utf8 command-line parameter. This may shed some light on how it works. In my opinion, it assigns the € representation in UTF-8 to longchar variable using some UTF to default CP conversion which breaks the content, ignoring the previous fix-codepage statement.

Guys, I've been trying to experiment with -cpstream and -cpinternal parameters and found that when it comes to setting them to UTF-16/UTF-32 I can set -cpstream to to UTF-16 and that's it. For other cases it produces "Case table for code page UTF-16 and case name BASIC was not found in convmap.cp (1038)" error message or silenlty abends.
Do you have any idea how to fix it?

I know, I had the same problems :(.
Try setting the same value for all these cp parameters, unless having them different is your goal.

Try setting the same value for all these cp parameters

Errors are the same for this case too.

I cannot change -cpstream/-cpinternal parameters on a customer's VM either, so I cannot experiment with it.
Moreover, testcases/copy_lob/tests/bom.p fails on this VM because output produced for UTF-8 is different (using default -cpstream/-cpinternal).

Greg, what should we do about these parameters if we cannot be sure how they work?

I cannot change -cpstream/-cpinternal parameters on a customer's VM either, so I cannot experiment with it.

Is the testing issue caused by the virtual machine not having the needed codepages installed? Otherwise I don't understand why OE would not honor these command line specifications.

Is the testing issue caused by the virtual machine not having the needed codepages installed?

Some cases claim that "case table" for this particular case is missing in convmap.cp (quick googling didn't tell me how to add something to it). Some cases abend. And some lead to artefacts in screen output.

Please provide the list of codepages/scenarios which need testing (and which you cannot test).

Marian: I think we need your team to do this testing.

Greg Shah wrote:

Please provide the list of codepages/scenarios which need testing (and which you cannot test).

Marian: I think we need your team to do this testing.

• the supported codepages/collation combinations: i18n/4gl/cp_collation.xml
• the accepted conversion between codepages for codepage-convert: i18n/4gl/cp_conversion.xml
• the accepted conversion between codepages for chr: i18n/4gl/chr_conversion.xml
• the accepted conversion between codepages for input stream: i18n/4gl/is_conversion.xml

i18n/4gl/cp_collation.xml i18n/4gl/cp_conversion.xml i18n/4gl/chr_conversion.xml i18n/4gl/is_conversion.xml

Marian, these are theoretically allowed conversions. The problem is that I cannot set -cpstream/-cpinternal parameters to test how they work. I either get an error that "case table" for this particular case is missing in convmap.cp or it just abends.
Do you think you'll be able to set -cpstream/-cpinternal with a working result? If so, I'll make a testcase to be tested with all -cpstream/-cpinternal combinations.

Stanislav Lomany wrote:

i18n/4gl/cp_collation.xml i18n/4gl/cp_conversion.xml i18n/4gl/chr_conversion.xml i18n/4gl/is_conversion.xml

Marian, these are theoretically allowed conversions. The problem is that I cannot set -cpstream/-cpinternal parameters to test how they work. I either get an error that "case table" for this particular case is missing in convmap.cp or it just abends.

What are the values you're trying to use for the codepages/collation tables - for cpstream and cpinternal?

Do you think you'll be able to set -cpstream/-cpinternal with a working result? If so, I'll make a testcase to be tested with all -cpstream/-cpinternal combinations.

Yes, the idea is that if the combination is valid then there shouldn't be any error although is quite rare to have different codepages exactly because of the conversion overhead - the client uses the codepage appropriate for the user, the server often use the same codepage as client or at times some UTF if the database needs to support various codepages.

What are the values you're trying to use for the codepages/collation tables - for cpstream and cpinternal?

I want to test this ones:
UTF-8
UTF-16
UTF-16LE
UTF-16BE
UTF-32
UTF-32LE
UTF-32BE

Stanislav: please summarize any work done for this task

I didn't produce any testcases and used the existing ones listed here. The problems I encountered are mentioned here - I didn't manage to find answers.

Greg Shah wrote:

The BOM handling has a testcase already written. See testcases/copy_lob/tests/bom.p.

Greg/Stanislav, I don't see any testcases/copy_lob/tests/bom.p testcases in testcases/copy_lob. There are 6 tests classes in testcases/copy_lob/memptr. Is this task referring to the uast testacases project?

Edit:

Never mind, found the tests cases in testcases/tests/copy_lob

Got the following after running testcases/tests/copy_lob/

Test run finished after 9879 ms
[       113 containers found      ]
[         0 containers skipped    ]
[       113 containers started    ]
[         0 containers aborted    ]
[       113 containers successful ]
[         0 containers failed     ]
[       736 tests found           ]
[         0 tests skipped         ]
[       736 tests started         ]
[         0 tests aborted         ]
[       209 tests successful      ]
[       527 tests failed          ]

After creating the following path in deploy/client/tests/copy_lob/support/data, I get the following results:

Test run finished after 6680 ms
[       113 containers found      ]
[         0 containers skipped    ]
[       113 containers started    ]
[         0 containers aborted    ]
[       113 containers successful ]
[         0 containers failed     ]
[       736 tests found           ]
[         0 tests skipped         ]
[       736 tests started         ]
[         0 tests aborted         ]
[       390 tests successful      ]
[       346 tests failed          ]

As you work through issues in these tests, please update LOB Tests with details of the test status and how to run them in FWD.

After solving all configuration issues, I was left only with assertion errors which should be investigated.

Test run finished after 17719 ms
[       113 containers found      ]
[         0 containers skipped    ]
[       113 containers started    ]
[         0 containers aborted    ]
[       113 containers successful ]
[         0 containers failed     ]
[       736 tests found           ]
[         0 tests skipped         ]
[       736 tests started         ]
[         0 tests aborted         ]
[       523 tests successful      ]
[       213 tests failed          ]

However there is still a configuration issue which I don't know how to solve with the following class TestFileInvalidSrc, TestFileInvalidSrcDB, TestFileInvalidTrg and TestFileInvalidTrgDB, there are hard coded Windows paths in 4gl code.

Also update the LOB Tests wiki.

From the wiki:

All files from tests/copy_lob/support/data should be moved to deploy/client/tests/copy_lob/support/data.

Even with this idea, we should still change any Windows style pathnames to use relative paths and / as the path separator. / works on Windows and Linux so it is more portable. Please change the 4GL code as needed and check it in.

Greg Shah wrote:

From the wiki:

All files from tests/copy_lob/support/data should be moved to deploy/client/tests/copy_lob/support/data.

Even with this idea, we should still change any Windows style pathnames to use relative paths and / as the path separator. / works on Windows and Linux so it is more portable. Please change the 4GL code as needed and check it in.

Looking at the tests more closely, I think there is an actual intend in adding an invalid path.

copy-lob from file 'E:/temp/invalid_file.txt' to oHandle::logo no-error. this operation should raise, 2 errors: Cannot find or open file E:/temp/invalid_file.txt, errno = 2. (43), Copy-lob failed to copy file 'E:/temp/invalid_file.txt' to large object.

But assertion fails because this only one error thrown. So this is a actual problem with fwd that should be investigated.

First major problem found, it is common for around 100 tests.

copy-lob from emptyMemptr to customer.logo no-error.

When trying to write to a buffer field, the following error will be raised com.goldencode.p2j.util.ErrorConditionException: Update of buffer field object requires a transaction. (7369)

Eduard Soltan wrote:

Committed on 6457a, rev. 16066.

- Emit Transaction.FULL parameter when target of copy-lob operation is a persistent buffer.

Can't this be done by runtime when the target is a persistent buffer?

Constantin Asofiei wrote:

Eduard Soltan wrote:

Committed on 6457a, rev. 16066.

- Emit Transaction.FULL parameter when target of copy-lob operation is a persistent buffer.

Can't this be done by runtime when the target is a persistent buffer?

I will look into it. I observed how transaction is handled in buffer-copy or other similar operation, and in all the transaction is open at block level.

Eduard Soltan wrote:

I will look into it. I observed how transaction is handled in buffer-copy or other similar operation, and in all the transaction is open at block level.

Ah, you mean at the surrounding block, and not at COPY-LOB. That makes sense.

I am not really sure how the conversion happens in UTF-8 character to ISO-8859-1 in 4GL.

Test 1¶

-Cpinternal iso8859-1, Cpstream iso8859-1

define variable inputLongchar1   as longchar no-undo.
inputLongchar1 = "CHR(14844588, "UTF-8", "UTF-8").

copy-lob from inputLongchar1 to file 'out.data' no-error.

The output in 'out.data' is size 3 bytes E2 | 82 | AC

Test 2¶

-Cpinternal iso8859-1, Cpstream iso8859-1

define variable inputLongchar1   as longchar no-undo.
inputLongchar1 = "CHR(14844588, "UTF-8", "UTF-8").

copy-lob from inputLongchar1 to file 'out.data' convert target codepage 'utf-8' no-error.

The output in 'out.data' is size 6 bytes C3 | A2 | C2 | 82 | C2 | AC

Test 3¶

-Cpinternal iso8859-1, Cpstream utf-8

define variable inputLongchar1   as longchar no-undo.
inputLongchar1 = "CHR(14844588, "UTF-8", "UTF-8").

copy-lob from inputLongchar1 to file 'out.data' no-error.

The output in 'out.data' is size 6 bytes C3 | A2 | C2 | 82 | C2 | AC

Test 4¶

-Cpinternal iso8859-1, Cpstream utf-8

define variable inputLongchar1   as longchar no-undo.
inputLongchar1 = "CHR(14844588, "UTF-8", "UTF-8").

copy-lob from inputLongchar1 to file 'out.data' convert target codepage 'utf-8' no-error.

The output in 'out.data' is size 6 bytes E2 | 82 | AC

Constantin Asofiei wrote:

Review for 6457a rev 16070:

• Stream.java - why the reason for the commented code?

define variable auxMemptr         as memptr  no-undo.
set-size (auxMemptr) = 5.
put-string(auxMemptr, 1) = "test".

copy-lob from auxMemptr to file 'out.tmp' no-error.

In fwd the copy-lob executes correctly, writing 5 bytes in the target file and the last one is null. And because of the nullQuirk, read from the file does not happen correctly. I will try to look the original testcase, why it was added.

• longchar.java
  • why "" + StringHelper. ?

Because if value was null, after this operation we would get something very weird in longchar value (ex: null ).

Eduard Soltan wrote:

• longchar.java
  • why "" + StringHelper. ?

Because if value was null, after this operation we would get something very weird in longchar value (ex: null ).

rightAlignText can not return null. So "" + is no-op.

• block_properties.rules has the XML prolog removed.
• LobCopy - no history entry
• longchar.write - what if the existing longchar instance already has the codepage set - does this get changed?

Please fix the above, rebase and start conversion and runtime testing.

Constantin Asofiei wrote:

• longchar.write - what if the existing longchar instance already has the codepage set - does this get changed?

No, it doesn't. If I already have a code page on the target longchar, and a different code page in convert target codepage 'ISO8859-1'. phrase I should get an error. target codepage conflicts with the longchars's fixed code page

Eduard, please address the review and - is there anything else for this task?

Constantin Asofiei wrote:

Eduard, please address the review and - is there anything else for this task?

I addressed the review and tested conversion and runtime on Chui regression test project and on a large GUI application. I think another gui application should be tested.

Otherwise there are 7 tests that require some code-page changes.

Eduard Soltan wrote:

Otherwise there are 7 tests that require some code-page changes.

You mean there is still work for standalone unit tests for this task?

Constantin Asofiei wrote:

Eduard Soltan wrote:
You mean there is still work for standalone unit tests for this task?

Not on this task, the root case of the remaining tests failures is the same as #4766.

Another customer application converts and runs successfully with 6457a. ✅

conversion and runtime on Chui regression passed ✅
large GUI application conversion and unit tests and smoke tests passed ✅

Is any other tests necessary?

6457a was rebased to trunk rev. 16358.

6457a rev 16365 fixes a XML header problem.