FWD » Core Development » Database

The remaining work:

• Encoding behavior needs better testing (at least #6623 must be fixed).
• BOM handling for file source/target not implemented yet
• Some remaining issues (see #4768-8)

The BOM handling has a testcase already written. See testcases/copy_lob/tests/bom.p.

The BOM handling has a testcase already written. See testcases/copy_lob/tests/bom.p.

OK, the first issue I met in bom.p surprisingly was that the following sequence leads to invalid output file content in 4GL because it runs some conversion due to convert option. FWD doesn't do it and the output file is fine UTF-8.

inputLongchar = ?.
fix-codepage(inputLongchar) = 'UTF-8'.
inputLongchar = "A" + CHR(14844588, "UTF-8", "UTF-8") + "A".
copy-lob from inputLongchar to file 'copy_lob/output/out.tmp' convert target codepage 'UTF-8' no-error.

Stanislav, looking at the 14844588 value, in hex is e282ac. See this: https://community.progress.com/s/article/P181822 Please experiment with -cpstream and see what happens. More, in the docs https://documentation.progress.com/output/ua/OpenEdge_latest/pdsoe/PLUGINS_ROOT/com.openedge.pdt.langref.help/rfi1424920632352.html it states:

If either the source or the target object is a file, the target's code page defaults to -cpstream.

On a side note, test abends in OE on Windows if the target path does not exist on disk.

Stanislav Lomany wrote:

Do you have any idea what and how 4GL produces?

Try launching the 4GL using -cpinternal utf8 command-line parameter. This may shed some light on how it works. In my opinion, it assigns the € representation in UTF-8 to longchar variable using some UTF to default CP conversion which breaks the content, ignoring the previous fix-codepage statement.

Guys, I've been trying to experiment with -cpstream and -cpinternal parameters and found that when it comes to setting them to UTF-16/UTF-32 I can set -cpstream to to UTF-16 and that's it. For other cases it produces "Case table for code page UTF-16 and case name BASIC was not found in convmap.cp (1038)" error message or silenlty abends.
Do you have any idea how to fix it?

I know, I had the same problems :(.
Try setting the same value for all these cp parameters, unless having them different is your goal.

Try setting the same value for all these cp parameters

Errors are the same for this case too.

I cannot change -cpstream/-cpinternal parameters on a customer's VM either, so I cannot experiment with it.
Moreover, testcases/copy_lob/tests/bom.p fails on this VM because output produced for UTF-8 is different (using default -cpstream/-cpinternal).

Greg, what should we do about these parameters if we cannot be sure how they work?

I cannot change -cpstream/-cpinternal parameters on a customer's VM either, so I cannot experiment with it.

Is the testing issue caused by the virtual machine not having the needed codepages installed? Otherwise I don't understand why OE would not honor these command line specifications.

Is the testing issue caused by the virtual machine not having the needed codepages installed?

Some cases claim that "case table" for this particular case is missing in convmap.cp (quick googling didn't tell me how to add something to it). Some cases abend. And some lead to artefacts in screen output.

Please provide the list of codepages/scenarios which need testing (and which you cannot test).

Marian: I think we need your team to do this testing.

Greg Shah wrote:

Please provide the list of codepages/scenarios which need testing (and which you cannot test).

Marian: I think we need your team to do this testing.

• the supported codepages/collation combinations: i18n/4gl/cp_collation.xml
• the accepted conversion between codepages for codepage-convert: i18n/4gl/cp_conversion.xml
• the accepted conversion between codepages for chr: i18n/4gl/chr_conversion.xml
• the accepted conversion between codepages for input stream: i18n/4gl/is_conversion.xml

i18n/4gl/cp_collation.xml i18n/4gl/cp_conversion.xml i18n/4gl/chr_conversion.xml i18n/4gl/is_conversion.xml

Marian, these are theoretically allowed conversions. The problem is that I cannot set -cpstream/-cpinternal parameters to test how they work. I either get an error that "case table" for this particular case is missing in convmap.cp or it just abends.
Do you think you'll be able to set -cpstream/-cpinternal with a working result? If so, I'll make a testcase to be tested with all -cpstream/-cpinternal combinations.

Stanislav Lomany wrote:

i18n/4gl/cp_collation.xml i18n/4gl/cp_conversion.xml i18n/4gl/chr_conversion.xml i18n/4gl/is_conversion.xml

Marian, these are theoretically allowed conversions. The problem is that I cannot set -cpstream/-cpinternal parameters to test how they work. I either get an error that "case table" for this particular case is missing in convmap.cp or it just abends.

What are the values you're trying to use for the codepages/collation tables - for cpstream and cpinternal?

Do you think you'll be able to set -cpstream/-cpinternal with a working result? If so, I'll make a testcase to be tested with all -cpstream/-cpinternal combinations.

Yes, the idea is that if the combination is valid then there shouldn't be any error although is quite rare to have different codepages exactly because of the conversion overhead - the client uses the codepage appropriate for the user, the server often use the same codepage as client or at times some UTF if the database needs to support various codepages.

What are the values you're trying to use for the codepages/collation tables - for cpstream and cpinternal?

I want to test this ones:
UTF-8
UTF-16
UTF-16LE
UTF-16BE
UTF-32
UTF-32LE
UTF-32BE