Harness

Currently harness only supports hardcoded ISO-8859-1 character encoding for terminal sessions. In order to test different code pages or unicode text procesing it needs to support different character encodings for remote ssh terminal. As an option detecting remote terminal encoding could be done automatically.

Code Review Task Branch 9489a Revision 35

I have no objections, it looks reasonable.

I do wonder if (in the future) we will need different encodings for different driver threads or tests. It isn't something to solve now.

Eugenie: Please review.

Greg Shah wrote:

Eugenie: Please review.

I have no objections. It works fine with wide char linked native libp2j.so. The good point is we can just use different config options instead of rebuilding harness for particular tasks.

Greg Shah wrote:

Eugenie: Please use this with your #7657 testing.

Yes, I have already use this version instead of harness base/trunk.

Tomasz Domin wrote:

Based on Eugenie's report in #7657-216 I think there is a problem with conversion of multi-byte charsets coming from underlying terminal in Harness.
The problem is probably caused by non-stream based conversion of a byte stream. Bytes are incoming from underlying session in chunks up to Terminal.BUFFER_SIZE length.
Then conversion to Character stream occurs using a classic new String(byte []byteArray, String charset) call, which gives wrong result in case multi-byte character appears on byteArray end and does not fully fit in byteArray. In such case underlying layer default conversion rules produce non-decodable Unicode character 0xFFFD when CharsetDecoder try to decode incomplete character for multi-byte character encodings (like UTF-16 or UTF-8).
The solution would be to switch from byte array based character decoding to stream based character decoding. The difference is how CharsetDecoder behaves on chunks borders - for byte array decoding it assumes no more bytes would come, for stream based processing it will wait for next byte to come and produce non-decodable Unicode character 0xFFFD only when underlying stream is closed.

As support for different terminal encodings would be incomplete without that change I will continue my work here.

How about to increase the reading buffer size(Terminal.BUFFER_SIZE) to 8192 to cover full screen data (80x24x4bytes) with single buffer?

In wide char version the size of full screen in bytes is doubling.

Eugenie Lyzenko wrote:

How about to increase the reading buffer size(Terminal.BUFFER_SIZE) to 8192 to cover full screen data (80x24x4bytes) with single buffer?

In wide char version the size of full screen in bytes is doubling.

I don't think that would solve a general case, AFAIR terminal is not producing line based output so buffer is not filled with line and there is no direct relation here, it is producing stream of bytes to be decoded by VT320 emulator - where we get that line based output, but not earlier. IMHO increasing buffer size would only reduce probability the issue would happen.

I've created task branch 9489b based on 9489a. I had a working solution based on PipedInput/OutputStream but that requires additional threads I'd prefer to avoid.

Tomasz Domin wrote:

I've created task branch 9489b based on 9489a. I had a working solution based on PipedInput/OutputStream but that requires additional threads I'd prefer to avoid.

I've implemented java.nio based stream decoding in harness in 9489b revision 36.
Its under testing, but Eugenie you may try it for #7657.
ps. I've actually reduced buffer size as I needed two additional buffers and I didn't want memory consumption increase.

Eugenie - did you find the change working for you ? especially regarding #7657

Tomasz Domin wrote:

Eugenie - did you find the change working for you ? especially regarding #7657

Yes, your fixes are working for harness with wide char capable FWD, thanks a lot. The option -encoding utf-8 should be use when testing wide char FWD. The issue with 0xFFFD is not happening anymore.

Code Review Task Branch 9489b Revisions 36 and 37

In Terminal, shouldn't we have some code to handle cases where the Charset.forName() fails (bad charset name) or setup of the CharsetDecoder fails?

Otherwise I have no objections.

Greg Shah wrote:

Code Review Task Branch 9489b Revisions 36 and 37

In Terminal, shouldn't we have some code to handle cases where the Charset.forName() fails (bad charset name) or setup of the CharsetDecoder fails?

Otherwise I have no objections.

Initially I was thinking about checking charset on parameters parsing, then I left as I thought it would fail anyway in test initialization. But after I tested it now it seems bad charset only terminates Driver thread causing unnecessary delays to be handled by testing script.

Committed revision 9489b/38 with changes to check charset at beginning of test execution to fail fast in case its invalid.
Please review.

• for 7657a with following Harness options: -encoding utf-8
• for trunk with following Harness options: with and without -encoding utf-8 option

Results are identical. I guess its safe to merge it back to harness/trunk