The XCODEd output is always 1 byte larger than the input. The byte is 0x13 and it is prefixed as the first byte in the XCODEd file. It might be a length of the encoded bytes. If so, other tests of XCODE should show this and the value is probably variable length too. It may just be a simple integrity check mechanism.

hello.p (18 bytes, ASCII encoded)

message "Hello!".

hello.p in hexadecimal notation:

0000000  6d 65 73 73 61 67 65 20 22 48 65 6c 6c 6f 21 22
0000010  2e 0a

encrypted_src/hello.p.encrypted_with_test (19 bytes, key is "test")

0000000  13 0b 7b d3 15 61 01 07 57 32 1b de 68 8c 37 72
0000010  2b e1 9b

encrypted_src/hello.p.encrypted_with_Progress (19 bytes, key is "Progress")

0000000  13 1b 60 76 44 5b 66 1c 2b 0a 60 4d d5 d5 57 80
0000010  83 15 27

encrypted_src/hello.p.encrypted_with_default_key (19 bytes, output is the same as using the key "Progress")

0000000  13 1b 60 76 44 5b 66 1c 2b 0a 60 4d d5 d5 57 80
0000010  83 15 27

We thus know the following:

• Dropping the fixed lead byte of 0x13 means that the encoded output has one encoded byte for every input byte, with no padding.
• The output is a function of the input key.
• The result is reversible (given the same key, the encoded text can be decoded into the original text).

The most obvious cipher approach to implement such output is the XOR Cipher. For this approach to be of any use, the byte being used for encoding ("adding" via xor to each input byte) must not be the same for the entire file. Otherwise decoding the input would be as simple as trying 256 possible byte values against the entire file and one of them would work.

This means that the encoding byte must change as the input is encoded. Using the above data:

• The first character of the input text is "m" which is 0x6d (binary 0110 1101).
• The first character of non-lead byte output using key "test" is 0x0b (binary 0000 1011).
• The xor value for the first character must be 0x66 (binary 0110 0110, lowercase "f" in ASCII).
• The second character of the input text is "e" which is 0x65 (binary 0110 0101).
• The second character of non-lead byte output using key "test" is 0x7b (binary 0111 1011).
• The xor value for the second character must be 0x1e (binary 0001 0110, which is a non-visual "record separator" character in ASCII).

This tells us that the simple approach of rotating through the key text is not used. In fact, the key text is not used without modification.

Knowing how ENCODE works, the CRC-16 would be expected to modify the key in a circular buffer and whatever byte was modified would be used as the xor value to be "added" to encode the next byte of the text.

Useful articles on CRC:

https://en.wikipedia.org/wiki/Cyclic_redundancy_check
http://www.qtcentre.org/threads/24881-CRC-16-0xA001-polynomial
https://stackoverflow.com/questions/23638939/crc-16-ibm-reverse-lookup-in-c
http://www.sunshine2k.de/articles/coding/crc/understanding_crc.html

The next step will be to modify the ENCODE implementation to generate xor bytes based on the initial key text. ENCODE uses a 16 byte circular buffer that additively modifies the resulting hash based on the CRC-16-IBM (with reversed polynomial 0xA001) as it walks through the input text from left to right.

We can easily test if there is a limit to the size of the key text that is honored, which may tell us the size of the buffer used. This can be determined by changing key values at the end of longer keys (i.e. using two keys with the same prefix text but differing deep in the key).

XCODE Replacement -- Specification and Implementation Plan (revision 3)¶

Building on my work from #3363-4 here is the plan and code specification that allows us to exactly duplicate the xcode results both encoding and decoding. The plan includes integration into the conversion pipeline and configuration so that xcode'd files will be automatically handled.

• XCODE Replacement -- Specification and Implementation Plan (revision 3)
  • 1. Status
  • 2. Specification
    • 2.1 File format
    • 2.2 Key derivation
    • 2.3 Initial vector
    • 2.4 The CRC primitive
    • 2.5 Encoding
    • 2.6 Decoding
    • 2.7 Detecting an XCODEd file
    • 2.8 Input reading -- text mode, and why it matters
    • 2.9 Test vectors
    • 2.10 Behavior of the command-line utility
  • 3. How the specification was determined
    • 3.1 Principles
    • 3.2 The corpora
    • 3.3 The harness
    • 3.4 The hypothesis framework
    • 3.5 Recovering the initial vector from captured output
    • 3.6 How the zero-byte rule was found
    • 3.7 Reproducing the whole thing
  • 4. Phase 2 -- Java implementation
    • 4.1 Encoding defaults
  • 5. Phase 3 -- conversion pipeline integration
    • 5.1 Target branch
    • 5.2 Integration points
    • 5.3 Configuration
  • 6. Phase 4 -- failure handling
  • 7. Phase 5 -- tests
  • 8. Risks and open items
  • 9. Sequencing

1. Status¶

The specification is complete. Every open question identified when this task was opened has been answered against a real OpenEdge 11.6.3 (64-bit) installation on Windows, and the algorithm below reproduces real xcode output byte for byte over 1025 independent checks covering every key length from 0 to 8 and beyond, input sizes from 0 to 100000 bytes, and every byte value that can appear in a source file.

Phase 1 is therefore complete and the implementation work can begin. Two findings changed the shape of the plan and are called out where they apply: the text-mode input read (section 2.8), which makes xcode lossy and forces a change to the Phase 5 assertion, and the -d path behavior (section 6), which affects the command-line tool.

2. Specification¶

This section is written to be implemented from directly. Everything in it is confirmed by measurement; nothing is inferred or assumed.

2.1 File format¶

An XCODEd file is a single leading byte 0x13 followed by one enciphered byte for every byte of input. There is no header, no trailer, no padding and no checksum.

+--------+----------------------------------------------+
| 0x13   | ciphertext, exactly as many bytes as input   |
+--------+----------------------------------------------+

Consequences worth stating plainly, because they constrain what an implementation can offer:

• Output length is always input length + 1. A zero-byte input produces a one-byte output containing only the lead byte.
• There is no room for a MAC or checksum, so a wrong key cannot be detected. Decoding with the wrong key yields silent garbage.
• 0x13 is DC3/XOFF, which cannot begin valid 4GL source, so it doubles as the detection signature.

2.2 Key derivation¶

The cipher consumes exactly 8 key bytes. They are produced by taking an 8-byte buffer preloaded with the default key Progress and copying the user key over it including the C string terminator, which lands in the buffer and leaves the remaining default-key bytes in place.

derive_key(user_key):
   buf = "Progress"                      /* 8 bytes, the default key */
   n   = min(length(user_key), 8)

   for i in 0 .. n-1:
      buf[i] = user_key[i]

   if length(user_key) < 8:
      buf[length(user_key)] = 0          /* the string terminator */

   return buf                            /* always exactly 8 bytes */

Worked examples, each confirmed against real xcode:

The surviving tail is a genuine quirk rather than an accident of our reading: a four-character key really does leave ess from Progress in the buffer, and that tail participates fully in the key schedule.

2.3 Initial vector¶

Eight iterations build a 16-byte state from the 8 key bytes. The starting index is taken from the low three bits of the fourth key byte, so two keys differing only at byte 3 produce completely different output.

init_iv(buf):
   ki = buf[3] AND 7
   iv[0 .. 15] = 0

   for n in 0 .. 7:
      if ki == 8:
         ki = 0

      m = (ki + 3) AND 7

      if buf[ki] == 0:
         a = 17 + n                      /* NOTE: plus the iteration count */
      else:
         a = (buf[ki] + ki) AND 127

      iv[n]      = a
      iv[ki + 8] = buf[m] XOR a

      ki = ki + 1

   return iv

The 17 + n term is the subtle part and the single easiest thing to get wrong. A zero key byte does not contribute a fixed constant; it contributes 17 plus the loop iteration counter. Because a zero byte can only enter the buffer as the string terminator, and because that terminator frequently lands on iteration 0 for four-character keys, an implementation that hardcodes a bare 17 will pass a casual test and then fail for most other key lengths. We lost considerable time to exactly that, and section 3.6 records how it was found.

The values are small and worth tabulating, since they make a good unit test on their own:

An 8-byte key contains no zero byte, so the branch never fires and any value would do -- which is precisely why the error is invisible at length 8.

2.4 The CRC primitive¶

XCODE uses ordinary CRC-16/ARC -- reversed polynomial 0xA001, processed least significant bit first. This is the same primitive already implemented for the ENCODE built-in under #27, though the surrounding usage differs: XCODE seeds the accumulator with 0x7FED and walks its input forward, where ENCODE seeds with 0x11 and iterates its array in reverse.

The bit-level form needs no table:

crc16(crc, value):
   crc = crc XOR (value AND 0xFF)

   repeat 8 times:
      if (crc AND 1) != 0:
         crc = (crc >> 1) XOR 0xA001
      else:
         crc = crc >> 1

   return crc AND 0xFFFF

The usual 256-entry table form is equivalent and faster; entry i of the table is the result of running the loop above over a starting accumulator of i with no input byte.

2.5 Encoding¶

encode(plaintext, user_key):
   buf = derive_key(user_key)
   iv  = init_iv(buf)
   crc = 0x7FED

   output = [0x13]                       /* the lead byte */

   for n in 0 .. length(plaintext)-1:
      m = n AND 15
      a = iv[15 - m]
      c = iv[(a + m) AND 15]

      output.append(plaintext[n] XOR c)

      crc   = crc16(crc, plaintext[n])   /* the PLAINTEXT byte */
      crc   = crc16(crc, c)
      iv[m] = crc AND 0xFF

   return output

Note the two-stage index: the state is read once to obtain an index, and that index selects the byte actually used for the XOR. The state is then overwritten at position m with the low byte of the running CRC, so the keystream is data dependent from the second byte onward.

2.6 Decoding¶

Decoding differs from encoding by exactly one line, and that line is the one implementations get wrong.

decode(blob, user_key):
   /* blob[0] is the 0x13 lead byte and is not enciphered */
   buf = derive_key(user_key)
   iv  = init_iv(buf)
   crc = 0x7FED

   plaintext = []

   for n in 0 .. length(blob)-2:
      m = n AND 15
      a = iv[15 - m]
      c = iv[(a + m) AND 15]

      p = blob[n + 1] XOR c
      plaintext.append(p)

      crc   = crc16(crc, p)              /* the RECOVERED PLAINTEXT byte */
      crc   = crc16(crc, c)
      iv[m] = crc AND 0xFF

   return plaintext

The cipher is not self-inverse. The CRC feedback consumes the plaintext byte in both directions, so a decoder must recover the plaintext byte first and feed that into the CRC. Feeding the ciphertext byte instead produces a correct first byte and garbage from the second onward -- a failure mode that looks like a wrong key rather than a coding error. Encode and decode should therefore be written as two functions that differ in that one line, and deliberately not collapsed into a single "run it again" implementation.

2.7 Detecting an XCODEd file¶

Test whether the first byte is 0x13. This is exact for any valid 4GL source, since 0x13 cannot begin a readable program in any encoding we support. A plaintext file whose first byte happened to be 0x13 would be a false positive, but such a file is not valid 4GL, so the limitation is acceptable and should simply be documented.

2.8 Input reading -- text mode, and why it matters¶

Real xcode on Windows reads its input in text mode, which is lossy in two distinct ways. Both were measured, not inferred.

Carriage returns. A CR that is immediately followed by LF is discarded; the LF survives. A CR in any other position survives untouched. It is a CRLF-to-LF collapse, not "CR stripping" -- a distinction that matters because getting it wrong would corrupt every line of a converted project.

The 0x1A byte. The read stops at the first 0x1A (DOS end-of-file). Everything from that byte onward is silently discarded, with no error and no warning. This is far more destructive than the newline handling:

A source file beginning with 0x1A encodes to nothing but the lead byte.

The cipher itself is fully byte transparent: every one of the 256 byte values round-trips exactly. All of the loss is in the read. The practical consequences are:

• decode(xcode(f)) recovers what xcode read, which for a file containing CRLF or 0x1A is not the file on disk. This is the reason the Phase 5 assertion is stated as it is in section 8.
• A customer's existing XCODEd files that originally contained 0x1A are already truncated. No decoder can recover that content; it was never encoded. This should be documented rather than worked around.
• Decoded source will have LF line endings even where the original had CRLF. That is harmless for conversion.

2.9 Test vectors¶

All measured from real xcode. The plaintext is hello.p -- the 18 bytes message "Hello!". followed by a single LF.

Three of these are load-bearing as regression tests and should be in the unit suite from the first commit:

• The abcdefg vector exercises 17 + n at iteration 3. An implementation with a hardcoded 17 fails it.
• The abcdefghXYZ vector confirms truncation at 8 bytes.
• The empty-key vector exercises a zero byte at buffer position 0.

Far larger vector sets are available in the archived captures; see section 3.

2.10 Behavior of the command-line utility¶

Established by measurement, and relevant to anyone writing a compatible tool:

• -k accepts an empty argument, a key containing spaces, and punctuation.
• -l lowercases the output filename only. Content is unaffected.
• Several files in one invocation produce byte-identical output to the same files run separately, so cipher state genuinely resets per file.
• -d <dir> appends the input path as given, not the basename. Passing corpus\x.p with -d out makes xcode attempt out\corpus\x.p and fail if that subdirectory does not exist. Invoke it from the input's own directory with bare filenames, or create the tree.

3. How the specification was determined¶

This section exists so the work can be repeated, audited or extended without repeating the dead ends. Everything described here is in phase1/ and runs unattended.

3.1 Principles¶

1. Every test file is valid 4GL. Byte-level payloads that could not survive as bare statements are carried inside .i include files, embedded in a 4GL comment, and pulled into a .p by an ordinary include reference -- which leaves both files independently valid and independently xcodeable.
2. One corpus serves determination and regression. The captured outputs become the golden vectors for the implementation, and the .p / .i / .cls files become the conversion round-trip fixtures. There is no second fixture set to keep in step.
3. Every open question is a named hypothesis with a mechanical verdict, so the work ends in a decision rather than an impression.
4. Corpus integrity is checked before anything runs. The corpus deliberately mixes LF and CRLF, high-bit bytes and a 0x1A; results from a corrupted corpus look perfectly valid while being worthless.

3.2 The corpora¶

Three corpora, each generated deterministically by a script and staged where the OpenEdge system can read it.

The design of the first corpus is worth recording because it is what made the lead-byte question decidable. Group A holds size-controlled files at 0, 1, 2, 4, 17, 18, 19, 64, 254, 255, 256, 257, 300, 511, 512, 65535, 65536, 65537 and 100000 bytes. Those sizes straddle the 8- and 16-bit boundaries deliberately: a length must change across that range and a constant cannot, so the group separates the two hypotheses whichever one is true. The 18-byte member is byte-identical to the original hello.p sample, which makes it a regression anchor -- its captured output must reproduce the vector already held, so a bad transfer or a wrong invocation is caught immediately instead of being mistaken for a discovery.

Group D isolates 0x1A in a file of its own rather than mixing it into a general byte-range payload. That decision paid for itself: when the byte-transparency test failed, the cause was immediately unambiguous instead of being blamed on the whole range.

3.3 The harness¶

The corpus is staged where the OpenEdge system can read it, the capture runs there, and the results are collected afterward. Corpus integrity is verified by SHA-256 on the development host both before and after, so the capture side is never relied on for it -- and a corpus that failed to arrive byte for byte stops the run rather than producing plausible nonsense.

3.4 The hypothesis framework¶

verify_results.py reports a verdict on each of the following and exits non-zero unless all are confirmed. Each verdict carries the consequence of a refutation, so a failure states what must change rather than merely that something is wrong.

3.5 Recovering the initial vector from captured output¶

This is the technique that made the key schedule tractable, and it is general enough to be worth stating separately. It recovers the initial vector without knowing the key at all.

Given a known plaintext and its ciphertext, the XOR stream is immediately known:

c[n] = plaintext[n] XOR ciphertext[n]

The CRC feedback consumes only plaintext[n] and c[n], both of which are now known, so the entire CRC sequence is computable -- and therefore every byte the algorithm writes into the state is known too:

W[n] = crc_after_step_n AND 0xFF,   written to position n AND 15

The only unknowns are the 16 original state bytes, and they are progressively overwritten by those known values. That leaves a small constraint system. Two structural facts make it solvable:

• From step 16 onward, every position the algorithm reads has already been overwritten. Those steps therefore constrain the stream but say nothing about the original state. A longer payload does not help; only the first 16 steps carry information.
• For steps 0 to 7 the index source iv[15-n] has not yet been overwritten, so it is an original state byte -- identical for every payload encoded under the same key. For steps 8 to 15 the index source has been overwritten, so it is known outright.

The second fact yields a free diagnostic. Because q(n) is payload independent for the first eight steps, comparing the XOR streams of several payloads under one key reveals directly which positions were read: a position in c[n] that is constant across payloads means the step read a still-original byte, and a position that varies means it read an already-overwritten one. That pins the index and hands back the state byte.

The solver is a depth-first search that branches only on a position some replay actually needs, so unread positions stay unknown rather than being guessed. It was validated against keys whose state was already known by other means; every position it determined matched exactly.

Two cautions learned the hard way. A solver that fills unread positions with a default and then reports "positions all solutions agree on" produces confident nonsense -- agreement on the filler is an artifact. And an under-determined state can point at a wrong answer that looks clean: with eight payloads one position read as 0x00 and implied a zero-byte contribution of 0x63; with thirty-two it read 0x77 and gave the correct 20. Determinacy must be established, not assumed.

3.6 How the zero-byte rule was found¶

Worth recording as a method, because the initial evidence pointed the wrong way.

The buffer rule of section 2.2 was confirmed for four-byte and eight-byte keys and appeared refuted for lengths 1, 2, 3, 5, 6 and 7. An exhaustive search over every unknown tail byte, crossed with all eight possible starting indices, found no candidate. A grid search over buffer rules, index rules, loop counts and wrap moduli found nothing better. Two independent captures were bit-identical, so the behavior was deterministic and a real rule had to exist.

The break came from tabulating which keys worked: exactly those where the starting index equalled the key length, plus every key of 8 bytes or more. That is logically impossible for a single fixed rule -- the same rule cannot work for one 7-character key and fail for another. It therefore could not be the buffer that varied, and had to be a term that depended on position within the loop.

Working the algebra by hand on two failing keys, using iv[k+8] = buf[(k+3) AND 7] XOR A(k) and the recovered state, recovered every buffer byte -- including the surviving Progress remnants -- confirming the buffer was right all along. That isolated the deviation to the single zero-byte term, which reduced the problem to sweeping one parameter. Each key then yielded a unique value, and the pattern was immediate once the values were indexed by the loop iteration at which the zero byte falls rather than by its position in the buffer.

3.7 Reproducing the whole thing¶

cd phase1
python3 xcode_ref.py                    # executable spec; must self-test clean
python3 gen_corpus.py --stage <staging directory>

   ... then, on the OpenEdge system:
   run_xcode.cmd -selftest              # validates the runner; needs no xcode
   run_xcode.cmd

python3 verify_results.py -r captures/<latest>/results

Run the self-test first. It verifies the corpus, proves the hashing tool works, and checks that the case matrix parses -- including the awkward cases, an empty key and a key containing a space -- without needing xcode and without changing anything. It is the cheapest way to discover that a capture would have been worthless.

Captures are archived to phase1/captures/<corpus>-<timestamp>/ holding the results, the exact corpus that produced them, the case matrix and a SHA-256 manifest over all of it -- a capture is only interpretable next to its inputs. The generators archive automatically before wiping a staging directory, because a capture is the evidence behind the specification and losing one to a routine regenerate would be both expensive and silent.

phase1/README.md carries the full inventory and the environment-specific traps.

4. Phase 2 -- Java implementation¶

New class com.goldencode.p2j.util.XCodeCipher. The util package is on both the conversion and the runtime classpath and sits beside the existing CRC work from #27.

XCODE is a conversion-time concern -- there is no 4GL XCODE statement -- so unlike #27 this does not belong inside SecurityOps.

• encode(byte[], String) and decode(byte[], String), kept as separate implementations for the reason in section 2.6.
• deriveKey(String) implementing section 2.2, and the 17 + n rule of section 2.3.
• isXCoded(byte[]) for detection.
• XCodeInputStream extends FilterInputStream for streaming decode. State is 16 bytes, so no buffering is required and files of any size are handled without reading them whole.
• textModeRead(byte[]) implementing section 2.8, needed to reproduce Windows xcode output in tests. It should not be applied on the decode path, which is byte exact by nature.
• Extract a shared single-byte crc16(int crc, int value) primitive rather than duplicating the polynomial loop. The existing SecurityOps.crc16(short[], int) cannot be reused directly because it iterates its array in reverse and seeds differently.

4.1 Encoding defaults¶

XCodeTool, the command-line utility, should mirror the real interface (-d, -k, -l, -) plus the decode mode Progress does not ship.

Encoding is OpenEdge compatible by default, which means applying the text-mode read of section 2.8 -- collapsing CRLF to LF and stopping at any 0x1A -- so that our output is byte identical to what xcode itself would have produced from the same source. That is the behavior a user expects from a drop-in replacement, and it is what makes our output interchangeable with an existing toolchain.

A -b (binary) option disables the translation and encodes the file exactly as it sits on disk. This is the honest choice when the input is not Windows-authored source, or when the caller wants a faithful round trip rather than compatibility, and it avoids silently truncating a file at a 0x1A.

The two modes differ only for files containing CRLF or 0x1A; for the large majority of source they produce identical output. Whichever mode is used, the tool should report when the translation actually changed the input -- a file that shrank on the way in is worth a line of output, and a file truncated by a 0x1A is worth a warning.

Decoding never applies the translation. Decode is byte exact by nature: it recovers precisely what was encoded, and inventing line endings on the way out would corrupt content that the format preserved faithfully.

5. Phase 3 -- conversion pipeline integration¶

5.1 Target branch¶

This work should be written in branch 11747a, not 3363a.

Two reasons. 11747a carries pending changes to the very file the integration touches -- FileScope there is some 185 lines longer than in 3363a, and its reader construction has changed in a way that matters (see below) -- so writing against 3363a would guarantee a merge conflict in exactly the wrong place. And 11747a is the branch needed for a project that actually has XCODEd files, so putting the support there is what makes it usable.

5.2 Integration points¶

Both confirmed by inspection of 11747a:

• src/com/goldencode/p2j/preproc/FileScope.java, the private getFileReader() at line 1052. A single choke point with exactly two callers, at lines 563 and 620, covering both main programs and include files.
• src/com/goldencode/p2j/uast/AstGenerator.java, prepareDataStream() at line 1123 -- the preprocess == false branch, which opens a FileInputStream at line 1136.

In 11747a, getFileReader() no longer returns a bare InputStreamReader. It now reads:

return new SurrogateFilterReader(new InputStreamReader(new FileInputStream(fileName),
                                                       charset));

where SurrogateFilterReader is a private static inner class of FileScope (line 1073). The decode therefore has to go inside both wrappers, at the InputStream layer:

return new SurrogateFilterReader(
   new InputStreamReader(XCodeInputStream.wrap(new FileInputStream(fileName), xcodeKey),
                         charset));

Critical design point: the decode must sit between FileInputStream and InputStreamReader. Putting it at the Reader layer would hand the charset decoder enciphered bytes and corrupt any non-ASCII source; putting it outside SurrogateFilterReader would bypass the surrogate handling that 11747a adds. Wrapping at the byte layer keeps both of those intact and lets them operate on the plaintext, which is what they were written for.

XCodeInputStream.wrap should sniff the leading 0x13 and return the stream unchanged when the file is not XCODEd, so the same call site serves both cases and no caller needs to know which it has.

5.3 Configuration¶

The key is configured the same way source-charset already is, which is a close precedent worth following rather than inventing something parallel: a global parameter that a profile can specialize, overridable per directory and per file.

1. Global. Add an xcode-key parameter to p2j.cfg.xml, read through Configuration.getParameter. Because profiles specialize configuration naturally, a project that uses one key throughout needs nothing further, and a project with a per-profile key gets that for free.
2. Per directory and per file. Add the key to UastHints as a hint, alongside source-charset. UastHints already initializes sourceCharset from Configuration.getParameter(SOURCE_CHARSET) and then lets the .hints sidecar override it; xcode-key should follow exactly that shape. Directory-level scope comes free through the existing UastHints.DIR_HINTS mechanism (directory plus the hints postfix), so a directory of files sharing a key needs one sidecar rather than one per file.

Concretely, mirroring the existing pieces:

Two details the source-charset path already settles, and which should be copied rather than re-litigated. An explicitly configured value wins over the sidecar -- that is what lets the standalone driver work at all, since it has no conversion configuration to read from. And UastHintsWorker.HintsReader.getUastString already exposes generic hints to the pattern engine, so rule code can read the key without further plumbing if it ever needs to.

An ordered list of candidate keys was considered and rejected. Per-directory and per-file hints express "different parts of this project use different keys" directly and legibly, where a list expresses it only as "try these until one appears to work" -- which, given that a wrong key cannot be detected (section 2.1), means guessing. The hint mechanism says which key belongs to which file; a list would only say which keys exist.

6. Phase 4 -- failure handling¶

Because no integrity check is possible (section 2.1), a wrong key produces silent garbage that will surface as a bewildering parser error hundreds of lines into what looks like binary noise. This needs designing deliberately rather than bolting on.

• A post-decode plausibility heuristic -- printable-character ratio and control-character scan -- applied before the decoded bytes reach the preprocessor.
• A diagnostic that names the file and the key that was used together with where that key came from -- the global parameter, a directory hint or a file hint -- rather than letting a parse failure stand as the only symptom. With per-file hints the question is almost always "which hint applied to this file", so the answer belongs in the message.
• Log which files were decoded, at info level, so a conversion run makes it evident that XCODEd input was involved.
• Fail loudly on the first file whose decode does not pass the heuristic, rather than continuing and burying the cause under a hundred parser errors. There is no key list to fall back through by design (section 5.3), so a failure here means the configured key is wrong for that file, which is a precise and actionable thing to say.

Two limitations to document rather than solve. A plaintext file whose first byte is 0x13 is a false positive for the sniff; it is not valid 4GL, so this is acceptable. And a source file that contained a 0x1A before being XCODEd is already truncated -- the content was never encoded and cannot be recovered. If a decoded file ends abruptly mid-construct, that is the likely explanation and the diagnostic should say so.

7. Phase 5 -- tests¶

The Phase 1 corpora are the fixture set and the captured outputs are the golden vectors. No separate fixtures are created.

The primary assertion is byte-level fidelity of the source, not of the generated Java:

For every (original, xcoded, key) triple, XCodeCipher.decode(xcoded, key) must equal textModeRead(original) byte for byte.

This is a correction to the assertion originally specified for this task. A plain comparison against the file on disk cannot hold, because xcode itself discards the CR of every CRLF pair and everything from any 0x1A onward (section 2.8). The loss is xcode's, not the decoder's, and textModeRead is exactly the difference. For the large majority of real source -- LF endings, no 0x1A -- the two statements coincide.

Comparing generated Java is deliberately not the test. It would conflate a decode defect with any unrelated change in conversion output, and would only detect problems that survive as far as code generation.

Beyond that:

• The eight measured vectors of section 2.9 as explicit golden-file tests, including the three called out as load-bearing.
• A round-trip property test over random payloads crossed with random key lengths.
• An explicit regression test for the not-self-inverse trap of section 2.6: a decoder that feeds the ciphertext byte to the CRC must fail it.
• An explicit regression test for 17 + n: an implementation with a hardcoded 17 must fail on the abcdefg vector.
• End to end: xcode a procedure together with its include, run the FWD conversion over them, and assert the source the preprocessor saw matches textModeRead of the originals.

8. Risks and open items¶

On placement: the Phase 1 harness currently lives in the task working directory. Since the corpora double as the Phase 5 fixture set, they should move into 11747a alongside the tests when Phase 2 begins, following the branch decision of section 5.1. Exactly where depends on how the round-trip test is wired, so that decision is deferred rather than guessed at now.

9. Sequencing¶

1. Phase 1 -- COMPLETE. The specification in section 2 is settled and measured.
2. Phase 2 -- implement XCodeCipher and XCodeTool against section 2, with the vectors of section 2.9 as unit tests from the first commit.
3. Phase 3 -- integrate at the two InputStream choke points and add the configuration surface.
4. Phase 4 -- failure handling and diagnostics.
5. Phase 5 -- the round-trip and end-to-end tests.

Phases 2 through 4 are largely independent of one another now that the specification is fixed, and could be worked in parallel if that is useful.

XCODE -- implementation complete through Phase 5¶

This is an update on the plan posted previously. All five phases are now complete.

It reports only what has changed since #3363-5: two additions to the specification, two changes to the plan itself, and the implementation as delivered. Everything not mentioned here stands as previously described -- the file format, key derivation, initial vector, CRC primitive, encode and decode, detection, the text-mode read, the test vectors, and the whole account of how the specification was determined are unchanged and were confirmed correct by the implementation.

1. Two additions to the specification¶

Both were measured on the same OpenEdge 11.6.3 installation, and both were established while designing the test suite rather than during the original investigation.

1.1 How OpenEdge itself consumes an XCODEd file (new section 2.11)¶

Four things follow, and two of them changed decisions:

• A non-default key is supplied per compile, as COMPILE <file> XCODE "<key>" .... Without the phrase the compiler reads ciphertext as source and fails. This is useful corroboration for the configuration design already described in section 5.3: OpenEdge also treats the key as a property of the individual compilation rather than of the installation, so a per-file hint is consistent with how the 4GL already works rather than an invention of ours.
• XCODEd include files are decoded too, and a single XCODEd include changes the behavior of an otherwise plaintext program. The include path is therefore a first-class part of the feature, which is what makes FileScope.getFileReader() -- the choke point both main programs and includes pass through -- the right integration point.
• OpenEdge never writes a preprocessor listing for XCODEd input. The compile succeeds and produces r-code, but the listing is empty regardless of the key and regardless of whether the XCODEd part was the program or one of its includes. This is evidently deliberate; writing the listing would spill the protected source to disk. It has a direct consequence for testing, covered in section 3 below.
• OpenEdge decodes XCODEd source transparently at compile time with no option needed for the default key, which confirms the format we implemented is exactly what the compiler consumes.

1.2 Decoded source reaches the conversion cache, deliberately (new section 2.12)¶

FWD conversion writes a .cache file holding the preprocessed source, and for XCODEd input that content is decoded. A project that XCODEd its source will therefore find the plaintext written to disk by a conversion run. OpenEdge, as above, declines to do the equivalent.

This is accepted and intended, and no suppression or encryption of the cache will be implemented. XCODE is not a meaningful protection mechanism in the first place, and more decisively: to convert an XCODEd file at all one must already hold the key, so withholding the plaintext from whoever supplied the key protects nothing. Recorded so the behavior is a documented decision rather than an oversight discovered later.

2. Phase 4 is deliberately empty¶

The previous post specified a post-decode plausibility heuristic, a diagnostic naming the file and the key's origin, and failing loudly on the first bad decode. None of it was built, and that is a decision rather than an omission.

The heuristic was written and measured first. Judging decoded content on NUL bytes and the proportion of control characters, it caught 61 of 62 wrong-key decodes across the corpus, the single miss being a one-byte file. It was then removed, for two reasons that outweigh the diagnostic value:

• It could refuse a legitimate file, with no override. A source file carrying substantial binary content inside a comment is indistinguishable from a wrong-key decode by this measure -- roughly 10% control characters against random data's 11%. Only the NUL test separates them, and that alone is weak on short files. No threshold separates the two cases, so the choice was strict-and-occasionally-wrong or lax-and-usually-useless. One file in our own corpus was rejected by it.
• It put a heuristic on the critical path of a pipeline that otherwise fails deterministically. Being wrong in either direction is a different quality of behavior from the rest of the conversion.

A wrong key therefore produces the same cryptic parse failure it always would -- honest, deterministic, and leaving the door open to a diagnostic later if experience shows one is needed. Two designs are recorded in section 6 should that happen: emit the message as a warning and let the content proceed, which never blocks a valid conversion; or make the check opt-in through a configuration parameter.

The two limitations previously listed remain documented rather than solved: a plaintext file whose first byte is 0x13 is a false positive for the sniff, and a source file that contained a 0x1A before being XCODEd is already truncated and unrecoverable.

3. Phase 5 is a Harness suite, not the unit tests previously described¶

The previous post described the assertion and a set of unit-style tests, with the fixtures moving into the FWD branch alongside test/. That has changed in three ways.

The suite is in the Testcases project, at tests/conversion/xcode/, not in the FWD source tree. It follows the layout and conventions of the preprocessor suite in tests/conversion/preprocessor/.

It is driven by the Harness, with no ABLUnit and no JUnit. The reasons the preprocessor suite recorded for rejecting ABLUnit apply here and more strongly: a wrong key and a 0x1A truncation cannot be expressed as an assertion; XCODE finishes before the lexer sees a byte, so there is no 4GL API for a test method to call; and byte-exact output is a comparison problem, which for a binary artifact it doubly is. A separate JUnit conformance suite proved unnecessary once the measured vectors were expressed as harness tests driving XCodeTool -- that keeps one runner and, more importantly, makes the tests assert agreement with OpenEdge rather than with ourselves.

There are three levels rather than one assertion.

Levels 1 and 2 use a binary comparison deliberately. The harness text comparison treats LF, a bare CR and CRLF alike as line terminators, which is right for a listing and actively harmful for ciphertext -- it would let real corruption sit in the repository without any test failing, which has already happened once in the preprocessor suite.

Level 3 is differential, and that is forced by the finding in section 1.1 above. OpenEdge writes no listing for XCODEd input, so no baseline can be captured for one. Baselining the plaintext twin from OpenEdge does not work either: FWD and OpenEdge disagree about blank line emission, which has nothing to do with XCODE and would fail the test for an unrelated reason. Both halves are therefore run through the same preprocessor and compared to each other, which cancels every such deviation and leaves exactly one variable -- whether decoding changed the result. Neither file is committed; both are produced live by one ant target, so there is no baseline to drift when the preprocessor legitimately changes, and nothing asserts that FWD agrees with itself about preprocessing.

3.1 Result¶

133 Harness tests, all passing, in roughly 50 seconds. Coverage carries the whole measured corpus: input sizes from 0 to 100000 bytes, every key length from 0 to 8, truncation beyond 8, key case and punctuation, byte transparency, the line-ending and 0x1A probes, and realistic 4GL sources including an include.

The suite was verified to be capable of failing: corrupting one byte of one baseline turns the run red.

4. What was built¶

The configuration surface is as previously described and was implemented unchanged: a global xcode-key parameter in p2j.cfg.xml, overridable per directory and per file through UastHints, mirroring source-charset piece for piece.

Three things were needed that the plan did not anticipate:

1. The standalone Preprocessor driver gained an -xcodekey option. It has no conversion configuration to read a hint from, exactly as with -charset, and the Phase 5 suite drives that driver.
2. FileScope.getFileReader() now also throws IOException, because the sniff reads a byte. Both callers previously caught only FileNotFoundException and were widened. That is precisely scoped -- FileInputStream could only ever throw FileNotFoundException -- but it does mean a genuine read error is reported as "could not open" rather than propagating, which was judged better than aborting a conversion run.
3. XCodeTool creates its output directory tree rather than refusing. Since -d appends the input path as given, any input named by a relative path lands several directories down, and requiring the caller to precreate that tree made -d unusable for the case it exists to serve.

5. Three defects found by testing rather than review¶

Each is worth recording, because none would have been caught by the layer that introduced it.

A null key decoded as the empty key. The first implementation collapsed null to "" -- defensible in isolation, and wrong the moment the pipeline uses null to mean "no key hint applies here". The empty string is a distinct, legitimate key yielding the buffer \0rogress, while a null key must select the default. Every default-key file decoded into garbage. The 65-case conformance fixture passed throughout, because it never exercised a null key; the fault appeared only when the real pipeline was run end to end. This is now section 4.1, and the suite pins it explicitly.

The conformance fixture could not have pinned that defect. It recorded only the key's value, so "no -k given" and "-k Progress" were indistinguishable -- the same key, but not the same code path, and only the former exercises the default. The fixture gained a column recording whether -k was actually passed, and levels 1 and 2 each have a -nokey ant target so the distinction is structural rather than a convention. v_default and b_default cover the absent key, v_progress and k_prog_p0 the explicit Progress, and v_empty and b_empty the empty key.

A sensitivity check that could not fail. The first negative control against the suite overwrote a baseline byte with the value it already held and reported success. A negative control that silently does nothing is indistinguishable from a passing suite; with a real mutation the harness correctly reports failure.

6. Risks, updated¶

7. What is not done¶

• Nothing is committed, in either branch.
• No diagnostic for a wrong key, by the decision in section 2 above.
• UNIX xcode is unmeasured. Everything here describes the Windows utility. A UNIX xcode plausibly reads in binary, which would change its output for CRLF sources; one capture would settle it.
• High-bit key bytes are unmeasured, being codepage dependent to pass through a shell. Worth a follow-up only if a customer uses one.
• Only OpenEdge 11.6.3 is measured. Nothing suggests version sensitivity, and each capture records the version, so the question stays answerable.

I've committed the changes in 11747a as revision 16721. The commit message:

Added XCODE support, so a project whose 4GL source was obscured with the XCODE utility converts without being decoded by hand first. XCodeCipher implements the cipher and its key schedule, XCodeInputStream decodes as a stream and XCodeTool provides a command line encoder and decoder, defaulting to OpenEdge-compatible output with -b for binary. Decoding is wired in at FileScope.getFileReader(), between FileInputStream and InputStreamReader. The key comes from a global xcode-key parameter, overridable per profile, per directory and per file through UastHints, with Options and a new Preprocessor -xcodekey option for the standalone driver, which has no conversion configuration to read a hint from. Three details are load-bearing and easy to get wrong: a zero key byte contributes 17 plus the loop iteration counter rather than a fixed constant, the cipher is not self-inverse because the CRC feedback always consumes the plaintext byte, and a null key selects the default key while the empty string is a distinct key in its own right.

I committed the tools needs to re-capture baselines in testcases revision 1894:

Added tests/conversion/xcode/tools, the tooling needed to re-capture the XCODE baselines and regenerate the suite from them, so that neither depends on a working directory outside this project. xcode_ref.py is an independent implementation of the cipher which self-tests against every fixture in the suite; it is what allows a capture to be verified without trusting FWD, which is the whole reason for capturing from OpenEdge rather than generating baselines ourselves. make_capture_kit.py assembles a self-contained directory to copy to the OpenEdge machine -- the 65 inputs, the case to key matrix, a SHA-256 manifest and run_xcode.cmd, which needs only cmd.exe and nothing installed -- and verify_capture.py checks what comes back before it is adopted as a baseline, confirming that every output has the right shape, that no case is missing, that the specification reproduces every captured file, and reporting any case that differs from the committed baseline. regen_suite.py rebuilds the harness tests and the test plan from the support files, with a --check mode that reports drift without writing, since 133 near-identical XML files are exactly what drifts when edited by hand and an unregistered test is silently never run. tools/analysis holds the programs that derived the specification and built the corpora in the first place: the initial-vector and key-schedule solvers, the three corpus generators, the conformance fixture builder, the capture archiver, a mock utility for testing the harness itself, and the batch runner linter. None of those is needed for a routine re-capture and most expect the archived captures, which are held outside this project because of their size; they are here so the work is recreatable and not merely repeatable. xcode_pp_keyed.xml is regenerated so its key quoting matches the other 132 tests, which is the only behavioral change -- the suite still passes 133 of 133.

The captures (actual data files) are archived in ~/secure/shared/projects/p2j/xcode/xcode_captures_from_redmine_3363_20260828.tar.gz.