FWD » Core Development » Base Language

From #5915-104:

FWD implementation for ENCRYPT produces different result than 4GL when the cipher requires IV and the provided IV is shorter than required. This includes AES in CBC/CFB/OFB modes and IV shorter than 16, DES3/DES in CBC/CFB/OFB modes and IV is shorter than 8. It is unclear what IV 4GL provides to the cipher in these cases. Maybe it is something like described in #5915-40.

The runtime implementation of this is in SecurityOps.encrypt()/decrypt(). There should be no conversion changes needed (the conversion occurs in rules/convert/builtin_functions.rules, search on prog.kw_encrypt or prog.kw_decrypt to find where we define the conversion mapping.

I have read some existing tests under testcases/security/security_policy and I tried to understand them. I tried many things but I couldn't recreate the issue. I used AES_CFB_128 and the IV was set to 2 (shorter than 16) on the following example. I executed the code below and it gave me identical results with FWD and OpenEdge.

def var k      as raw       no-undo.
def var iv     as raw       no-undo.
def var s      as character no-undo.
def var encr   as memptr    no-undo.
def var decr   as memptr    no-undo.
def var enchar as longchar  no-undo.
def var dechar as longchar  no-undo.

s = 'Hello world!'.
k = generate-pbe-key('k').
put-byte(iv, 1) = 2.

message 'Key: '              + string(k).
message 'IV length: '        + string(length(iv)).
message 'IV: '               + string(iv).
message 'IV string length: ' + string(length(string(iv))).

encr = encrypt(s, k, iv, 'AES_CFB_128').
decr = decrypt(encr, k, iv, 'AES_CFB_128').

enchar = base64-encode(encr).
message 'Encoded string: ' + string(enchar).

copy-lob from decr to dechar.
message 'Decoded string: ' + string(dechar).

I would like to ask for some help recreating the issue (if the issue still exists).

From #5915-40:

The problem is with generate-pbe-key, where AES_CBC_256 is used for this key (32 length). There is this interesting note here: https://community.progress.com/s/question/0D54Q00007ztLiuSAE/generatepbekey-inconsistent-between-net-for-large-keys

Just in case anyone comes across this and needs an answer. OpenEdge uses the OpenSSL encryption library under the hood. Specifically, the GENERATE-PBE-KEY() method returns the same output as OpenSSL's EVP_BytesToKey method. There's a C# version available here on Github (not my code).

With the OpenSSL notes here: https://www.openssl.org/docs/manmaster/man3/EVP_BytesToKey.html

KEY DERIVATION ALGORITHM
The key and IV is derived by concatenating D_1, D_2, etc until enough data is available for the key and IV. D_i is defined as:

        D_i = HASH^count(D_(i-1) || data || salt)
where || denotes concatenation, D_0 is empty, HASH is the digest algorithm in use, HASH^1(data) is simply HASH(data), HASH^2(data) is HASH(HASH(data)) and so on.

The initial bytes are used for the key and the subsequent bytes for the IV.

This may explain this comment:

      // Generate tail bytes, which is not  PBKDF1 compliant but 4GL does it in some way.
      // It is unclear how exactly it is implemented so we use the first 8 bytes of the
      // previous step as  a salt for a next chunk.

I think the issue in this task was fixed in trunk rev 13659. A customer's scenario which triggered this solution was:

def var lcdata as longchar.
def var lc as longchar.
def var rkey as raw.
def var ckey as char.
def var mMemData as memptr.
def var mMemPtr as memptr.

lcdata = "<encrypted-data>".
ckey = "<key>".

security-policy:symmetric-encryption-algorithm = "AES_CBC_256":u.
security-policy:symmetric-encryption-iv = ?.
rKey = generate-pbe-key(cKey).

// set the encryption key
security-policy:symmetric-encryption-key = rKey.
mMemData      = base64-decode(lcData).
set-size(mMemPtr) = 0.
mMemPtr = decrypt(mMemData, rKey).
if get-size(mMemPtr) = 0 
   then lc = "n/a".
   else copy-lob from mMemPtr to lc.

/* clear memory space after use */
set-size(mMemPtr) = 0.
set-size(mMemData) = 0.

message string(lc).

I don't recall exactly how <encrypted-data> and <key> was computed by the customer's code.

As far as I rememeber after the Constantin's fix we are 100% binary compatible with 4GL with respect to the encryption key. and the problem is with short IV only.
May be the analysys of the OpenSSL source code can give a clue.

I tried many combinations with AES_CBC_256. Some of them can be found below. I keep getting identical results with FWD and OpenEdge.

This example is copied and modified from #5915-39:

def var lcdata as longchar.
def var lc as longchar.
def var rkey as raw.
def var iv as raw.
def var ckey as char.
def var mMemData as memptr.
def var mMemPtr as memptr.

lcdata = "ZQSXDX+ZT4vlMRmgQr4JzcG9+Lw==".
ckey = "852564e3-27eb-3a99-3814-bb1830a2aa63".
put-string(iv, 1) = '6'.

security-policy:symmetric-encryption-algorithm = "AES_CBC_256":u.
security-policy:symmetric-encryption-iv = iv.
rKey = generate-pbe-key(cKey).

message string(lcdata).
message ckey.
message string(rkey).
message string(iv).

// set the encryption key
security-policy:symmetric-encryption-key = rKey.
lcData        = substring(lcData, 6).  // skip ZQSXD chars
mMemData      = base64-decode(lcData).
set-size(mMemPtr) = 0.
mMemPtr = decrypt(mMemData, rKey).
if get-size(mMemPtr) = 0 
   then lc = "n/a".
   else copy-lob from mMemPtr to lc.

/* clear memory space after use */
set-size(mMemPtr) = 0.
set-size(mMemData) = 0.

message string(lcData).
message string(lc).

Another example:

def var k      as raw       no-undo.
def var iv     as raw       no-undo.
def var s      as character no-undo.
def var encr   as memptr    no-undo.
def var decr   as memptr    no-undo.
def var enchar as longchar  no-undo.
def var dechar as longchar  no-undo.

security-policy:symmetric-encryption-algorithm = 'AES_CBC_256'.
s = 'Hello world!'.
k = generate-pbe-key('1').
put-string(iv, 1) = '2'.

message 'Key: '              + string(k).
message 'IV length: '        + string(length(iv)).
message 'IV: '               + string(iv).
message 'IV string length: ' + string(length(string(iv))).

security-policy:symmetric-encryption-iv = iv.
encr = encrypt(s, k).
decr = decrypt(encr, k).

enchar = base64-encode(encr).
message 'Encoded string: ' + string(enchar).

copy-lob from decr to dechar.
message 'Decoded string: ' + string(dechar).

Constantin Asofiei wrote:

I think the issue in this task was fixed in trunk rev 13659.

I think there is a big chance that the bug has already been fixed.

I'm OK with closing this task. Before we do that, please do/consider the following:

• Gather all of your various testcases and make sure the different variants are all represented the set of programs.
• Make changes to ensure that the processing is 100% automated. There should be no interactive usage.
• Let's determine how to locate/integrate these in the testcases project.
• What do we need to do to ensure that all of the encryption/decryption tests can be run as a single set?

Thank you Igor for your help regarding the test. I modified it a bit and added it under testcases/security/security_policy/attributes/test-encryption-iv.p. I forgot to mention that my p2j project is on the branch 6421a, which has a fix for #6421. The test has identical results with FWD and OE.