Base64-decoded strings cuts off at hex value 00


#1

Hello,

We have two base64-encoded strings whose raw bits need to be concatenated together, and then the resulting binary be base64-encoded again. In python, we would do

$ python
import base64
str_a = 'l70aCVczVOG6WhZy6G6y7A=='
str_b = 'BnltAWAAdGBZuNgm’
base64.b64decode(str_a).encode(‘hex’)
'97bd1a09573354e1ba5a1672e86eb2ec’
base64.b64decode(str_b).encode(‘hex’)
'06796d016000746059b8d826’
concatenated_bits = base64.b64decode(str_a) + base64.b64decode(str_b)
concatenated_bits.encode(‘hex’)
'97bd1a09573354e1ba5a1672e86eb2ec06796d016000746059b8d826’
base64.b64encode(concatenated_bits)
‘l70aCVczVOG6WhZy6G6y7AZ5bQFgAHRgWbjYJg==’

We’re trying to port this logic over to the Fastly VCL.

declare local var.str_a STRING;
declare local var.str_b STRING;
declare local var.str_c STRING;
set var.str_a = “l70aCVczVOG6WhZy6G6y7A==”;
set var.str_b = “BnltAWAAdGBZuNgm”;
set var.str_c = digest.base64_decode(var.str_a) + digest.base64_decode(var.str_b);
set var.str_c = digest.base64(var.str_c);
set resp.http.x-str-a = var.str_a;
set resp.http.x-str-b = var.str_b;
set resp.http.x-str-c = var.str_c;

However, it seems like the results aren’t as expected.

$ curl -I -H “Host: vcl-experiments.mntd.netnonssl.global.fastly.net | grep x-str
x-str-a: l70aCVczVOG6WhZy6G6y7A==
x-str-b: BnltAWAAdGBZuNgm
x-str-c: l70aCVczVOG6WhZy6G6y7AZ5bQFg

Taking a closer look at the result for str_c:

$ python
import base64
base64.b64decode(‘l70aCVczVOG6WhZy6G6y7AZ5bQFg’).encode(‘hex’)
‘97bd1a09573354e1ba5a1672e86eb2ec06796d0160’

Here are the expected and observed values:
97bd1a09573354e1ba5a1672e86eb2ec06796d016000746059b8d826
97bd1a09573354e1ba5a1672e86eb2ec06796d0160

It looks like the data got cut off at the value 00. I’m guessing when we ran:
set var.str_c = digest.base64_decode(var.str_a) + digest.base64_decode(var.str_b);
The concatenation of the raw bytes hit the 00 and interpreted it as an “end of string” character. This is how C strings behave, after all. In fact, this behavior is present even if we do a simple encode-decode without concatenation:

set resp.http.x-str-d = digest.base64(digest.base64_decode(var.str_b));
x-str-d: BnltAWA=

Is there a way to get the result that we want, or is this calculation impossible on Fastly? If we had a digest.base64decode_hex() and a digest.base64_hex(), I could get around this problem. But right now these functions seem to only take ascii-encoded strings as arguments, which terminate with 00.


#2

Hi James,

You are correct: at the moment strings in VCL are C-like and are NUL (0x00) terminated.

Unfortunately this means that you can’t base64 decode strings which might contain NUL characters. Any bytes following the NUL byte will be ignored.

Could you explain your use case a little? Why would you concatenate two binary strings?

For popular cases, such as digest.awsv4_hmac() for AWSv4 message authentication we have built a separate function to take the calculation out of VCL and into C.

Regards, Léon