Possible WASM related, JavaScript and Rust have UTF-8 related issues when reading backend responses

Compute
1

I have a very simple Compute example based off of both the Rust and JavaScript starter projects.

I have two backends, one is httpbin and one is a shopify mock data endpoint. I’m testing with these two backend URL / paths specifically:

https://httpbin.org/anything/asdfasdfasdfsda

and

https://mock.shop/api?query=%7B%20product(id%3A%20%22gid%3A%2F%2Fshopify%2FProduct%2F7982905098262%22)%20%7B%20id%20title%20description%20featuredImage%20%7B%20id%20url%20%7D%20%7D%7D

The httpbin response is completely fine and I have no issues accessing the response body with .json(), however I’m having issues with the shopify endpoint for reasons I don’t understand

I’ve tested this with both Rust and JavaScript, but I’ll stick with JavaScript error messages for now.

When I attempt to read the response body as JSON, e.g.

...snip snip...
async function handleRequest(event) {
... snip snip...
backendResponse = await fetch(event.request, {
    backend: "shopify",
  });
  const backendResponseBody = await backendResponse.json();

(or as .text())

I see something like this:

Error while running request handler: malformed UTF-8 character sequence at offset 1
Stack:
  handleRequest@<stdin>:1608:61
  async*bin/index.js/<@<stdin>:1513:64

I’ll add that if I don’t attempt to read the response body (e.g. I don’t do await backendResponse.json();) and simply return the backend response directly then I see no error, and the response is correct as I’d expect to see it.

For the particular use case I’m investigating I need to be able to access and modify this response body, so I need to be able to read it somehow. Note that httpbin and the shopify mock api are just for examples that I can share.

Am I missing something? Originally I ran into the issue with Rust and assumed it was my rusty Rust that I was running into the issue, but again the same error in the same situations seems to be happening in JS / WASM.

Thanks for any advice!

2

I wondered if this could have something to do with the chunked response type for whatever reason so I attempted to read the body in “manual mode”, e.g.

async function readChunkedBody(body) {

  const reader = body.getReader();
  let allDone = false;
  let finalOutput = '';

  do {
      let { value, done } = await reader.read();
      allDone = done;

      if (!allDone) {
          const chunkString = new TextDecoder('ascii').decode(value);                
          try {                                                                                      
              finalOutput = `${finalOutput}${chunkString}`;                        
          } catch(error) {                
            console.log(error);
          }
      }
  } while (!allDone);

  return finalOutput;

}

If I try UTF-8 encoding with the TextReader I see:

Screenshot 2024-06-12 at 4.24.55 PM

I’m guessing I’m barking up the completely wrong tree and this is just a limitation somewhere, perhaps even documented, with the response size or something along those lines, but I’m unsure at this point.

Thanks again for your help!

3

I have found a solution thanks to this blog post: Get started with Fastly Compute and JavaScript | Antoine Brossault

Apparently in some situations for reasons that are non-obvious to me, you can’t simply pass the request on as I was doing in my original example. You need to specify the actual URL for fetch. Again, I’m not sure why this works in some situations but not others but that seems to be the case.