Missing response packets for HTTPS+IPv6 requests

Network Services
1

I’ve noticed that, on a particular network (both on their wifi and their wired connections), requests that go over HTTPS and IPv6 to sites using the Fastly CDN are regularly slow.

It seems the slowness falls into ‘buckets’: some requests are healthy (~100ms), some take around 3.5s, some seem to hang more or less indefinitely.

Looking with wireshark, it seems a response packet is not received, and the delay is due to waiting for the retransmission timeout. Notice packet 6 has the unexpected Seq=4285, then after a ~3.5s delay the data comes in over a couple of packets:

    1   0.000000 2a02:166b:71:c2:921b:132e:5dc7:bef8 → 2a04:4e42::644 TCP 94 46134 → 443 [SYN] Seq=0 Win=64800 Len=0 MSS=1440 SACK_PERM TSval=1260631689 TSecr=0 WS=128
    2   0.014288 2a04:4e42::644 → 2a02:166b:71:c2:921b:132e:5dc7:bef8 TCP 94 443 → 46134 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1440 SACK_PERM TSval=961702020 TSecr=1260631689 WS=512
    3   0.014334 2a02:166b:71:c2:921b:132e:5dc7:bef8 → 2a04:4e42::644 TCP 86 46134 → 443 [ACK] Seq=1 Ack=1 Win=64896 Len=0 TSval=1260631704 TSecr=961702020
    4   0.019496 2a02:166b:71:c2:921b:132e:5dc7:bef8 → 2a04:4e42::644 TLSv1 603 Client Hello
    5   0.032726 2a04:4e42::644 → 2a02:166b:71:c2:921b:132e:5dc7:bef8 TCP 86 443 → 46134 [ACK] Seq=1 Ack=518 Win=143872 Len=0 TSval=961702039 TSecr=1260631709
    6   0.034821 2a04:4e42::644 → 2a02:166b:71:c2:921b:132e:5dc7:bef8 TCP 579 [TCP Previous segment not captured] 443 → 46134 [PSH, ACK] Seq=4285 Ack=518 Win=143872 Len=493 TSval=961702042 TSecr=1260631709 [TCP segment of a reassembled PDU]
    7   0.034830 2a02:166b:71:c2:921b:132e:5dc7:bef8 → 2a04:4e42::644 TCP 98 [TCP Dup ACK 3#1] 46134 → 443 [ACK] Seq=518 Ack=1 Win=64896 Len=0 TSval=1260631724 TSecr=961702039 SLE=4285 SRE=4778
    8   3.487915 2a04:4e42::644 → 2a02:166b:71:c2:921b:132e:5dc7:bef8 TLSv1.3 1294 [TCP Retransmission] , Server Hello, Change Cipher Spec, Application Data
    9   3.487935 2a02:166b:71:c2:921b:132e:5dc7:bef8 → 2a04:4e42::644 TCP 98 46134 → 443 [ACK] Seq=518 Ack=1209 Win=63744 Len=0 TSval=1260635177 TSecr=961705447 SLE=4285 SRE=4778
   10   3.500350 2a04:4e42::644 → 2a02:166b:71:c2:921b:132e:5dc7:bef8 TCP 306 [TCP Retransmission] 443 → 46134 [ACK] Seq=1209 Ack=518 Win=143872 Len=220 TSval=961705507 TSecr=1260635177 [TCP segment of a reassembled PDU]
   11   3.500368 2a02:166b:71:c2:921b:132e:5dc7:bef8 → 2a04:4e42::644 TCP 98 46134 → 443 [ACK] Seq=518 Ack=1429 Win=63616 Len=0 TSval=1260635190 TSecr=961705507 SLE=4285 SRE=4778
   12   3.500916 2a04:4e42::644 → 2a02:166b:71:c2:921b:132e:5dc7:bef8 TCP 1294 [TCP Retransmission] 443 → 46134 [ACK] Seq=1429 Ack=518 Win=143872 Len=1208 TSval=961705507 TSecr=1260635177 [TCP segment of a reassembled PDU]
   13   3.500934 2a02:166b:71:c2:921b:132e:5dc7:bef8 → 2a04:4e42::644 TCP 98 46134 → 443 [ACK] Seq=518 Ack=2637 Win=62464 Len=0 TSval=1260635190 TSecr=961705507 SLE=4285 SRE=4778
   14   3.513191 2a04:4e42::644 → 2a02:166b:71:c2:921b:132e:5dc7:bef8 TCP 1294 [TCP Retransmission] 443 → 46134 [ACK] Seq=2637 Ack=518 Win=143872 Len=1208 TSval=961705519 TSecr=1260635190 [TCP segment of a reassembled PDU]
   15   3.513214 2a02:166b:71:c2:921b:132e:5dc7:bef8 → 2a04:4e42::644 TCP 98 46134 → 443 [ACK] Seq=518 Ack=3845 Win=61312 Len=0 TSval=1260635202 TSecr=961705519 SLE=4285 SRE=4778
   16   3.513313 2a04:4e42::644 → 2a02:166b:71:c2:921b:132e:5dc7:bef8 TLSv1.3 526 [TCP Retransmission] , Application Data, Application Data, Application Data, Application Data
   17   3.513322 2a02:166b:71:c2:921b:132e:5dc7:bef8 → 2a04:4e42::644 TCP 86 46134 → 443 [ACK] Seq=518 Ack=4778 Win=61056 Len=0 TSval=1260635203 TSecr=961705519
   18   3.514560 2a02:166b:71:c2:921b:132e:5dc7:bef8 → 2a04:4e42::644 TLSv1.3 166 Change Cipher Spec, Application Data
   19   3.514700 2a02:166b:71:c2:921b:132e:5dc7:bef8 → 2a04:4e42::644 TLSv1.3 172 Application Data
   20   3.515190 2a02:166b:71:c2:921b:132e:5dc7:bef8 → 2a04:4e42::644 TLSv1.3 153 Application Data
   21   3.526794 2a04:4e42::644 → 2a02:166b:71:c2:921b:132e:5dc7:bef8 TCP 86 443 → 46134 [ACK] Seq=4778 Ack=598 Win=143872 Len=0 TSval=961705534 TSecr=1260635204
   22   3.527112 2a04:4e42::644 → 2a02:166b:71:c2:921b:132e:5dc7:bef8 TCP 86 443 → 46134 [ACK] Seq=4778 Ack=684 Win=143872 Len=0 TSval=961705534 TSecr=1260635204
   23   3.527412 2a04:4e42::644 → 2a02:166b:71:c2:921b:132e:5dc7:bef8 TLSv1.3 145 Application Data
   24   3.527545 2a02:166b:71:c2:921b:132e:5dc7:bef8 → 2
a04:4e42::644 TLSv1.3 117 Application Data
   25   3.528179 2a04:4e42::644 → 2a02:166b:71:c2:921b:132e:5dc7:bef8 TCP 86 443 → 46134 [ACK] Seq=4837 Ack=751 Win=143872 Len=0 TSval=961705534 TSecr=1260635204
   26   3.528505 2a04:4e42::644 → 2a02:166b:71:c2:921b:132e:5dc7:bef8 TLSv1.3 527 Application Data
   27   3.528812 2a02:166b:71:c2:921b:132e:5dc7:bef8 → 2a04:4e42::644 TCP 86 46134 → 443 [FIN, ACK] Seq=782 Ack=5278 Win=64128 Len=0 TSval=1260635218 TSecr=961705535
   28   3.540839 2a04:4e42::644 → 2a02:166b:71:c2:921b:132e:5dc7:bef8 TCP 86 443 → 46134 [ACK] Seq=5278 Ack=782 Win=143872 Len=0 TSval=961705547 TSecr=1260635217
   29   3.541850 2a04:4e42::644 → 2a02:166b:71:c2:921b:132e:5dc7:bef8 TCP 86 443 → 46134 [ACK] Seq=5278 Ack=783 Win=143872 Len=0 TSval=961705549 TSecr=1260635218
   30   3.542440 2a04:4e42::644 → 2a02:166b:71:c2:921b:132e:5dc7:bef8 TLSv1.3 110 Application Data
   31   3.542457 2a02:166b:71:c2:921b:132e:5dc7:bef8 → 2a04:4e42::644 TCP 74 46134 → 443 [RST] Seq=783 Win=0 Len=0
   32   3.542628 2a04:4e42::644 → 2a02:166b:71:c2:921b:132e:5dc7:bef8 TCP 86 443 → 46134 [FIN, ACK] Seq=5302 Ack=783 Win=143872 Len=0 TSval=961705549 TSecr=1260635218
   33   3.542642 2a02:166b:71:c2:921b:132e:5dc7:bef8 → 2a04:4e42::644 TCP 74 46134 → 443 [RST] Seq=783 Win=0 Len=0

This is likely some problem with the infrastructure at this site, but what would be good next steps to diagnose / narrow down the problem?

2

Hi @raboof

Thanks for reaching out.

Just to let you know that I’m going to enquire internally about what debugging steps you could take and get back to you.

1 Like
3

Hi @raboof

Below is the feedback I received but I would also recommend reaching out to support@fastly.com (so you may provide some of the details as requested below). This will also help with tracking your enquiry and improve the support feedback loop/effectiveness.

I have some initial questions:

  • Is this a Fastly customer or a single end user reporting an issue from a particular network?
  • If it’s a customer, can we get their Customer/Service ID?
  • Do we know the edge POP or site the user is trying to hit?
  • Does this issue happen across all sites via Fastly or just 1?

My initial thoughts of what we would need to continue debugging are:

  • MTR/Traceroutes via ipv6 from client > fastly
  • Can the user send across the pcap file for further analysis?
  • Does IPV4 testing result in the same issues or all good on that side?
  • Client IP/ASN

If you’re able to provide some of the requested details (again, I’d recommend sending to support@fastly.com and referencing this reply) then we can dig into this more deeply for you.

Thanks!