记一次网络访问异常 ERR_CONNECTION_RESET 排查
145
|
杂活
|

1205 字
|
7 分钟

最近访问 chatgpt.com 的时候，不间断的报 ERR_CONNECTION_RESET，ChatGPT 桌面应用也报 net::ERR_CONNECTION_RESET，这个问题很恼火，非常影响日常生活（迫真），于是开始了排查。

首先以前出现过，我用 EasyTier 的时候把 IP 网段设置为 172.16.0.0/16，然后学校某个内网服务也在这个网段上导致访问异常的情况，于是我首先开始排查 chatgpt.com 的 DNS，看它解析到什么 IP 上，结果如下：

Server:         192.168.6.1
Address:        192.168.6.1#53

Name:   chatgpt.com
Address: 104.18.32.47
Name:   chatgpt.com
Address: 172.64.155.209

有时候两条记录会反过来，也就是 172.* 在前面，104.* 在后面。结合我体感观察，是一段时间报连接重置，一段时间正常，怀疑可能跟 DNS 解析顺序相关。

经实际排查确认了这个猜想，当 172.* 记录在前面的时候访问异常，104.* 在前面时访问正常。验证：

  • 当地址解析到 104.* 时连接正常：

      coder@dev-container:~$ curl -4 -v --resolve chatgpt.com:443:104.18.32.47 https://chatgpt.com/ -o /dev/null
  * Added chatgpt.com:443:104.18.32.47 to DNS cache
  * Hostname chatgpt.com was found in DNS cache
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                                   Dload  Upload   Total   Spent    Left  Speed
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 104.18.32.47:443...
  ...
  * Connected to chatgpt.com (104.18.32.47) port 443

  • 当地址解析到 172.* 时连接被异常重置：

      coder@dev-container:~$ curl -4 -v --resolve chatgpt.com:443:172.64.155.209 https://chatgpt.com/ -o /dev/null
  * Added chatgpt.com:443:172.64.155.209 to DNS cache
  * Hostname chatgpt.com was found in DNS cache
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                                   Dload  Upload   Total   Spent    Left  Speed
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 172.64.155.209:443...
  * ALPN: curl offers h2,http/1.1
  } [5 bytes data]
  * TLSv1.3 (OUT), TLS handshake, Client hello (1):
  } [1566 bytes data]
  *  CAfile: /etc/ssl/certs/ca-certificates.crt
  *  CApath: /etc/ssl/certs
  * Recv failure: Connection reset by peer
  * TLS connect error: error:00000000:lib(0)::reason(0)
  * OpenSSL SSL_connect: Connection reset by peer in connection to chatgpt.com:443 
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  * closing connection #0
  curl: (35) Recv failure: Connection reset by peer

接下来就是抓包 TCP 连接了，结果如下：

coder@dev-container:~$ sudo tcpdump -ni any 'host 172.64.155.209 and tcp port 443'
tcpdump: WARNING: any: That device doesn't support promiscuous mode
(Promiscuous mode not supported on the "any" device)
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
16:42:36.390774 eth0  Out IP 172.21.0.5.55664 > 172.64.155.209.443: Flags [S], seq 363907265, win 64240, options [mss 1460,sackOK,TS val 1944989153 ecr 0,nop,wscale 10], length 0
16:42:36.482537 eth0  In  IP 172.64.155.209.443 > 172.21.0.5.55664: Flags [S.], seq 3758968428, ack 363907266, win 65535, options [mss 1400,sackOK,TS val 3957740349 ecr 1944989153,nop,wscale 13], length 0
16:42:36.482568 eth0  Out IP 172.21.0.5.55664 > 172.64.155.209.443: Flags [.], ack 1, win 63, options [nop,nop,TS val 1944989245 ecr 3957740349], length 0
16:42:36.485747 eth0  Out IP 172.21.0.5.55664 > 172.64.155.209.443: Flags [P.], seq 1:1572, ack 1, win 63, options [nop,nop,TS val 1944989248 ecr 3957740349], length 1571
16:42:36.524450 eth0  In  IP 172.64.155.209.443 > 172.21.0.5.55664: Flags [R], seq 3758968429, win 54380, length 0
16:42:36.525041 eth0  In  IP 172.64.155.209.443 > 172.21.0.5.55664: Flags [R.], seq 1, ack 1572, win 1246, length 0
16:42:36.525067 eth0  In  IP 172.64.155.209.443 > 172.21.0.5.55664: Flags [R.], seq 1, ack 1572, win 1246, length 0
16:42:36.577201 eth0  In  IP 172.64.155.209.443 > 172.21.0.5.55664: Flags [.], ack 1, win 16, options [nop,nop,TS val 3957740444 ecr 1944989245], length 0
16:42:36.577222 eth0  Out IP 172.21.0.5.55664 > 172.64.155.209.443: Flags [R], seq 363907266, win 0, length 0
16:42:36.577270 eth0  In  IP 172.64.155.209.443 > 172.21.0.5.55664: Flags [.], ack 1389, win 16, options [nop,nop,TS val 3957740444 ecr 1944989248], length 0
16:42:36.577280 eth0  Out IP 172.21.0.5.55664 > 172.64.155.209.443: Flags [R], seq 363908654, win 0, length 0
16:42:36.577367 eth0  In  IP 172.64.155.209.443 > 172.21.0.5.55664: Flags [.], ack 1572, win 16, options [nop,nop,TS val 3957740444 ecr 1944989248], length 0
16:42:36.577370 eth0  Out IP 172.21.0.5.55664 > 172.64.155.209.443: Flags [R], seq 363908837, win 0, length 0

分析抓包结果发现，RST 相应不是由 172.* 正常发出的，而是中间链路伪造的，关键证据是：在收到 RST 约 52ms 后，172.* 仍发来了正常的 ACK。

由于本机先收到了伪造的 RST，因此认为连接已失效，连接被立即关闭，哪怕后续收到了正常的 ACK，也会被认为是连接异常，而回复对方 RST。

通过对多个外部 DNS 服务器的解析记录作为对比，发现一些 DNS 服务器将 172.* 记录放在前面，另一些放后面，解决方案就是让 chatgpt.com 这个域名仅使用后者这些 DNS 服务器完成解析。

上一篇
下一篇