Network Connection Debugging

When I set up a Kubernetes cluster with Rancher, some connections failed for no obvious reason. The culprit turned out to be a firewall rule. This is the workflow I use to track down problems like that.

1. Reproduce the problem

First, check whether a connection works at all and where it fails:

# HTTP(S): verbose output shows DNS, TCP and TLS steps
curl -v https://registry-1.docker.io/v2/

# Plain TCP: is the port reachable?
nc -vz registry-1.docker.io 443

2. Check the route

ip route get 203.0.113.10
traceroute -n -I 203.0.113.10

3. Look at open connections

ss -tanp

Connections stuck in SYN-SENT usually mean packets are dropped on the way.

4. Capture the traffic

sudo tcpdump -i eth0 -w capture.pcap port 443

Modern tcpdump versions capture full packets by default, so you don't need to set a snap length with -s.

5. Analyse the capture

• tcptrace gives a quick summary per connection: tcptrace capture.pcap, or tcptrace -o5 capture.pcap for the details of a single connection.
• Wireshark is the most convenient option: open capture.pcap and filter, for example, with tcp.analysis.retransmission or tcp.flags.reset == 1 to spot problems.

The actual problem in my case

The firewall only allowed return traffic to the ephemeral port range 32,768–65,535. The hosts, however, used local ports between 10,000 and 20,000 for their outgoing connections – so the replies were dropped.

You can check the local port range on Linux with:

cat /proc/sys/net/ipv4/ip_local_port_range

If connections fail seemingly at random, comparing this range with your firewall rules is worth a look.