The goal of this article is to review and understand the node network interfaces, and how network communications works in a Kubernetes cluster.
My server has a kubeadm installation with Flannel as a CNI plugin. By default, Flannel has the IP range ‘10.244.0.0/16‘ for the Pods network. Each Kubernetes node will have a subnet of ‘/24’ for Pods on that node.
Let’s start by listing my Kubernetes Control Plane node network interfaces which have an assigned IP address.
root@vps-2153e875:~# ifconfig | grep -B1 ' inet '
cni0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.244.0.1 netmask 255.255.255.0 broadcast 10.244.0.255
--
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 135.125.174.14 netmask 255.255.255.255 broadcast 0.0.0.0
--
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.244.0.0 netmask 255.255.255.255 broadcast 0.0.0.0
--
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0From the previous capture, we can see that ‘ens3‘ (135.125.174.14) is the main network interface of the server, which has a public range IP assigned. The network interfaces ‘cni0‘ (10.244.0.1) and ‘flannel.1‘ (10.244.0.0) are network interfaces which were created when I installed Kubernetes on this server.
- cni0 (Bridge): Handles intra-node traffic. It connects all pods on the same node so they can talk to each other directly at Layer 2.
- flannel.1 (VXLAN Device): Handles inter-node traffic. It is a VXLAN Tunnel Endpoint (VTEP) that encapsulates the original pod packet into a UDP packet to “tunnel” it across the physical network to the destination node.
Given my Kubernetes installation has a single node, then the node routing table looks as follows.
root@vps-2153e875:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 135.125.174.1 0.0.0.0 UG 100 0 0 ens3
10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0
135.125.174.1 0.0.0.0 255.255.255.255 UH 100 0 0 ens3On a Kubernetes installation with 2 nodes, the routing table on the Control Plane node would show as follows. Network packets destined to 10.244.0.0/24 (this node Pods) will go through the cni0 interface, and packets destined to 10.244.1.0/24 (the other node Pods) will go through the flannel.1 interface. Any other network packet destined for external IPs will use the server interface ens3, and it will go through the gateway ‘135.125.174.1‘.
root@vps-2153e875:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 135.125.174.1 0.0.0.0 UG 100 0 0 ens3
10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0
10.244.1.0 10.244.1.0 255.255.255.0 UG 0 0 0 flannel.1
135.125.174.1 0.0.0.0 255.255.255.255 UH 100 0 0 ens3With the following command we have the flannel.1 VXLAN interface configuration details.
root@vps-2153e875:~# ip -d link show type vxlan
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether 52:dc:f2:f0:c2:2d brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 68 maxmtu 65535
info: Using default fan map value (33)
vxlan id 1 local 135.125.174.14 dev ens3 srcport 0 0 dstport 8472 nolearning ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536
root@vps-2153e875:~# netstat -ptuan | grep -i 8472
udp 0 0 0.0.0.0:8472 0.0.0.0:* -Here is the summary of the previous output:
flannel.1: This is the virtual interface acting as a VTEP (VXLAN Tunnel Endpoint). It encapsulates pod-to-pod traffic in UDP packets to allow communication across different physical nodes.
mtu 1450: The Maximum Transmission Unit is set to 1450 bytes. This is 50 bytes lower than the standard 1500-byte Ethernet MTU to account for the VXLAN encapsulation overhead (IP + UDP + VXLAN headers).
vxlan id 1: This is the VNI (VXLAN Network Identifier). Flannel typically uses ID 1 by default to identify its virtual network segment.
local 135.125.174.14: This is the host IP address of our node. It is the source IP used for the outer UDP tunnel header.
dev ens3: This specifies that the VXLAN traffic is physically routed through the ens3 network interface.
dstport 8472: This is the UDP port used for the VXLAN tunnel. While the IANA standard for VXLAN is 4789, Flannel uses 8472 by default.
nolearning: Indicates that this interface does not dynamically learn MAC addresses from incoming traffic; instead, the flanneld daemon populates the forwarding database (FDB) and ARP tables manually based on cluster data.
The full output of the ifconfig or ip command also shows other network interfaces called ‘veth‘. These interfaces are “virtual cables” used in Kubernetes to connect Pods to the host node’s network. Because each Pod is isolated in its own network namespace, it requires a bridge to communicate with the rest of the cluster. A veth device always exists as a pair of interconnected virtual interfaces. One end of the pair (named eth0 inside the Pod) resides within the Pod’s network namespace. The other end (appearing as vethxxxx on the node) resides in the host’s “root” network namespace.
root@vps-2153e875:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether fa:16:3e:d7:34:75 brd ff:ff:ff:ff:ff:ff
altname enp0s3
inet 135.125.174.14/32 metric 100 scope global dynamic ens3
valid_lft 74401sec preferred_lft 74401sec
inet6 2001:41d0:701:1100::43e5/128 scope global
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fed7:3475/64 scope link
valid_lft forever preferred_lft forever
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 52:dc:f2:f0:c2:2d brd ff:ff:ff:ff:ff:ff
inet 10.244.0.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::50dc:f2ff:fef0:c22d/64 scope link
valid_lft forever preferred_lft forever
4: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether 1e:8c:09:c5:ea:f6 brd ff:ff:ff:ff:ff:ff
inet 10.244.0.1/24 brd 10.244.0.255 scope global cni0
valid_lft forever preferred_lft forever
inet6 fe80::1c8c:9ff:fec5:eaf6/64 scope link
valid_lft forever preferred_lft forever
5: veth8b4a677f@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default qlen 1000
link/ether 16:a0:70:21:9b:a5 brd ff:ff:ff:ff:ff:ff link-netns cni-810f9b65-9809-e471-0ba3-c8d9560937bd
inet6 fe80::14a0:70ff:fe21:9ba5/64 scope link
valid_lft forever preferred_lft forever
6: veth4555af4e@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default qlen 1000
link/ether da:17:c4:67:25:23 brd ff:ff:ff:ff:ff:ff link-netns cni-7f5857ae-3ae5-4fd1-e2a3-db82c77ecacd
inet6 fe80::d817:c4ff:fe67:2523/64 scope link
valid_lft forever preferred_lft forever
7: veth6246e369@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default qlen 1000
link/ether 6a:32:1b:61:ae:73 brd ff:ff:ff:ff:ff:ff link-netns cni-62eee174-f142-3bad-2d5d-833fe1b3eb7e
inet6 fe80::6832:1bff:fe61:ae73/64 scope link
valid_lft forever preferred_lft forever
8: veth7a0388b7@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default qlen 1000
link/ether be:ce:5b:c3:d2:0b brd ff:ff:ff:ff:ff:ff link-netns cni-43bbe06a-5b8b-2cb5-809b-c22bcbdeb115
inet6 fe80::bcce:5bff:fec3:d20b/64 scope link
valid_lft forever preferred_lft foreverLet’s now run the ifconfig command in the Pod’s network namespace to see the eth0 IP address assigned to each existing Pod.
root@vps-2153e875:~# ip -all netns exec ifconfig | grep -B2 -A1 ' inet '
netns: cni-43bbe06a-5b8b-2cb5-809b-c22bcbdeb115
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.244.0.36 netmask 255.255.255.0 broadcast 10.244.0.255
inet6 fe80::6c2d:ffff:fe6d:8ad0 prefixlen 64 scopeid 0x20<link>
--
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
--
netns: cni-62eee174-f142-3bad-2d5d-833fe1b3eb7e
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.244.0.35 netmask 255.255.255.0 broadcast 10.244.0.255
inet6 fe80::3870:38ff:fe7a:f295 prefixlen 64 scopeid 0x20<link>
--
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
--
netns: cni-7f5857ae-3ae5-4fd1-e2a3-db82c77ecacd
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.244.0.34 netmask 255.255.255.0 broadcast 10.244.0.255
inet6 fe80::a0a1:b2ff:fe90:2ea7 prefixlen 64 scopeid 0x20<link>
--
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
--
netns: cni-810f9b65-9809-e471-0ba3-c8d9560937bd
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.244.0.33 netmask 255.255.255.0 broadcast 10.244.0.255
inet6 fe80::cc00:71ff:fee5:4390 prefixlen 64 scopeid 0x20<link>
--
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>Running the route command on the Pod’s network namespace, we can see the routing table for each of the existing Pods. Pods on the network ‘10.244.0.0/24’ can communicate with each other locally without using any gateway, while communication with other Pods in the network range ‘10.244.0.0/16’ will use the gateway ‘10.244.0.1’ (cni0 bridge). Any other connection will use the default gateway, which is also the gateway ‘10.244.0.1’ (cni0 bridge).
root@vps-2153e875:~# ip -all netns exec route -n
---
netns: cni-43bbe06a-5b8b-2cb5-809b-c22bcbdeb115
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.244.0.1 0.0.0.0 UG 0 0 0 eth0
10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.244.0.0 10.244.0.1 255.255.0.0 UG 0 0 0 eth0
---
netns: cni-62eee174-f142-3bad-2d5d-833fe1b3eb7e
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.244.0.1 0.0.0.0 UG 0 0 0 eth0
10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.244.0.0 10.244.0.1 255.255.0.0 UG 0 0 0 eth0
---
netns: cni-7f5857ae-3ae5-4fd1-e2a3-db82c77ecacd
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.244.0.1 0.0.0.0 UG 0 0 0 eth0
10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.244.0.0 10.244.0.1 255.255.0.0 UG 0 0 0 eth0
---
netns: cni-810f9b65-9809-e471-0ba3-c8d9560937bd
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.244.0.1 0.0.0.0 UG 0 0 0 eth0
10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.244.0.0 10.244.0.1 255.255.0.0 UG 0 0 0 eth0Now let’s see, step-by-step, the traffic flow when a Pod from one node (e.g. Pod with IP 10.244.0.2) communicates with another Pod from another node (e.g. Pod with IP 10.244.1.2).
1. Exit from Source Pod: The packet leaves the source pod’s network namespace via its eth0 interface and enters the host’s root namespace through a virtual ethernet pair (veth).
2. Local Bridge Processing: The packet reaches the cni0 bridge. The node’s routing table identifies that the destination IP belongs to a subnet assigned to a different node.
3. Encapsulation (Overlay): The traffic is directed to the flannel.1 virtual interface (VTEP). Flannel encapsulates the original packet within a UDP packet (defaulting to port 8472 for VXLAN). Flannel knows where other pods are by using the distributed database (etcd) to store node IP/subnet information, allowing the local flanneld agent on each node to look up the correct destination node for a pod’s IP.
- The inner header has the Pod IPs.
- The outer header has the Source Node and Destination Node physical IP addresses.
4. Network Transit: The encapsulated packet travels across the physical network (the “underlay”) using the nodes’ primary network interfaces (e.g., eth0).
5. Arrival and Decapsulation: On the destination node, the packet is received by the flannel.1 interface. It unwraps (decapsulates) the UDP packet to reveal the original pod-to-pod packet.
6. Delivery to Destination Pod: The decapsulated packet is forwarded to the local cni0 bridge, which then routes it to the specific pod via its corresponding veth interface.
Adib Ahmed Akhtar