有次我們從 Grafana 上看到 worker node 上的流量異常增加,從 node_exporter 上有收集到是哪個 network interface 造成的,但是找不到實際上是哪個 pod 使用的,後續研究後才產生這篇
CNI:calico
找到該 interfance 名稱後 grep 出來找到 link-netnsid 的 idd
ip a | grep -A 2 calicd52f3066ff 134355: calicd52f3066ff@if4:
mtu 1400 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 13 <- inet6 fe80::ecee:eeff:feee:eeee/64 scope link 找到步驟一的 id 的 cni id
ip netns cni-97834ea1-9c70-98cb-213c-434cbd6a2795 (id: 15) cni-3c1a7a21-222c-15ce-acf6-5580990d0926 (id: 13) <------ this cni-dd2ec7d7-9966-dc39-595d-1bbf88411e8b (id: 10) cni-f7f8f37a-1c06-d54c-d647-6217f7545928 (id: 9) cni-cfb02643-930f-846a-8a47-5b741055db89 (id: 14) cni-f270b11f-eb6f-fd81-1b2e-e367fccc5301 (id: 64) cni-ca00e01e-5f02-2e6d-bd68-0eb11ce8e813 (id: 62) cni-4bcd07a6-e5e8-7939-06c4-928d20790ef5 (id: 63) cni-6b9b4b7c-1a66-5c4d-50b0-2ef23d09779d (id: 61) cni-1c6c5253-88f5-a423-5049-cca06468fd93 (id: 60) cni-b0c84d67-73b1-9d50-63f2-5d084776fabc (id: 59) cni-919cf6fc-fafc-fb86-0d96-d0e9ce3535bf (id: 58) cni-322c0a51-ff60-d6bc-c851-ce960afd3a92 (id: 56) cni-d695f49d-2cb7-4087-2df4-e344f85e88a2 (id: 55) cni-a42cef60-4628-6fe9-ab34-3ee7dec9947a (id: 54) cni-172333bc-0138-e3f0-03d7-5f1e4b5028f7 (id: 53)
使用該 cni id 執行
ip a
拿到該 pod 的 ipnsenter --net=/var/run/netns/cni-3c1a7a21-222c-15ce-acf6-5580990d0926 ip a 1: lo:
mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: tunl0@NONE: mtu 1480 qdisc noop state DOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 4: eth0@if134355: mtu 1400 qdisc noqueue state UP group default link/ether be:a4:fc:80:7b:f0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.30.222.61/32 brd 172.30.222.61 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::bca4:fcff:fe80:7bf0/64 scope link valid_lft forever preferred_lft forever 最後使用 pod ip 找到實際的 pod
k get pods -A -o wide |grep 172\.30\.222\.61 prod-hk-02 lc-data-server-f9d685864-2z9zd 2/2 Running 0 164m 172.30.222.61 prod-aws-hk-k8s-node04