k8s 之NetworkPolicy介绍使用
在 Kubernetes 中要实现容器之间网络的隔离，是通过一个专门的 API 对象 NetworkPolicy（网络策略）来实现的，要让网络策略生效，就需要特定的网络插件支持，目前已经实现了 NetworkPolicy 的网络插件包括 Calico、Weave 和 kube-router 等项目，但是并不包括 Flannel 项目。所以说，如果想要在使用 Flannel 的同时还使用 NetworkPolicy 的话，你就需要再额外安装一个网络插件，比如 Calico 项目，来负责执行 NetworkPolicy。本测试环境使用的是 Calico 网络插件,可以直接使用
默认情况下 Pod 是可以接收来自任何发送方的请求，也可以向任何接收方发送请求。如果要对这个情况作出限制，就必须通过 NetworkPolicy 对象来指定。
以下定义了一个网络策略资源清单文件,内容如下：
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:name: network-policynamespace: default
spec:podSelector: matchLabels:app: nginxpolicyTypes:- Ingress- Egressingress:- from:- ipBlock:cidr: 10.233.0.0/16except:- 10.233.93.0/24- namespaceSelector:matchLabels:project: test- podSelector:matchLabels:app: busybox- ports:- protocol: TCPport: 80egress:- to:- ipBlock:cidr: 10.233.93.0/24ports:- protocol: TCPport: 5978
Ingress参数解释：
spec:podSelector: matchLabels:app: nginx 定义了pod对象，表示当前ns中标签为app: nginx的pod
如果想要NetworkPolicy作用于当前ns中的所有pod，则用如下形式：
spec:podSelector: {}
spec:policyTypes: 网络策略的类型- Ingress- Egress
每个 NetworkPolicy 包含一个 policyTypes 列表，可以是一个 Ingress、Egress 或者都包含，该字段表示给当前策略是否应用于所匹配的 Pod 的入口流量、出口流量或者二者都包含，如果没有指定 policyTypes，则默认情况下表示 Ingress 入口流量，如果配置了任何出口流量规则，则将指定为 Egress。
ingress: 配置pod的ingress策略- from:- ipBlock: 配置允许的网络访问cidr: 10.233.0.0/16except: 不允许某个网络访问- 10.233.93.0/24- namespaceSelector: 允许标签为project: test的ns下的所有的pod访问matchLabels:project: test- podSelector: 此处代表当前ns下的pod，默认是允许同一ns下的pod互通，添加了此处选项代表只允许当前ns下label为app: busybox的pod访问label为app: nginx的pod应用matchLabels:app: busybox- ports: #允许被访问的端口- protocol: TCPport: 80一旦 Pod 被 NetworkPolicy 选中，那么这个 Pod 就会进入“拒绝所有”（Deny All）的状态，即这个 Pod 既不允许被外界访问，也不允许对外界发起访问，所以 NetworkPolicy 定义的规则，其实就是“白名单”了。
ingress测试
在default的ns下启动两个web服务的pod，如下：
[root@master ~]# kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx 1/1 Running 0 9d 10.233.96.3 node2 <none> <none>
web 1/1 Running 0 9d 10.233.96.4 node2 <none> <none>
创建networkpolicy，如下：
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:name: denynetwork
spec:podSelector:matchLabels:app: nginx #被访问pod的label标签policyTypes:- Egress- Ingress 添加规则类型ingress:- from:- podSelector: 此处代表当前ns下的pod，默认是允许同一ns下的pod互通，添加了此处选项代表只允许当前ns下label为app: web的pod访问label为app: nginx的pod应用matchLabels:app: web创建如下：
[root@master ~]# kubectl get networkpolicy
NAME POD-SELECTOR AGE
denynetwork app=nginx 9d 可以看到被访问的pod标签
[root@master ~]#
podSelector测试：
##
如上，web pod位于node2上，进入pod的网络命令空间[root@node2 ~]# crictl ps | grep web
dcf892535ed93 3f8a00f137a0d 9 days ago Running count 0 205811ac7b0f0 web
[root@node2 ~]#
[root@node2 ~]# crictl inspect dcf892535ed93 | grep -i pid"pid": 16814,"pid": 1"type": "pid"
[root@node2 ~]#
[root@node2 ~]# nsenter -t 16814 -n bash #进入pod的网络命令空间
[root@node2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default link/ether 9a:6b:07:ab:e4:0d brd ff:ff:ff:ff:ff:ff link-netnsid 0inet 10.233.96.4/32 scope global eth0valid_lft forever preferred_lft foreverinet6 fe80::986b:7ff:feab:e40d/64 scope link valid_lft forever preferred_lft forever
[root@node2 ~]# ping 10.233.96.3 ##此处可以看到无放访问nginx pod
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
^C
--- 10.233.96.3 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 5999ms[root@node2 ~]# ##
#
为web pod添加app: web的标签，如下：
root@master ~]# kubectl label po web app=web
pod/web labeled
[root@master ~]#
再次测试访问nginx，如下：
root@master ~]# ssh node2
Last login: Sat Mar 4 14:26:28 2023 from 192.168.5.240[root@node2 ~]# nsenter -t 16814 -n bash
[root@node2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default link/ether 9a:6b:07:ab:e4:0d brd ff:ff:ff:ff:ff:ff link-netnsid 0inet 10.233.96.4/32 scope global eth0valid_lft forever preferred_lft foreverinet6 fe80::986b:7ff:feab:e40d/64 scope link valid_lft forever preferred_lft forever
[root@node2 ~]# ping 10.233.96.3
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
64 bytes from 10.233.96.3: icmp_seq=1 ttl=63 time=0.333 ms
64 bytes from 10.233.96.3: icmp_seq=2 ttl=63 time=0.127 ms
^C
--- 10.233.96.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.127/0.230/0.333/0.103 ms[root@node2 ~]# curl 10.233.96.3 #可以正常访问服务
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@node2 ~]#
namespaceSelector测试
编辑networkpolicy.yaml文件
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:name: denynetwork
spec:podSelector:matchLabels:app: nginx #被访问pod的label标签policyTypes:- Egress- Ingress 添加规则类型ingress:- from:- namespaceSelector: 添加了此处选项代表只允许label为prject:test的ns下的pod访问label为app: nginx的pod应用matchLabels:project: test创建如下：
[root@master ~]# kubectl get networkpolicy
NAME POD-SELECTOR AGE
denynetwork app=nginx 9d 可以看到被访问的pod标签
创建ns以及测试pod
#创建ns
[root@master ~]# kubectl create ns test##创建测试pod
apiVersion: v1
kind: Pod
metadata:name: busyboxnamespace: test ##选择test的ns
spec:containers:- name: busyboximage: docker.io/library/busybox:latestimagePullPolicy: IfNotPresentcommand: ["init"] 启动pod如下：
[root@master ~]# kubectl get po -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 0 9d 10.233.90.4 node1 <none> <none>
[root@master ~]# #
进入busybox的网络命令，如下：
[root@node1 ~]# crictl ps | grep busybox
755e001322ef6 2fb6fc2d97e10 9 days ago Running busybox 0 b5e1dfe3d3746 busybox
[root@node1 ~]#
[root@node1 ~]#
[root@node1 ~]# crictl inspect 755e001322ef6| grep -i pid"pid": 44236,"pid": 1"type": "pid"[root@node1 ~]# nsenter -t 44236 -n bash
[root@node1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default link/ether 42:2b:78:3a:6a:c1 brd ff:ff:ff:ff:ff:ff link-netnsid 0inet 10.233.90.4/32 scope global eth0valid_lft forever preferred_lft foreverinet6 fe80::402b:78ff:fe3a:6ac1/64 scope link valid_lft forever preferred_lft forever
[root@node1 ~]# ping 10.233.96.3
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
^C
--- 10.233.96.3 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3000ms[root@node1 ~]# ##为test ns添加标签，如下：
[root@master ~]# kubectl label ns test project=test
namespace/test labeled
[root@master ~]# 再次测试如下：[root@node1 ~]#
[root@node1 ~]# nsenter -t 44236 -n bash
[root@node1 ~]# ping 10.233.96.3
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
64 bytes from 10.233.96.3: icmp_seq=1 ttl=62 time=2.35 ms
64 bytes from 10.233.96.3: icmp_seq=2 ttl=62 time=1.46 ms
^C
--- 10.233.96.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.468/1.910/2.352/0.442 ms
[root@node1 ~]# curl 10.233.96.3
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@node1 ~]#
ipBlock测试
编辑networkpolicy.yaml文件：
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:name: denynetworknamespace: default
spec:podSelector: matchLabels:app: nginxpolicyTypes:- Ingress- Egressingress:- from:- ipBlock:cidr: 10.233.0.0/16 ##允许访问的网段except:- 10.233.93.0/24 #排除此网段（此网段位于node1，所以测试pod也位于node1）创建测试pod
##创建pod，如下：
[root@master ~]# kubectl get po -o wide -n test
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 0 9d 10.233.90.4 node1 <none> <none>#
进入pod的网络命名空间
[root@master ~]# ssh node1
Last login: Sat Mar 4 14:55:41 2023 from 192.168.5.240
[root@node1 ~]# crictl ps | grep busybox
755e001322ef6 2fb6fc2d97e10 9 days ago Running busybox 0 b5e1dfe3d3746 busybox
[root@node1 ~]#
[root@node1 ~]# crictl inspect 755e001322ef6 | grep -i pid"pid": 44236,"pid": 1"type": "pid"
[root@node1 ~]#
[root@node1 ~]#
[root@node1 ~]# nsenter -t 44236 -n bash
[root@node1 ~]#
[root@node1 ~]# ping 10.233.96.3
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
^C
--- 10.233.96.3 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5000ms[root@node1 ~]# #
修改networkpolicy，取消expect参数
[root@master ~]# kubectl edit networkpolicy denynetwork
networkpolicy.networking.k8s.io/denynetwork edited
[root@master ~]# kubectl describe networkpolicy denynetwork
Name: denynetwork
Namespace: default
Created on: 2023-02-22 22:39:54 +0800 CST
Labels: <none>
Annotations: <none>
Spec:PodSelector: app=nginxAllowing ingress traffic:To Port: <any> (traffic allowed to all ports)From:IPBlock:CIDR: 10.233.0.0/16Except: #已经去掉了10.233.93.0/24的网段的限制Allowing egress traffic:<none> (Selected pods are isolated for egress connectivity)Policy Types: Egress, Ingress
[root@master ~]# ##
再此测试如下：[root@node1 ~]#
[root@node1 ~]# ping 10.233.96.3
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
64 bytes from 10.233.96.3: icmp_seq=1 ttl=62 time=2.76 ms
64 bytes from 10.233.96.3: icmp_seq=2 ttl=62 time=1.55 ms
^C
--- 10.233.96.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.556/2.158/2.761/0.604 ms
[root@node1 ~]# curl 10.233.96.3
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@node1 ~]# egress参数解释
每个 NetworkPolicy 包含一个 egress 规则的白名单列表。每个规则都允许匹配 to 和 port 部分的流量。比如我们这里示例规则的配置：
egress:- to:- ipBlock:cidr: 10.0.0.0/24 ##允许访问10.0.0.0段的80端口ports:- protocol: TCPport: 80表示 Kubernetes 会拒绝被隔离 Pod 对外发起任何请求，除非请求的目的地址属于 10.0.0.0/24 网段，并且访问的是该网段地址的 80 端口。
创建networkpolicy测试：
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:name: denynetworknamespace: default
spec:podSelector: matchLabels:app: nginxpolicyTypes:- Ingress- Egressegress:- to:- ipBlock:cidr: 10.233.90.0/24 ##允许访问的网段ports:- protocol: TCPport: 80 允许被访问网络的端口#创建策略查看
[root@master ~]# kubectl describe networkpolicy denynetwork
Name: denynetwork
Namespace: default
Created on: 2023-02-22 22:39:54 +0800 CST
Labels: <none>
Annotations: <none>
Spec:PodSelector: app=nginxAllowing ingress traffic:<none> (Selected pods are isolated for ingress connectivity)Allowing egress traffic:To Port: 80/TCPTo:IPBlock:CIDR: 10.233.90.0/24Except: Policy Types: Ingress, Egress
[root@master ~]# ##找到app=nginx的pod并访问
[root@master ~]# kubectl get po -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
egress-test 1/1 Running 0 70s 10.233.90.6 node1 <none> <none>[root@master ~]# kubectl get po -o wide | grep nginx
nginx 1/1 Running 0 9d 10.233.96.3 node2 <none> <none>进入app=nginx的pod的命令空间访问
[root@node2 ~]# crictl ps | grep nginx
27fd22bdf596d 3f8a00f137a0d 9 days ago Running count 0 cb2afd0d86bcf nginx
[root@node2 ~]# crictl inspect 27fd22bdf596d| grep -i pid"pid": 11243,"pid": 1"type": "pid"
[root@node2 ~]# nsenter -t 11243 -n bash
[root@node2 ~]#
[root@node2 ~]# ping 10.233.90.6 因为只允许了tcp协议，所以ping不通
PING 10.233.90.6 (10.233.90.6) 56(84) bytes of data.
^C
--- 10.233.90.6 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 2999ms[root@node2 ~]# curl 10.233.90.6 #如下可正常访问80端口
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@node2 ~]# ##编辑networkpolicy ，将80端口改为81
[root@master ~]# kubectl describe networkpolicy denynetwork
Name: denynetwork
Namespace: default
Created on: 2023-02-22 22:39:54 +0800 CST
Labels: <none>
Annotations: <none>
Spec:PodSelector: app=nginxNot affecting ingress trafficAllowing egress traffic:To Port: 81/TCPTo:IPBlock:CIDR: 10.233.90.0/24Except: Policy Types: Egress
[root@master ~]# #再次访问10.233.90.6的80端口如下：
[root@node2 ~]# curl 10.233.90.6 无法访问
^C
[root@node2 ~]# ##编辑networkpolicy，放通整个网段测试
[root@master ~]# kubectl edit networkpolicy denynetwork
networkpolicy.networking.k8s.io/denynetwork edited
[root@master ~]# kubectl describe networkpolicy denynetwork
Name: denynetwork
Namespace: default
Created on: 2023-02-22 22:39:54 +0800 CST
Labels: <none>
Annotations: <none>
Spec:PodSelector: app=nginxNot affecting ingress trafficAllowing egress traffic:To Port: <any> (traffic allowed to all ports)To:IPBlock:CIDR: 10.233.90.0/24Except: Policy Types: Egress
[root@master ~]# ##再次访问10.233.90.6，如下：
[root@node2 ~]# ping 10.233.90.6
PING 10.233.90.6 (10.233.90.6) 56(84) bytes of data.
64 bytes from 10.233.90.6: icmp_seq=1 ttl=62 time=1.56 ms
^C
--- 10.233.90.6 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.560/1.560/1.560/0.000 ms
[root@node2 ~]#
[root@node2 ~]# curl 10.233.90.6
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@node2 ~]#
但是无法访问其他节点的pod，即使是同网段和节点，如下：
[root@master ~]# kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx 1/1 Running 0 9d 10.233.96.3 node2 <none> <none>
web 1/1 Running 0 9d 10.233.96.4 node2 <none> <none>
[root@node2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default link/ether 56:52:52:7c:82:81 brd ff:ff:ff:ff:ff:ff link-netnsid 0inet 10.233.96.3/32 scope global eth0valid_lft forever preferred_lft foreverinet6 fe80::5452:52ff:fe7c:8281/64 scope link valid_lft forever preferred_lft forever
[root@node2 ~]# ping 10.233.96.4
PING 10.233.96.4 (10.233.96.4) 56(84) bytes of data.
^C
--- 10.233.96.4 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3000ms[root@node2 ~]# curl 10.233.96.4
^C
[root@node2 ~]#
