grokking hard

code smarter, not harder

Trying out NetworkPolicy on Kubernetes running on Docker for macOS

Posted at — 2023-Jan-16

A problem

I was tempting to try out NetworkPolicy on Kubernetes as a secure way to protect traffic in and out the namespaces but never managed to actuall do it.

Today a friend of mine who was pursuing “Certified Kubernetes Application Developer” asked me a problem related to it. He was using a small set of exercises to practice Kubernetes. The problem statement was:

Create an nginx deployment of 2 replicas, expose it via a ClusterIP service on port 80. Create a NetworkPolicy so that only pods with labels ‘access: granted’ can access the deployment and apply it

The guide also included the solution and he tried it on his test environment and it didn’t work. The deployed network policy did not take effect at all.

At first, I thought it might be because he did not deploy a “deny-all” policy as described in the page. This practice seemed like a good practice that almost all serious production-ready solution used (e.g CouchBase Operator).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

---
apiVersion: v1
kind: Namespace
metadata:
  name: $NAMESPACE
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all-except-dns
  namespace: $NAMESPACE
  # Prevent any traffic for all pods apart from that we specifically allow.
  # This is taken from the Kubernetes documentation with a tweak to allow DNS.
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  egress:
  # Allow DNS resolution, primarily for Couchbase pods but it triggers other
  # hard to debug connection issues sometimes too so be aware if this is moved to
  # the later rules.
  - ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
---

But my friend tried that and it still DIDN’T work. Hmm! So I thought “Interesting! Let’s try!”.

A first look

I was using a MacBook Pro and installed Docker for macOS. I didn’t want to use any managed Kubernetes cluster available online because I didn’t want to waste time on them. I chose to use Kind as it allowed to set up a Kubernetes cluster on the existing Docker Engine.

kind is a tool for running local Kubernetes clusters using Docker container “nodes”.
kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI.

1
2
3

$> brew install kind
$> kind create cluster
$> kubectl cluster-info --context kind-kind

Once I had a Kubernetes cluster running, I ran the same set of commands suggested by the exercise and, interestingly, reproduced the issue.

Grokking it a little harder

I decided to read the Kubernetes’ NetworkPolicy page 1 very carefully again. The “Aha!” moment came:

Make sure you’ve configured a network provider with network policy support. There are a number of network providers that support NetworkPolicy, including:

Okay! So NetworkPolicy did not come out-of-the-box in a Kubernetes cluster. And to top it off, the exercise DID NOTICE THAT as well:

[..] Note that network policies may not be enforced by default, depending on your k8s implementation. E.g. Azure AKS by default won’t have policy enforcement, the cluster must be created with an explicit support for netpol https://docs.microsoft.com/en-us/azure/aks/use-network-policies#overview-of-network-policy [..]

Shame on me!

In order to test the NetworkPolicy, I had to see if the cluster created by Kind supported network policy? Turned out that it was supported…eventually.

According to the GitHub issue "#842 NetworkPolicy support" (which was still open as of this writing), Kind could provide support for network policies by installing Calico. The most voted comment (thankfully!) showed how to do it.

To sum it up:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

# IMPORTANT: This guide worked on 2023-01-16

# [1]: First create a Kind cluster with default CNI disabled.
$> cat <<'_YAML_' |  kind create cluster --name test --config -
# this config file contains all config fields with comments
# NOTE: this is not a particularly useful config file
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true
  podSubnet: "192.168.0.0/16"
_YAML_

# [2]: Install Calico v3.25
$> kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml

# [3]: It may take a while for Calico and Kubernetes to settle.
# Use the following command to watch until all Pods reach "Running" state.
$> kubectl get pods -n kube-system

# Once all the Pods reached the state "Running", it was time to test.

$> kubectl create deployment nginx --image=nginx --replicas=2
$> kubectl expose deployment nginx --port=80
$> cat <<_YAML_ | kubectl apply -f-
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: access-nginx # pick a name
spec:
  podSelector:
    matchLabels:
      app: nginx # selector for the pods
  ingress: # allow ingress traffic
  - from:
    - podSelector: # from pods
        matchLabels: # with this label
          access: granted
_YAML_
$> kubectl run busybox 
  --image=busybox --rm -it --restart=Never -- wget -O- http://nginx:80 --timeout 2
  # This should not work. --timeout is optional here. But it helps to get answer more quickly (in seconds vs minutes)
$> kubectl run busybox --image=busybox --rm -it --restart=Never --labels=access=granted -- wget -O- http://nginx:80 --timeout 2 # This should be fine

Key Takeaways