Storage Network: 10GbE on VLAN 10 via NMState

The bootstrap deployed ArgoCD and cert-manager gave me trusted TLS. The cluster is still running on the onboard 1GbE NICs only. Each node has a Mellanox CX4121C with dual 10GbE SFP28 ports sitting idle — connected to the CRS317 switch but unconfigured at the OS level.

This post enables the storage network: NMState operator for declarative node networking, and per-node NNCPs (NodeNetworkConfigurationPolicies) that assign static IPs on VLAN 10. This is the path Ceph replication traffic will use.

What is NMState?

Kubernetes doesn’t manage host networking. Pods get their own network via CNI, but the node’s physical NICs — the ones connected to the switch — are outside Kubernetes control. On a cloud provider, the hypervisor handles this. On bare metal, you need something.

NMState is an operator that brings host network configuration into the Kubernetes API. You write a NodeNetworkConfigurationPolicy (NNCP) — a declarative spec of what the node’s network should look like — and NMState’s daemon running on each node applies it via NetworkManager. If the config fails (wrong interface name, bad IP), NMState rolls it back automatically. No SSH-ing into nodes, no nmcli by hand, no risk of cutting yourself off.

This is why I deferred NIC configuration to day 2 instead of baking it into the agent-config ISO. If a bond config or VLAN sub-interface has a typo in agent-config, you regenerate the ISO and reinstall from scratch. With NMState NNCPs, you edit the YAML, push to Git, and ArgoCD reapplies. If it breaks, NMState rolls back within 4 minutes.

The operator

NMState is in OKDerators, but its bundle is currently broken on OKD 4.20. The NMState operator bundle contains an ImageStream resource (image.openshift.io/v1) — a build artifact from the OpenShift pipeline that isn’t needed at runtime. OKD’s OLM doesn’t support ImageStreams, so the InstallPlan fails immediately with UnsupportedResource. The CSV, CRDs, and RBAC are all fine — it’s just the ImageStream blocking the install.

I filed an issue and sent the fix — stopping the build pipeline from injecting that ImageStream into the bundle in the first place, since merged upstream. When I set this up the bundle was still broken, though, so NMState comes from operatorhubio-catalog instead:

1
namespace: openshift-nmstate
2
channel: alpha
3
source: operatorhubio-catalog
4
sourceNamespace: openshift-marketplace

The operator chart creates the namespace (with pod security labels), OperatorGroup, Subscription, and the NMState CR that triggers daemon deployment on every node:

1
apiVersion: nmstate.io/v1
2
kind: NMState
3
metadata:
4
  name: nmstate
5
spec: {}

Once the NMState CR is applied, the operator deploys a nmstate-handler DaemonSet — one pod per node, responsible for applying NNCPs.

The NNCPs

One NNCP per node. Each selects a single node via kubernetes.io/hostname and configures both CX4121C ports:

Storage network topology — per-node NNCPs, VLAN 10, 10GbE via CRS317

1
storageInterface: enp1s0f0np0
2
unusedInterface: enp1s0f1np1
3

4
nodes:
5
  - name: node4
6
    hostname: node4.okd.sudops.pl
7
    ip: 192.168.10.2
8
    prefix: 24
9
  - name: node5
10
    hostname: node5.okd.sudops.pl
11
    ip: 192.168.10.3
12
    prefix: 24
13
  - name: node6
14
    hostname: node6.okd.sudops.pl
15
    ip: 192.168.10.4
16
    prefix: 24

The template generates one NNCP per node:

1
{{- range .Values.nodes }}
2
---
3
apiVersion: nmstate.io/v1
4
kind: NodeNetworkConfigurationPolicy
5
metadata:
6
  name: storage-{{ .name }}
7
spec:
8
  nodeSelector:
9
    kubernetes.io/hostname: {{ .hostname }}
10
  desiredState:
11
    interfaces:
12
      - name: {{ $.Values.storageInterface }}
13
        type: ethernet
14
        state: up
15
        ipv4:
16
          enabled: true
17
          dhcp: false
18
          address:
19
            - ip: {{ .ip }}
20
              prefix-length: {{ .prefix }}
21
        ipv6:
22
          enabled: false
23
      - name: {{ $.Values.unusedInterface }}
24
        type: ethernet
25
        state: up
26
        ipv4:
27
          enabled: false
28
        ipv6:
29
          enabled: false
30
{{- end }}

Why per-node NNCPs, not one for all?

Each node needs a different static IP. A single NNCP with nodeSelector: node-role.kubernetes.io/worker would apply the same IP to all three nodes — IP conflict. Per-node NNCPs with hostname selectors give each node its own address.

Why is port 2 configured?

Port 2 (enp1s0f1np1) is brought up but has no IP. Two reasons: NMState manages both ports on the CX4121C, and having port 2 in a known state (up, no IP, no DHCP) prevents NetworkManager from grabbing a DHCP lease on it. When LACP bonding is added later (day 2 operation), port 2 becomes the second bond member — it needs to be up and managed.

Why no LACP bond yet?

Bonding belongs in a separate NNCP update after the single-port storage network is validated. The sequence: single port works, Ceph runs on it, add bond member, verify bond failover. One change at a time. If I bonded now and Ceph had issues, I wouldn’t know if it’s a Ceph problem or a bond problem.

The traffic path

The storage network is fully isolated — designed in the network architecture post and implemented in the network implementation post:

Node sends plain Ethernet frames on enp1s0f0np0
CRS317 switch tags them as VLAN 10 (pvid=10 on the access port)
VLAN 10 is trunk-carried to the CCR2004 router
Router’s firewall drops all forwarded Backnet traffic — no internet, no inter-VLAN routing
Router’s 192.168.10.1 exists for diagnostics only (ping from router)

Ceph replication between nodes stays entirely within the CRS317 switch fabric at 10GbE line rate. The traffic never touches the router unless you’re debugging.

Verification

After ArgoCD syncs the NNCPs:

oc get nncp

1
NAME            STATUS      REASON
2
storage-node4   Available   SuccessfullyConfigured
3
storage-node5   Available   SuccessfullyConfigured
4
storage-node6   Available   SuccessfullyConfigured

Cross-node ping from node4 over the storage network:

1
[core@node4 ~]$ ping -c2 192.168.10.3 && ping -c2 192.168.10.4
2
PING 192.168.10.3 (192.168.10.3) 56(84) bytes of data.
3
64 bytes from 192.168.10.3: icmp_seq=1 ttl=64 time=0.196 ms
4
64 bytes from 192.168.10.3: icmp_seq=2 ttl=64 time=0.081 ms
5

6
--- 192.168.10.3 ping statistics ---
7
2 packets transmitted, 2 received, 0% packet loss, time 1047ms
8
rtt min/avg/max/mdev = 0.081/0.138/0.196/0.057 ms
9
PING 192.168.10.4 (192.168.10.4) 56(84) bytes of data.
10
64 bytes from 192.168.10.4: icmp_seq=1 ttl=64 time=0.296 ms
11
64 bytes from 192.168.10.4: icmp_seq=2 ttl=64 time=0.233 ms
12

13
--- 192.168.10.4 ping statistics ---
14
2 packets transmitted, 2 received, 0% packet loss, time 1022ms
15
rtt min/avg/max/mdev = 0.233/0.264/0.296/0.031 ms

Sub-millisecond latency across the CRS317 switch fabric. All three nodes reachable on the storage network — this is the path Rook-Ceph will use for OSD replication in the next post.

What was committed

Two entries in values.yaml:

1
nmstate-operator:
2
  enabled: true
3
  path: components/operators/nmstate
4
  namespace: openshift-nmstate
5
  syncWave: "1"
6

7
nmstate-nncp:
8
  enabled: true
9
  path: components/cluster-config/nmstate-nncp
10
  namespace: default
11
  syncWave: "2"

Operator in wave 1, NNCPs in wave 2 (depends on the operator being ready). One git push, ArgoCD syncs both, all three nodes get their storage IPs.

Next: Rook-Ceph — NVMe fast pool on the storage network. (coming soon)