Bootstrap: GitOps Before Everything

The 3-node cluster is running. Before touching storage, networking, or any operator, the management layer goes in. Everything from here on should flow through Git — commit, push, ArgoCD syncs, done.

The operator source: OKDerators

OKD ships with community-operators — the OperatorHub.io catalog built for vanilla Kubernetes. For OKD-native operators, there’s OKDerators — built and tested for OKD/SCOS, maintained by the OKD working group. The plan from the start: use OKDerators for everything it provides.

1
apiVersion: operators.coreos.com/v1alpha1
2
kind: CatalogSource
3
metadata:
4
  name: okderators
5
  namespace: openshift-marketplace
6
spec:
7
  sourceType: grpc
8
  image: quay.io/okderators/catalog-index:4.20
9
  displayName: OKDerators
10
  publisher: OKD Community
11
  updateStrategy:
12
    registryPoll:
13
      interval: 60m

The Software Catalog jumps from 284 to 307 operators:

OKD Software Catalog before OKDerators — 284 operators

OKD Software Catalog after OKDerators — 307 operators, OKD Optional category added

GitOps operator

1
apiVersion: operators.coreos.com/v1alpha1
2
kind: Subscription
3
metadata:
4
  name: gitops-operator
5
  namespace: openshift-operators
6
spec:
7
  channel: alpha
8
  installPlanApproval: Automatic
9
  name: gitops-operator
10
  source: okderators
11
  sourceNamespace: openshift-marketplace

Auto-provisions openshift-gitops namespace, a cluster-scoped ArgoCD instance, a Route, and the console plugin.

oc get route openshift-gitops-server -n openshift-gitops -o jsonpath='{.spec.host}'
# openshift-gitops-server-openshift-gitops.apps.okd.sudops.pl

oc get secret openshift-gitops-cluster -n openshift-gitops \
  -o jsonpath='{.data.admin\.password}' | base64 -d && echo

ArgoCD login page — "Let's get stuff deployed!"

RBAC

The operator may not auto-grant cluster-admin to the ArgoCD service accounts. Without it, ArgoCD can’t create CRDs, namespaces, or ClusterRoles:

1
apiVersion: rbac.authorization.k8s.io/v1
2
kind: ClusterRoleBinding
3
metadata:
4
  name: openshift-gitops-cluster-admin
5
roleRef:
6
  apiGroup: rbac.authorization.k8s.io
7
  kind: ClusterRole
8
  name: cluster-admin
9
subjects:
10
  - kind: ServiceAccount
11
    name: openshift-gitops-argocd-application-controller
12
    namespace: openshift-gitops
13
  - kind: ServiceAccount
14
    name: openshift-gitops-argocd-server
15
    namespace: openshift-gitops

Note (cluster-admin is broad, and that's fine here)

Yeah, cluster-admin is broad. It’s a homelab — I’m the only user, and the Git repo is the real access control. If you can push to master, you own the cluster. For a team setup, scope this down.

Connect the Git repo

Private repo at github.com/sudoom/homelab. Deploy key for read-only SSH access:

ssh-keygen -t ed25519 -f ~/.ssh/argocd-homelab -C "argocd-deploy-key" -N ""

Add the public key in GitHub (Settings → Deploy Keys), then create the Secret:

KEY_B64=$(base64 -w0 < ~/.ssh/argocd-homelab)
URL_B64=$(echo -n 'git@github.com:sudoom/homelab.git' | base64 -w0)
TYPE_B64=$(echo -n 'git' | base64 -w0)

cat <<EOF | oc apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: repo-creds
  namespace: openshift-gitops
  labels:
    argocd.argoproj.io/secret-type: repo-creds
type: Opaque
data:
  type: ${TYPE_B64}
  url: ${URL_B64}
  sshPrivateKey: ${KEY_B64}
EOF

Repo structure

App-of-Apps pattern with Helm charts. At bootstrap, it’s mostly a skeleton:

1
homelab/
2
  bootstrap/
3
    phase0/                          # Manual (applied once)
4
    root-app/                        # Root Application (Helm)
5
      Chart.yaml
6
      values.yaml
7
      templates/applications.yaml
8
  components/                        # One chart per component
9
    cluster-config/node-labels/
10
    operators/nmstate/
11
    operators/cert-manager/
12
    operators/rook-ceph/
13
    ...

The template

The root Application renders one ArgoCD Application per enabled entry:

1
{{- range $name, $app := .Values.applications }}
2
{{- if $app.enabled }}
3
---
4
apiVersion: argoproj.io/v1alpha1
5
kind: Application
6
metadata:
7
  name: {{ $name }}
8
  namespace: {{ $.Values.global.argocdNamespace }}
9
  annotations:
10
    argocd.argoproj.io/sync-wave: {{ $app.syncWave | quote }}
11
  finalizers:
12
    - resources-finalizer.argocd.argoproj.io
13
spec:
14
  project: default
15
  source:
16
    repoURL: {{ $.Values.global.repoURL }}
17
    targetRevision: {{ $.Values.global.targetRevision }}
18
    path: {{ $app.path }}
19
    helm:
20
      valueFiles:
21
        - values.yaml
22
  destination:
23
    server: {{ $.Values.global.clusterURL }}
24
    namespace: {{ $app.namespace }}
25
  syncPolicy:
26
    automated:
27
      prune: true
28
      selfHeal: true
29
    syncOptions:
30
      - ServerSideApply=true
31
      - RespectIgnoreDifferences=true
32
      - SkipDryRunOnMissingResource=true
33
    retry:
34
      limit: 5
35
      backoff:
36
        duration: 5s
37
        factor: 2
38
        maxDuration: 3m
39
{{- end }}
40
{{- end }}

Every syncPolicy option earns its place:

prune: true — remove a manifest from Git, ArgoCD deletes it from the cluster. Git is the source of truth.
selfHeal: true — manual cluster edits get reverted to match Git. No drift.
No CreateNamespace=true. Each component chart includes an explicit Namespace resource — full control over labels, annotations, pod security settings.
ServerSideApply=true — avoids the 262 KB annotation limit that kills client-side apply on large operator CRDs.
RespectIgnoreDifferences=true — stops ArgoCD from fighting operators that patch their own resources.
SkipDryRunOnMissingResource=true — lets CRs sync before their CRDs are registered. Retry handles the timing.
Retry backoff — 5 retries, 5s → 10s → 20s → 40s → 80s. Bridges the gap between operator install and CRD availability.

The first app: node labels

The simplest component — failure domain labels from the compute design:

1
nodes:
2
  - hostname: node4.okd.sudops.pl
3
    labels:
4
      topology.kubernetes.io/zone: fd-a
5
  - hostname: node5.okd.sudops.pl
6
    labels:
7
      topology.kubernetes.io/zone: fd-b
8
  - hostname: node6.okd.sudops.pl
9
    labels:
10
      topology.kubernetes.io/zone: fd-c

1
{{- range .Values.nodes }}
2
---
3
apiVersion: v1
4
kind: Node
5
metadata:
6
  name: {{ .hostname }}
7
  labels:
8
    {{- range $key, $value := .labels }}
9
    {{ $key }}: {{ $value | quote }}
10
    {{- end }}
11
{{- end }}

At bootstrap time, this is the only enabled app in values.yaml:

1
global:
2
  repoURL: git@github.com:sudoom/homelab.git
3
  targetRevision: master
4
  argocdNamespace: openshift-gitops
5
  clusterURL: https://kubernetes.default.svc
6

7
applications:
8
  node-labels:
9
    enabled: true
10
    path: components/cluster-config/node-labels
11
    namespace: default
12
    syncWave: "0"
13

14
  # Remaining components enabled as Day 2 subposts progress:
15
  # nmstate-operator, cert-manager-operator, rook-ceph-operator (wave 1)
16
  # nmstate-nncp, cert-manager-config (wave 2)
17
  # ingress-controller, api-server (wave 3)
18
  # ceph-cluster, ceph-storage-classes (wave 3-4)

App-of-Apps planned sync wave order — operators first, config second, consumers third

Apply the root Application

This is the one Application I apply by hand — it points ArgoCD at the bootstrap/root-app chart, which renders everything else:

1
apiVersion: argoproj.io/v1alpha1
2
kind: Application
3
metadata:
4
  name: root-app
5
  namespace: openshift-gitops
6
spec:
7
  project: default
8
  source:
9
    repoURL: git@github.com:sudoom/homelab.git
10
    targetRevision: master
11
    path: bootstrap/root-app
12
    helm:
13
      valueFiles:
14
        - values.yaml
15
  destination:
16
    server: https://kubernetes.default.svc
17
    namespace: openshift-gitops
18
  syncPolicy:
19
    automated:
20
      prune: true
21
      selfHeal: true

Last manual oc apply:

oc apply -f bootstrap/phase0/05_root_application.yaml

ArgoCD picks it up, renders node-labels, labels all three nodes. One app running, the rest enabled as each Day 2 subpost is written.

Five manual commands, then Git

oc apply -f bootstrap/phase0/01_catalog_source.yaml
oc apply -f bootstrap/phase0/02_gitops_operator.yaml
oc apply -f bootstrap/phase0/03_cluster_admin_rbac.yaml
oc apply -f bootstrap/phase0/04_credential_template.yaml
oc apply -f bootstrap/phase0/05_root_application.yaml

After that: create a chart, add to values.yaml, commit, push. No oc apply, no helm install. If something breaks, git revert.

Next: cert-manager — the first component deployed entirely through this pipeline.