Kubernetes Secrets Management with Vault Secrets Operator

Kubernetes
Kubernetes Secrets Management with Vault Secrets Operator

The Leaky Ladder: How Teams End Up Here

Most teams climb the same rungs. Secrets start baked into Docker images or .env files passed as environment variables. Someone commits a credential to git, the on-call rotation gets interesting, and the team moves to Kubernetes Secrets — only to discover they’re base64-encoded YAML sitting unencrypted in etcd, routinely committed inside Helm values files.

The next rung is a manual sync job: a CI pipeline that pulls from a secret store and runs kubectl apply. This works until the job fails silently, a certificate expires mid-weekend, or a database password rotates and half the pods crash at 3 AM.

Vault Secrets Operator (VSO) is the rung most teams wish they’d started on.

What VSO Is and How It Works

VSO is a Kubernetes operator — a controller that runs inside your cluster and watches custom resources you define. It authenticates to a HashiCorp Vault server (self-hosted or HCP Vault Dedicated), fetches secrets, and writes them as standard Kubernetes Secrets. Your pods keep consuming env references and envFrom blocks. They never need to know Vault exists.

This is VSO’s core value proposition: Vault’s powerful secret engine, delivered through the K8s-native Secret API that every Deployment, StatefulSet, and Helm chart already speaks.

Four CRDs drive the whole system:

  • VaultConnection — the address and TLS config of your Vault server
  • VaultAuth — how VSO authenticates (Kubernetes JWT auth is the default; AWS IAM, GCP, and AppRole are also supported)
  • VaultStaticSecret — syncs a KV v1 or v2 path into a named K8s Secret on a configurable refresh interval
  • VaultDynamicSecret — requests a short-lived credential (database login, AWS STS token), manages its lease, and renews or re-issues before expiry
  • VaultPKISecret — requests a TLS certificate from Vault PKI and re-requests it before a configurable expiry offset

The Sync Model: Why It Beats the Alternatives

Vault Agent Injector

The classic HashiCorp approach injects a Vault Agent sidecar into every pod. The sidecar authenticates to Vault and writes secrets to a shared in-memory volume. It works, but you pay in CPU and memory for every pod, manage per-workload agent config templates, and end up with sidecar sprawl across hundreds of deployments.

VSO is centralized. One operator manages secrets for the entire cluster. No sidecars, no per-pod config.

Secrets Store CSI Driver

The CSI driver mounts Vault secrets as files inside a pod volume — solid for workloads that read config from disk. But it does not create Kubernetes Secrets. Any app that uses env or envFrom requires code changes. VSO writes real K8s Secrets, so existing workloads need zero modification.

External Secrets Operator (ESO)

ESO is deliberately provider-agnostic: AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, and Vault in one unified API. If you’re multi-cloud with no single secret backend, ESO is attractive.

Where ESO falls short is dynamic secrets. ESO treats Vault like a static KV store. VSO understands Vault leases natively — it tracks TTLs, renews before expiry, and requests a fresh credential when renewal is no longer possible. You lose the most powerful half of Vault’s feature set if you go ESO.

The Migration Path

Step 1: Audit Existing Secrets

Before touching a manifest, find what you’re dealing with:

kubectl get secrets -A -o json | \
  jq '.items[] | select(.type != "kubernetes.io/service-account-token") | {name: .metadata.name, ns: .metadata.namespace}'

Categorize each secret: static (API keys that rotate infrequently), dynamic (database credentials that should rotate per deployment), or PKI (TLS certs with a hard expiry date). The category determines which VSO CRD you’ll use.

Step 2: Install VSO

helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault-secrets-operator hashicorp/vault-secrets-operator \
  --namespace vault-secrets-operator-system \
  --create-namespace

VSO requires a running Vault instance. HCP Vault Dedicated removes the self-hosted operational burden — VSO connects identically either way.

Step 3: Configure Vault Auth

Enable Kubernetes auth in Vault and map a service account to a Vault policy:

vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://${K8S_HOST}:443"

vault write auth/kubernetes/role/my-app \
  bound_service_account_names=my-app-sa \
  bound_service_account_namespaces=production \
  policies=my-app-policy \
  ttl=1h

Then create the corresponding VaultAuth resource in the cluster:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: my-app-auth
  namespace: production
spec:
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: my-app
    serviceAccount: my-app-sa

Step 4: Replace Static Secrets

For an API key stored at secret/data/my-app/config in KV v2:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: my-app-config
  namespace: production
spec:
  vaultAuthRef: my-app-auth
  mount: secret
  type: kv-v2
  path: my-app/config
  destination:
    name: my-app-config
    create: true
  refreshAfter: 1h

VSO creates the K8s Secret immediately and re-syncs every hour. If the value changes in Vault, pods see the update within the refresh window — no CI job required.

Step 5: Upgrade to Dynamic Credentials

This is where VSO earns its place. For short-lived database credentials issued by Vault’s database secrets engine:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: my-app-db-creds
  namespace: production
spec:
  vaultAuthRef: my-app-auth
  mount: database
  path: creds/my-app-role
  destination:
    name: my-app-db-creds
    create: true
  rolloutRestartTargets:
    - kind: Deployment
      name: my-app

rolloutRestartTargets is the standout feature: when the lease expires and VSO fetches new credentials, it triggers a rolling restart of the target Deployment automatically. No humans, no cron jobs, no “why is the database timing out at 3 AM” incidents.

Step 6: Remove the Old Secrets

Once VSO owns the K8s Secret and your workloads are consuming it successfully, delete the manually managed resource and remove any CI/CD steps that were creating it. The secret now has exactly one source of truth: Vault.

Operational Tips

Namespace isolation: Create one VaultAuth per namespace, mapped to a narrow Vault policy. Never use a single auth role that can read all secrets — minimize blast radius from a compromised service account.

Immutability by design: VSO continuously reconciles. If someone edits the K8s Secret directly with kubectl, VSO overwrites it on the next sync. This enforces immutability at the secret layer automatically.

Certificate automation: Set expiryOffset: 72h on VaultPKISecret resources so VSO requests a renewed certificate three days before the current one expires. Combine with rolloutRestartTargets for zero-touch TLS rotation.

Observability: VSO exposes Prometheus metrics at :8080/metrics. Watch vso_secret_sync_total and vso_secret_sync_errors_total to catch sync failures before your pods notice.

The Bottom Line

Kubernetes Secrets are not secret management — they’re secret storage with no rotation, no audit trail, and no dynamic credential support. VSO closes all three gaps without forcing workloads to change how they consume secrets. The migration is incremental: audit, install, authenticate, define, remove. Most teams can move a production namespace in an afternoon.

The painful leak that usually triggers this migration is avoidable. The path is well-lit.

Sources