Encrypting CI/CD Artifacts and Data Streams with Vault Transit

Vault
Encrypting CI/CD Artifacts and Data Streams with Vault Transit

The Problem With Naive Encryption at Scale

When most teams think “encrypt this,” they reach for a KMS call or pipe the whole file through OpenSSL. That works fine for a 10 KB config file. It breaks badly when your artifact is a 2 GB Docker layer tarball or your Kafka topic is ingesting 50,000 events per second.

The failure modes are predictable: out-of-memory crashes in your CI runner, timeouts on KMS API calls that expect small payloads, and either skipping encryption entirely or protecting only a fraction of what actually needs coverage.

Vault’s Transit secrets engine solves this with a pattern called envelope encryption—and once you understand it, you will see why it is the right default for any non-trivial encryption workload.

Envelope Encryption in 60 Seconds

The Transit engine is Vault’s “encryption as a service” primitive. You send plaintext in, you get ciphertext back. But Transit is not designed to handle multi-gigabyte blobs directly—it is designed to protect keys, not payloads.

Envelope encryption works like this:

  1. Generate a random Data Encryption Key (DEK)—a short, symmetric key (e.g., AES-256).
  2. Encrypt your large artifact or stream with the DEK locally (fast, in-process).
  3. Send only the DEK to Vault’s Transit engine to be wrapped by your Key Encryption Key (KEK).
  4. Store the encrypted DEK alongside the artifact. Discard the plaintext DEK.

To decrypt, reverse the steps: unwrap the DEK from Vault, decrypt the artifact locally. Vault never touches the actual payload. Your CI runner never loads gigabytes into a Vault API call.

Artifact (2 GB) ──► AES-GCM encrypt (local) ──► encrypted artifact
                         ▲
                    DEK (32 bytes)
                         │
                    Vault Transit (wraps DEK)
                         │
                    wrapped DEK (stored with artifact)

This is exactly how AWS KMS, Google Cloud KMS, and Azure Key Vault work under the hood. Vault Transit lets you run the same pattern on-prem, in hybrid environments, or wherever your compliance boundary demands it.

Setting Up Vault Transit for Artifact Encryption

Enable the Transit engine and create a named key:

vault secrets enable transit

vault write -f transit/keys/ci-artifacts \
  type=aes256-gcm96

The aes256-gcm96 type gives you authenticated encryption with an automatic nonce—no footguns around IV reuse.

A minimal shell pattern for encrypting a build artifact in CI:

# 1. Generate a random DEK
DEK=$(openssl rand -base64 32)

# 2. Encrypt the artifact locally
openssl enc -aes-256-gcm -pbkdf2 \
  -pass pass:"$DEK" \
  -in build-output.tar.gz \
  -out build-output.tar.gz.enc

# 3. Wrap the DEK with Vault Transit
WRAPPED_DEK=$(vault write -field=ciphertext \
  transit/encrypt/ci-artifacts \
  plaintext=$(echo -n "$DEK" | base64))

# 4. Store WRAPPED_DEK alongside the encrypted artifact
echo "$WRAPPED_DEK" > build-output.tar.gz.enc.key

# 5. Discard the plaintext DEK
unset DEK

Your pipeline ships build-output.tar.gz.enc and build-output.tar.gz.enc.key to artifact storage. Without Vault access, neither file is useful.

To decrypt in a downstream stage:

# Unwrap the DEK
DEK=$(vault write -field=plaintext \
  transit/decrypt/ci-artifacts \
  ciphertext=$(cat build-output.tar.gz.enc.key) \
  | base64 --decode)

# Decrypt the artifact
openssl enc -d -aes-256-gcm -pbkdf2 \
  -pass pass:"$DEK" \
  -in build-output.tar.gz.enc \
  -out build-output.tar.gz

unset DEK

Access to Vault is governed by your existing AppRole or OIDC policies—no separate key management infrastructure required.

Streaming Workloads: Kafka and Beyond

High-throughput event streams present a different challenge. You cannot wrap every individual Kafka message through a Vault API call—latency and rate limits would destroy your throughput.

The envelope pattern still applies, but the DEK scope changes: instead of one DEK per artifact, you use one DEK per time window or partition batch.

A Practical Kafka Pattern

Producer               Vault Transit            Consumer
   │                        │                       │
   ├── fetch DEK for ──────►│                       │
   │   window               │                       │
   │◄── wrapped DEK ────────┤                       │
   │                        │                       │
   ├── encrypt events w/ DEK                        │
   ├── publish {wrapped_DEK + ciphertext_batch} ───►│
   │                        │                       │
   │                        │◄── unwrap DEK ────────┤
   │                        ├── plaintext DEK ──────►
   │                        │   decrypt events ─────►

A producer fetches a fresh DEK at the start of each window (e.g., every 5 minutes or 10,000 messages). It encrypts locally and embeds the wrapped DEK in the message header or a sidecar metadata topic. Consumers unwrap on first read and cache the plaintext DEK for the window duration.

This keeps Vault API call volume proportional to your window count, not your message count—manageable at any throughput.

Key Rotation Without Re-encrypting Everything

One of Transit’s most operationally valuable features is key versioning. When you rotate:

vault write -f transit/keys/ci-artifacts/rotate

Vault increments the key version. New encrypt operations use the latest version. Old wrapped DEKs can still be unwrapped with their original version—Transit tracks all versions until you explicitly retire them. Your artifact store does not need to be re-encrypted en masse.

When you are ready to enforce the new key version and retire old material:

vault write transit/keys/ci-artifacts/config \
  min_decryption_version=2 \
  min_encryption_version=2

This is a fundamentally better operational model than rotating a raw symmetric key and scrambling to re-encrypt a 500 GB artifact store over a weekend.

Access Control: Separating Encrypt from Decrypt

A minimal policy for a CI producer role that can only encrypt:

path "transit/encrypt/ci-artifacts" {
  capabilities = ["update"]
}

A deployment consumer that needs to decrypt:

path "transit/decrypt/ci-artifacts" {
  capabilities = ["update"]
}

Keep these roles separate. A compromised build runner should not be able to decrypt production artifacts. This separation is trivial in Vault but impossible when you are using a shared symmetric key baked into pipeline environment variables.

Operational Pitfalls to Avoid

  • Nonce reuse: If you call raw AES-GCM outside Vault’s managed endpoint, never reuse a nonce with the same key. Vault’s Transit endpoint handles this automatically—prefer it where you can.
  • DEK lifetime: Shorter-lived DEKs shrink your blast radius. For streaming, a 5-minute window is a reasonable default. For artifacts, per-build DEKs are ideal.
  • Audit logs: Vault logs every Transit operation. Route these to your SIEM. An unexpected spike in decrypt calls is a meaningful signal worth alerting on.
  • Network topology: If you are running on Azure with HCP Vault Dedicated, hub-and-spoke network peering lets you centralize Transit access across multiple spoke VNets without exposing Vault’s API surface to the public internet.

The Takeaway

Vault Transit and envelope encryption give platform teams a clean, auditable path to protecting large artifacts and high-throughput streams without loading multi-gigabyte payloads into API calls or re-encrypting everything on key rotation. The pattern is straightforward: encrypt locally with a DEK, protect the DEK with Vault. Everything else—access control, key versioning, audit trails—comes from Vault’s existing machinery at no extra configuration cost.

If your team is already using Vault for secret rotation, adding Transit for data-at-rest and data-in-motion encryption is a low-friction step with a high return on compliance posture.

Sources