Skip to main content
Welcome. This site supports keyboard navigation and screen readers. Press ? at any time for keyboard shortcuts. Press [ to focus the sidebar, ] to focus the content. High-contrast themes are available via the toolbar.
serard@dev00:~/cv

Part 10: Secrets Bridging from ISecretStore to k8s Secrets

"Two paths from the secret store to the cluster. Build-time for the simple cases. Runtime sync for the rotating cases. Both go through one interface."

Why

homelab-docker Part 43 defined ISecretStore with five providers (file, age, sops, Vault, Bitwarden). The K8s.Dsl plugin needs to push those secrets into Kubernetes as Secret objects so workloads can mount them. There are two ways to do this:

  1. Build-time injection: at apply time, the K8sBundleWriter reads the secret value from ISecretStore and emits a Secret manifest with the value baked in. The manifest is then kubectl applyed to the cluster. Simple, fast, works for static secrets.
  2. Runtime sync: install External Secrets Operator in the cluster, configure it with a custom SecretStore that talks to the underlying ISecretStore, and let it sync the values into native Secret objects on a schedule. Handles rotation, supports auditing, decouples the secret lifecycle from the apply lifecycle.

Both have a place. Build-time is right for things that never rotate (the GitLab admin password, the MinIO root credentials). Runtime sync is right for things that do rotate (TLS certs, OAuth tokens, database credentials with short TTLs).

The thesis: K8s.Dsl supports both paths via the same ISecretStore interface. The bundle writer handles build-time. A new ExternalSecretsProvider Helm release handles runtime sync. The user picks per-secret which path applies.

Path 1: build-time injection

When a contributor declares a Secret manifest with a Source reference, the bundle writer reads the secret store at apply time and inlines the value:

[Builder]
public sealed record SecretManifest
{
    public required string Name { get; init; }
    public required string Namespace { get; init; }
    public required string Source { get; init; }   // ISecretStore key, e.g. "GITLAB_ROOT_PASSWORD"
    public string DataKey { get; init; } = "value";   // the key inside the Secret's data dict
    public string SyncMode { get; init; } = "build-time";   // "build-time" | "external-secrets"
}

A contributor uses it like this:

public void Contribute(KubernetesBundle bundle)
{
    bundle.Secrets.Add(new SecretManifest
    {
        Name = "gitlab-root",
        Namespace = "gitlab",
        Source = "GITLAB_ROOT_PASSWORD",
        DataKey = "password",
        SyncMode = "build-time"
    });
}

The bundle writer's secret stage runs late in the apply phase, after all other manifests have been generated. It walks every SecretManifest with SyncMode == "build-time", reads the secret, and produces a real Kubernetes Secret object:

[Injectable(ServiceLifetime.Singleton)]
public sealed class BuildTimeSecretMaterializer
{
    private readonly ISecretStore _secrets;
    private readonly IK8sBundleWriter _writer;
    private readonly IHomeLabEventBus _events;
    private readonly IClock _clock;

    public async Task<Result> MaterializeAsync(KubernetesBundle bundle, DirectoryInfo outputDir, CancellationToken ct)
    {
        var buildTime = bundle.Secrets.Where(s => s.SyncMode == "build-time").ToList();
        if (buildTime.Count == 0) return Result.Success();

        var materialized = new List<RawManifest>();
        foreach (var s in buildTime)
        {
            var value = await _secrets.ReadAsync(s.Source, ct);
            if (value.IsFailure)
                return Result.Failure($"secret '{s.Source}' not found in store: {string.Join(", ", value.Errors)}");

            materialized.Add(new RawManifest
            {
                ApiVersion = "v1",
                Kind = "Secret",
                Metadata = new() { Name = s.Name, Namespace = s.Namespace, Annotations = new() { ["k8sdsl.source"] = s.Source } },
                Type = "Opaque",
                Data = new Dictionary<string, string>
                {
                    [s.DataKey] = Convert.ToBase64String(Encoding.UTF8.GetBytes(value.Value))
                }
            });

            await _events.PublishAsync(new K8sSecretMaterialized(s.Name, s.Namespace, "build-time", _clock.UtcNow), ct);
        }

        // Write the materialized secrets to a separate file with restrictive permissions
        var secretsPath = Path.Combine(outputDir.FullName, "k8s", "secrets.yaml");
        await _writer.WriteRawManifestsAsync(materialized, secretsPath, ct);

        if (!OperatingSystem.IsWindows())
            File.SetUnixFileMode(secretsPath, UnixFileMode.UserRead | UnixFileMode.UserWrite);

        return Result.Success();
    }
}

Two things to notice:

  1. The secret value never appears in the typed bundle. It is materialized at apply time into a separate file (secrets.yaml) with 0600 permissions. The rest of the bundle (the Deployment, the Service, etc.) is freely viewable; only secrets.yaml is restricted.
  2. The materialized file is not committed to the GitOps repo. When ArgoCD is in use, the secrets file is excluded from the repo and applied directly via kubectl apply on the host. The GitOps repo references the secret by name only.

This solves the GitOps-with-secrets problem cleanly: the declarative state in the repo references the secret name, and the secret value comes from ISecretStore via the host's privileged path.

Path 2: External Secrets Operator (runtime sync)

For secrets that rotate, build-time injection is not enough. You need the cluster to re-read the secret periodically and update the in-cluster Secret object when the source value changes. The standard tool for this is External Secrets Operator, which supports many backends out of the box (Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, etc.) and can be extended with custom providers.

K8s.Dsl ships an IHelmReleaseContributor that installs External Secrets Operator and a custom ClusterSecretStore resource that points at HomeLab's ISecretStore:

[Injectable(ServiceLifetime.Singleton)]
public sealed class ExternalSecretsHelmReleaseContributor : IHelmReleaseContributor
{
    public string TargetCluster => "*";

    public bool ShouldContribute() =>
        _config.K8s?.Secrets?.RuntimeSync?.Enabled ?? false;

    public void Contribute(KubernetesBundle bundle)
    {
        bundle.HelmReleases.Add(new HelmReleaseSpec
        {
            Name = "external-secrets",
            Namespace = "external-secrets-system",
            Chart = "external-secrets/external-secrets",
            Version = "0.10.4",
            RepoUrl = "https://charts.external-secrets.io",
            CreateNamespace = true,
            Values = new()
            {
                ["installCRDs"] = true,
                ["webhook"] = new Dictionary<string, object?> { ["replicaCount"] = 1 },
                ["certController"] = new Dictionary<string, object?> { ["replicaCount"] = 1 }
            }
        });
    }
}

Plus an IK8sManifestContributor that creates a ClusterSecretStore pointing at a small bridge service running inside the lab (not in the cluster) that exposes ISecretStore as a Vault-compatible HTTP API:

[Injectable(ServiceLifetime.Singleton)]
public sealed class HomeLabClusterSecretStoreContributor : IK8sManifestContributor
{
    public string TargetCluster => "*";

    public bool ShouldContribute() => _config.K8s?.Secrets?.RuntimeSync?.Enabled ?? false;

    public void Contribute(KubernetesBundle bundle)
    {
        var bridgeUrl = _config.K8s?.Secrets?.RuntimeSync?.BridgeUrl ?? "http://homelab-secret-bridge.lab:8200";

        bundle.CrdInstances.Add(new RawManifest
        {
            ApiVersion = "external-secrets.io/v1",
            Kind = "ClusterSecretStore",
            Metadata = new() { Name = "homelab" },
            Spec = new Dictionary<string, object?>
            {
                ["provider"] = new Dictionary<string, object?>
                {
                    ["vault"] = new Dictionary<string, object?>
                    {
                        ["server"] = bridgeUrl,
                        ["path"] = "secret",
                        ["version"] = "v2",
                        ["auth"] = new Dictionary<string, object?>
                        {
                            ["tokenSecretRef"] = new Dictionary<string, object?>
                            {
                                ["name"] = "homelab-bridge-token",
                                ["key"] = "token",
                                ["namespace"] = "external-secrets-system"
                            }
                        }
                    }
                }
            }
        });
    }
}

The "bridge service" is a small ASP.NET Core process that runs on the host (or in the lab's docker stack) and exposes ISecretStore over a Vault-compatible HTTP API. External Secrets Operator inside the cluster talks to it. The bridge enforces an instance scope (only secrets for this HomeLab instance are accessible) and uses a token from the secret store itself for authentication.

The full bridge implementation is ~200 lines and ships in the K8sDsl.Lib NuGet. It is started by HomeLab's compose deploy step alongside the other DevLab services, on the host VM that the cluster's nodes can reach.

A user who wants runtime sync flips one config field:

k8s:
  secrets:
    runtime_sync:
      enabled: true
      bridge_url: http://homelab-secret-bridge.acme.lab:8200

After that, every contributor that declares SyncMode = "external-secrets" produces an ExternalSecret CRD instance instead of a build-time materialized Secret:

public void Contribute(KubernetesBundle bundle)
{
    bundle.CrdInstances.Add(new RawManifest
    {
        ApiVersion = "external-secrets.io/v1",
        Kind = "ExternalSecret",
        Metadata = new() { Name = "acme-api-db", Namespace = "acme-prod" },
        Spec = new Dictionary<string, object?>
        {
            ["refreshInterval"] = "1h",
            ["secretStoreRef"] = new Dictionary<string, object?>
            {
                ["name"] = "homelab",
                ["kind"] = "ClusterSecretStore"
            },
            ["target"] = new Dictionary<string, object?>
            {
                ["name"] = "acme-api-db-secret"
            },
            ["data"] = new[]
            {
                new Dictionary<string, object?>
                {
                    ["secretKey"] = "password",
                    ["remoteRef"] = new Dictionary<string, object?>
                    {
                        ["key"] = "ACME_DB_PASSWORD"
                    }
                }
            }
        }
    });
}

External Secrets Operator polls the bridge every hour, fetches the latest value, and creates / updates the acme-api-db-secret Kubernetes Secret. Workloads mount it normally. When the value rotates in the underlying ISecretStore, the cluster picks up the change within the refresh interval.

When to use which

Scenario Path
GitLab admin password (set once, rotated yearly via day-2 verb) build-time
MinIO root credentials (set at provisioning, rotated rarely) build-time
TLS server cert + key (rotated quarterly by cert-manager) build-time (cert-manager handles rotation)
Postgres user password (rotated by CloudNativePG operator on schedule) runtime sync — operator updates, ESO picks up
External API token (rotated by upstream provider) runtime sync
Workload-specific credential issued per deploy build-time

The default is build-time. Runtime sync is opt-in per secret. The user does not need ESO running unless they have a secret that benefits from it.

The wiring

Both paths are [Injectable]. The build-time materializer runs in the K8sApplyStage (after manifests are generated, before they are applied). The runtime sync contributors run in the standard K8sGenerateStage. The cluster lifecycle does not change.

K8sSecretMaterialized events are published for both paths so the audit log captures every secret access. The audit log is a subscriber on the event bus (from homelab-docker Part 09) and writes to a JSONL file per instance.

The test

[Fact]
public async Task build_time_materialization_writes_secret_yaml_with_base64_value()
{
    var fs = new MockFileSystem();
    var secrets = new InMemorySecretStore();
    await secrets.WriteAsync("GITLAB_ROOT_PASSWORD", "s3cret-123", default);

    var bundle = new KubernetesBundle();
    bundle.Secrets.Add(new SecretManifest
    {
        Name = "gitlab-root", Namespace = "gitlab", Source = "GITLAB_ROOT_PASSWORD", DataKey = "password"
    });

    var writer = new K8sBundleWriter(fs);
    var materializer = new BuildTimeSecretMaterializer(secrets, writer, new RecordingEventBus(), new FakeClock(DateTimeOffset.UtcNow));

    var result = await materializer.MaterializeAsync(bundle, new DirectoryInfo("/lab"), default);

    result.IsSuccess.Should().BeTrue();
    fs.FileExists("/lab/k8s/secrets.yaml").Should().BeTrue();
    var content = fs.File.ReadAllText("/lab/k8s/secrets.yaml");
    content.Should().Contain("name: gitlab-root");
    content.Should().Contain($"password: {Convert.ToBase64String(Encoding.UTF8.GetBytes("s3cret-123"))}");
}

[Fact]
public async Task missing_secret_returns_failure_with_clear_message()
{
    var bundle = new KubernetesBundle();
    bundle.Secrets.Add(new SecretManifest { Name = "x", Namespace = "y", Source = "DOES_NOT_EXIST" });

    var materializer = new BuildTimeSecretMaterializer(
        new InMemorySecretStore(),
        Mock.Of<IK8sBundleWriter>(),
        new RecordingEventBus(),
        new FakeClock(DateTimeOffset.UtcNow));

    var result = await materializer.MaterializeAsync(bundle, new DirectoryInfo("/x"), default);

    result.IsFailure.Should().BeTrue();
    result.Errors.Should().Contain(e => e.Contains("DOES_NOT_EXIST"));
}

[Fact]
public void external_secrets_contributor_does_not_run_when_runtime_sync_disabled()
{
    var bundle = new KubernetesBundle();
    var contributor = new ExternalSecretsHelmReleaseContributor(
        Options.Create(new HomeLabConfig
        {
            K8s = new() { Secrets = new() { RuntimeSync = new() { Enabled = false } } }
        }));

    contributor.ShouldContribute().Should().BeFalse();
    if (contributor.ShouldContribute()) contributor.Contribute(bundle);

    bundle.HelmReleases.Should().NotContain(h => h.Name == "external-secrets");
}

What this gives you that hand-managed Kubernetes Secrets don't

Hand-managed: kubectl create secret generic gitlab-root --from-literal=password=... from a script that hard-codes the value, or sops --decrypt secrets.yaml | kubectl apply -f - from a script that hopes the user has the right age key. Both work. Both lose audit. Neither integrates with rotation.

A typed two-path bridge gives you, for the same surface area:

  • One ISecretStore interface for both paths
  • Build-time materialization that produces a real Secret manifest with the value inlined
  • Runtime sync via External Secrets Operator with a HomeLab-aware bridge
  • Per-secret choice so the user picks the right path for each kind of secret
  • Audit events on every materialization
  • Tests for both paths against fake stores

The bargain pays back the first time a database password rotates and you do not have to redeploy anything because External Secrets Operator picked up the change on its own.

Cross-links

⬇ Download