Part 19: cert-manager for TLS Automation
"Every Ingress in dev should be served over TLS for the same reason every Ingress in production should: because the bug is the missing TLS, and you only find it when you test with TLS."
Why
A real Kubernetes ingress serves traffic over HTTPS. The cert comes from somewhere — Let's Encrypt via HTTP-01 in production, or a private CA in dev. cert-manager is the standard way to automate cert issuance and renewal in Kubernetes. It watches Certificate resources (or Ingress objects with the cert-manager.io/cluster-issuer annotation) and obtains certs from a configured Issuer or ClusterIssuer.
For HomeLab K8s, the ClusterIssuer points at the same self-signed CA that HomeLab Docker generates (from homelab-docker Part 34). This means: one CA per HomeLab instance is trusted by every cluster service AND by the host machine's browser AND by the developer's curl. There is no second CA, no second trust enrollment, no second confusion.
The thesis: K8s.Dsl ships a CertManagerHelmReleaseContributor plus a HomeLabCaClusterIssuerContributor that imports the existing HomeLab CA into the cluster as a ClusterIssuer. Every Ingress gets a cert by adding one annotation. The whole flow is automatic.
The shape
[Injectable(ServiceLifetime.Singleton)]
public sealed class CertManagerHelmReleaseContributor : IHelmReleaseContributor
{
public string TargetCluster => "*";
public bool ShouldContribute() => true; // always installed
public void Contribute(KubernetesBundle bundle)
{
bundle.HelmReleases.Add(new HelmReleaseSpec
{
Name = "cert-manager",
Namespace = "cert-manager",
Chart = "jetstack/cert-manager",
Version = "v1.16.2",
RepoUrl = "https://charts.jetstack.io",
CreateNamespace = true,
Wait = true,
Values = new()
{
["installCRDs"] = true,
["replicaCount"] = _config.K8s?.Topology == "k8s-ha" ? 2 : 1,
["webhook"] = new Dictionary<string, object?>
{
["replicaCount"] = _config.K8s?.Topology == "k8s-ha" ? 2 : 1
},
["cainjector"] = new Dictionary<string, object?>
{
["replicaCount"] = _config.K8s?.Topology == "k8s-ha" ? 2 : 1
}
}
});
}
}
[Injectable(ServiceLifetime.Singleton)]
public sealed class HomeLabCaClusterIssuerContributor : IK8sManifestContributor
{
private readonly ITlsCertificateProvider _tls;
private readonly IFileSystem _fs;
public string TargetCluster => "*";
public async Task ContributeAsync(KubernetesBundle bundle, CancellationToken ct)
{
// 1. Read the HomeLab CA cert + key from the well-known location
var caCertPath = Path.Combine(_config.Workdir, "data", "certs", "ca.crt");
var caKeyPath = Path.Combine(_config.Workdir, "data", "certs", "ca.key");
var caCert = await _fs.File.ReadAllBytesAsync(caCertPath, ct);
var caKey = await _fs.File.ReadAllBytesAsync(caKeyPath, ct);
// 2. Create a Kubernetes Secret in the cert-manager namespace containing the CA
bundle.CrdInstances.Add(new RawManifest
{
ApiVersion = "v1",
Kind = "Secret",
Metadata = new() { Name = "homelab-ca-key-pair", Namespace = "cert-manager" },
Type = "kubernetes.io/tls",
Data = new Dictionary<string, string>
{
["tls.crt"] = Convert.ToBase64String(caCert),
["tls.key"] = Convert.ToBase64String(caKey)
}
});
// 3. Create a CA-type ClusterIssuer that uses the secret
bundle.CrdInstances.Add(new RawManifest
{
ApiVersion = "cert-manager.io/v1",
Kind = "ClusterIssuer",
Metadata = new() { Name = "homelab-ca" },
Spec = new Dictionary<string, object?>
{
["ca"] = new Dictionary<string, object?>
{
["secretName"] = "homelab-ca-key-pair"
}
}
});
}
// The synchronous Contribute is a wrapper that calls the async one
public void Contribute(KubernetesBundle bundle)
=> ContributeAsync(bundle, CancellationToken.None).GetAwaiter().GetResult();
}The CA cert and key are stored on the host machine in data/certs/ (the HomeLab Docker convention). The contributor reads them, base64-encodes them, and creates a Secret of type kubernetes.io/tls in the cluster. Then it creates a ClusterIssuer of type ca that points at the secret.
The CA private key is now inside the cluster. This is intentionally a private development concern: the cluster has the key because it needs to sign certs for every Ingress. The trust radius is one HomeLab instance per developer, not multiple users.
The ingress annotation
Once the ClusterIssuer is in place, every Ingress object gets a cert by adding one annotation:
public void Contribute(KubernetesBundle bundle)
{
bundle.Ingresses.Add(new IngressManifest
{
Name = "acme-api",
Namespace = "acme-prod",
IngressClassName = "nginx",
Annotations = new()
{
["cert-manager.io/cluster-issuer"] = "homelab-ca"
},
Rules = new[] { /* host rules */ },
Tls = new[]
{
new IngressTls { Hosts = new[] { "api.acme.lab" }, SecretName = "acme-api-tls" }
}
});
}When the Ingress is applied:
- cert-manager sees the annotation and the
Tlsblock - It creates a
Certificateresource forapi.acme.labwithSecretName=acme-api-tls - The certificate is signed by the
homelab-caClusterIssuer - The signed cert is stored in the
acme-api-tlsSecret - nginx-ingress (or Traefik) picks up the secret and serves HTTPS
The whole flow takes about 10 seconds from kubectl apply to working HTTPS. No human intervention.
Why the same CA as HomeLab Docker
Every developer who runs HomeLab adds the HomeLab CA to their OS trust store via homelab tls trust (from homelab-docker Part 34). After that, every cert signed by that CA is trusted by every browser, every curl, every dotnet HttpClient, every Java truststore (with one extra step), and every Go HTTP client on the host machine.
K8s.Dsl reuses this CA. Every cert issued by cert-manager via the homelab-ca ClusterIssuer is automatically trusted by the host machine, because the CA is the same one. The developer never sees a "self-signed cert" warning. The browser shows the green padlock. curl works without --insecure. dotnet HttpClient works without ServerCertificateCustomValidationCallback.
If we used a separate CA per cluster, the developer would have to enrol that CA into their OS trust store every time they spun up a new cluster. Reusing the existing one means zero trust enrolment work for the cluster certs.
What about Let's Encrypt?
Some users will ask: "can I use Let's Encrypt instead of a self-signed CA?". The answer is no — Let's Encrypt requires a public DNS resolvable hostname, and *.acme.lab is not public. If you have a real domain (*.acme.frenchexdev.com) and you want Let's Encrypt for dev, K8s.Dsl supports it via a different ClusterIssuer you opt into:
k8s:
cert_manager:
issuer: lets-encrypt # or "homelab-ca" (default)
email: ops@example.com # only meaningful with lets-encryptThe LetsEncryptClusterIssuerContributor creates an Issuer with the acme provider pointing at LE's production endpoint and uses DNS-01 against the user's real DNS provider (Cloudflare, Route53, etc.) via cert-manager's solver plugins. This is the same flow production uses, and validating it in dev catches a class of bugs (wrong DNS provider config, wrong solver, wrong rate-limit handling) that cannot be tested with self-signed CAs.
What this gives you that manual cert distribution doesn't
A manual approach is kubectl create secret tls foo-tls --cert=foo.crt --key=foo.key for every Ingress, plus a script to renew them every year. The cert paths are hard-coded. The renewal is forgotten. The first time a cert expires, the Ingress goes 503 and nobody knows why.
Typed cert-manager + ClusterIssuer contributors give you, for the same surface area:
- Automatic cert issuance for every Ingress with the right annotation
- Automatic renewal by cert-manager (default: 30 days before expiry)
- The same CA as the host machine so no extra trust enrolment is needed
- An optional Let's Encrypt path for users with real DNS
- Tests for the contributor configuration
The bargain pays back the first time you load https://api.acme.lab in your browser and see a green padlock without ever having clicked "advanced → proceed".