Skip to main content
Welcome. This site supports keyboard navigation and screen readers. Press ? at any time for keyboard shortcuts. Press [ to focus the sidebar, ] to focus the content. High-contrast themes are available via the toolbar.
serard@dev00:~/cv

Part 12: Choosing the Distribution — kubeadm vs k3s

"Pick kubeadm if you want to learn real Kubernetes operations. Pick k3s if you want to ship something that fits in 700 MB of RAM. Either is real."

Why

Part 04 made the case that kubeadm and k3s are both real Kubernetes. This part is the implementation: how K8s.Dsl exposes the choice as a config field, and how the plugin's two IClusterDistribution implementations make the rest of the lib agnostic to which one is in use.

The thesis: IClusterDistribution is a small plugin contract with two K8s.Dsl-shipped implementations (KubeadmClusterDistribution and K3sClusterDistribution). The user picks via k8s.distribution: kubeadm | k3s. Every other contributor in K8s.Dsl is distribution-agnostic; only the bootstrap and upgrade saga branches.

The shape

public interface IClusterDistribution
{
    string Name { get; }
    string DefaultVersion { get; }
    Task<Result<ClusterBootstrapResult>> BootstrapControlPlaneAsync(ControlPlaneSpec spec, CancellationToken ct);
    Task<Result> JoinWorkerAsync(WorkerJoinSpec spec, CancellationToken ct);
    Task<Result<KubeconfigBundle>> ExportKubeconfigAsync(string clusterName, CancellationToken ct);
    Task<Result> UpgradeAsync(UpgradeSpec spec, CancellationToken ct);
    Task<Result> DestroyAsync(string clusterName, CancellationToken ct);
    Task<Result<ClusterHealth>> HealthCheckAsync(CancellationToken ct);
}

public sealed record ControlPlaneSpec(
    string ClusterName,
    string ApiAdvertiseAddress,
    int ApiPort,
    string PodSubnet,
    string ServiceSubnet,
    string Version,
    bool IsHa,
    string? ApiVip);

public sealed record WorkerJoinSpec(
    string ClusterName,
    string NodeName,
    string ControlPlaneEndpoint,
    string JoinToken,
    string CaCertHash);

public sealed record ClusterBootstrapResult(
    KubeconfigBundle Kubeconfig,
    string JoinToken,
    string CaCertHash,
    DateTimeOffset BootstrappedAt);

The interface is small — six methods — and every implementation returns Result<T>. The distribution does not own the VMs (those come from the topology resolver) or the kubeconfig store (that is a separate plugin contract). It owns the bootstrap: from "an empty Vagrant VM" to "a usable Kubernetes API server".

KubeadmClusterDistribution

[Injectable(ServiceLifetime.Singleton)]
public sealed class KubeadmClusterDistribution : IClusterDistribution
{
    public string Name => "kubeadm";
    public string DefaultVersion => "v1.31.4";

    private readonly KubeadmClient _kubeadm;
    private readonly KubectlClient _kubectl;
    private readonly IVosBackend _vagrant;
    private readonly IHomeLabEventBus _events;
    private readonly IClock _clock;

    public async Task<Result<ClusterBootstrapResult>> BootstrapControlPlaneAsync(ControlPlaneSpec spec, CancellationToken ct)
    {
        await _events.PublishAsync(new ClusterBootstrapStarted(spec.ClusterName, "kubeadm", _clock.UtcNow), ct);

        // 1. SSH into the first control-plane VM and run kubeadm init
        var initConfig = KubeadmConfigGenerator.GenerateInitConfig(spec);
        var configPath = $"/tmp/kubeadm-init-config-{spec.ClusterName}.yaml";

        await _vagrant.SshFileWriteAsync($"{spec.ClusterName}-cp-1", configPath, initConfig, ct);

        var initResult = await _vagrant.SshCommandAsync(
            $"{spec.ClusterName}-cp-1",
            $"sudo kubeadm init --config {configPath} --upload-certs",
            ct);
        if (initResult.IsFailure) return initResult.Map<ClusterBootstrapResult>();

        // 2. Parse the join token from the kubeadm output
        var (joinToken, caCertHash) = KubeadmOutputParser.ParseJoinDetails(initResult.Value.StdOut);

        // 3. Pull the kubeconfig out of the VM
        var kubeconfigContent = await _vagrant.SshCommandAsync(
            $"{spec.ClusterName}-cp-1",
            "sudo cat /etc/kubernetes/admin.conf",
            ct);
        if (kubeconfigContent.IsFailure) return kubeconfigContent.Map<ClusterBootstrapResult>();

        var kubeconfig = KubeconfigParser.ParseAdminConfig(kubeconfigContent.Value.StdOut, spec.ClusterName);

        await _events.PublishAsync(new ClusterBootstrapCompleted(spec.ClusterName, _clock.UtcNow), ct);
        return Result.Success(new ClusterBootstrapResult(kubeconfig, joinToken, caCertHash, _clock.UtcNow));
    }

    public async Task<Result> JoinWorkerAsync(WorkerJoinSpec spec, CancellationToken ct)
    {
        var cmd = $"sudo kubeadm join {spec.ControlPlaneEndpoint} " +
                  $"--token {spec.JoinToken} " +
                  $"--discovery-token-ca-cert-hash sha256:{spec.CaCertHash}";

        return await _vagrant.SshCommandAsync(spec.NodeName, cmd, ct).Map();
    }

    // ... ExportKubeconfigAsync, UpgradeAsync, DestroyAsync, HealthCheckAsync similar
}

The KubeadmClient is the typed [BinaryWrapper("kubeadm")] wrapper that the distribution composes with. We will see it in Part 14.

K3sClusterDistribution

[Injectable(ServiceLifetime.Singleton)]
public sealed class K3sClusterDistribution : IClusterDistribution
{
    public string Name => "k3s";
    public string DefaultVersion => "v1.31.4+k3s1";

    private readonly K3sClient _k3s;
    private readonly KubectlClient _kubectl;
    private readonly IVosBackend _vagrant;
    private readonly IHomeLabEventBus _events;
    private readonly IClock _clock;
    private readonly IOptions<HomeLabConfig> _config;

    public async Task<Result<ClusterBootstrapResult>> BootstrapControlPlaneAsync(ControlPlaneSpec spec, CancellationToken ct)
    {
        await _events.PublishAsync(new ClusterBootstrapStarted(spec.ClusterName, "k3s", _clock.UtcNow), ct);

        var disableArgs = BuildDisableArgs(_config.Value.K8s!.K3s);
        var clusterInit = spec.IsHa ? "--cluster-init" : "";

        var installCmd =
            $"curl -sfL https://get.k3s.io | " +
            $"INSTALL_K3S_VERSION={spec.Version} " +
            $"K3S_NODE_NAME={spec.ClusterName}-cp-1 " +
            $"sh -s - server {disableArgs} {clusterInit} " +
            $"--tls-san={spec.ApiAdvertiseAddress} " +
            $"--cluster-cidr={spec.PodSubnet} " +
            $"--service-cidr={spec.ServiceSubnet}";

        var result = await _vagrant.SshCommandAsync($"{spec.ClusterName}-cp-1", installCmd, ct);
        if (result.IsFailure) return result.Map<ClusterBootstrapResult>();

        // Wait for the kubeconfig file to appear
        await Task.Delay(TimeSpan.FromSeconds(5), ct);

        var tokenResult = await _vagrant.SshCommandAsync(
            $"{spec.ClusterName}-cp-1",
            "sudo cat /var/lib/rancher/k3s/server/node-token",
            ct);
        if (tokenResult.IsFailure) return tokenResult.Map<ClusterBootstrapResult>();

        var kubeconfigResult = await _vagrant.SshCommandAsync(
            $"{spec.ClusterName}-cp-1",
            "sudo cat /etc/rancher/k3s/k3s.yaml",
            ct);
        if (kubeconfigResult.IsFailure) return kubeconfigResult.Map<ClusterBootstrapResult>();

        var kubeconfig = KubeconfigParser.ParseK3sConfig(kubeconfigResult.Value.StdOut, spec.ClusterName, spec.ApiAdvertiseAddress);

        await _events.PublishAsync(new ClusterBootstrapCompleted(spec.ClusterName, _clock.UtcNow), ct);
        return Result.Success(new ClusterBootstrapResult(
            Kubeconfig: kubeconfig,
            JoinToken: tokenResult.Value.StdOut.Trim(),
            CaCertHash: "",   // k3s uses node-token, not a CA hash
            BootstrappedAt: _clock.UtcNow));
    }

    public async Task<Result> JoinWorkerAsync(WorkerJoinSpec spec, CancellationToken ct)
    {
        var installCmd =
            $"curl -sfL https://get.k3s.io | " +
            $"INSTALL_K3S_VERSION={_config.Value.K8s!.Version} " +
            $"K3S_URL=https://{spec.ControlPlaneEndpoint}:6443 " +
            $"K3S_TOKEN={spec.JoinToken} " +
            $"K3S_NODE_NAME={spec.NodeName} " +
            $"sh -s - agent";

        return await _vagrant.SshCommandAsync(spec.NodeName, installCmd, ct).Map();
    }

    private static string BuildDisableArgs(K3sConfig? cfg)
    {
        var disable = new List<string>();
        if (cfg?.DisableTraefik ?? true) disable.Add("--disable=traefik");
        if (cfg?.DisableServicelb ?? true) disable.Add("--disable=servicelb");
        if (cfg?.DisableLocalStorage ?? false) disable.Add("--disable=local-storage");
        return string.Join(" ", disable);
    }

    // ... ExportKubeconfigAsync, UpgradeAsync, DestroyAsync, HealthCheckAsync similar
}

The k3s distribution disables Traefik and servicelb by default — both because we install our own ingress controller via Part 18 and because the bundled Traefik conflicts with the multi-instance pattern.

How the user picks

# config-homelab.yaml
k8s:
  distribution: kubeadm     # or "k3s"
  version: "v1.31.4"
  topology: k8s-multi
  k3s:
    disable_traefik: true       # only meaningful when distribution: k3s
    disable_servicelb: true
    disable_local_storage: false
  kubeadm:
    pod_subnet: "10.244.0.0/16"  # only meaningful when distribution: kubeadm
    service_subnet: "10.96.0.0/12"
    feature_gates: []
    audit_policy_path: null

The two sub-blocks (k3s and kubeadm) are option-tagged: only the active one is meaningful, but both are validated against the schema. A user who picks distribution: k3s and sets kubeadm.pod_subnet gets a clear "this field is ignored when distribution is k3s" warning at validate time — not silent acceptance.

The composition root

DI registers both implementations and a IClusterDistributionResolver that picks one based on config:

[Injectable(ServiceLifetime.Singleton)]
public sealed class ClusterDistributionResolver
{
    private readonly IEnumerable<IClusterDistribution> _distributions;
    private readonly IOptions<HomeLabConfig> _config;

    public IClusterDistribution Resolve()
    {
        var name = _config.Value.K8s?.Distribution ?? "kubeadm";
        var match = _distributions.SingleOrDefault(d => d.Name == name);
        if (match is null)
            throw new InvalidOperationException(
                $"unknown distribution: {name}. Installed distributions: {string.Join(", ", _distributions.Select(d => d.Name))}");
        return match;
    }
}

The resolver is a thin layer because the choice is per-instance and read once per pipeline run. The K8sCreateRequestHandler calls it once and caches the result for the duration of the run.

The pluggability angle

A future plugin can ship its own distribution by implementing IClusterDistribution and shipping it as a NuGet:

[Injectable(ServiceLifetime.Singleton)]
public sealed class TalosClusterDistribution : IClusterDistribution
{
    public string Name => "talos";
    public string DefaultVersion => "v1.9.0";

    // ... uses talosctl wrapper instead of kubeadm/k3s
}

FrenchExDev.HomeLab.Plugin.K8sDsl.Talos is a NuGet that:

  1. Adds a Packer overlay for the Talos node image (Packer.Talos.Node)
  2. Registers TalosClusterDistribution via [Injectable]
  3. Adds a manifest entry clusterDistributions: ["talos"]

After installation, distribution: talos becomes a valid choice. The K8s.Dsl core does not need to change. Everything else in the plugin (the manifest contributors, the Helm release contributors, the kubeconfig store, the topology resolver) is distribution-agnostic and works unchanged.

The same pattern applies to k0s, Microk8s, RKE2, and any future distribution. K8s.Dsl ships kubeadm and k3s; the rest is community.

The test

[Fact]
public async Task kubeadm_distribution_calls_kubeadm_init_with_correct_config()
{
    var vagrant = new ScriptedVosBackend();
    vagrant.OnSshFileWrite("acme-cp-1", "/tmp/kubeadm-init-config-acme.yaml", _ => { });
    vagrant.OnSshCommand("acme-cp-1",
        cmd => cmd.Contains("kubeadm init"),
        exitCode: 0,
        stdout: "kubeadm join 192.168.60.10:6443 --token abc123.def456 --discovery-token-ca-cert-hash sha256:cafebabe");
    vagrant.OnSshCommand("acme-cp-1",
        cmd => cmd.Contains("admin.conf"),
        exitCode: 0,
        stdout: TestData.MinimalKubeconfig());

    var distribution = new KubeadmClusterDistribution(
        Mock.Of<KubeadmClient>(),
        Mock.Of<KubectlClient>(),
        vagrant,
        new RecordingEventBus(),
        new FakeClock(DateTimeOffset.UtcNow));

    var spec = new ControlPlaneSpec(
        ClusterName: "acme",
        ApiAdvertiseAddress: "192.168.60.10",
        ApiPort: 6443,
        PodSubnet: "10.244.0.0/16",
        ServiceSubnet: "10.96.0.0/12",
        Version: "v1.31.4",
        IsHa: false,
        ApiVip: null);

    var result = await distribution.BootstrapControlPlaneAsync(spec, default);

    result.IsSuccess.Should().BeTrue();
    result.Value.JoinToken.Should().Be("abc123.def456");
    result.Value.CaCertHash.Should().Be("cafebabe");
}

[Fact]
public async Task k3s_distribution_passes_disable_args_for_traefik_and_servicelb()
{
    var vagrant = new ScriptedVosBackend();
    string? capturedCommand = null;
    vagrant.OnSshCommand("acme-cp-1",
        cmd => { capturedCommand = cmd; return cmd.Contains("get.k3s.io"); },
        exitCode: 0, stdout: "");
    vagrant.OnSshCommand("acme-cp-1", cmd => cmd.Contains("node-token"), exitCode: 0, stdout: "node-token-xyz");
    vagrant.OnSshCommand("acme-cp-1", cmd => cmd.Contains("k3s.yaml"), exitCode: 0, stdout: TestData.MinimalK3sConfig());

    var config = Options.Create(new HomeLabConfig
    {
        K8s = new K8sConfig
        {
            Distribution = "k3s",
            Version = "v1.31.4+k3s1",
            K3s = new K3sConfig { DisableTraefik = true, DisableServicelb = true }
        }
    });

    var distribution = new K3sClusterDistribution(
        Mock.Of<K3sClient>(), Mock.Of<KubectlClient>(),
        vagrant, new RecordingEventBus(), new FakeClock(DateTimeOffset.UtcNow), config);

    var result = await distribution.BootstrapControlPlaneAsync(StandardSpec("acme"), default);

    result.IsSuccess.Should().BeTrue();
    capturedCommand.Should().Contain("--disable=traefik");
    capturedCommand.Should().Contain("--disable=servicelb");
}

[Fact]
public void resolver_returns_kubeadm_when_config_says_so()
{
    var distributions = new IClusterDistribution[]
    {
        new KubeadmClusterDistribution(/* ... */),
        new K3sClusterDistribution(/* ... */)
    };
    var resolver = new ClusterDistributionResolver(distributions,
        Options.Create(new HomeLabConfig { K8s = new() { Distribution = "kubeadm" } }));

    resolver.Resolve().Name.Should().Be("kubeadm");
}

What this gives you that hand-rolled kubeadm init scripts don't

The hand-rolled approach is a bash script that runs kubeadm init with hard-coded flags, parses the output for the join token, copies the kubeconfig to the host via scp, and breaks the moment the kubeadm output format changes.

A typed IClusterDistribution with two implementations gives you, for the same surface area:

  • One config field to switch between distributions
  • Two parallel implementations that follow the same six-method contract
  • Plugin extensibility for k0s, Talos, Microk8s, RKE2 — none of which require K8s.Dsl core changes
  • Architecture tests that prevent the distribution from leaking into other contributors
  • Tests that exercise both distributions against fakes without spawning real VMs

The bargain pays back the first time a security review says "we cannot ship a daemon" and you switch the cluster from kubeadm to k3s with one config field, watching everything else (manifests, ingress, cert-manager, ArgoCD, the workloads) come up identically.

Cross-links

⬇ Download