Skip to main content
Welcome. This site supports keyboard navigation and screen readers. Press ? at any time for keyboard shortcuts. Press [ to focus the sidebar, ] to focus the content. High-contrast themes are available via the toolbar.
serard@dev00:~/cv

Part 51: Running Many Labs Side-By-Side

"Multiple instances by isolation, not by convention. The architecture rejects two labs that could collide." — HomeLab/doc/PHILOSOPHY.md

Why

A homelab user typically wants more than one lab running at once:

  • dev-single — a throwaway single-VM lab they iterate on while developing HomeLab itself
  • prod-multi — the real four-VM DevLab that hosts the actual GitLab repo
  • ha-stage — an HA topology used to validate Reference Architecture changes before they touch production
  • pr-1234 — an ephemeral lab spun up by CI to validate every HomeLab pull request end to end

These four instances should coexist on the same workstation without colliding on subnets, VM names, DNS entries, certs, or docker networks. The naive approach (run them sequentially, tear down between runs) is too slow. The right approach is structural isolation: every resource gets a per-instance prefix, every subnet is allocated from a non-overlapping range, every DNS entry is namespaced.

The thesis of this part is: HomeLab assigns each instance a unique name; the name is the prefix for every resource the instance creates. The subnet allocator gives each instance a non-overlapping /24. The DNS provider namespaces hostnames by instance. Two instances cannot accidentally collide because the allocator refuses to.

The shape: instance scoping

public sealed record InstanceScope(string Name, string Subnet, string TldPrefix);

public interface IInstanceRegistry
{
    Task<Result<InstanceScope>> AcquireAsync(string name, CancellationToken ct);
    Task<Result> ReleaseAsync(string name, CancellationToken ct);
    Task<Result<IReadOnlyList<InstanceScope>>> ListAsync(CancellationToken ct);
}

[Injectable(ServiceLifetime.Singleton)]
public sealed class FileBasedInstanceRegistry : IInstanceRegistry
{
    private readonly string _registryPath = Path.Combine(
        Environment.GetFolderPath(Environment.SpecialFolder.UserProfile),
        ".homelab",
        "instances.json");

    public async Task<Result<InstanceScope>> AcquireAsync(string name, CancellationToken ct)
    {
        var existing = await LoadAsync(ct);
        if (existing.Any(i => i.Name == name))
            return Result.Success(existing.Single(i => i.Name == name));   // already exists; idempotent

        // Allocate a new subnet from a private range
        var allocated = AllocateSubnet(existing);
        if (allocated.IsFailure) return allocated.Map<InstanceScope>();

        // TLD prefix is the instance name (mostly for hostname namespacing)
        var scope = new InstanceScope(
            Name: name,
            Subnet: allocated.Value,
            TldPrefix: name);

        var updated = existing.Append(scope).ToList();
        await SaveAsync(updated, ct);
        return Result.Success(scope);
    }

    private static Result<string> AllocateSubnet(IEnumerable<InstanceScope> existing)
    {
        // Allocate from 192.168.{56-95}.0/24 (40 instances max in this range)
        var taken = existing.Select(i => int.Parse(i.Subnet.Split('.')[2])).ToHashSet();
        for (var third = 56; third <= 95; third++)
        {
            if (!taken.Contains(third))
                return Result.Success($"192.168.{third}");
        }
        return Result.Failure<string>("no free subnet in 192.168.{56..95}");
    }
}

The registry is a small JSON file in ~/.homelab/instances.json. Each instance has a unique name and a non-overlapping /24. Acquiring a new instance is atomic: read, allocate, write. Two homelab init commands cannot get the same subnet because the registry serialises the writes.

Per-instance resource naming

Once a scope is acquired, every resource the instance creates is prefixed with the instance name:

Resource Without scope With scope prod-multi
VM names gateway, platform, data, obs prod-multi-gateway, prod-multi-platform, prod-multi-data, prod-multi-obs
Subnet 192.168.56 192.168.57 (allocated by registry)
DNS hostname (default TLD) gitlab.frenchexdev.lab gitlab.prod-multi.lab
Cert wildcard *.frenchexdev.lab *.prod-multi.lab
Docker network platform prod-multi-platform
Compose project devlab prod-multi-devlab
MinIO buckets gitlab-artifacts prod-multi-gitlab-artifacts
Vagrant box names (per-image, shared) (per-image, shared)
Working directory ./ ./prod-multi/

The prefix is applied by every contributor at composition time. The contributors do not branch on instance — they just call _scope.PrefixOf("gateway") and get back "prod-multi-gateway".

public interface IInstanceScope
{
    string Name { get; }
    string Subnet { get; }
    string TldPrefix { get; }
    string PrefixOf(string baseName) => $"{Name}-{baseName}";
    string IpOf(int hostByte) => $"{Subnet}.{hostByte}";
    string HostnameOf(string baseName) => $"{baseName}.{TldPrefix}.lab";
}

The contributors take IInstanceScope from DI and use it everywhere they would otherwise hard-code a name.

DNS namespacing

In the multi-instance world, gitlab.frenchexdev.lab is ambiguous — which instance? The fix is to namespace hostnames by instance: gitlab.dev-single.lab, gitlab.prod-multi.lab, gitlab.ha-stage.lab. Each instance still serves the same services on different hostnames.

The DNS provider adds entries scoped to the instance:

public async Task<Result> AddInstanceEntriesAsync(InstanceScope scope, IReadOnlyList<(string Service, int HostByte)> services, CancellationToken ct)
{
    foreach (var (svc, host) in services)
    {
        var hostname = scope.HostnameOf(svc);   // gitlab.prod-multi.lab
        var ip = scope.IpOf(host);              // 192.168.57.10
        var result = await AddAsync(hostname, ip, ct);
        if (result.IsFailure) return result;
    }
    return Result.Success();
}

Two instances side-by-side end up with two sets of hostnames in the hosts file (or in PiHole), each pointing to its own subnet. The browser knows which instance you mean by the URL.

Cert namespacing

Each instance gets its own CA and its own wildcard cert. They are not shared. This means every instance is independently trustable, every cert can be rotated independently, and revoking one instance does not affect any other.

// In the TLS init handler
var ca = await provider.GenerateCaAsync($"HomeLab CA - {scope.Name}", ct);
var cert = await provider.GenerateCertAsync(ca.Value, $"*.{scope.TldPrefix}.lab", new[]
{
    $"{scope.TldPrefix}.lab",
    $"*.{scope.TldPrefix}.lab"
}, ct);

The user runs homelab tls trust --instance prod-multi to enrol just that instance's CA. They can revoke dev-single's CA from the OS trust store without affecting prod-multi.

CI ephemeral instances

CI for HomeLab itself uses ephemeral instances. Every PR pipeline does:

homelab init --name pr-$CI_MERGE_REQUEST_IID --topology single --ephemeral
homelab vos up
# ... run tests against the ephemeral instance ...
homelab vos destroy --force
homelab instance release pr-$CI_MERGE_REQUEST_IID

The --ephemeral flag tells the registry that this instance is short-lived: its working directory lives in /tmp/, its VMs are tagged for cleanup, and a periodic janitor removes any ephemeral instance older than 24 hours regardless of whether the PR finished.

The runner that runs these tests is the runner inside prod-multi DevLab — dogfood loop #1 in action. The runner spins up a pr-1234 instance, the PR's HomeLab build runs against it, tests pass or fail, the instance is destroyed. None of this disturbs the running prod-multi because they have different subnets, different VM names, different DNS namespaces, different docker networks.

The Mermaid diagram

Diagram
Three permanent instances plus one ephemeral PR instance share the same workstation without collision — the registry refuses any overlapping subnet.

Three permanent instances + one ephemeral, all coexisting. None can collide because the registry refused to give them overlapping subnets.

The test

[Fact]
public async Task instance_registry_allocates_distinct_subnets()
{
    var registry = new FileBasedInstanceRegistry(Path.GetTempFileName());

    var first  = await registry.AcquireAsync("a", default);
    var second = await registry.AcquireAsync("b", default);
    var third  = await registry.AcquireAsync("c", default);

    first.Value.Subnet.Should().NotBe(second.Value.Subnet);
    second.Value.Subnet.Should().NotBe(third.Value.Subnet);
    first.Value.Subnet.Should().NotBe(third.Value.Subnet);
}

[Fact]
public async Task acquiring_existing_name_returns_same_scope_idempotently()
{
    var registry = new FileBasedInstanceRegistry(Path.GetTempFileName());
    var first  = await registry.AcquireAsync("dev", default);
    var second = await registry.AcquireAsync("dev", default);

    second.Value.Should().BeEquivalentTo(first.Value);
}

[Fact]
public async Task subnet_pool_exhaustion_returns_failure()
{
    var registry = new FileBasedInstanceRegistry(Path.GetTempFileName());
    for (var i = 0; i < 40; i++)
        await registry.AcquireAsync($"i{i}", default);

    var overflow = await registry.AcquireAsync("overflow", default);
    overflow.IsFailure.Should().BeTrue();
    overflow.Errors.Should().Contain(e => e.Contains("no free subnet"));
}

[Fact]
[Trait("category", "e2e")]
[Trait("category", "slow")]
public async Task three_topologies_can_coexist_on_one_workstation()
{
    using var dev   = await TestLab.NewAsync(name: "dev-single",  topology: "single");
    using var prod  = await TestLab.NewAsync(name: "prod-multi",  topology: "multi");
    using var stage = await TestLab.NewAsync(name: "ha-stage",    topology: "ha");

    await dev.UpAllAsync();
    await prod.UpAllAsync();
    await stage.UpAllAsync();

    dev.Service("gitlab").Should().BeHealthy();
    prod.Service("gitlab").Should().BeHealthy();
    stage.Service("gitlab-rails").Should().BeHealthy();

    dev.Subnet.Should().NotBe(prod.Subnet);
    prod.Subnet.Should().NotBe(stage.Subnet);
}

The last test is the proof that three full topologies coexist. It is slow (~90 minutes), runs nightly. When it passes, multi-instance is intact.

What this gives you that bash doesn't

A bash script that runs "two homelabs" is two copies of the script in different directories with hand-edited subnets and VM-name prefixes. Every hand-edit is a future bug.

A typed IInstanceRegistry with deterministic subnet allocation gives you, for the same surface area:

  • One JSON registry in ~/.homelab/instances.json that tracks every instance
  • Atomic acquisition that refuses overlapping subnets
  • Per-instance resource naming applied by every contributor
  • DNS namespacing with per-instance hostnames
  • Cert namespacing with per-instance CAs
  • Ephemeral instance support for CI
  • Tests that prove three topologies coexist

The bargain pays back the first time you spin up a pr-1234 ephemeral instance to validate a HomeLab change without disturbing the prod-multi that holds your real source code.

Cross-links

⬇ Download