Skip to main content
Welcome. This site supports keyboard navigation and screen readers. Press ? at any time for keyboard shortcuts. Press [ to focus the sidebar, ] to focus the content. High-contrast themes are available via the toolbar.
serard@dev00:~/cv

Part 31: The Compose File for DevLab

"There is no toy compose file. Every service is something I would lose work without."

Why

Part 30 decided which VMs exist. This part decides what runs on them. The compose file (or files, in multi-VM topology) is the heart of DevLab — the actual services HomeLab provisions. We have one compose file per VM in the multi-VM topology and one combined file in the single-VM topology, all produced by the same set of contributors.

The thesis of this part is: DevLab's compose files are produced by composing N typed IComposeFileContributors, one per service, all of them topology-agnostic. The contributors do not know how many VMs there are; they just contribute their service to a ComposeFile. The pipeline decides which contributors run on which VM based on the topology.

The shape

public sealed class ComposeFile
{
    public string Version { get; set; } = "3.8";
    public Dictionary<string, ComposeService> Services { get; set; } = new();
    public Dictionary<string, ComposeNetwork> Networks { get; set; } = new();
    public Dictionary<string, ComposeVolume> Volumes { get; set; } = new();
    public Dictionary<string, ComposeSecret> Secrets { get; set; } = new();
}

public sealed record ComposeService
{
    public string? Image { get; set; }
    public string? Restart { get; set; }
    public Dictionary<string, string> Environment { get; set; } = new();
    public List<string> Volumes { get; set; } = new();
    public List<string> Ports { get; set; } = new();
    public List<string> Networks { get; set; } = new();
    public Dictionary<string, ComposeDependency> DependsOn { get; set; } = new();
    public ComposeHealthcheck? HealthCheck { get; set; }
    public IReadOnlyDictionary<string, string>? Labels { get; set; }
    public string? Hostname { get; set; }
    public List<string> Secrets { get; set; } = new();
    public ComposeDeploy? Deploy { get; set; }
}

The contributor for GitLab:

[Injectable(ServiceLifetime.Singleton)]
public sealed class GitLabComposeContributor : IComposeFileContributor
{
    private readonly HomeLabConfig _config;
    private readonly IPlacementResolver _placement;

    public GitLabComposeContributor(IOptions<HomeLabConfig> config, IPlacementResolver placement)
    {
        _config = config.Value;
        _placement = placement;
    }

    public string TargetVm => _placement.For("gitlab");  // "main" in single, "platform" in multi/ha

    public void Contribute(ComposeFile compose)
    {
        compose.Services["gitlab"] = new ComposeService
        {
            Image = $"gitlab/gitlab-ce:{_config.GitLab.Version ?? "16.11.0-ce.0"}",
            Restart = "always",
            Hostname = $"gitlab.{_config.Acme.Tld}",
            Environment = new()
            {
                ["GITLAB_OMNIBUS_CONFIG"] = string.Empty,  // The actual config is in /etc/gitlab/gitlab.rb mounted below
            },
            Volumes = new()
            {
                "./gitlab/config:/etc/gitlab",
                "./gitlab/logs:/var/log/gitlab",
                "./gitlab/data:/var/opt/gitlab",
                "./certs:/etc/gitlab/ssl:ro"
            },
            Ports = new() { "22:22" },     // SSH for git pushes; HTTPS goes through Traefik
            Networks = new() { "platform" },
            HealthCheck = new ComposeHealthcheck
            {
                Test = new[] { "CMD", "curl", "-fk", "https://localhost:443/-/health" },
                Interval = "30s",
                Timeout = "10s",
                Retries = 5,
                StartPeriod = "10m"   // GitLab takes a while to come up
            },
            DependsOn = new()
            {
                ["postgres"] = new() { Condition = "service_healthy" },
                ["minio"]    = new() { Condition = "service_healthy" }
            },
            Labels = new TraefikLabels()
                .Enable()
                .Router("gitlab", r => r
                    .Rule($"Host(`gitlab.{_config.Acme.Tld}`)")
                    .EntryPoints("websecure")
                    .Tls(certResolver: "default"))
                .Build()
        };

        compose.Networks["platform"] ??= new ComposeNetwork { Driver = "bridge" };
    }
}

The contributor adds one service. The service is fully specified: image, restart policy, env, volumes, ports, networks, healthcheck, depends_on, Traefik labels. Everything is generated from the typed config.

All the DevLab contributors

Contributor Service Target VM (multi)
TraefikComposeContributor traefik gateway
PiholeComposeContributor pihole gateway
GitLabComposeContributor gitlab platform
GitLabRunnerComposeContributor gitlab-runner platform
BagetComposeContributor baget platform
PostgresComposeContributor postgres data
MinioComposeContributor minio data
MeilisearchComposeContributor meilisearch data
PrometheusComposeContributor prometheus obs
GrafanaComposeContributor grafana obs
LokiComposeContributor loki obs
AlertmanagerComposeContributor alertmanager obs
StaticDocsComposeContributor docs-site gateway
VagrantRegistryComposeContributor vagrant-registry platform

Fourteen contributors. Each one is [Injectable], each one is small (~30–60 lines), each one is independently testable. The list grows when DevLab gains a new service; nothing else changes.

Per-VM file generation

The Generate stage walks the contributors, partitions them by TargetVm, and produces one ComposeFile per VM:

[Injectable(ServiceLifetime.Singleton)]
public sealed class PerVmComposeGenerator
{
    private readonly IEnumerable<IComposeFileContributor> _contributors;
    private readonly IBundleWriter _writer;

    public async Task<Result<IReadOnlyDictionary<string, string>>> GenerateAsync(HomeLabPlan plan, DirectoryInfo outputDir, CancellationToken ct)
    {
        var result = new Dictionary<string, string>();

        foreach (var vm in plan.Machines)
        {
            var compose = new ComposeFile();
            var contributorsForVm = _contributors.Where(c => c.TargetVm == vm.Role);

            foreach (var c in contributorsForVm)
                c.Contribute(compose);

            var path = await _writer.WriteComposeAsync(compose, outputDir, fileName: $"compose.{vm.Role}.yaml", ct);
            if (path.IsFailure) return path.Map<IReadOnlyDictionary<string, string>>();
            result[vm.Name] = path.Value.FullPath;
        }

        return Result.Success<IReadOnlyDictionary<string, string>>(result);
    }
}

Single-VM topology has one VM, so all 14 contributors run for it and produce one compose file. Multi-VM topology has four VMs and produces four compose files. HA has more VMs and even more compose files. The contributor list does not change.

The wiring

The generator is consumed by the Generate stage. The Apply stage then walks each generated compose file and runs IContainerEngine.RunComposeAsync(file, projectName) on the corresponding VM, using the engine wrapper from Part 17.

The test

public sealed class DevLabComposeTests
{
    [Fact]
    public void single_topology_runs_all_contributors_for_one_vm()
    {
        var contributors = StandardContributors(); // all 14
        var generator = new PerVmComposeGenerator(contributors);
        var plan = SinglePlan();

        var result = generator.GenerateAsync(plan, TestOutputDir(), default).Result;

        result.IsSuccess.Should().BeTrue();
        result.Value.Should().ContainSingle();   // one VM, one compose file

        var compose = LoadCompose(result.Value.Single().Value);
        compose.Services.Should().HaveCount(14); // all services in one file
    }

    [Fact]
    public void multi_topology_partitions_contributors_across_four_vms()
    {
        var contributors = StandardContributors();
        var generator = new PerVmComposeGenerator(contributors);
        var plan = MultiPlan();

        var result = generator.GenerateAsync(plan, TestOutputDir(), default).Result;

        result.Value.Should().HaveCount(4);

        var gateway  = LoadCompose(result.Value["devlab-gateway"]);
        var platform = LoadCompose(result.Value["devlab-platform"]);
        var data     = LoadCompose(result.Value["devlab-data"]);
        var obs      = LoadCompose(result.Value["devlab-obs"]);

        gateway.Services.Should().Contain(s => s.Key == "traefik");
        gateway.Services.Should().Contain(s => s.Key == "pihole");
        platform.Services.Should().Contain(s => s.Key == "gitlab");
        platform.Services.Should().Contain(s => s.Key == "gitlab-runner");
        platform.Services.Should().Contain(s => s.Key == "baget");
        data.Services.Should().Contain(s => s.Key == "postgres");
        data.Services.Should().Contain(s => s.Key == "minio");
        obs.Services.Should().Contain(s => s.Key == "prometheus");
        obs.Services.Should().Contain(s => s.Key == "grafana");
    }

    [Fact]
    public void gitlab_contributor_emits_traefik_labels_with_correct_host()
    {
        var compose = new ComposeFile();
        var c = new GitLabComposeContributor(
            Options.Create(StandardConfig() with { Acme = new() { Tld = "lab" } }),
            Mock.Of<IPlacementResolver>());

        c.Contribute(compose);

        compose.Services["gitlab"].Labels!["traefik.http.routers.gitlab.rule"]
            .Should().Be("Host(`gitlab.lab`)");
    }
}

What this gives you that bash doesn't

A hand-written docker-compose.yaml for a real GitLab + runner + Postgres + MinIO + Traefik + observability + DNS stack is a thousand-line file with three indentation styles, two trailing whitespace bugs, and at least one volume mount that points at a directory that does not exist. Adding a new service is cat >> and a prayer. Removing a service is vim and a regret.

A typed IComposeFileContributor per service gives you, for the same surface area:

  • One contributor per service, ~30–60 lines each, independently testable
  • Topology-agnostic — the same contributor runs in single, multi, and HA
  • Per-VM file generation that partitions contributors by TargetVm
  • Typed Traefik labels instead of stringly-typed key-value pairs
  • Tests that lock per-service contributors and per-topology distribution

The bargain pays back the first time you add a new service to DevLab — you write one new contributor, register it, and the per-VM compose files update on the next homelab vos init.

Cross-links

⬇ Download