Skip to main content
Welcome. This site supports keyboard navigation and screen readers. Press ? at any time for keyboard shortcuts. Press [ to focus the sidebar, ] to focus the content. High-contrast themes are available via the toolbar.
serard@dev00:~/cv

Part III: Design Time -- Downloading gitlab.rb

Phase 1 of the pipeline runs once per GitLab release. It talks to the GitLab API, figures out which versions matter, downloads one file per version, and writes it to disk. The next time you build, the source generator picks it up automatically.

The GitLab Tags API

GitLab's repository hosts the Omnibus configuration templates in the omnibus-gitlab project. Each release is tagged with a version string like 18.10.1+ce.0 (Community Edition) or 18.10.1+ee.0 (Enterprise Edition). The design tool uses the GitLab Repository Tags API to enumerate all tags:

GET /api/v4/projects/gitlab-org%2Fomnibus-gitlab/repository/tags
    ?per_page=100
    &order_by=version
    &sort=desc

The response is paginated via Link headers. The collector follows rel="next" links until all tags are retrieved.

Version Filtering

Not all tags are relevant. The design tool applies two filters:

  1. Stable CE only: Keep tags ending with +ce.0, exclude anything containing rc (release candidates)
  2. Latest patch per minor: From 18.10.0, 18.10.1, 18.10.2, keep only 18.10.2 -- the latest patch carries all configuration from earlier patches
var pipeline = new DesignPipeline<string>()
    .UseHttpDownload(version =>
        $"https://gitlab.com/gitlab-org/omnibus-gitlab/-/raw/" +
        $"{version}%2Bce.0/files/gitlab-config-template/gitlab.rb.template")
    .UseSave()
    .Build();

return await new DesignPipelineRunner<string>
{
    ItemCollector = new GitLabTagsVersionCollector(
        "gitlab-org%2Fomnibus-gitlab",
        tagToVersion: tag =>
        {
            // Tags: "18.10.1+ce.0", "18.10.0+ee.0", "18.10.0+rc43.ce.0"
            // Keep only stable CE: ends with "+ce.0", no "rc"
            if (!tag.EndsWith("+ce.0") || tag.Contains("rc"))
                return null;
            return tag.Replace("+ce.0", ""); // "18.10.1"
        }),
    Pipeline = pipeline,
    KeySelector = v => v,
    ItemFilter = VersionFilters.LatestPatchPerMinor,
    OutputDir = outputDir,
    OutputFilePattern = "gitlab-{key}.rb",
    AuthTokenEnvVar = "GITLAB_TOKEN",
}.RunAsync(args);

The tagToVersion lambda converts raw tags to clean version strings: "18.10.1+ce.0" becomes "18.10.1". Tags that don't match the CE stable pattern return null and are discarded.

VersionFilters.LatestPatchPerMinor groups versions by (major, minor) and keeps only the highest patch. This is the same filter used by the Docker Compose Bundle for compose-spec releases and the Traefik Bundle for Traefik releases.

The Version Collector

GitLabTagsVersionCollector implements the shared IVersionCollector interface. It handles GitLab-specific concerns:

public sealed class GitLabTagsVersionCollector : IVersionCollector
{
    private readonly string _projectPath;
    private readonly HttpClient _httpClient;
    private readonly Func<string, string?> _tagToVersion;

    public async Task<IReadOnlyList<string>> CollectVersionsAsync(
        CancellationToken cancellationToken = default)
    {
        var versions = new List<string>();
        var url = $"{_baseUrl}/api/v4/projects/{_projectPath}" +
                  "/repository/tags?per_page=100&order_by=version&sort=desc";

        while (url is not null)
        {
            using var request = new HttpRequestMessage(HttpMethod.Get, url);
            using var response = await _httpClient.SendAsync(request, cancellationToken);
            response.EnsureSuccessStatusCode();

            var json = await response.Content.ReadAsStringAsync(cancellationToken);
            var items = JsonSerializer.Deserialize<JsonElement>(json);

            if (items.ValueKind == JsonValueKind.Array)
            {
                foreach (var item in items.EnumerateArray())
                {
                    var tag = item.GetProperty("name").GetString();
                    if (tag is null) continue;

                    var version = _tagToVersion(tag);
                    if (!string.IsNullOrEmpty(version))
                        versions.Add(version);
                }
            }

            url = ParseNextLink(response.Headers);
        }

        versions.Sort((a, b) => CompareVersions(a, b));
        return versions;
    }
}

Key implementation details:

  • Pagination: Parses the Link response header for rel="next" to follow paginated results
  • Authentication: Reads GITLAB_TOKEN from the environment for private access (optional for public projects, but useful for rate limit increases)
  • Semantic sorting: Sorts versions numerically (9.2.10 before 15.0.5 before 18.10.1), not lexicographically

The collector uses the Tags API (/repository/tags) rather than the Releases API because GitLab's Releases API only returns releases that have been explicitly created, while tags include all version markers.

Download URL Construction

Each version maps to a specific file path in the GitLab repository:

https://gitlab.com/gitlab-org/omnibus-gitlab/-/raw/
    {version}%2Bce.0/
    files/gitlab-config-template/gitlab.rb.template

The %2B is a URL-encoded +. So version 18.10.1 becomes the tag 18.10.1+ce.0, which URL-encodes to 18.10.1%2Bce.0. The file path within the repository is always files/gitlab-config-template/gitlab.rb.template.

The DesignPipeline handles the HTTP download with retry logic and writes the content to disk.

The Resource Files

After running the design tool, the resources/ directory contains one .rb file per version:

resources/
├── gitlab-9.2.10.rb       (68 KB,  ~2100 lines)
├── gitlab-15.0.5.rb       (82 KB,  ~2500 lines)
├── gitlab-15.1.7.rb       (82 KB,  ~2510 lines)
├── gitlab-15.2.5.rb       (83 KB,  ~2520 lines)
├── ...
├── gitlab-18.9.7.rb       (89 KB,  ~2700 lines)
├── gitlab-18.10.1.rb      (90 KB,  ~2720 lines)
└── (~80 files total, ~7 MB)

These files are committed to the repository and included as AdditionalFiles in the main library's .csproj:

<ItemGroup>
  <AdditionalFiles Include="..\..\..\resources\gitlab-*.rb" />
</ItemGroup>

The Source Generator discovers them via context.AdditionalTextsProvider, filters by filename pattern (gitlab-*.rb), and extracts the version from the filename:

var rbFiles = context.AdditionalTextsProvider
    .Where(static f =>
    {
        var name = System.IO.Path.GetFileName(f.Path);
        return name.StartsWith("gitlab-") && name.EndsWith(".rb");
    });

Version Extraction

The parser extracts the version string from the filename:

public static string ExtractVersion(string filename)
{
    // "gitlab-18.10.1.rb" → "18.10.1"
    var name = Path.GetFileNameWithoutExtension(filename);
    if (name.StartsWith("gitlab-"))
        return name.Substring("gitlab-".Length);
    return name;
}

This version string becomes the identity of each parsed model and drives the SinceVersion/UntilVersion metadata in the merged output.

Idempotency and Refresh

The design tool is idempotent. Running it again:

  1. Re-collects all tags from the API
  2. Filters to latest patch per minor
  3. Checks which files already exist in resources/
  4. Downloads only missing versions

When GitLab 18.11.0 ships, the tool adds one file: gitlab-18.11.0.rb. The next dotnet build picks it up automatically -- the Source Generator sees a new AdditionalFile and re-runs.

Existing files are never modified. If a .rb file needs to be updated (rare -- only if the template at a given tag changes after the fact), you delete the local file and re-run the tool.

What the Parser Sees

Here's a representative excerpt from gitlab-18.10.1.rb -- the input that Phase 2 must parse:

################################################################################
## GitLab NGINX
##! Docs: https://docs.gitlab.com/omnibus/settings/nginx.html
################################################################################

# nginx['enable'] = true
# nginx['client_max_body_size'] = '250m'
# nginx['redirect_http_to_https'] = false
# nginx['redirect_http_to_https_port'] = 80

##! Most root users won't need this setting.
# nginx['listen_port'] = nil

##! Most root users won't need this setting.
# nginx['listen_https'] = nil

# nginx['proxy_set_headers'] = {
#   "Host" => "$http_host_with_default",
#   "X-Real-IP" => "$remote_addr",
#   "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
#   "X-Forwarded-Proto" => "https",
#   "X-Forwarded-Ssl" => "on",
#   "Upgrade" => "$http_upgrade",
#   "Connection" => "$connection_upgrade"
# }

Commented-out Ruby. Bracket-notation keys. Multi-line hashes. Doc comments with ##!. Section headers with ###. This is what the parser in Part IV must handle -- and it does, across all 80 versions.

⬇ Download