The Source Generator Pipeline and SchemaReader

Generator Entry Point

[Generator]
public sealed class GitLabCiBundleGenerator : IIncrementalGenerator
{
    public void Initialize(IncrementalGeneratorInitializationContext context)
    {
        var schemaFiles = context.AdditionalTextsProvider
            .Where(static f =>
                System.IO.Path.GetFileName(f.Path).StartsWith("gitlab-ci-") &&
                f.Path.EndsWith(".json"));

        context.RegisterSourceOutput(schemaFiles.Collect(), static (ctx, files) =>
        {
            if (files.IsDefaultOrEmpty || files.Length == 0)
                return;

            var ns = "FrenchExDev.Net.GitLab.Ci.Yaml";
            Generate(ctx, ns, files);
        });
    }
}

The generator watches for AdditionalFiles matching the pattern gitlab-ci-*.json. When these files change, the generator re-runs. This is the incremental aspect — changes to unrelated files don't trigger regeneration.

The trigger comes from the main library's .csproj:

<ItemGroup>
    <AdditionalFiles Include="schemas\gitlab-ci-*.json" />
    <EmbeddedResource Include="schemas\gitlab-ci-*.json" />
</ItemGroup>

And the descriptor class that activates the generator:

[GitLabCiBundle]
public partial class GitLabCiBundleDescriptor;

The Generation Pipeline

The Generate method orchestrates a five-stage pipeline:

Diagram — The Generate method runs a five-stage pipeline — parse each schema, merge into a UnifiedSchema, then emit models, fluent builders, and SinceVersion/UntilVersion metadata in turn.

Here's the actual implementation:

private static void Generate(SourceProductionContext ctx, string ns,
    ImmutableArray<AdditionalText> files)
{
    try
    {
        // Stage 1: Parse each schema file
        var schemas = new List<SchemaModel>();
        foreach (var file in files)
        {
            ctx.CancellationToken.ThrowIfCancellationRequested();
            var text = file.GetText(ctx.CancellationToken);
            if (text is null) continue;

            var version = SchemaReader.ExtractVersion(
                System.IO.Path.GetFileName(file.Path));
            var schema = SchemaReader.Parse(text.ToString(), version);
            schemas.Add(schema);
        }

        if (schemas.Count == 0) return;

        // Stage 2: Merge all versions
        var unified = SchemaVersionMerger.Merge(schemas);

        // Stage 5: Version metadata (emitted first for dependency reasons)
        ctx.AddSource("GitLabCiSchemaVersions.g.cs",
            SourceText.From(VersionMetadataEmitter.Emit(ns, unified), Encoding.UTF8));

        // Stage 3: Root model + builder
        ctx.AddSource("GitLabCiFile.g.cs",
            SourceText.From(ModelClassEmitter.EmitGitLabCiFile(ns, unified), Encoding.UTF8));

        var rootBuilderModel = BuilderHelper.CreateRootBuilderModel(
            ns, "GitLabCiFile", unified.RootProperties);
        ctx.AddSource("GitLabCiFileBuilder.g.cs",
            SourceText.From(BuilderEmitter.Emit(rootBuilderModel), Encoding.UTF8));

        // Stage 3+4: Definition models + builders + inline classes
        var emittedClasses = new HashSet<string>();
        emittedClasses.Add("GitLabCiFile");

        foreach (var item in ModelClassEmitter.EmitDefinitions(ns, unified))
        {
            ctx.CancellationToken.ThrowIfCancellationRequested();

            var className = item.FileName;
            if (className.EndsWith(".g.cs"))
                className = className.Substring(0, className.Length - ".g.cs".Length);

            // Skip duplicate class emissions
            if (!emittedClasses.Add(className)) continue;

            ctx.AddSource(item.FileName,
                SourceText.From(item.Source, Encoding.UTF8));

            var builderModel = FindBuilderModel(ns, className, unified);
            if (builderModel is not null)
            {
                ctx.AddSource($"{className}Builder.g.cs",
                    SourceText.From(BuilderEmitter.Emit(builderModel), Encoding.UTF8));
            }
        }
    }
    catch (System.Exception ex)
    {
        // Error recovery: emit a comment file so the build doesn't silently fail
        ctx.AddSource("GenerateError.g.cs",
            SourceText.From(
                $"// Generator error: {ex.GetType().Name}: {ex.Message}\n" +
                $"// {ex.StackTrace?.Replace("\n", "\n// ")}\n",
                Encoding.UTF8));
    }
}

Key design decisions:

Duplicate prevention — The emittedClasses set prevents the same class from being emitted twice. This can happen when allOf-resolved definitions produce the same inline class from multiple code paths.
Error recovery — If any exception occurs during generation, the generator emits a comment file with the error details rather than failing silently. This makes debugging much easier.
Cancellation support — The generator checks CancellationToken in each loop iteration, supporting IDE responsiveness.

Generated Output

The generator produces 61 files:

1 GitLabCiSchemaVersions.g.cs (version metadata + attributes)
1 GitLabCiFile.g.cs + 1 GitLabCiFileBuilder.g.cs (root model)
~29 definition model files (GitLabCiJob.g.cs, GitLabCiArtifacts.g.cs, ...)
~29 builder files (GitLabCiJobBuilder.g.cs, GitLabCiArtifactsBuilder.g.cs, ...)

All files are marked // <auto-generated/> and use #nullable enable.

Finding Builder Models

The generator needs to find the correct property list for each builder. This is non-trivial because properties can come from:

Top-level definitions
Inline objects nested inside properties
oneOf object variants

The FindBuilderModel method searches all three locations recursively:

private static BuilderEmitModel? FindBuilderModel(
    string ns, string className, UnifiedSchema schema)
{
    // Check top-level definitions
    foreach (var kvp in schema.Definitions)
    {
        var defClassName = NamingHelper.DefinitionToClassName(kvp.Key);
        if (defClassName == className)
            return BuilderHelper.CreateBuilderModel(
                ns, className, kvp.Value.Properties);
    }

    // Check inline objects (recursive)
    return FindInlineBuilderModel(
        ns, className, schema.Definitions, schema.RootProperties);
}

The recursive search handles deeply nested inline objects — for example, GitLabCiJobTemplateEnvironmentConfigKubernetesManagedResources is an inline object four levels deep.

SchemaReader: Parsing JSON Schema

The SchemaReader is the most complex component in the source generator. It transforms a JSON Schema document into a structured SchemaModel that the rest of the pipeline can work with.

The Parse Method

internal static class SchemaReader
{
    public static SchemaModel Parse(string json, string version)
    {
        using var doc = JsonDocument.Parse(json);
        var root = doc.RootElement;
        var model = new SchemaModel { Version = version };

        // Parse root properties (reserved keys: stages, variables, etc.)
        if (root.TryGetProperty("properties", out var rootProps))
            model.RootProperties = ParseProperties(rootProps, "GitLabCi");

        // Support both "definitions" (draft-07) and "$defs" (2020-12)
        JsonElement defs = default;
        var hasDefs = root.TryGetProperty("$defs", out defs) ||
                      root.TryGetProperty("definitions", out defs);

        if (hasDefs)
        {
            // First pass: parse all definitions
            foreach (var def in defs.EnumerateObject())
            {
                var defModel = ParseDefinition(def.Name, def.Value);
                if (defModel is not null)
                    model.Definitions[def.Name] = defModel;
            }

            // Second pass: resolve allOf references
            foreach (var def in defs.EnumerateObject())
            {
                if (def.Value.TryGetProperty("allOf", out var allOf))
                    ResolveAllOf(def.Name, allOf, model.Definitions);
            }
        }

        return model;
    }
}

The two-pass approach is necessary because allOf references can point forward — a definition might reference another definition that hasn't been parsed yet in a single-pass approach.

Version Extraction

public static string ExtractVersion(string filename)
{
    var name = Path.GetFileNameWithoutExtension(filename);
    var prefix = "gitlab-ci-v";
    if (name.StartsWith(prefix))
        return name.Substring(prefix.Length);
    return name;
}

Given filename gitlab-ci-v18.10.0.json, this returns "18.10.0".

Property Parsing

Each property in the schema can take many forms. The ParseProperty method handles them all:

private static PropertyModel ParseProperty(string name, JsonElement element,
    string parentClassName)
{
    var pm = new PropertyModel
    {
        JsonName = name,
        CSharpName = NamingHelper.ToPascalCase(name)
    };

    // Description (prefer "description", fall back to "markdownDescription")
    if (element.TryGetProperty("description", out var desc))
        pm.Description = desc.GetString();
    else if (element.TryGetProperty("markdownDescription", out var mdDesc))
        pm.Description = mdDesc.GetString();

    // Deprecated flag
    if (element.TryGetProperty("deprecated", out var dep) && dep.GetBoolean())
        pm.IsDeprecated = true;

    // $ref — direct reference to another definition
    if (element.TryGetProperty("$ref", out var refProp))
    {
        var refName = ExtractRef(refProp.GetString()!);
        pm.Ref = refName;
        pm.Type = MapRefToType(refName);
        return pm;
    }

    // oneOf — union type
    if (element.TryGetProperty("oneOf", out var oneOf))
    {
        ParseOneOf(pm, oneOf, parentClassName);
        return pm;
    }

    // anyOf — treated same as oneOf
    if (element.TryGetProperty("anyOf", out var anyOf))
    {
        ParseOneOf(pm, anyOf, parentClassName);
        return pm;
    }

    // type — direct type specification
    if (element.TryGetProperty("type", out var typeEl))
    {
        if (typeEl.ValueKind == JsonValueKind.Array)
        {
            // Multi-type: ["string", "null"] etc.
            var types = typeEl.EnumerateArray()
                .Select(t => t.GetString()!).ToList();
            pm.Type = MapMultiType(types);
        }
        else
        {
            var typeName = typeEl.GetString()!;

            // Inline object with properties → generate a class
            if (typeName == "object" &&
                element.TryGetProperty("properties", out var objProps))
            {
                var className = parentClassName + pm.CSharpName;
                pm.Type = PropertyType.InlineObject;
                pm.InlineClassName = className;
                pm.InlineObjectProperties = ParseProperties(objProps, className);
                return pm;
            }

            pm.Type = MapSingleType(typeName);

            // Array with items definition
            if (typeName == "array" &&
                element.TryGetProperty("items", out var items))
            {
                ParseArrayItems(pm, items, parentClassName);
            }
        }
    }

    // Enum values
    if (element.TryGetProperty("enum", out var enumEl))
    {
        pm.EnumValues = new List<string>();
        foreach (var e in enumEl.EnumerateArray())
            if (e.ValueKind == JsonValueKind.String)
                pm.EnumValues.Add(e.GetString()!);
    }

    return pm;
}

The priority order matters:

$ref is checked first — if present, the property is a reference to another definition
oneOf/anyOf — union types require special handling
type — direct type declaration, with special cases for inline objects and arrays
enum — recorded but doesn't change the type

The MapRefToType Switch

This is the core mapping that determines how 40+ type-alias definitions become C# types:

private static PropertyType MapRefToType(string refName)
{
    return refName switch
    {
        "string_or_list" or "stringOrList" => PropertyType.StringOrList,
        "string_file_list" => PropertyType.StringOrList,
        "script" or "optional_script" or "before_script" or "after_script"
            => PropertyType.StringOrList,
        "tags" or "filter_refs" => PropertyType.Array,
        "image" => PropertyType.String,
        "services" or "rules" or "includeRules" or "steps" => PropertyType.Array,
        "identity" or "when" or "workflowName" or "if" or "timeout" or "start_in"
            => PropertyType.String,
        "interruptible" => PropertyType.Boolean,
        "retry_max" => PropertyType.Integer,
        "retry" or "retry_errors" => PropertyType.Object,
        "configInputs" or "jobInputs" or "inputs" => PropertyType.Object,
        "globalVariables" or "jobVariables" or "rulesVariables" => PropertyType.Object,
        "id_tokens" or "secrets" => PropertyType.Object,
        "allow_failure" => PropertyType.Boolean,
        "cache" => PropertyType.Array,
        "filter" => PropertyType.Object,
        "parallel" or "parallel_matrix" => PropertyType.Object,
        "include_item" => PropertyType.StringOrObject,
        "rulesNeeds" => PropertyType.Array,
        "changes" or "exists" => PropertyType.StringOrList,
        "stepName" or "stepNamedStrings" or "stepNamedValues" => PropertyType.String,
        "step" or "stepGitReference" or "stepOciReference" or "stepFuncReference"
            => PropertyType.Object,
        "!reference" => PropertyType.Array,
        _ => PropertyType.Ref  // Not a type alias → real definition
    };
}

When the switch returns PropertyType.Ref, it means the definition is a real object with properties, and the generator will create a C# class for it. All other return values indicate type aliases that map to primitive C# types.

Schema Data Model

The intermediate representation used between parsing and emission:

internal sealed class SchemaModel
{
    public string Version { get; set; } = "";
    public Dictionary<string, DefinitionModel> Definitions { get; set; } = new();
    public List<PropertyModel> RootProperties { get; set; } = new();
}

internal sealed class DefinitionModel
{
    public string Name { get; set; } = "";
    public string? Description { get; set; }
    public List<PropertyModel> Properties { get; set; } = new();
    public bool IsNullableType { get; set; }
}

internal sealed class PropertyModel
{
    public string JsonName { get; set; } = "";
    public string CSharpName { get; set; } = "";
    public string? Description { get; set; }
    public bool IsDeprecated { get; set; }
    public PropertyType Type { get; set; } = PropertyType.String;
    public string? Ref { get; set; }
    public PropertyModel? Items { get; set; }
    public List<string>? EnumValues { get; set; }
    public bool IsRequired { get; set; }

    // For inline objects
    public string? InlineClassName { get; set; }
    public List<PropertyModel>? InlineObjectProperties { get; set; }

    // For oneOf/anyOf[string, object]
    public string? OneOfObjectClassName { get; set; }
    public List<PropertyModel>? OneOfObjectProperties { get; set; }
}

internal enum PropertyType
{
    String,
    Integer,
    Number,
    Boolean,
    StringOrBoolean,
    StringOrInteger,
    Array,
    Object,
    InlineObject,
    Ref,
    StringOrList,
    StringOrObject,
}

PropertyType to C# Type Mapping

PropertyType	C# Type	Example
`String`	`string?`	`image`, `when`, `timeout`
`Integer`	`int?`	`retry_max`
`Number`	`double?`	(rare)
`Boolean`	`bool?`	`interruptible`, `allow_failure`
`StringOrBoolean`	`bool?`	Union types losing string variant
`StringOrInteger`	`int?`	Union types losing string variant
`Array`	`List<object>?`	`stages`, `tags`, `services`
`Array` (with typed items)	`List<T>?`	`List<string>?` for scripts
`Object`	`Dictionary<string, object?>?`	`variables`, `secrets`
`InlineObject`	`ClassName?`	Nested object → generated class
`Ref`	`ClassName?`	Reference to definition → generated class
`StringOrList`	`List<string>?`	`script`, `before_script`
`StringOrObject`	`object?` or `ClassName?`	`include`, `environment`

OneOf Resolution: The Complete Logic

The ParseOneOf method handles all union type combinations:

private static void ParseOneOf(PropertyModel pm, JsonElement oneOf,
    string parentClassName)
{
    var items = oneOf.EnumerateArray().ToList();

    var hasString = false;
    var hasArray = false;
    var hasObject = false;
    var hasInteger = false;
    var hasBoolean = false;
    var hasNull = false;
    var hasRef = false;
    string? refName = null;
    JsonElement? objectElement = null;

    foreach (var item in items)
    {
        if (item.TryGetProperty("$ref", out var r))
        {
            hasRef = true;
            refName = ExtractRef(r.GetString()!);
        }
        if (item.TryGetProperty("type", out var typeProp) &&
            typeProp.ValueKind == JsonValueKind.String)
        {
            var t = typeProp.GetString();
            if (t == "string") hasString = true;
            else if (t == "array") hasArray = true;
            else if (t == "object") { hasObject = true; objectElement = item; }
            else if (t == "integer") hasInteger = true;
            else if (t == "boolean") hasBoolean = true;
            else if (t == "null") hasNull = true;
        }
    }

    // Priority 1: string + object with properties → inline config class
    if (hasString && hasObject && objectElement.HasValue &&
        objectElement.Value.TryGetProperty("properties", out var objProps))
    {
        var className = parentClassName + pm.CSharpName + "Config";
        pm.Type = PropertyType.StringOrObject;
        pm.OneOfObjectClassName = className;
        pm.OneOfObjectProperties = ParseProperties(objProps, className);
        return;
    }

    // Priority 2-4: primitive unions
    if (hasString && hasInteger) { pm.Type = PropertyType.StringOrInteger; return; }
    if (hasString && hasBoolean) { pm.Type = PropertyType.StringOrBoolean; return; }
    if (hasString && hasArray) { pm.Type = PropertyType.StringOrList; return; }

    // Priority 5: null + ref → nullable ref
    if (hasNull && hasRef && refName is not null)
    {
        pm.Ref = refName;
        pm.Type = MapRefToType(refName);
        return;
    }

    // Priority 6: refs only → use first ref
    if (hasRef && refName is not null)
    {
        var mappedType = MapRefToType(refName);
        pm.Ref = mappedType == PropertyType.Ref ? refName : null;
        pm.Type = mappedType;
        return;
    }

    // Priority 7: null + something → resolve the non-null type
    if (hasNull && items.Count == 2)
    {
        var nonNull = items.First(i =>
            !(i.TryGetProperty("type", out var t) &&
              t.ValueKind == JsonValueKind.String &&
              t.GetString() == "null"));
        if (nonNull.TryGetProperty("$ref", out var r2))
        {
            pm.Ref = ExtractRef(r2.GetString()!);
            pm.Type = PropertyType.Ref;
        }
        return;
    }

    // Fallback
    pm.Type = PropertyType.String;
}

Array Items Parsing

When a property is type: array, the items sub-schema determines the element type:

private static void ParseArrayItems(PropertyModel pm, JsonElement items,
    string parentClassName)
{
    // $ref items → List<ReferencedType>
    if (items.TryGetProperty("$ref", out var itemRef))
    {
        pm.Items = new PropertyModel
        {
            Ref = ExtractRef(itemRef.GetString()!),
            Type = PropertyType.Ref
        };
    }
    // Typed items
    else if (items.TryGetProperty("type", out var itemType))
    {
        if (itemType.ValueKind == JsonValueKind.Array)
        {
            var itemTypes = itemType.EnumerateArray()
                .Select(t => t.GetString()!).ToList();
            pm.Items = new PropertyModel { Type = MapMultiType(itemTypes) };
        }
        // Inline object items → List<InlineClass>
        else if (itemType.GetString() == "object" &&
                 items.TryGetProperty("properties", out var itemObjProps))
        {
            var itemClassName = parentClassName + pm.CSharpName + "Item";
            pm.Items = new PropertyModel
            {
                Type = PropertyType.InlineObject,
                InlineClassName = itemClassName,
                InlineObjectProperties = ParseProperties(itemObjProps, itemClassName)
            };
        }
        else
        {
            pm.Items = new PropertyModel
            {
                Type = MapSingleType(itemType.GetString()!)
            };
        }
    }
    // oneOf/anyOf items
    else if (items.TryGetProperty("oneOf", out var itemOneOf))
    {
        pm.Items = new PropertyModel { Type = PropertyType.String };
        ParseOneOf(pm.Items, itemOneOf, parentClassName + pm.CSharpName);
    }
    else if (items.TryGetProperty("anyOf", out var itemAnyOf))
    {
        pm.Items = new PropertyModel { Type = PropertyType.String };
        ParseOneOf(pm.Items, itemAnyOf, parentClassName + pm.CSharpName);
    }
}

This recursive approach handles deeply nested structures. For example, an array of objects where each object has a property that's itself an array of objects — the parser follows the nesting all the way down.

`[` or `Alt+S`	Focus sidebar navigation
`]` or `Alt+C`	Focus main content
`↑` `↓`	Navigate between sidebar items
`Enter`	Open page / toggle section
`Space`	Toggle section expand/collapse
`Escape`	Close overlay / sidebar

`Ctrl+K`	Open search
`?`	Show this help

`Ctrl+=` or `Ctrl+↑`	Increase font size
`Ctrl+−` or `Ctrl+↓`	Decrease font size
`f`	Open console font selector

`Ctrl+⇧+=` or `Ctrl+⇧+↑`	Browser zoom in
`Ctrl+⇧+−` or `Ctrl+⇧+↓`	Browser zoom out
`Ctrl+⇧+0`	Reset browser zoom

`Tab`	Focus a diagram or image
`Enter`	Open full size overlay
`+` `−`	Zoom in / out (in overlay)
`Escape`	Close overlay, return focus

The Source Generator Pipeline and SchemaReader📋

Generator Entry Point📋

The Generation Pipeline📋

Generated Output📋

Finding Builder Models📋

SchemaReader: Parsing JSON Schema📋

The Parse Method📋

Version Extraction📋

Property Parsing📋

The MapRefToType Switch📋

Schema Data Model📋

PropertyType to C# Type Mapping📋

OneOf Resolution: The Complete Logic📋

Array Items Parsing📋

SchemaVersionMerger: Multi-Version Intelligence📋