DistributedTask.Dsl: Saga-Driven Distributed Processing from C# Attributes📋

The Premise📋

You describe what your distributed task does. The compiler produces how it orchestrates, retries, compensates, and reports.

DistributedTask.Dsl is an attribute-based DSL that decorates C# classes to define saga-patterned distributed tasks. A Roslyn Source Generator reads the attributes at compile time and emits production-ready saga orchestrators, typed step contexts, queue consumers, API controllers, SignalR hubs, retry policies, and DI registration — with zero runtime reflection.

This series builds two complete distributed processing pipelines — a file-to-ZIP archiver and a video transcoding pipeline — one concept at a time, showing every attribute and every line of generated code.

Table of Contents📋

Part I: The Problem📋

The boilerplate behind distributed file processing. Upload, queue, process, compensate, track progress, handle failures — and do it all again for every new task. The case for a declarative alternative.

Part II: The DSL📋

The attribute surface: [DistributedTask], [SagaStep], [FileUploadStep], [CustomStep], [StepPlacement], [ParallelStepGroup], [StepCondition], [RetryPolicy], [Cancellable]. How each attribute maps to a [MetaConcept] in the M3 meta-metamodel. The ListeningStrategy enum — client-chosen notifications. Typed step data ValueObjects instead of JSON bags.

Part III: The ZIP Example📋

A complete walk-through of the file-to-ZIP use case. ~35 lines of attributes, ~20 lines of developer overrides. The [StepPlacement] split between API and Worker. The orchestrator kernel shared by both sides. Sequence and state diagrams.

Part IV: The Video Pipeline📋

A more complex example: video transcoding with fan-out/fan-in. [ParallelStepGroup] for concurrent transcoding. [StepCondition] for conditional 4K encoding. Differentiated retry policies. Partial compensation of completed parallel steps.

Part V: The Data Model📋

The DistributedTaskInstance aggregate root modeled with Entity.Dsl. SagaStepRecord, StepConditionSnapshot, TaskAuditEntry. Generated EF Core configurations, repositories, unit-of-work. Automatic archival rotation.

Part VI: Generated Code📋

Every generated artifact: the orchestrator kernel, API and Worker orchestrators, typed step data ValueObjects, worker messages, queue consumers, API controllers, domain events, Polly policies, SignalR hubs, webhook notifiers, reconciliation services, archive services, and DI registration. The Generation Gap table — who writes what.

Part VII: Modes & Infrastructure📋

InProcess mode (BackgroundService + Channel<T>) for development. Distributed mode (real queues) for production. The abstraction layer: IS3Client, IQueuePublisher, ITaskProgressTracker, IDistributedLock, ISagaStateSerializer.

Part VIII: Resilience📋

Polly integration per step via [RetryPolicy]. Idempotency keys and deduplication. Redis distributed locking with RedLock. Cancellation with compensation. Compensation retry and failure recording with trace IDs. S3 orphan reconciliation.

Part IX: Observability📋

OpenTelemetry spans per step. Trace context propagation through queue messages. Structured logging. Metrics: step duration, failure rate, queue depth. Generated health checks.

Part X: Listening Strategy📋

The client picks how it receives progress updates — Polling, SSE, WebSocket, SignalR, or Webhook — in the submit request. The system generates all five transports, all powered by a single UpdateAsync call through Redis pub/sub. Polling is always available as fallback.

Part XI: Testing📋

Testing sagas as callback orchestrators. In-memory implementations for S3, queues, and distributed locks. Compensation chain testing. Retry simulation. Concurrent worker testing. Snapshot testing of generated code.

Part XII: Workflow Integration📋

The Workflow DSL extracted from Diem as a standalone DSL. Bidirectional composition: a workflow transition triggers a distributed task; a completed task triggers a workflow transition.

Part XIII: Toward BPM📋

The DistributedTask DSL as a building block for Business Process Modeling. Tracked state, condition snapshots for audit, saga chaining with [DependsOn], shared process variables. A teaser for the full BPM article.

Part XIV: Deployment📋

Graceful worker shutdown. Rolling updates and saga versioning. Horizontal scaling with distributed locks.

Part XV: Comparison📋

DistributedTask.Dsl vs MassTransit sagas, Temporal, Durable Functions, Hangfire, and Camunda/Zeebe. The amplification ratio: ~55 lines of developer code → ~700+ lines of generated infrastructure. Compile-time diagnostics table.

Part XVI: Security📋

The IDistributedTaskAuthorizationHandler interface. Generated controller guards. The [RequiresRole] shortcut.