Objects, Streaming and Task Budgets📋

Pump-Me separates three boundaries: the native process, the provider's business data and the task's final answer. Each boundary has its own completeness and size concerns.

The seven-field envelope📋

Defaults are 20 items, 8192 UTF-8 bytes of compact JSON and 600 seconds for native execution. The public output budget accepts 1024–8192 bytes. The walkthrough selects 2048 bytes for its final task.

Success does not mean every preview record was retained. Truncation.Applied requires inspection. TotalCount=null means unknown, not zero; its scope also matters. At task level, projection counts describe the final projected items, while individual providers describe their own domain counts.

Useful PowerShell objects📋

A provider can construct a domain PSCustomObject, with named properties and an optional PSTypeName. This makes composition clear:

[pscustomobject]@{
    WorktreeCount = $worktrees.Data.WorktreeCount
    Branch = $status.Data.Branch
    TodoMatchingLines = $todos.Data.MatchLines
}

The full walkthrough guards the TODO count with Data.Complete. Properties make local code easier to understand; they do not establish static type safety. The engine's envelope construction applies bounding and sanitization. JSON transports values and properties; a JSON round trip does not retain PowerShell runtime type behavior.

The observation contract gives these properties their meaning. A field named Count is insufficient unless its unit, scope and completeness are defined. Provider fixtures verify that meaning, including null or incomplete results; an optional PSTypeName supplies a PowerShell type name, without enforcing a schema.

Local object composition means calculating the final answer before serialization. It does not mean sending a worktree object's formatted text to another native executable through standard input. Registered object transforms use the separate Pump-Me -PumpId pipeline interface, with recognition and transform callbacks.

For native options, use -NativeArguments. A wrapper parameter named -Filter is a PowerShell scriptblock filter; a CLI's --filter belongs in the native argument list. -Filter receives the provider's .Data and must return one boolean. -Map receives that data and returns a projection. They act after the provider returns its envelope and before the wrapper's final sanitization and bounding pass; they cannot recover records already omitted by the provider.

Use -Filter and -Map for suitable local selection and transformation of those retained results. When a task needs an exact count across the entire native stream, put that reduction in the provider's parser. Choosing the shape of .Data can require streaming parsing and domain logic before bounding; the contract is implemented at that level.

Streaming is a bounded reduction📋

Invoke-OutputPumpNative supports State, OnRecord and final Parse:

1. The process produces records from stdout and stderr.
2. OnRecord(record,state) updates counters and a bounded preview. It emits no objects.
3. The engine waits for process completion and stream EOF.
4. Parse(state,capture) finalizes domain data with exit, timeout and record-loss information.
5. The envelope applies item, field, secret-redaction and JSON limits.

Keep reducer state bounded independently of the output envelope. Keeping every record in State would consume memory even if the final JSON were tiny.

The worktree reducer retains one current record, a count and at most MaxItems completed records. Ripgrep's reducer counts matching-line events and retains selected lines. A line with two TODO occurrences counts as one matching line.

A preview limit bounds retained examples. A JSON limit bounds transport. Neither should stop an otherwise valid streaming analysis. An oversized record can prevent analysis of that record; the engine reports RecordLimit, failure and an unknown total rather than treating the clipped record as complete.

A timeout is different: it limits native execution and terminates the process tree when exceeded. Failure to finish cannot certify a complete analysis. No timeout or truncation triggers an automatic relaunch.

Exact counts with omitted previews📋

The executed ripgrep fixture has three matching lines and a one-line preview. Data.Complete=true and Data.MatchLines=3 remain useful while truncation explains the omitted lines. See streaming-preview.json.

If a path cannot be read, the exact count is null and completion is false. The observed count can describe processed records, but does not prove the total. Parser failure similarly cannot certify completion. Malformed-stream evidence demonstrates that distinction.

This is why an agent should inspect the reason for truncation. Narrow the search when it needs omitted content or complete analysis is unavailable. It need not rerun merely to recover examples when an already complete count answers the question.

One task envelope📋

Invoke-OutputPumpTask consumes emitted intermediate envelopes instead of dumping them into the final projection. Invoke-OutputPumpStep records a result even when the task assigns it to a local variable. The task deduplicates repeated emission of the same envelope instance.

The generated caller invokes the aggregator. The task body returns ordinary domain objects to project into Data.Items. It must not serialize each intermediate result itself: a JSON string is ordinary data, not a recognizable pump envelope.

The task prioritizes status and issues before the projection. A large projection can disappear; failure counters and the aggregate outcome remain. The issue list retains at most three entries before further byte budgeting, and prioritizes failures over successful preview omissions. OmittedIssues makes the loss explicit.

Timeout takes precedence in the aggregate outcome, followed by failures/script errors, then unsupported results. A successful outcome can still carry truncation.

The 1 KiB evidence records twenty steps, nineteen successful truncated previews followed by a failure, while preserving failure status and explicit omissions. Assigned failures, timeouts and duplicate emission have separate fixtures in Sources and Verification.

Previous: Building a Reusable Provider · Next: Existing Pumps in Practice