✍️ Blog Post

How to Build a Multi-Agent Pipeline with OpenClaw

•7 min read

Most automation tools give you one agent doing one thing. OpenClaw gives you a fleet. Once you understand how to wire agents together into a pipeline — spawning subagents, passing state between them, handling failures gracefully — you stop thinking about individual tasks and start designing systems.

I've been running multi-agent pipelines on OpenClaw for several months now. This guide covers the patterns that actually work in production: how to spawn agents, how to coordinate their output, how to avoid the common failure modes, and how to build pipelines that self-correct instead of silently dying.

What Is a Multi-Agent Pipeline?

A multi-agent pipeline is a workflow where a parent (orchestrator) agent breaks a complex task into sub-tasks, delegates each to a child (worker) agent, collects results, and either synthesizes the output or routes it downstream.

In OpenClaw, this maps directly to the sessions_spawn tool. The orchestrator calls sessions_spawn to create isolated child sessions, each with its own task. Children report back through their session output, which the orchestrator reads via sessions_history or sessions_send.

The key insight: OpenClaw's session model is the pipeline primitive. You don't need external message brokers, shared queues, or custom IPC. Sessions are the unit of work, and the gateway manages lifecycle.

Step 1: Design Your Pipeline Shape

Before writing any config, decide your pipeline topology. There are three common shapes:

  • Fan-out / gather: One orchestrator spawns N parallel workers, waits for all, then synthesizes. Best for research, batch processing, parallel builds.
  • Chain: Worker A feeds output to Worker B feeds output to Worker C. Best for staged transformations — scrape, then analyze, then format.
  • Map-reduce: Fan-out with aggregation logic. Each worker processes one item from a list; orchestrator merges all results into a single artifact.

Most real pipelines are hybrids. A research pipeline might fan out to five scrapers (fan-out), then run each scraper result through a summarizer (chain per item), then merge all summaries (reduce).

Sketch your shape before coding. It determines how you handle state, timeouts, and failures.

Step 2: Write the Orchestrator Skill

The orchestrator is just an OpenClaw skill — a SKILL.md that describes what the agent does and how it coordinates children. Here's a minimal orchestrator template:

# SKILL.md — Research Orchestrator Purpose: Coordinate parallel web research across multiple topics. Spawn one worker per topic, collect summaries, write final brief. Inputs: - topics: list of research queries (passed in task message) - output_path: where to write the merged brief Flow: 1. Parse topics from task message 2. Spawn one subagent per topic via sessions_spawn with mode=run 3. Yield — wait for all workers to complete 4. Collect each worker output via sessions_history 5. Merge and write to output_path 6. Report done

Keep SKILL.md declarative — describe the intent, not the implementation. The agent figures out execution. The more concrete your SKILL.md, the more reliably the agent follows the pattern.

Step 3: Spawn Workers with sessions_spawn

Here's how the orchestrator spawns workers in practice. Each call to sessions_spawn is a non-blocking fire-and-forget when used with mode: "run". The parent does not wait for the child to finish before continuing.

The essential parameters:

  • mode: "run" — one-shot background execution. Worker runs, completes, terminates.
  • label — human-readable session name. Use it to find sessions later via sessions_list.
  • cleanup: "keep" — session persists after completion so you can read its history.
  • context: "fork" — passes current transcript context to the child. Use sparingly; it adds overhead. Only needed when the child requires prior conversation history.
  • runTimeoutSeconds — hard timeout for the worker. Essential for production pipelines.

Call sessions_yield after spawning all workers to hand control back to the gateway. The orchestrator will wake up when workers complete or when you send a follow-up message.

Step 4: Collect and Merge Results

After yielding, the orchestrator collects worker output via sessions_history. The pattern: filter sessions by label, read the last assistant message, treat that as the worker deliverable.

Collect step — orchestrator SKILL.md instructions: After yield: 1. Call sessions_list with label filter matching your worker prefix 2. For each session, call sessions_history with limit=5 to get last messages 3. Extract the final assistant message text as the summary 4. Check each result for error signals before treating as valid output 5. Concatenate valid summaries into a single document 6. Synthesize: write a merge prompt that covers all topics and highlights key findings 7. Write final output to target path using write tool

The merge step is where you add synthesis logic. Don't just concatenate raw outputs. Give the orchestrator a merge prompt: "Here are five research summaries. Write a single coherent brief that covers all topics, eliminates redundancy, and highlights the three most important findings."

Step 5: Handle Failures Without Breaking the Pipeline

Workers fail. Networks timeout. APIs rate-limit. A robust pipeline handles partial failure without bringing down the whole run.

Three patterns that work in production:

  • Timeout plus fallback: Set runTimeoutSeconds on each spawned worker. If a worker times out, log the miss and continue with remaining results.
  • Retry by re-spawn: After collecting, check which worker sessions have empty or error outputs. Re-spawn just those workers with the same task. One retry round handles most transient failures.
  • Partial synthesis: Design your merge prompt to work with incomplete inputs. "Given these N summaries (some may be missing), write the best brief you can" beats hard-failing when worker 3 of 7 errored.

The most common failure mode: a worker session completes but its final message is a tool error, not content output. Always inspect the last message for error signals before treating it as valid output. A simple check — does the message start with a capital letter and contain more than 50 characters? — filters out most tool error strings.

Step 6: Pass State Between Pipeline Stages

For chain pipelines where Stage A feeds Stage B feeds Stage C, you need to pass structured state between stages. File-based state works cleanly in OpenClaw:

## Stage A worker task message pattern: ## "Process the input. Write your JSON output to ## /tmp/pipeline-RUNID/stage-a-output.json using the write tool." ## Stage B worker task message pattern: ## "Read /tmp/pipeline-RUNID/stage-a-output.json. ## Analyze the data. Write your output to ## /tmp/pipeline-RUNID/stage-b-output.json."

Use a run-scoped directory with a unique ID such as a timestamp. Pass the path in the task message. This keeps stages decoupled — each stage reads its input path, writes its output path, and doesn't need to know about sibling workers.

For lightweight state (fewer than 10 fields), you can pass structured JSON directly in the task message string. Workers parse it from their initial message without needing filesystem reads.

Advanced Pattern: Self-Healing Pipelines

The most powerful pattern I've built: the orchestrator monitors its own worker pool and re-routes failed work automatically.

The implementation:

  1. Orchestrator writes a run manifest to a JSON file listing all workers and their expected output paths before spawning.
  2. After the gather step, orchestrator reads the manifest and checks which output files are present and non-empty.
  3. For missing outputs, orchestrator re-spawns the worker with an adjusted task — sometimes including a hint about what the previous attempt returned.
  4. After one retry round, orchestrator synthesizes whatever is available and notes missing items in the output report.

This handles 80% of real-world failure modes — transient API errors, rate limits, model timeouts — without human intervention. The remaining 20% (structural failures, missing credentials, broken tools) surfaces clearly in the output report so you can fix the root cause.

I've been running a competitive intelligence pipeline with this pattern for three months. Out of roughly 200 runs, fewer than five have required manual intervention. The rest resolved themselves through the retry layer.

FAQ

How many workers can I spawn in parallel? In practice, 5 to 10 concurrent workers is a reliable range. More than that and you may hit gateway request queuing. For large batches, fan-out in waves: spawn 8, yield, collect, spawn next 8.

Can workers spawn their own sub-workers? Yes — OpenClaw supports recursive agent spawning. Workers can call sessions_spawn themselves. Depth is governed by your gateway config. Use with care; deep trees are hard to debug and can run away if task decomposition loops.

How do I pass API credentials to workers? Workers inherit the gateway's configured credentials. You don't pass credentials in task messages. If a worker needs a specific key not in the gateway config, write it to a temp file and pass the file path in the task message.

What's the difference between mode: "run" and the default? mode: "run" is one-shot background execution — the session runs its task and terminates. Without it, the spawned session stays alive waiting for more input. Always use mode: "run" for pipeline workers.

How do I debug a worker that produced wrong output? Call sessions_history with includeTools: true. You'll see every tool call and its result. Compare the sequence to what you expected — that's almost always where the divergence is. The most common culprit is a tool that returned partial data that the agent treated as complete.

What to Build Next

Once you're comfortable with the basic pipeline pattern, the natural next step is adding observability — knowing what each worker did, how long it took, and where time was spent. The OpenClaw Agent Observability Guide covers exactly that, including how to build a lightweight dashboard that tracks worker completion rates across runs.

For more on structuring the skills that workers execute, the Custom Skills Developer Guide has the full template system including how to write skills that produce structured JSON outputs — critical for pipeline merge steps where the orchestrator needs to parse worker results programmatically.

And if you want a working production example, the Competitive Intelligence Automation guide walks through a real five-worker research pipeline end to end, including the manifest file structure and the merge prompt that consistently produces coherent briefs from partial data.

Multi-agent pipelines are where OpenClaw shifts from a productivity tool to infrastructure. Once you have the pattern down, the question changes from "how do I do this task" to "how many workers should I throw at it." That's the right question to be asking.