Shared State, Memory & Durable Execution

The last lesson was about what flows between agents. This one is about what they all read from and write to: shared state. As soon as more than one agent can change the same thing, you inherit problems that people who build distributed systems have dealt with for decades. These are clashing writes, stale reads and lost progress after a crash. The good news is that the same solutions work.

State is not the same as messages

Beginners picture a multi-agent system as agents chatting. Real systems model it as one typed state object. The steps, which are agents, tools and checkers, read it and update it. Messages are just one field of that state. The state holds the goal, the plan, the status of each task, the results so far, the budget used and some flags. This is the core idea of graph-based frameworks such as LangGraph. The workflow is a graph of steps over one shared, typed state, and every step returns a partial update.

from typing import TypedDict, Annotated
from operator import add

def merge_findings(a: dict, b: dict) -> dict:
    """Reducer: how concurrent updates to 'findings' combine."""
    return {**a, **b}                      # keyed by task_id, so parallel workers never collide

class RunState(TypedDict):
    goal: str                              # pinned, never overwritten
    plan: list[dict]
    findings: Annotated[dict, merge_findings]     # parallel-safe
    errors: Annotated[list[str], add]             # append-only
    tokens_used: Annotated[int, add]
    status: str                            # "planning" | "working" | "synthesizing" | "done"

# Each node returns only the keys it changes:
def worker_node(state: RunState) -> dict:
    result = do_work(state)
    return {"findings": {result.task_id: result.data}, "tokens_used": result.tokens}

Reducers: what a concurrent update means

Say three workers finish at once, and each returns an update to the same field. Something must say how those updates merge. Without a clear rule you get last-writer-wins, and results are lost without a sound. Choose a reducer for each field. Use append for logs and errors. Use merge-by-key for findings. Use sum for counters. Use max or min for scores. Use single-writer-only for fields like the goal, which nobody except the lead should overwrite.

The blackboard pattern

A blackboard is a shared workspace. Specialist agents add partial solutions to it. A controller decides who acts next, based on what is on the board right now. It suits problems where the path is not known in advance and different specialists can build on each other's progress, such as diagnosis, planning and investigation. The trade-off: the board becomes a shared context that everyone reads. So it needs the same trimming and hygiene as any context window.

Checkpoints and durable execution

A multi-agent run can last minutes or hours and cost real money. If the process dies at minute 40 and you start over, you have doubled the bill and annoyed the user. Save the state after every step. That is a checkpoint. Then a run can resume where it stopped, be inspected, be rewound to an earlier step, or pause for a person. Frameworks offer checkpointers. Workflow engines like Temporal offer durable execution: your orchestration code is replayed from a recorded history after a crash. Either way, the rule is the same. Long-running agent work must be resumable.

• Make every step safe to repeat, or guard side effects with an idempotency key. Resuming must never send the email twice.
• Checkpoint before actions that cannot be undone, and not only after.
• Store the version of the state schema, so old checkpoints can be migrated or rejected cleanly.
• Give each run a steady ID you can pass to support: show me run 8f3a.

Concurrency traps to design out

• Clashing writes: two workers edit the same file or record. Fix it with a single owner for each resource, or with version numbers (optimistic locking).
• Read-then-act races: an agent checks is this ticket unassigned, and acts after another agent has assigned it. Fix it with one atomic claim operation instead of check and then set.
• Repeated work: two workers pick up the same sub-task. Fix it with task claims, or a queue that delivers each task exactly once.
• Stale context: a worker reasons over a snapshot that is already out of date. Fix it by versioning snapshots and checking again before it commits a write.

Memory across runs

Keep run state, which is this job, apart from long-term memory, which is what the system has learned across jobs. Long-term memory tends to come in three kinds. Episodic memory is what happened in past runs. Semantic memory is facts about users, products and preferences. Procedural memory is how to do things, such as refined prompts and playbooks. Decide for each agent whether memory is private or shared, and who may write it. An agent that can write shared memory can poison every other agent that reads it. So treat memory writes like database writes: checked, credited to someone and reversible.