A2A Task Idempotency and Recovery: Duplicates, Dropped Streams, and Stalled Tasks

A2A Task Idempotency and Recovery: Duplicates, Dropped Streams, and Stalled Tasks

Networks drop, clients retry, processes restart. The protocol does not absorb these failures for you: it provides messageId, contextId, and two recovery channels, and leaves exactly-once effects to the implementation. This guide shows the patterns that put those three to work, plus a reference endpoint and probes that verify them by counting business executions — the results printed in section 4 come from actually running them.

State and operation semantics are on the Task object page and in the Task runtime guide ; the release checklist for restart and retries is section 5 of the production readiness guide .

Recovery decision tree

After a dropped connection or a response timeout, the client first determines where the task went, then acts:

stream dropped / response timeout
  ├─ already terminal (COMPLETED / FAILED / CANCELED / REJECTED)?
  │    └─ yes: GetTask for the terminal state and artifacts, done
  ├─ subscription available (capabilities.streaming)?
  │    └─ yes: SubscribeToTask and continue from the event stream
  │         └─ when the server cannot resume from an arbitrary point,
  │            trust the events received and deduplicate repeats by
  │            an implementation-provided stable event ID, then reconcile with GetTask
  ├─ polling only?
  │    └─ GetTask with backoff until terminal or stalled
  └─ stalled: still SUBMITTED / WORKING past the business cap
       ├─ client: CancelTask, or resend the request in a new task (same contextId)
       └─ server: move the task to FAILED with a stated reason, never WORKING forever

Do not read “the stream ended” as “the task ended”: a dropped connection only means you can no longer see it; the task may still be progressing server-side.

1. Duplicate messages: this implementation uses messageId as an idempotency key

This implementation adopts server-side message deduplication. The sender generates messageId and preserves it on retries; the server returns the registered outcome within its configured scope and retention window. The specification makes this optional — “Send Message operations MAY be idempotent. Agents may utilize the messageId to detect duplicate messages” (section 3.3.1) — so a retry against a server without deduplication executes the work twice; check the server’s policy before relying on it. Get operations are naturally idempotent and Cancel Task is idempotent by the same section; Send is the one that needs a decision.

ScenarioClient behaviorServer policy in this implementation
Resending the same message after a timeoutResend verbatim; messageId unchangedRecognize the duplicate; return the first processed result
Continuing in the same conversation (same task)New messageId + taskId (optionally the matching contextId)Advance the task with the new input
Follow-up after a task endedSame contextId, new message without taskIdCreate a new task (terminal tasks cannot restart)
The task a retry targets is already terminalDo not resend with that taskId; redo on a new task in the same contextId with a new messageIdUnsupportedOperationError: terminal tasks accept no further messages (section 3.1.1) — the verbatim-retry replay above applies only to messages that do not carry a terminal taskId

Implementation notes for this design:

  1. This server uses (taskId or contextId, messageId) as the idempotency key, storing a pointer to the first outcome (task ID, response, or error);
  2. The idempotency window spans the whole period “before the client gives up”, not a few seconds;
  3. Business side effects (external calls, database writes) happen after the key is registered. That order prevents a second execution when the response was lost but the work happened; it cannot prevent the mirror image — a crash between registration and the side effect leaves a key with no outcome. The sweep in section 3 fails such work explicitly and the redo moves to a new messageId; exactly-once across processes would require transactional storage. Section 4 runs this gap for real;
  4. A retry only hits the key if it lands in the same scope. Follow-ups carry the taskId by construction; a first message should carry a contextId from the start (or the server opens a messageId-only window for scopeless firsts), otherwise each retry mints a new context and executes as new work. The reference endpoint keys scopeless first messages by messageId alone;
  5. For idempotency across messages, use contextId: new tasks in the same context can carry an upstream business key whose dedup scope your implementation defines.

The protocol does not guarantee exactly-once at the business level — referenceTaskIds and contextId carry correlation only; dedup policy is yours. Section 5 of the production readiness guide lists the same boundary.

2. Duplicate delivery: three dedup points on the receiving side

Three places on the recovery path can see “the same event a second time”:

LocationDedup key
Streamed events (after reconnect)Implementation-provided stable event IDs; no universal sequence is defined
Push notificationsImplementation-defined notification IDs; Task ID + state alone can merge distinct updates (section 4 of Push receiver hardening )
Artifact chunksImplementation-provided chunk IDs; artifactId, append, and lastChunk alone cannot detect duplicates

append and lastChunk describe aggregation and completion, not chunk identity. Repeated chunks require implementation-defined deduplication; when identity is unavailable, reconcile with a GetTask snapshot.

3. Restart recovery: give accepted tasks an outcome

After a server restart, accepted tasks have only two acceptable outcomes: continue per the product contract, or fail with an explicit result. The pattern:

startup -> load persisted task records
  ├─ task with a registered idempotency key -> replay the recorded result, never re-run side effects
  ├─ task already executing -> resume, or move to FAILED (reason: "server restart")
  └─ unrecoverable -> FAILED with a readable reason; never parked back in WORKING
       waiting for an executor that will not come

Write “where tasks go after a restart” into the product contract so clients can decide between retrying and giving up; this is item 3 of the checklist in section 5 of the production readiness guide . Phase 2 of the run in section 4 shows the sweep behaving this way: after a crash between registration and the side effect, the accepted task is FAILED on restart, the ledger still holds nothing for that key, and the redo executes once on a new key.

4. A reference endpoint and the probes, run

Equal-looking responses are exactly what a duplicated side effect hides behind, so these probes do not compare responses: they count business executions in an append-only ledger and reconcile the recorded outcomes. The endpoint under test is a single-file A2A JSON-RPC server implementing the policy of section 1 and the recovery sweep of section 3.

The short path through this section: save the two files below — a2a-idempotency-reference.py and check-idempotency-recovery.sh — into one directory, and from that same directory run bash check-idempotency-recovery.sh . work with python3, curl, and jq available; read the PASS/FAIL lines down to the final summary:. The assertions count executions in work/ledger.jsonl; a FAIL line names the exact expectation it broke. Save the server first:

#!/usr/bin/env python3
"""Reference A2A endpoint for idempotency and recovery probes (v1.0.1, JSON-RPC binding).

A single-file teaching server, not a production agent. It implements the policy of
the guide: (taskId or contextId, messageId) as the idempotency key, business side
effects recorded to an append-only ledger, task records persisted to a state file,
and a startup sweep that gives interrupted tasks a definite FAILED outcome.
Accepted work executes on its own thread from the moment the request is
registered; SSE handlers only observe and relay, so a dropped connection can
neither cancel nor skip the work.

Implemented methods (POST /, JSON-RPC 2.0):
  SendMessage (blocking), SendStreamingMessage (SSE), SubscribeToTask (SSE),
  GetTask. CancelTask and push notifications are not implemented;
  configuration/metadata are accepted and ignored.

Message text drives the scenarios (the business is a word counter):
  "please fail"  -> the business operation fails; the task ends FAILED
  "please crash" -> the key and task are persisted, then the process exits
                    before the business runs (the crash gap)
  "slowly"       -> the task stays WORKING for ~4 s before completing

Environment switches for the negative controls:
  A2A_DEDUP=off    skip the key registry: duplicates execute twice
  A2A_PERSIST=off  keep state in memory only: a restart forgets keys and tasks

Usage: a2a-idempotency-reference.py --port 9999 --state state.json --ledger ledger.jsonl
"""
from __future__ import annotations

import argparse
import copy
import datetime
import json
import os
import sys
import threading
import time
import uuid
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

LOCK = threading.RLock()
STATE = {"tasks": {}, "keys": {}}          # persisted shape
DONE: dict[str, threading.Event] = {}      # in-memory terminal events, not persisted
STATE_PATH = None
LEDGER_PATH = None
PERSIST = os.environ.get("A2A_PERSIST") != "off"
DEDUP = os.environ.get("A2A_DEDUP") != "off"

TERMINAL = ("TASK_STATE_COMPLETED", "TASK_STATE_FAILED", "TASK_STATE_CANCELED", "TASK_STATE_REJECTED")


def now() -> str:
    return datetime.datetime.now(datetime.timezone.utc).isoformat()


def log(message: str) -> None:
    print(f"[{datetime.datetime.now().strftime('%H:%M:%S')}] {message}", file=sys.stderr, flush=True)


# ---------------------------------------------------------------- state and ledger

def load() -> None:
    if PERSIST and os.path.exists(STATE_PATH):
        with open(STATE_PATH, encoding="utf-8") as fh:
            loaded = json.load(fh)
        STATE["tasks"] = loaded.get("tasks", {})
        STATE["keys"] = loaded.get("keys", {})
    # Startup sweep: every accepted task gets an outcome. In-flight tasks cannot
    # resume (the executor is gone), so they move to FAILED with a readable reason.
    for task in STATE["tasks"].values():
        if task["status"]["state"] not in TERMINAL:
            task["status"] = failed_status("interrupted by server restart; resend with a new messageId to redo the work")
    save()
    for task in STATE["tasks"].values():
        DONE[task["id"]] = threading.Event()
        DONE[task["id"]].set()


def save() -> None:
    if not PERSIST:
        return
    tmp = f"{STATE_PATH}.tmp"
    with open(tmp, "w", encoding="utf-8") as fh:
        json.dump({"tasks": STATE["tasks"], "keys": STATE["keys"]}, fh)
    os.replace(tmp, STATE_PATH)  # atomic on the same filesystem


def ledger(key: str, task_id: str, outcome: str, words: int | None) -> None:
    """Record one business execution. This file is what the probes assert against."""
    line = {"ts": now(), "key": key, "taskId": task_id, "outcome": outcome}
    if words is not None:
        line["words"] = words
    with open(LEDGER_PATH, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(line) + "\n")
        fh.flush()


# ---------------------------------------------------------------- task helpers

def failed_status(reason: str) -> dict:
    return {
        "state": "TASK_STATE_FAILED",
        "timestamp": now(),
        "message": {
            "messageId": f"recovery-{uuid.uuid4()}",
            "role": "ROLE_AGENT",
            "parts": [{"text": reason}],
        },
    }


def snapshot(task: dict) -> dict:
    """The wire shape of a task, as a deep copy: internal bookkeeping fields never
    leave the server, and later mutations never leak into an earlier snapshot."""
    return copy.deepcopy({k: v for k, v in task.items() if k != "input"})


# ---------------------------------------------------------------- business operation

def run_business(text: str) -> tuple[str, int | None, str | None]:
    """Returns (outcome, words, artifact_text). The only place side effects happen."""
    if "please fail" in text:
        return "failed", None, None
    words = len(text.split(":", 1)[1].split()) if ":" in text else len(text.split())
    return "completed", words, str(words)


# ---------------------------------------------------------------- JSON-RPC plumbing

def rpc_error(code: int, message: str, reason: str, **metadata) -> dict:
    return {
        "jsonrpc": "2.0",
        "id": None,
        "error": {
            "code": code,
            "message": message,
            "data": [{
                "@type": "type.googleapis.com/google.rpc.ErrorInfo",
                "reason": reason,
                "domain": "a2a-protocol.org",
                "metadata": {k: str(v) for k, v in metadata.items()},
            }],
        },
    }


# ---------------------------------------------------------------- send path

def prepare(params: dict):
    """Resolve, replay, or register one idempotency key. No side effects happen here.

    Returns (task, key, text, mode, error):
      mode "terminal"  task is done (replay or instant result); task is the answer
      mode "join"      a duplicate of work still in flight; wait on DONE[task.id]
      mode "run"       registered; the caller dispatches the business to its own
                       thread (blocking callers then wait on DONE; streaming
                       callers send the first event while it runs)
      mode "slow"      registered; as "run" but the work takes ~4 s
    """
    msg = params.get("message") or {}
    message_id = msg.get("messageId") or ""
    if not message_id:
        return None, None, None, None, rpc_error(-32602, "messageId is required", "INVALID_PARAMS")
    text = " ".join(p.get("text", "") for p in msg.get("parts", []) if isinstance(p, dict))

    with LOCK:
        task = None
        if msg.get("taskId"):
            task = STATE["tasks"].get(msg["taskId"])
            if task is None:
                return None, None, None, None, rpc_error(
                    -32001, "Task not found", "TASK_NOT_FOUND", taskId=msg["taskId"])
            if task["status"]["state"] in TERMINAL:
                # Messages cannot be accepted for terminal tasks (spec 3.1.1).
                return None, None, None, None, rpc_error(
                    -32004, "Task is in a terminal state", "UNSUPPORTED_OPERATION",
                    taskId=task["id"], state=task["status"]["state"])
            if msg.get("contextId") and msg["contextId"] != task["contextId"]:
                return None, None, None, None, rpc_error(
                    -32602, "contextId does not match the task's context", "INVALID_PARAMS")

        # Key scope: the task for follow-ups, the context for new work, and the
        # messageId alone for a first message that carries neither.
        scope = msg.get("taskId") or msg.get("contextId") or "*first-message*"
        key = f"{scope}/{message_id}"

        if DEDUP and key in STATE["keys"]:
            replay = STATE["tasks"][STATE["keys"][key]]
            if replay["status"]["state"] in TERMINAL:
                return snapshot(replay), key, text, "terminal", None
            return snapshot(replay), key, text, "join", None

        # Register the key and persist BEFORE any side effect. A crash after this
        # point leaves accepted work with an outcome of nothing yet — the startup
        # sweep turns that into an explicit FAILED, never into silent re-execution.
        context_id = msg.get("contextId") or (task["contextId"] if task else f"ctx-{uuid.uuid4().hex[:10]}")
        if task is None:
            task = {
                "id": f"task-{uuid.uuid4().hex[:12]}",
                "contextId": context_id,
                "input": text,
                "artifacts": [],
            }
            STATE["tasks"][task["id"]] = task
            DONE[task["id"]] = threading.Event()
        if DEDUP:
            STATE["keys"][key] = task["id"]
        task["status"] = {"state": "TASK_STATE_WORKING", "timestamp": now()}
        save()

        if "please crash" in text:
            log(f"crash injection: key {key} persisted, exiting before the business runs")
            os._exit(70)

        return snapshot(task), key, text, ("slow" if "slowly" in text else "run"), None


def complete(task_id: str, key: str, text: str) -> dict:
    """Run the business for a prepared task and return the final snapshot."""
    if "slowly" in text:
        time.sleep(4)  # outside the lock: slow work must not block other requests
    with LOCK:
        task = STATE["tasks"][task_id]
        outcome, words, artifact_text = run_business(text)
        ledger(key, task["id"], outcome, words)
        if outcome == "completed":
            task["artifacts"].append({
                "artifactId": f"artifact-{uuid.uuid4().hex[:10]}",
                "name": "word count",
                "parts": [{"text": artifact_text}],
            })
            task["status"] = {"state": "TASK_STATE_COMPLETED", "timestamp": now()}
        else:
            task["status"] = failed_status("business error (injected by 'please fail')")
        DONE[task["id"]].set()
        save()
        return snapshot(task)


def join(task_id: str) -> dict:
    """A duplicate of in-flight work: wait for the outcome, return the same task."""
    DONE[task_id].wait(timeout=30)
    with LOCK:
        return snapshot(STATE["tasks"][task_id])


def dispatch(task_id: str, key: str, text: str) -> None:
    """Start the business for a prepared task on its own thread.

    Execution begins when the request is accepted, not when a connection delivers
    an event: a stream cut before the first event still executes the work exactly
    once, and blocking callers simply wait on the same DONE event.
    """
    def worker() -> None:
        try:
            complete(task_id, key, text)
        except Exception as exc:  # an accepted task must still reach a terminal state
            with LOCK:
                task = STATE["tasks"].get(task_id)
                if task is not None and task["status"]["state"] not in TERMINAL:
                    task["status"] = failed_status(f"executor error: {exc}")
                    DONE[task_id].set()
                    save()
    threading.Thread(target=worker, daemon=True).start()


# ---------------------------------------------------------------- HTTP layer

class Handler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def log_message(self, fmt, *args):  # quiet; the ledger is the record that matters
        pass

    def do_POST(self) -> None:
        length = int(self.headers.get("Content-Length") or 0)
        request = json.loads(self.rfile.read(length) or b"{}")
        method = request.get("method", "")
        rid = request.get("id")
        params = request.get("params") or {}

        def respond(payload: dict) -> None:
            payload["id"] = rid
            body = json.dumps(payload).encode()
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)

        def start_stream() -> None:
            """Open the SSE response; the connection closes when the handler returns."""
            self.send_response(200)
            self.send_header("Content-Type", "text/event-stream")
            self.send_header("Cache-Control", "no-cache")
            self.send_header("Connection", "close")
            self.close_connection = True
            self.end_headers()

        def write_event(payload: dict) -> None:
            """One SSE event. A broken pipe means the client cut the stream."""
            self.wfile.write(f"data: {json.dumps({'jsonrpc': '2.0', 'id': rid, 'result': payload})}\n\n".encode())
            self.wfile.flush()

        def updates_of(task_id: str) -> list[dict]:
            """The artifact and status events of a finished task, in order."""
            with LOCK:
                task = STATE["tasks"][task_id]
                rest = [{"artifactUpdate": {
                    "taskId": task["id"], "contextId": task["contextId"], "artifact": artifact}}
                    for artifact in task["artifacts"]]
                rest.append({"statusUpdate": {
                    "taskId": task["id"], "contextId": task["contextId"], "status": task["status"]}})
            return rest

        def stream(first: dict | None, rest: list[dict]) -> None:
            try:
                start_stream()
                if first is not None:
                    write_event(first)
                for payload in rest:
                    write_event(payload)
            except (BrokenPipeError, ConnectionResetError):
                log("stream cut by the client; the task keeps running server-side")

        def stream_live(first: dict, task_id: str) -> None:
            """Send the first event now; terminal updates follow when they happen."""
            try:
                start_stream()
                write_event(first)
                DONE[task_id].wait(timeout=30)
                for payload in updates_of(task_id):
                    write_event(payload)
            except (BrokenPipeError, ConnectionResetError):
                log("stream cut by the client; the task keeps running server-side")

        if method in ("SendMessage", "SendStreamingMessage"):
            first, key, text, mode, error = prepare(params)
            if error is not None:
                respond(error)
            elif method == "SendMessage":
                # SendMessageResponse wraps the payload: result = {"task": ...}
                # (GetTask returns the Task itself; proto v1.0.1).
                if mode == "terminal":
                    respond({"jsonrpc": "2.0", "result": {"task": first}})
                elif mode == "join":
                    respond({"jsonrpc": "2.0", "result": {"task": join(first["id"])}})
                else:
                    dispatch(first["id"], key, text)  # execution does not need this connection
                    respond({"jsonrpc": "2.0", "result": {"task": join(first["id"])}})
            else:  # SendStreamingMessage: the task event goes out first, updates follow
                if mode == "terminal":
                    stream({"task": first}, [])
                else:
                    if mode != "join":
                        dispatch(first["id"], key, text)  # before any connection I/O
                    stream_live({"task": first}, first["id"])
            return

        if method == "GetTask":
            with LOCK:
                task = STATE["tasks"].get(params.get("id") or "")
                if task is None:
                    respond(rpc_error(-32001, "Task not found", "TASK_NOT_FOUND", taskId=params.get("id")))
                else:
                    respond({"jsonrpc": "2.0", "result": snapshot(task)})
            return

        if method == "SubscribeToTask":
            with LOCK:
                task = STATE["tasks"].get(params.get("id") or "")
                if task is None:
                    respond(rpc_error(-32001, "Task not found", "TASK_NOT_FOUND", taskId=params.get("id")))
                    return
                if task["status"]["state"] in TERMINAL:
                    respond(rpc_error(-32004, "Task is in a terminal state", "UNSUPPORTED_OPERATION",
                                      taskId=task["id"], state=task["status"]["state"]))
                    return
                current = snapshot(task)  # deep copy under the lock: one consistent moment
                task_id = task["id"]
            # The first event is the current Task, sent at subscription time (spec
            # 3.1.6) — not the stale state of a finished task.
            stream_live({"task": current}, task_id)
            return

        respond(rpc_error(-32601, "Method not found", "METHOD_NOT_FOUND", method=method))


def main() -> None:
    global STATE_PATH, LEDGER_PATH
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--port", type=int, default=9999)
    parser.add_argument("--state", required=True)
    parser.add_argument("--ledger", required=True)
    args = parser.parse_args()
    STATE_PATH = args.state
    LEDGER_PATH = args.ledger
    load()
    server = ThreadingHTTPServer(("127.0.0.1", args.port), Handler)
    log(f"reference endpoint on 127.0.0.1:{args.port} "
        f"(dedup={'on' if DEDUP else 'off'}, persist={'on' if PERSIST else 'off'})")
    server.serve_forever()


if __name__ == "__main__":
    main()

The business is a word counter, and the message text drives the scenarios: please fail fails the business operation, please crash exits the process after the key is registered but before the business runs — the gap admitted in section 1 — and slowly keeps the task WORKING for four seconds so a connection can be cut mid-task. One structural rule matters more than any single scenario: the business is dispatched to its own thread the moment a request is accepted, before any connection I/O, so the SSE handlers only observe and relay. Execution of accepted work never depends on the connection that delivered it staying alive. The two environment switches break exactly one guarantee each, for the negative controls: A2A_DEDUP=off skips the key registry, A2A_PERSIST=off keeps state in memory only.

The harness starts and stops the endpoint, kills it for the crash and restart scenarios, and asserts from the ledger file. Save it as check-idempotency-recovery.sh:

#!/usr/bin/env bash
# Assert the idempotency-and-recovery contract of an A2A endpoint by counting
# business executions, not by comparing responses. Runs the full matrix from the
# guide against the reference endpoint: success, duplicate, concurrent duplicate,
# failure and redo, the crash gap, restart persistence, and real connection cuts
# before the first event and mid-task — plus the two negative controls (dedup
# off, persistence off), which must be DETECTED.
#
# Usage: bash check-idempotency-recovery.sh <reference-dir> <work-dir>
# Exit 0 iff every assertion holds. Dependencies: curl, jq, python3.
set -uo pipefail

REF=${1:?reference dir with a2a-idempotency-reference.py}
WORK=${2:?work dir for state and ledger files}
BASE=${A2A_BASE_URL:-}
[ -n "$BASE" ] || BASE=http://127.0.0.1:9999
PORT=${BASE##*:}
PY="$REF/a2a-idempotency-reference.py"
STATE="$WORK/state.json"
LEDGER="$WORK/ledger.jsonl"
SRV_LOG="$WORK/server.log"
PASS=0; FAIL=0

ok()   { printf 'PASS  %s\n' "$1"; PASS=$((PASS+1)); }
bad()  { printf 'FAIL  %s\n' "$1"; FAIL=$((FAIL+1)); }
check(){ if [ "$2" = "$3" ]; then ok "$1 ($2)"; else bad "$1: expected $3, got $2"; fi; }

send() { # messageId context text [taskId] -> response JSON
  local mid=$1 ctx=$2 text=$3 tid=${4:-}
  local body="{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"SendMessage\",\"params\":{\"message\":{"
  body+="\"role\":\"ROLE_USER\",\"parts\":[{\"text\":\"$text\"}],\"messageId\":\"$mid\""
  [ -n "$ctx" ] && body+=",\"contextId\":\"$ctx\""
  [ -n "$tid" ] && body+=",\"taskId\":\"$tid\""
  body+="}}}"
  curl -s --max-time 20 -H 'Content-Type: application/json' -d "$body" "$BASE"
}
stream() { # same args, SendStreamingMessage; prints raw SSE lines
  local mid=$1 ctx=$2 text=$3 tid=${4:-}
  local body="{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"SendStreamingMessage\",\"params\":{\"message\":{"
  body+="\"role\":\"ROLE_USER\",\"parts\":[{\"text\":\"$text\"}],\"messageId\":\"$mid\""
  [ -n "$ctx" ] && body+=",\"contextId\":\"$ctx\""
  [ -n "$tid" ] && body+=",\"taskId\":\"$tid\""
  body+="}}}"
  curl -sN --max-time 20 -H 'Content-Type: application/json' -d "$body" "$BASE"
}
get_task() { curl -s --max-time 10 -H 'Content-Type: application/json' \
  -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"GetTask\",\"params\":{\"id\":\"$1\"}}" "$BASE"; }
cut_before_first_event() { # messageId context text — a SendStreamingMessage whose
  # connection dies before a single response byte is read (a real early cut; the
  # request itself is delivered, so the server has accepted the work)
  python3 - "$BASE" "$1" "$2" "$3" <<'PYEOF'
import json, socket, struct, sys
base, mid, ctx, text = sys.argv[1:5]
host, port = base.split("//")[1].rsplit(":", 1)
body = json.dumps({"jsonrpc": "2.0", "id": 1, "method": "SendStreamingMessage",
                   "params": {"message": {"role": "ROLE_USER", "parts": [{"text": text}],
                                          "messageId": mid, "contextId": ctx}}}).encode()
head = ("POST / HTTP/1.1\r\nHost: %s:%s\r\nContent-Type: application/json\r\n"
        "Content-Length: %d\r\nConnection: close\r\n\r\n" % (host, port, len(body))).encode()
s = socket.create_connection((host, int(port)), timeout=5)
s.sendall(head + body)
# Abortive close (RST), not a graceful FIN: with a FIN the server's writes can
# still buffer and "succeed"; an RST makes the first write fail for real.
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
s.close()  # die before reading any response byte
PYEOF
}

executions() { grep -c "\"key\": \"$1\"" "$LEDGER" 2>/dev/null || true; }   # business count for one key
task_field() { printf '%s' "$1" | jq -r "$2"; }

start_server() { # [env assignments before the command]
  env "$@" python3 "$PY" --port "$PORT" --state "$STATE" --ledger "$LEDGER" 2>>"$SRV_LOG" &
  SRV_PID=$!
  for _ in $(seq 1 50); do curl -s -o /dev/null --max-time 1 "$BASE" && break; sleep 0.1; done
}
stop_server()  { [ -n "${SRV_PID:-}" ] && kill -9 "$SRV_PID" 2>/dev/null; wait "$SRV_PID" 2>/dev/null; SRV_PID=; }

trap stop_server EXIT
mkdir -p "$WORK"; : > "$LEDGER"; : > "$SRV_LOG"

echo "== phase 1: success, duplicate, concurrent, failure (fresh state) =="
rm -f "$STATE"; start_server

R=$(send k1 c-scenario1 "count words in: one two three")
check "success reaches COMPLETED" "$(task_field "$R" '.result.task.status.state')" "TASK_STATE_COMPLETED"
check "success artifact word count" "$(task_field "$R" '.result.task.artifacts[0].parts[0].text')" "3"
check "success executes the business once" "$(executions c-scenario1/k1)" "1"
T1=$(task_field "$R" '.result.task.id')

R=$(send k1 c-scenario1 "count words in: one two three")            # verbatim retry
check "duplicate replays the same task" "$(task_field "$R" '.result.task.id')" "$T1"
check "duplicate adds no business execution" "$(executions c-scenario1/k1)" "1"

send k2 c-scenario2 "count words in: one two three" >"$WORK/c1.json" &
P1=$!
send k2 c-scenario2 "count words in: one two three" >"$WORK/c2.json" &
P2=$!
wait "$P1" "$P2"
A=$(task_field "$(cat "$WORK/c1.json")" '.result.task.id'); B=$(task_field "$(cat "$WORK/c2.json")" '.result.task.id')
check "concurrent duplicates land on one task" "$([ "$A" = "$B" ] && echo same || echo "different ($A vs $B)")" "same"
check "concurrent duplicates execute once" "$(executions c-scenario2/k2)" "1"

R=$(send k3 c-scenario3 "please fail while you count words")
check "business failure ends FAILED" "$(task_field "$R" '.result.task.status.state')" "TASK_STATE_FAILED"
check "failed attempt is one recorded execution" "$(executions c-scenario3/k3)" "1"
T3=$(task_field "$R" '.result.task.id')
R=$(send k3 c-scenario3 "please fail while you count words")         # retry of a failure
check "retry of the failed message replays it" "$(task_field "$R" '.result.task.id')" "$T3"
check "retry of the failed message executes nothing" "$(executions c-scenario3/k3)" "1"
R=$(send k4 c-scenario3 "count words in: one two three")             # redo with a new messageId
check "redo with a new messageId completes" "$(task_field "$R" '.result.task.status.state')" "TASK_STATE_COMPLETED"
check "redo is its own single execution" "$(executions c-scenario3/k4)" "1"

echo "== phase 2: the crash gap (registered key, side effect never ran) =="
CTX=c-crash
send k5 "$CTX" "please crash before counting words" >"$WORK/crash.out" 2>&1
CRASH_RC=$?
if [ "$CRASH_RC" -eq 0 ] && [ -s "$WORK/crash.out" ]; then
  bad "crash injection: the server answered instead of dying"
else
  ok "crash injection: no response arrives (curl exit $CRASH_RC)"
fi
check "the crashed key has NO business execution" "$(executions "$CTX/k5")" "0"

start_server                                                        # restart on the same state
R=$(send k5 "$CTX" "please crash before counting words")            # same messageId
check "after restart the sweep FAILED the accepted work" "$(task_field "$R" '.result.task.status.state')" "TASK_STATE_FAILED"
check "replay of the crashed key still executes nothing" "$(executions "$CTX/k5")" "0"
R=$(send k6 "$CTX" "count words in: one two three")                 # redo with a new messageId
check "redo after the crash gap completes" "$(task_field "$R" '.result.task.status.state')" "TASK_STATE_COMPLETED"
check "redo after the crash gap executes once" "$(executions "$CTX/k6")" "1"

echo "== phase 3: restart keeps accepted outcomes =="
R=$(get_task "$T1")
check "a COMPLETED task survives the restarts" "$(task_field "$R" '.result.status.state')" "TASK_STATE_COMPLETED"
check "its artifact survived" "$(task_field "$R" '.result.artifacts[0].parts[0].text')" "3"
R=$(send k1 c-scenario1 "count words in: one two three")            # retry long after a restart
check "late duplicate replays without re-execution" "$(executions c-scenario1/k1)" "1"

echo "== phase 4: real connection cuts, then reconcile =="
echo "-- 4a: the connection dies before the first event --"
cut_before_first_event k7 c-cut-pre "slowly count words in: one two three four five"
R=$(send k7 c-cut-pre "slowly count words in: one two three four five")  # blocking replay
check "cut before the first event: the work still completed" \
  "$(task_field "$R" '.result.task.status.state')" "TASK_STATE_COMPLETED"
check "cut before the first event: the artifact is intact" \
  "$(task_field "$R" '.result.task.artifacts[0].parts[0].text')" "5"
check "cut before the first event: exactly one execution" "$(executions c-cut-pre/k7)" "1"
T7=$(task_field "$R" '.result.task.id')

echo "-- 4b: the connection dies after the first event, mid-task --"
curl -sN --max-time 2 -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"SendStreamingMessage","params":{"message":{"role":"ROLE_USER","parts":[{"text":"slowly count words in: one two three four five six"}],"messageId":"k8","contextId":"c-cut-mid"}}}' \
  "$BASE" >"$WORK/cut-mid.txt" 2>/dev/null
CUT_RC=$?
check "the mid-task cut is a real disconnect (curl exit 28)" "$CUT_RC" "28"
check "the cut client received exactly one event" "$(grep -c '^data: ' "$WORK/cut-mid.txt")" "1"
E1=$(sed -n 's/^data: //p' "$WORK/cut-mid.txt" | head -n 1)
T8=$(printf '%s' "$E1" | jq -r '.result.task.id')
check "the delivered first event is WORKING" "$(printf '%s' "$E1" | jq -r '.result.task.status.state')" "TASK_STATE_WORKING"
check "the delivered first event has no artifacts yet" "$(printf '%s' "$E1" | jq -r '.result.task.artifacts | length')" "0"

echo "-- 4c: SubscribeToTask mid-task: snapshot now, updates when they happen --"
SUB=$(curl -sN --max-time 15 -H 'Content-Type: application/json' \
  -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"SubscribeToTask\",\"params\":{\"id\":\"$T8\"}}" "$BASE")
S1=$(printf '%s' "$SUB" | sed -n 's/^data: //p' | head -n 1)
check "subscription first event is the task at subscription time" \
  "$(printf '%s' "$S1" | jq -r '.result.task.status.state')" "TASK_STATE_WORKING"
check "subscription first event carries no completed artifacts" \
  "$(printf '%s' "$S1" | jq -r '.result.task.artifacts | length')" "0"
check "subscription delivered the artifact update" "$(printf '%s' "$SUB" | grep -c 'artifactUpdate')" "1"
check "subscription ended COMPLETED" \
  "$(printf '%s' "$SUB" | sed -n 's/^data: //p' | tail -n 1 | jq -r '.result.statusUpdate.status.state')" "TASK_STATE_COMPLETED"

echo "-- 4d: reconcile the cut task --"
R=$(get_task "$T8")
check "reconciled task id unchanged" "$(task_field "$R" '.result.id')" "$T8"
check "reconciled task reached COMPLETED" "$(task_field "$R" '.result.status.state')" "TASK_STATE_COMPLETED"
check "reconciled artifact word count" "$(task_field "$R" '.result.artifacts[0].parts[0].text')" "6"
check "mid-task cut executed the business once" "$(executions c-cut-mid/k8)" "1"

echo "-- 4e: a terminal task refuses SubscribeToTask --"
SUBT=$(curl -s --max-time 10 -H 'Content-Type: application/json' \
  -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"SubscribeToTask\",\"params\":{\"id\":\"$T7\"}}" "$BASE")
check "terminal task refuses SubscribeToTask" "$(task_field "$SUBT" '.error.code')" "-32004"

stop_server

echo "== negative control A: A2A_DEDUP=off must be detected =="
rm -f "$STATE"; : > "$LEDGER"; start_server A2A_DEDUP=off
send nd1 c-neg "count words in: one two three" >/dev/null
send nd1 c-neg "count words in: one two three" >/dev/null
N=$(executions c-neg/nd1)
if [ "$N" -eq 2 ]; then ok "duplicate assertion catches it ($N executions, expected 1)"; else bad "dedup-off control: expected 2 executions, saw $N"; fi
stop_server

echo "== negative control B: A2A_PERSIST=off must be detected =="
rm -f "$STATE"; : > "$LEDGER"; start_server A2A_PERSIST=off
send np1 c-neg "count words in: one two three" >/dev/null
stop_server
start_server A2A_PERSIST=off
send np1 c-neg "count words in: one two three" >/dev/null
N=$(executions c-neg/np1)
if [ "$N" -eq 2 ]; then ok "restart assertion catches it ($N executions across restarts, expected 1)"; else bad "persist-off control: expected 2 executions, saw $N"; fi
stop_server

echo
echo "summary: $PASS passed, $FAIL failed"
[ "$FAIL" -eq 0 ]

Run it with python3, curl, and jq available:

$ bash check-idempotency-recovery.sh . work
== phase 1: success, duplicate, concurrent, failure (fresh state) ==
PASS  success reaches COMPLETED (TASK_STATE_COMPLETED)
PASS  success artifact word count (3)
PASS  success executes the business once (1)
PASS  duplicate replays the same task (task-ecad48f103f5)
PASS  duplicate adds no business execution (1)
PASS  concurrent duplicates land on one task (same)
PASS  concurrent duplicates execute once (1)
PASS  business failure ends FAILED (TASK_STATE_FAILED)
PASS  failed attempt is one recorded execution (1)
PASS  retry of the failed message replays it (task-3424828c3ec7)
PASS  retry of the failed message executes nothing (1)
PASS  redo with a new messageId completes (TASK_STATE_COMPLETED)
PASS  redo is its own single execution (1)
== phase 2: the crash gap (registered key, side effect never ran) ==
PASS  crash injection: no response arrives (curl exit 52)
PASS  the crashed key has NO business execution (0)
PASS  after restart the sweep FAILED the accepted work (TASK_STATE_FAILED)
PASS  replay of the crashed key still executes nothing (0)
PASS  redo after the crash gap completes (TASK_STATE_COMPLETED)
PASS  redo after the crash gap executes once (1)
== phase 3: restart keeps accepted outcomes ==
PASS  a COMPLETED task survives the restarts (TASK_STATE_COMPLETED)
PASS  its artifact survived (3)
PASS  late duplicate replays without re-execution (1)
== phase 4: real connection cuts, then reconcile ==
-- 4a: the connection dies before the first event --
PASS  cut before the first event: the work still completed (TASK_STATE_COMPLETED)
PASS  cut before the first event: the artifact is intact (5)
PASS  cut before the first event: exactly one execution (1)
-- 4b: the connection dies after the first event, mid-task --
PASS  the mid-task cut is a real disconnect (curl exit 28) (28)
PASS  the cut client received exactly one event (1)
PASS  the delivered first event is WORKING (TASK_STATE_WORKING)
PASS  the delivered first event has no artifacts yet (0)
-- 4c: SubscribeToTask mid-task: snapshot now, updates when they happen --
PASS  subscription first event is the task at subscription time (TASK_STATE_WORKING)
PASS  subscription first event carries no completed artifacts (0)
PASS  subscription delivered the artifact update (1)
PASS  subscription ended COMPLETED (TASK_STATE_COMPLETED)
-- 4d: reconcile the cut task --
PASS  reconciled task id unchanged (task-3ebb99c5f8b2)
PASS  reconciled task reached COMPLETED (TASK_STATE_COMPLETED)
PASS  reconciled artifact word count (6)
PASS  mid-task cut executed the business once (1)
-- 4e: a terminal task refuses SubscribeToTask --
PASS  terminal task refuses SubscribeToTask (-32004)
== negative control A: A2A_DEDUP=off must be detected ==
PASS  duplicate assertion catches it (2 executions, expected 1)
== negative control B: A2A_PERSIST=off must be detected ==
PASS  restart assertion catches it (2 executions across restarts, expected 1)

summary: 40 passed, 0 failed

That output is one real run of the two files above (macOS, Python 3.14, curl 8.7, jq 1.8, loopback only, 2026-10-31). Task IDs vary between runs; the counts and states do not. The 4a cut is an abortive close (RST), so the server’s first write fails instead of buffering; a graceful FIN can let a coupled implementation slip through the probe. The 40 assertions establish, for this endpoint:

  • Duplicates. A duplicate — sequential or concurrent — of one key produces exactly one ledger line and one task; in-flight duplicates join the running execution instead of starting a second one.
  • Failure and redo. A business failure is a definite recorded outcome: replaying the same messageId returns the same FAILED task with no new execution, and the redo happens on a new messageId in the same context.
  • The crash gap. It behaves as section 1 admits: after a crash between registration and the side effect, the ledger holds nothing for the key, the startup sweep moves the accepted task to FAILED, its replay does not execute, and a new messageId redoes the work once.
  • Restarts. Accepted outcomes survive kill -9: a COMPLETED task and its artifact are still there after a restart, and a late duplicate still replays without re-execution.
  • Reception. A cut connection loses nothing at either point: with the connection dead before the first event (RST, nothing read) or cut mid-task after the first event (curl exit 28), the accepted work still executes exactly once, and GetTask reconciles the same task id to COMPLETED with its artifact.
  • Resubscription. A mid-task SubscribeToTask returns the task as it was at subscription time — WORKING, no artifacts — as its first event, promptly rather than after the work finishes, then the artifact and terminal updates as they happen.
  • Terminal. A terminal task refuses the subscription with -32004.
  • Negative controls. With the registry off, the duplicate assertion sees two executions; with persistence off, the restart assertion sees the replay execute again — the probes detect a broken guarantee instead of passing vacuously.
  • Decisiveness. The two cut probes were checked against a deliberately regressed variant with the business back inside the stream handler: the pre-first-event cut left the registered task WORKING with zero executions in 6 of 6 trials (and a shallow-copy subscription showed WORKING with the finished artifact), so the assertions are decisive rather than vacuous.

Separately, the emitted wire shapes were checked with the trace validator in section 4 of the Task runtime guide : a full stream, a subscribed stream, a cut stream, and one blocking response each pass it. Capture a trace the same way as there: sed -n 's/^data: //p' stream.txt > trace.jsonl.

What this does not establish:

  • Anything about your server. The endpoint is a teaching reference with the dedup policy, retention, and sweep chosen by this guide; your implementation needs its own equivalent business record before these assertions mean anything.
  • Exactly-once for external side effects. The ledger is one process’s own file, appended and flushed but not fsynced; work shared across processes or external systems needs transactional storage or business-side deduplication, which is precisely what the protocol leaves open.
  • Production-grade behavior: sends are serialized by one process-wide lock (a production server scopes the mutex to the idempotency key), each accepted task runs on one unbounded daemon thread (a production server uses a bounded executor), events are not replayed from an arbitrary offset, and push notifications and CancelTask are not implemented.

5. Observability: make stalls visible

  • Stall alerts: the count of tasks still WORKING past the business P99 duration without events; on trigger, include the task-age distribution and the latest event;
  • Terminal distribution: COMPLETED / FAILED / CANCELED / REJECTED ratios, FAILED clustered by reason;
  • Duplicate rate: the share of messages hitting the idempotency key — it reflects both network quality and client retry policy;
  • Recovery metrics: the share of reconnects that retrieve a terminal state, and the number of tasks finalized (rather than left hanging) after a restart.

Field naming follows the logging conventions in section 4 of the production readiness guide ; task IDs are for correlation only, never for authorization.

Related pages

Version and verification notes

Protocol statements are checked against the pinned A2A v1.0.1 specification and the matching a2a.proto : Send Message MAY be idempotent and implementations may use messageId to detect duplicates (section 3.3.1); messages to terminal tasks return UnsupportedOperationError (section 3.1.1); a subscription’s first event is the current Task and terminal tasks refuse SubscribeToTask (section 3.1.6); SendMessage wraps its payload in result.task while GetTask returns the Task itself (proto, oneof payload). Where the two disagree, the proto wins (section 1.4).

The reference endpoint and harness in section 4 were run to completion on 2026-10-31 (macOS, Python 3.14, curl 8.7, jq 1.8, loopback only): 40 assertions passed, including real connection cuts before the first event (abortive RST close) and mid-task (curl timeout), a mid-task subscription whose first event is the task at subscription time, and a refused subscription on a terminal task; both negative controls were detected, and the captured streams and blocking response pass the Task runtime guide’s trace validator. The same cut probes were also run against a deliberately regressed variant (business inside the stream handler, shallow-copy subscription): the pre-first-event cut left the accepted task WORKING with zero executions in 6 of 6 trials, and the regressed subscription reported WORKING together with the finished artifact about 3.3 s late — which is why the fixed assertions are the ones printed above. That is runtime evidence for this reference implementation on one machine — not for your stack, not exactly-once for external side effects, and not production behavior. Deduplication windows, persistence, stall thresholds, and event deduplication IDs remain implementation policies. The harness was rerun on 2026-11-14 with Python 3.12.16 on macOS 12.7.6 (loopback): 40 passed, 0 failed, with the same counts and states.

Sources: A2A data structures , Life of a task .