A2A Task Idempotency and Recovery: Duplicates, Dropped Streams, and Stalled Tasks
Networks drop, clients retry, processes restart. The protocol does not absorb these failures for you: it provides messageId, contextId, and two recovery channels, and leaves exactly-once effects to the implementation. This guide shows the patterns that put those three to work, plus a reference endpoint and probes that verify them by counting business executions — the results printed in section 4 come from actually running them.
State and operation semantics are on the Task object page and in the Task runtime guide ; the release checklist for restart and retries is section 5 of the production readiness guide .
Recovery decision tree
After a dropped connection or a response timeout, the client first determines where the task went, then acts:
stream dropped / response timeout
├─ already terminal (COMPLETED / FAILED / CANCELED / REJECTED)?
│ └─ yes: GetTask for the terminal state and artifacts, done
├─ subscription available (capabilities.streaming)?
│ └─ yes: SubscribeToTask and continue from the event stream
│ └─ when the server cannot resume from an arbitrary point,
│ trust the events received and deduplicate repeats by
│ an implementation-provided stable event ID, then reconcile with GetTask
├─ polling only?
│ └─ GetTask with backoff until terminal or stalled
└─ stalled: still SUBMITTED / WORKING past the business cap
├─ client: CancelTask, or resend the request in a new task (same contextId)
└─ server: move the task to FAILED with a stated reason, never WORKING foreverDo not read “the stream ended” as “the task ended”: a dropped connection only means you can no longer see it; the task may still be progressing server-side.
1. Duplicate messages: this implementation uses messageId as an idempotency key
This implementation adopts server-side message deduplication. The sender generates messageId and preserves it on retries; the server returns the registered outcome within its configured scope and retention window. The specification makes this optional — “Send Message operations MAY be idempotent. Agents may utilize the messageId to detect duplicate messages” (section 3.3.1) — so a retry against a server without deduplication executes the work twice; check the server’s policy before relying on it. Get operations are naturally idempotent and Cancel Task is idempotent by the same section; Send is the one that needs a decision.
| Scenario | Client behavior | Server policy in this implementation |
|---|---|---|
| Resending the same message after a timeout | Resend verbatim; messageId unchanged | Recognize the duplicate; return the first processed result |
| Continuing in the same conversation (same task) | New messageId + taskId (optionally the matching contextId) | Advance the task with the new input |
| Follow-up after a task ended | Same contextId, new message without taskId | Create a new task (terminal tasks cannot restart) |
| The task a retry targets is already terminal | Do not resend with that taskId; redo on a new task in the same contextId with a new messageId | UnsupportedOperationError: terminal tasks accept no further messages (section 3.1.1) — the verbatim-retry replay above applies only to messages that do not carry a terminal taskId |
Implementation notes for this design:
- This server uses
(taskId or contextId, messageId)as the idempotency key, storing a pointer to the first outcome (task ID, response, or error); - The idempotency window spans the whole period “before the client gives up”, not a few seconds;
- Business side effects (external calls, database writes) happen after the key is registered. That order prevents a second execution when the response was lost but the work happened; it cannot prevent the mirror image — a crash between registration and the side effect leaves a key with no outcome. The sweep in section 3
fails such work explicitly and the redo moves to a new
messageId; exactly-once across processes would require transactional storage. Section 4 runs this gap for real; - A retry only hits the key if it lands in the same scope. Follow-ups carry the
taskIdby construction; a first message should carry acontextIdfrom the start (or the server opens amessageId-only window for scopeless firsts), otherwise each retry mints a new context and executes as new work. The reference endpoint keys scopeless first messages bymessageIdalone; - For idempotency across messages, use
contextId: new tasks in the same context can carry an upstream business key whose dedup scope your implementation defines.
The protocol does not guarantee exactly-once at the business level — referenceTaskIds and contextId carry correlation only; dedup policy is yours. Section 5 of the production readiness guide
lists the same boundary.
2. Duplicate delivery: three dedup points on the receiving side
Three places on the recovery path can see “the same event a second time”:
| Location | Dedup key |
|---|---|
| Streamed events (after reconnect) | Implementation-provided stable event IDs; no universal sequence is defined |
| Push notifications | Implementation-defined notification IDs; Task ID + state alone can merge distinct updates (section 4 of Push receiver hardening ) |
| Artifact chunks | Implementation-provided chunk IDs; artifactId, append, and lastChunk alone cannot detect duplicates |
append and lastChunk describe aggregation and completion, not chunk identity. Repeated chunks require implementation-defined deduplication; when identity is unavailable, reconcile with a GetTask snapshot.
3. Restart recovery: give accepted tasks an outcome
After a server restart, accepted tasks have only two acceptable outcomes: continue per the product contract, or fail with an explicit result. The pattern:
startup -> load persisted task records
├─ task with a registered idempotency key -> replay the recorded result, never re-run side effects
├─ task already executing -> resume, or move to FAILED (reason: "server restart")
└─ unrecoverable -> FAILED with a readable reason; never parked back in WORKING
waiting for an executor that will not comeWrite “where tasks go after a restart” into the product contract so clients can decide between retrying and giving up; this is item 3 of the checklist in section 5 of the production readiness guide . Phase 2 of the run in section 4 shows the sweep behaving this way: after a crash between registration and the side effect, the accepted task is FAILED on restart, the ledger still holds nothing for that key, and the redo executes once on a new key.
4. A reference endpoint and the probes, run
Equal-looking responses are exactly what a duplicated side effect hides behind, so these probes do not compare responses: they count business executions in an append-only ledger and reconcile the recorded outcomes. The endpoint under test is a single-file A2A JSON-RPC server implementing the policy of section 1 and the recovery sweep of section 3.
The short path through this section: save the two files below — a2a-idempotency-reference.py and check-idempotency-recovery.sh — into one directory, and from that same directory run bash check-idempotency-recovery.sh . work with python3, curl, and jq available; read the PASS/FAIL lines down to the final summary:. The assertions count executions in work/ledger.jsonl; a FAIL line names the exact expectation it broke. Save the server first:
#!/usr/bin/env python3
"""Reference A2A endpoint for idempotency and recovery probes (v1.0.1, JSON-RPC binding).
A single-file teaching server, not a production agent. It implements the policy of
the guide: (taskId or contextId, messageId) as the idempotency key, business side
effects recorded to an append-only ledger, task records persisted to a state file,
and a startup sweep that gives interrupted tasks a definite FAILED outcome.
Accepted work executes on its own thread from the moment the request is
registered; SSE handlers only observe and relay, so a dropped connection can
neither cancel nor skip the work.
Implemented methods (POST /, JSON-RPC 2.0):
SendMessage (blocking), SendStreamingMessage (SSE), SubscribeToTask (SSE),
GetTask. CancelTask and push notifications are not implemented;
configuration/metadata are accepted and ignored.
Message text drives the scenarios (the business is a word counter):
"please fail" -> the business operation fails; the task ends FAILED
"please crash" -> the key and task are persisted, then the process exits
before the business runs (the crash gap)
"slowly" -> the task stays WORKING for ~4 s before completing
Environment switches for the negative controls:
A2A_DEDUP=off skip the key registry: duplicates execute twice
A2A_PERSIST=off keep state in memory only: a restart forgets keys and tasks
Usage: a2a-idempotency-reference.py --port 9999 --state state.json --ledger ledger.jsonl
"""
from __future__ import annotations
import argparse
import copy
import datetime
import json
import os
import sys
import threading
import time
import uuid
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
LOCK = threading.RLock()
STATE = {"tasks": {}, "keys": {}} # persisted shape
DONE: dict[str, threading.Event] = {} # in-memory terminal events, not persisted
STATE_PATH = None
LEDGER_PATH = None
PERSIST = os.environ.get("A2A_PERSIST") != "off"
DEDUP = os.environ.get("A2A_DEDUP") != "off"
TERMINAL = ("TASK_STATE_COMPLETED", "TASK_STATE_FAILED", "TASK_STATE_CANCELED", "TASK_STATE_REJECTED")
def now() -> str:
return datetime.datetime.now(datetime.timezone.utc).isoformat()
def log(message: str) -> None:
print(f"[{datetime.datetime.now().strftime('%H:%M:%S')}] {message}", file=sys.stderr, flush=True)
# ---------------------------------------------------------------- state and ledger
def load() -> None:
if PERSIST and os.path.exists(STATE_PATH):
with open(STATE_PATH, encoding="utf-8") as fh:
loaded = json.load(fh)
STATE["tasks"] = loaded.get("tasks", {})
STATE["keys"] = loaded.get("keys", {})
# Startup sweep: every accepted task gets an outcome. In-flight tasks cannot
# resume (the executor is gone), so they move to FAILED with a readable reason.
for task in STATE["tasks"].values():
if task["status"]["state"] not in TERMINAL:
task["status"] = failed_status("interrupted by server restart; resend with a new messageId to redo the work")
save()
for task in STATE["tasks"].values():
DONE[task["id"]] = threading.Event()
DONE[task["id"]].set()
def save() -> None:
if not PERSIST:
return
tmp = f"{STATE_PATH}.tmp"
with open(tmp, "w", encoding="utf-8") as fh:
json.dump({"tasks": STATE["tasks"], "keys": STATE["keys"]}, fh)
os.replace(tmp, STATE_PATH) # atomic on the same filesystem
def ledger(key: str, task_id: str, outcome: str, words: int | None) -> None:
"""Record one business execution. This file is what the probes assert against."""
line = {"ts": now(), "key": key, "taskId": task_id, "outcome": outcome}
if words is not None:
line["words"] = words
with open(LEDGER_PATH, "a", encoding="utf-8") as fh:
fh.write(json.dumps(line) + "\n")
fh.flush()
# ---------------------------------------------------------------- task helpers
def failed_status(reason: str) -> dict:
return {
"state": "TASK_STATE_FAILED",
"timestamp": now(),
"message": {
"messageId": f"recovery-{uuid.uuid4()}",
"role": "ROLE_AGENT",
"parts": [{"text": reason}],
},
}
def snapshot(task: dict) -> dict:
"""The wire shape of a task, as a deep copy: internal bookkeeping fields never
leave the server, and later mutations never leak into an earlier snapshot."""
return copy.deepcopy({k: v for k, v in task.items() if k != "input"})
# ---------------------------------------------------------------- business operation
def run_business(text: str) -> tuple[str, int | None, str | None]:
"""Returns (outcome, words, artifact_text). The only place side effects happen."""
if "please fail" in text:
return "failed", None, None
words = len(text.split(":", 1)[1].split()) if ":" in text else len(text.split())
return "completed", words, str(words)
# ---------------------------------------------------------------- JSON-RPC plumbing
def rpc_error(code: int, message: str, reason: str, **metadata) -> dict:
return {
"jsonrpc": "2.0",
"id": None,
"error": {
"code": code,
"message": message,
"data": [{
"@type": "type.googleapis.com/google.rpc.ErrorInfo",
"reason": reason,
"domain": "a2a-protocol.org",
"metadata": {k: str(v) for k, v in metadata.items()},
}],
},
}
# ---------------------------------------------------------------- send path
def prepare(params: dict):
"""Resolve, replay, or register one idempotency key. No side effects happen here.
Returns (task, key, text, mode, error):
mode "terminal" task is done (replay or instant result); task is the answer
mode "join" a duplicate of work still in flight; wait on DONE[task.id]
mode "run" registered; the caller dispatches the business to its own
thread (blocking callers then wait on DONE; streaming
callers send the first event while it runs)
mode "slow" registered; as "run" but the work takes ~4 s
"""
msg = params.get("message") or {}
message_id = msg.get("messageId") or ""
if not message_id:
return None, None, None, None, rpc_error(-32602, "messageId is required", "INVALID_PARAMS")
text = " ".join(p.get("text", "") for p in msg.get("parts", []) if isinstance(p, dict))
with LOCK:
task = None
if msg.get("taskId"):
task = STATE["tasks"].get(msg["taskId"])
if task is None:
return None, None, None, None, rpc_error(
-32001, "Task not found", "TASK_NOT_FOUND", taskId=msg["taskId"])
if task["status"]["state"] in TERMINAL:
# Messages cannot be accepted for terminal tasks (spec 3.1.1).
return None, None, None, None, rpc_error(
-32004, "Task is in a terminal state", "UNSUPPORTED_OPERATION",
taskId=task["id"], state=task["status"]["state"])
if msg.get("contextId") and msg["contextId"] != task["contextId"]:
return None, None, None, None, rpc_error(
-32602, "contextId does not match the task's context", "INVALID_PARAMS")
# Key scope: the task for follow-ups, the context for new work, and the
# messageId alone for a first message that carries neither.
scope = msg.get("taskId") or msg.get("contextId") or "*first-message*"
key = f"{scope}/{message_id}"
if DEDUP and key in STATE["keys"]:
replay = STATE["tasks"][STATE["keys"][key]]
if replay["status"]["state"] in TERMINAL:
return snapshot(replay), key, text, "terminal", None
return snapshot(replay), key, text, "join", None
# Register the key and persist BEFORE any side effect. A crash after this
# point leaves accepted work with an outcome of nothing yet — the startup
# sweep turns that into an explicit FAILED, never into silent re-execution.
context_id = msg.get("contextId") or (task["contextId"] if task else f"ctx-{uuid.uuid4().hex[:10]}")
if task is None:
task = {
"id": f"task-{uuid.uuid4().hex[:12]}",
"contextId": context_id,
"input": text,
"artifacts": [],
}
STATE["tasks"][task["id"]] = task
DONE[task["id"]] = threading.Event()
if DEDUP:
STATE["keys"][key] = task["id"]
task["status"] = {"state": "TASK_STATE_WORKING", "timestamp": now()}
save()
if "please crash" in text:
log(f"crash injection: key {key} persisted, exiting before the business runs")
os._exit(70)
return snapshot(task), key, text, ("slow" if "slowly" in text else "run"), None
def complete(task_id: str, key: str, text: str) -> dict:
"""Run the business for a prepared task and return the final snapshot."""
if "slowly" in text:
time.sleep(4) # outside the lock: slow work must not block other requests
with LOCK:
task = STATE["tasks"][task_id]
outcome, words, artifact_text = run_business(text)
ledger(key, task["id"], outcome, words)
if outcome == "completed":
task["artifacts"].append({
"artifactId": f"artifact-{uuid.uuid4().hex[:10]}",
"name": "word count",
"parts": [{"text": artifact_text}],
})
task["status"] = {"state": "TASK_STATE_COMPLETED", "timestamp": now()}
else:
task["status"] = failed_status("business error (injected by 'please fail')")
DONE[task["id"]].set()
save()
return snapshot(task)
def join(task_id: str) -> dict:
"""A duplicate of in-flight work: wait for the outcome, return the same task."""
DONE[task_id].wait(timeout=30)
with LOCK:
return snapshot(STATE["tasks"][task_id])
def dispatch(task_id: str, key: str, text: str) -> None:
"""Start the business for a prepared task on its own thread.
Execution begins when the request is accepted, not when a connection delivers
an event: a stream cut before the first event still executes the work exactly
once, and blocking callers simply wait on the same DONE event.
"""
def worker() -> None:
try:
complete(task_id, key, text)
except Exception as exc: # an accepted task must still reach a terminal state
with LOCK:
task = STATE["tasks"].get(task_id)
if task is not None and task["status"]["state"] not in TERMINAL:
task["status"] = failed_status(f"executor error: {exc}")
DONE[task_id].set()
save()
threading.Thread(target=worker, daemon=True).start()
# ---------------------------------------------------------------- HTTP layer
class Handler(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def log_message(self, fmt, *args): # quiet; the ledger is the record that matters
pass
def do_POST(self) -> None:
length = int(self.headers.get("Content-Length") or 0)
request = json.loads(self.rfile.read(length) or b"{}")
method = request.get("method", "")
rid = request.get("id")
params = request.get("params") or {}
def respond(payload: dict) -> None:
payload["id"] = rid
body = json.dumps(payload).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def start_stream() -> None:
"""Open the SSE response; the connection closes when the handler returns."""
self.send_response(200)
self.send_header("Content-Type", "text/event-stream")
self.send_header("Cache-Control", "no-cache")
self.send_header("Connection", "close")
self.close_connection = True
self.end_headers()
def write_event(payload: dict) -> None:
"""One SSE event. A broken pipe means the client cut the stream."""
self.wfile.write(f"data: {json.dumps({'jsonrpc': '2.0', 'id': rid, 'result': payload})}\n\n".encode())
self.wfile.flush()
def updates_of(task_id: str) -> list[dict]:
"""The artifact and status events of a finished task, in order."""
with LOCK:
task = STATE["tasks"][task_id]
rest = [{"artifactUpdate": {
"taskId": task["id"], "contextId": task["contextId"], "artifact": artifact}}
for artifact in task["artifacts"]]
rest.append({"statusUpdate": {
"taskId": task["id"], "contextId": task["contextId"], "status": task["status"]}})
return rest
def stream(first: dict | None, rest: list[dict]) -> None:
try:
start_stream()
if first is not None:
write_event(first)
for payload in rest:
write_event(payload)
except (BrokenPipeError, ConnectionResetError):
log("stream cut by the client; the task keeps running server-side")
def stream_live(first: dict, task_id: str) -> None:
"""Send the first event now; terminal updates follow when they happen."""
try:
start_stream()
write_event(first)
DONE[task_id].wait(timeout=30)
for payload in updates_of(task_id):
write_event(payload)
except (BrokenPipeError, ConnectionResetError):
log("stream cut by the client; the task keeps running server-side")
if method in ("SendMessage", "SendStreamingMessage"):
first, key, text, mode, error = prepare(params)
if error is not None:
respond(error)
elif method == "SendMessage":
# SendMessageResponse wraps the payload: result = {"task": ...}
# (GetTask returns the Task itself; proto v1.0.1).
if mode == "terminal":
respond({"jsonrpc": "2.0", "result": {"task": first}})
elif mode == "join":
respond({"jsonrpc": "2.0", "result": {"task": join(first["id"])}})
else:
dispatch(first["id"], key, text) # execution does not need this connection
respond({"jsonrpc": "2.0", "result": {"task": join(first["id"])}})
else: # SendStreamingMessage: the task event goes out first, updates follow
if mode == "terminal":
stream({"task": first}, [])
else:
if mode != "join":
dispatch(first["id"], key, text) # before any connection I/O
stream_live({"task": first}, first["id"])
return
if method == "GetTask":
with LOCK:
task = STATE["tasks"].get(params.get("id") or "")
if task is None:
respond(rpc_error(-32001, "Task not found", "TASK_NOT_FOUND", taskId=params.get("id")))
else:
respond({"jsonrpc": "2.0", "result": snapshot(task)})
return
if method == "SubscribeToTask":
with LOCK:
task = STATE["tasks"].get(params.get("id") or "")
if task is None:
respond(rpc_error(-32001, "Task not found", "TASK_NOT_FOUND", taskId=params.get("id")))
return
if task["status"]["state"] in TERMINAL:
respond(rpc_error(-32004, "Task is in a terminal state", "UNSUPPORTED_OPERATION",
taskId=task["id"], state=task["status"]["state"]))
return
current = snapshot(task) # deep copy under the lock: one consistent moment
task_id = task["id"]
# The first event is the current Task, sent at subscription time (spec
# 3.1.6) — not the stale state of a finished task.
stream_live({"task": current}, task_id)
return
respond(rpc_error(-32601, "Method not found", "METHOD_NOT_FOUND", method=method))
def main() -> None:
global STATE_PATH, LEDGER_PATH
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--port", type=int, default=9999)
parser.add_argument("--state", required=True)
parser.add_argument("--ledger", required=True)
args = parser.parse_args()
STATE_PATH = args.state
LEDGER_PATH = args.ledger
load()
server = ThreadingHTTPServer(("127.0.0.1", args.port), Handler)
log(f"reference endpoint on 127.0.0.1:{args.port} "
f"(dedup={'on' if DEDUP else 'off'}, persist={'on' if PERSIST else 'off'})")
server.serve_forever()
if __name__ == "__main__":
main()The business is a word counter, and the message text drives the scenarios: please fail fails the business operation, please crash exits the process after the key is registered but before the business runs — the gap admitted in section 1 — and slowly keeps the task WORKING for four seconds so a connection can be cut mid-task. One structural rule matters more than any single scenario: the business is dispatched to its own thread the moment a request is accepted, before any connection I/O, so the SSE handlers only observe and relay. Execution of accepted work never depends on the connection that delivered it staying alive. The two environment switches break exactly one guarantee each, for the negative controls: A2A_DEDUP=off skips the key registry, A2A_PERSIST=off keeps state in memory only.
The harness starts and stops the endpoint, kills it for the crash and restart scenarios, and asserts from the ledger file. Save it as check-idempotency-recovery.sh:
#!/usr/bin/env bash
# Assert the idempotency-and-recovery contract of an A2A endpoint by counting
# business executions, not by comparing responses. Runs the full matrix from the
# guide against the reference endpoint: success, duplicate, concurrent duplicate,
# failure and redo, the crash gap, restart persistence, and real connection cuts
# before the first event and mid-task — plus the two negative controls (dedup
# off, persistence off), which must be DETECTED.
#
# Usage: bash check-idempotency-recovery.sh <reference-dir> <work-dir>
# Exit 0 iff every assertion holds. Dependencies: curl, jq, python3.
set -uo pipefail
REF=${1:?reference dir with a2a-idempotency-reference.py}
WORK=${2:?work dir for state and ledger files}
BASE=${A2A_BASE_URL:-}
[ -n "$BASE" ] || BASE=http://127.0.0.1:9999
PORT=${BASE##*:}
PY="$REF/a2a-idempotency-reference.py"
STATE="$WORK/state.json"
LEDGER="$WORK/ledger.jsonl"
SRV_LOG="$WORK/server.log"
PASS=0; FAIL=0
ok() { printf 'PASS %s\n' "$1"; PASS=$((PASS+1)); }
bad() { printf 'FAIL %s\n' "$1"; FAIL=$((FAIL+1)); }
check(){ if [ "$2" = "$3" ]; then ok "$1 ($2)"; else bad "$1: expected $3, got $2"; fi; }
send() { # messageId context text [taskId] -> response JSON
local mid=$1 ctx=$2 text=$3 tid=${4:-}
local body="{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"SendMessage\",\"params\":{\"message\":{"
body+="\"role\":\"ROLE_USER\",\"parts\":[{\"text\":\"$text\"}],\"messageId\":\"$mid\""
[ -n "$ctx" ] && body+=",\"contextId\":\"$ctx\""
[ -n "$tid" ] && body+=",\"taskId\":\"$tid\""
body+="}}}"
curl -s --max-time 20 -H 'Content-Type: application/json' -d "$body" "$BASE"
}
stream() { # same args, SendStreamingMessage; prints raw SSE lines
local mid=$1 ctx=$2 text=$3 tid=${4:-}
local body="{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"SendStreamingMessage\",\"params\":{\"message\":{"
body+="\"role\":\"ROLE_USER\",\"parts\":[{\"text\":\"$text\"}],\"messageId\":\"$mid\""
[ -n "$ctx" ] && body+=",\"contextId\":\"$ctx\""
[ -n "$tid" ] && body+=",\"taskId\":\"$tid\""
body+="}}}"
curl -sN --max-time 20 -H 'Content-Type: application/json' -d "$body" "$BASE"
}
get_task() { curl -s --max-time 10 -H 'Content-Type: application/json' \
-d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"GetTask\",\"params\":{\"id\":\"$1\"}}" "$BASE"; }
cut_before_first_event() { # messageId context text — a SendStreamingMessage whose
# connection dies before a single response byte is read (a real early cut; the
# request itself is delivered, so the server has accepted the work)
python3 - "$BASE" "$1" "$2" "$3" <<'PYEOF'
import json, socket, struct, sys
base, mid, ctx, text = sys.argv[1:5]
host, port = base.split("//")[1].rsplit(":", 1)
body = json.dumps({"jsonrpc": "2.0", "id": 1, "method": "SendStreamingMessage",
"params": {"message": {"role": "ROLE_USER", "parts": [{"text": text}],
"messageId": mid, "contextId": ctx}}}).encode()
head = ("POST / HTTP/1.1\r\nHost: %s:%s\r\nContent-Type: application/json\r\n"
"Content-Length: %d\r\nConnection: close\r\n\r\n" % (host, port, len(body))).encode()
s = socket.create_connection((host, int(port)), timeout=5)
s.sendall(head + body)
# Abortive close (RST), not a graceful FIN: with a FIN the server's writes can
# still buffer and "succeed"; an RST makes the first write fail for real.
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
s.close() # die before reading any response byte
PYEOF
}
executions() { grep -c "\"key\": \"$1\"" "$LEDGER" 2>/dev/null || true; } # business count for one key
task_field() { printf '%s' "$1" | jq -r "$2"; }
start_server() { # [env assignments before the command]
env "$@" python3 "$PY" --port "$PORT" --state "$STATE" --ledger "$LEDGER" 2>>"$SRV_LOG" &
SRV_PID=$!
for _ in $(seq 1 50); do curl -s -o /dev/null --max-time 1 "$BASE" && break; sleep 0.1; done
}
stop_server() { [ -n "${SRV_PID:-}" ] && kill -9 "$SRV_PID" 2>/dev/null; wait "$SRV_PID" 2>/dev/null; SRV_PID=; }
trap stop_server EXIT
mkdir -p "$WORK"; : > "$LEDGER"; : > "$SRV_LOG"
echo "== phase 1: success, duplicate, concurrent, failure (fresh state) =="
rm -f "$STATE"; start_server
R=$(send k1 c-scenario1 "count words in: one two three")
check "success reaches COMPLETED" "$(task_field "$R" '.result.task.status.state')" "TASK_STATE_COMPLETED"
check "success artifact word count" "$(task_field "$R" '.result.task.artifacts[0].parts[0].text')" "3"
check "success executes the business once" "$(executions c-scenario1/k1)" "1"
T1=$(task_field "$R" '.result.task.id')
R=$(send k1 c-scenario1 "count words in: one two three") # verbatim retry
check "duplicate replays the same task" "$(task_field "$R" '.result.task.id')" "$T1"
check "duplicate adds no business execution" "$(executions c-scenario1/k1)" "1"
send k2 c-scenario2 "count words in: one two three" >"$WORK/c1.json" &
P1=$!
send k2 c-scenario2 "count words in: one two three" >"$WORK/c2.json" &
P2=$!
wait "$P1" "$P2"
A=$(task_field "$(cat "$WORK/c1.json")" '.result.task.id'); B=$(task_field "$(cat "$WORK/c2.json")" '.result.task.id')
check "concurrent duplicates land on one task" "$([ "$A" = "$B" ] && echo same || echo "different ($A vs $B)")" "same"
check "concurrent duplicates execute once" "$(executions c-scenario2/k2)" "1"
R=$(send k3 c-scenario3 "please fail while you count words")
check "business failure ends FAILED" "$(task_field "$R" '.result.task.status.state')" "TASK_STATE_FAILED"
check "failed attempt is one recorded execution" "$(executions c-scenario3/k3)" "1"
T3=$(task_field "$R" '.result.task.id')
R=$(send k3 c-scenario3 "please fail while you count words") # retry of a failure
check "retry of the failed message replays it" "$(task_field "$R" '.result.task.id')" "$T3"
check "retry of the failed message executes nothing" "$(executions c-scenario3/k3)" "1"
R=$(send k4 c-scenario3 "count words in: one two three") # redo with a new messageId
check "redo with a new messageId completes" "$(task_field "$R" '.result.task.status.state')" "TASK_STATE_COMPLETED"
check "redo is its own single execution" "$(executions c-scenario3/k4)" "1"
echo "== phase 2: the crash gap (registered key, side effect never ran) =="
CTX=c-crash
send k5 "$CTX" "please crash before counting words" >"$WORK/crash.out" 2>&1
CRASH_RC=$?
if [ "$CRASH_RC" -eq 0 ] && [ -s "$WORK/crash.out" ]; then
bad "crash injection: the server answered instead of dying"
else
ok "crash injection: no response arrives (curl exit $CRASH_RC)"
fi
check "the crashed key has NO business execution" "$(executions "$CTX/k5")" "0"
start_server # restart on the same state
R=$(send k5 "$CTX" "please crash before counting words") # same messageId
check "after restart the sweep FAILED the accepted work" "$(task_field "$R" '.result.task.status.state')" "TASK_STATE_FAILED"
check "replay of the crashed key still executes nothing" "$(executions "$CTX/k5")" "0"
R=$(send k6 "$CTX" "count words in: one two three") # redo with a new messageId
check "redo after the crash gap completes" "$(task_field "$R" '.result.task.status.state')" "TASK_STATE_COMPLETED"
check "redo after the crash gap executes once" "$(executions "$CTX/k6")" "1"
echo "== phase 3: restart keeps accepted outcomes =="
R=$(get_task "$T1")
check "a COMPLETED task survives the restarts" "$(task_field "$R" '.result.status.state')" "TASK_STATE_COMPLETED"
check "its artifact survived" "$(task_field "$R" '.result.artifacts[0].parts[0].text')" "3"
R=$(send k1 c-scenario1 "count words in: one two three") # retry long after a restart
check "late duplicate replays without re-execution" "$(executions c-scenario1/k1)" "1"
echo "== phase 4: real connection cuts, then reconcile =="
echo "-- 4a: the connection dies before the first event --"
cut_before_first_event k7 c-cut-pre "slowly count words in: one two three four five"
R=$(send k7 c-cut-pre "slowly count words in: one two three four five") # blocking replay
check "cut before the first event: the work still completed" \
"$(task_field "$R" '.result.task.status.state')" "TASK_STATE_COMPLETED"
check "cut before the first event: the artifact is intact" \
"$(task_field "$R" '.result.task.artifacts[0].parts[0].text')" "5"
check "cut before the first event: exactly one execution" "$(executions c-cut-pre/k7)" "1"
T7=$(task_field "$R" '.result.task.id')
echo "-- 4b: the connection dies after the first event, mid-task --"
curl -sN --max-time 2 -H 'Content-Type: application/json' \
-d '{"jsonrpc":"2.0","id":1,"method":"SendStreamingMessage","params":{"message":{"role":"ROLE_USER","parts":[{"text":"slowly count words in: one two three four five six"}],"messageId":"k8","contextId":"c-cut-mid"}}}' \
"$BASE" >"$WORK/cut-mid.txt" 2>/dev/null
CUT_RC=$?
check "the mid-task cut is a real disconnect (curl exit 28)" "$CUT_RC" "28"
check "the cut client received exactly one event" "$(grep -c '^data: ' "$WORK/cut-mid.txt")" "1"
E1=$(sed -n 's/^data: //p' "$WORK/cut-mid.txt" | head -n 1)
T8=$(printf '%s' "$E1" | jq -r '.result.task.id')
check "the delivered first event is WORKING" "$(printf '%s' "$E1" | jq -r '.result.task.status.state')" "TASK_STATE_WORKING"
check "the delivered first event has no artifacts yet" "$(printf '%s' "$E1" | jq -r '.result.task.artifacts | length')" "0"
echo "-- 4c: SubscribeToTask mid-task: snapshot now, updates when they happen --"
SUB=$(curl -sN --max-time 15 -H 'Content-Type: application/json' \
-d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"SubscribeToTask\",\"params\":{\"id\":\"$T8\"}}" "$BASE")
S1=$(printf '%s' "$SUB" | sed -n 's/^data: //p' | head -n 1)
check "subscription first event is the task at subscription time" \
"$(printf '%s' "$S1" | jq -r '.result.task.status.state')" "TASK_STATE_WORKING"
check "subscription first event carries no completed artifacts" \
"$(printf '%s' "$S1" | jq -r '.result.task.artifacts | length')" "0"
check "subscription delivered the artifact update" "$(printf '%s' "$SUB" | grep -c 'artifactUpdate')" "1"
check "subscription ended COMPLETED" \
"$(printf '%s' "$SUB" | sed -n 's/^data: //p' | tail -n 1 | jq -r '.result.statusUpdate.status.state')" "TASK_STATE_COMPLETED"
echo "-- 4d: reconcile the cut task --"
R=$(get_task "$T8")
check "reconciled task id unchanged" "$(task_field "$R" '.result.id')" "$T8"
check "reconciled task reached COMPLETED" "$(task_field "$R" '.result.status.state')" "TASK_STATE_COMPLETED"
check "reconciled artifact word count" "$(task_field "$R" '.result.artifacts[0].parts[0].text')" "6"
check "mid-task cut executed the business once" "$(executions c-cut-mid/k8)" "1"
echo "-- 4e: a terminal task refuses SubscribeToTask --"
SUBT=$(curl -s --max-time 10 -H 'Content-Type: application/json' \
-d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"SubscribeToTask\",\"params\":{\"id\":\"$T7\"}}" "$BASE")
check "terminal task refuses SubscribeToTask" "$(task_field "$SUBT" '.error.code')" "-32004"
stop_server
echo "== negative control A: A2A_DEDUP=off must be detected =="
rm -f "$STATE"; : > "$LEDGER"; start_server A2A_DEDUP=off
send nd1 c-neg "count words in: one two three" >/dev/null
send nd1 c-neg "count words in: one two three" >/dev/null
N=$(executions c-neg/nd1)
if [ "$N" -eq 2 ]; then ok "duplicate assertion catches it ($N executions, expected 1)"; else bad "dedup-off control: expected 2 executions, saw $N"; fi
stop_server
echo "== negative control B: A2A_PERSIST=off must be detected =="
rm -f "$STATE"; : > "$LEDGER"; start_server A2A_PERSIST=off
send np1 c-neg "count words in: one two three" >/dev/null
stop_server
start_server A2A_PERSIST=off
send np1 c-neg "count words in: one two three" >/dev/null
N=$(executions c-neg/np1)
if [ "$N" -eq 2 ]; then ok "restart assertion catches it ($N executions across restarts, expected 1)"; else bad "persist-off control: expected 2 executions, saw $N"; fi
stop_server
echo
echo "summary: $PASS passed, $FAIL failed"
[ "$FAIL" -eq 0 ]Run it with python3, curl, and jq available:
$ bash check-idempotency-recovery.sh . work
== phase 1: success, duplicate, concurrent, failure (fresh state) ==
PASS success reaches COMPLETED (TASK_STATE_COMPLETED)
PASS success artifact word count (3)
PASS success executes the business once (1)
PASS duplicate replays the same task (task-ecad48f103f5)
PASS duplicate adds no business execution (1)
PASS concurrent duplicates land on one task (same)
PASS concurrent duplicates execute once (1)
PASS business failure ends FAILED (TASK_STATE_FAILED)
PASS failed attempt is one recorded execution (1)
PASS retry of the failed message replays it (task-3424828c3ec7)
PASS retry of the failed message executes nothing (1)
PASS redo with a new messageId completes (TASK_STATE_COMPLETED)
PASS redo is its own single execution (1)
== phase 2: the crash gap (registered key, side effect never ran) ==
PASS crash injection: no response arrives (curl exit 52)
PASS the crashed key has NO business execution (0)
PASS after restart the sweep FAILED the accepted work (TASK_STATE_FAILED)
PASS replay of the crashed key still executes nothing (0)
PASS redo after the crash gap completes (TASK_STATE_COMPLETED)
PASS redo after the crash gap executes once (1)
== phase 3: restart keeps accepted outcomes ==
PASS a COMPLETED task survives the restarts (TASK_STATE_COMPLETED)
PASS its artifact survived (3)
PASS late duplicate replays without re-execution (1)
== phase 4: real connection cuts, then reconcile ==
-- 4a: the connection dies before the first event --
PASS cut before the first event: the work still completed (TASK_STATE_COMPLETED)
PASS cut before the first event: the artifact is intact (5)
PASS cut before the first event: exactly one execution (1)
-- 4b: the connection dies after the first event, mid-task --
PASS the mid-task cut is a real disconnect (curl exit 28) (28)
PASS the cut client received exactly one event (1)
PASS the delivered first event is WORKING (TASK_STATE_WORKING)
PASS the delivered first event has no artifacts yet (0)
-- 4c: SubscribeToTask mid-task: snapshot now, updates when they happen --
PASS subscription first event is the task at subscription time (TASK_STATE_WORKING)
PASS subscription first event carries no completed artifacts (0)
PASS subscription delivered the artifact update (1)
PASS subscription ended COMPLETED (TASK_STATE_COMPLETED)
-- 4d: reconcile the cut task --
PASS reconciled task id unchanged (task-3ebb99c5f8b2)
PASS reconciled task reached COMPLETED (TASK_STATE_COMPLETED)
PASS reconciled artifact word count (6)
PASS mid-task cut executed the business once (1)
-- 4e: a terminal task refuses SubscribeToTask --
PASS terminal task refuses SubscribeToTask (-32004)
== negative control A: A2A_DEDUP=off must be detected ==
PASS duplicate assertion catches it (2 executions, expected 1)
== negative control B: A2A_PERSIST=off must be detected ==
PASS restart assertion catches it (2 executions across restarts, expected 1)
summary: 40 passed, 0 failed
That output is one real run of the two files above (macOS, Python 3.14, curl 8.7, jq 1.8, loopback only, 2026-10-31). Task IDs vary between runs; the counts and states do not. The 4a cut is an abortive close (RST), so the server’s first write fails instead of buffering; a graceful FIN can let a coupled implementation slip through the probe. The 40 assertions establish, for this endpoint:
- Duplicates. A duplicate — sequential or concurrent — of one key produces exactly one ledger line and one task; in-flight duplicates join the running execution instead of starting a second one.
- Failure and redo. A business failure is a definite recorded outcome: replaying the same
messageIdreturns the same FAILED task with no new execution, and the redo happens on a newmessageIdin the same context. - The crash gap. It behaves as section 1 admits: after a crash between registration and the side effect, the ledger holds nothing for the key, the startup sweep moves the accepted task to FAILED, its replay does not execute, and a new
messageIdredoes the work once. - Restarts. Accepted outcomes survive
kill -9: a COMPLETED task and its artifact are still there after a restart, and a late duplicate still replays without re-execution. - Reception. A cut connection loses nothing at either point: with the connection dead before the first event (RST, nothing read) or cut mid-task after the first event (curl exit 28), the accepted work still executes exactly once, and
GetTaskreconciles the same task id toCOMPLETEDwith its artifact. - Resubscription. A mid-task
SubscribeToTaskreturns the task as it was at subscription time —WORKING, no artifacts — as its first event, promptly rather than after the work finishes, then the artifact and terminal updates as they happen. - Terminal. A terminal task refuses the subscription with
-32004. - Negative controls. With the registry off, the duplicate assertion sees two executions; with persistence off, the restart assertion sees the replay execute again — the probes detect a broken guarantee instead of passing vacuously.
- Decisiveness. The two cut probes were checked against a deliberately regressed variant with the business back inside the stream handler: the pre-first-event cut left the registered task
WORKINGwith zero executions in 6 of 6 trials (and a shallow-copy subscription showedWORKINGwith the finished artifact), so the assertions are decisive rather than vacuous.
Separately, the emitted wire shapes were checked with the trace validator in section 4 of the Task runtime guide
: a full stream, a subscribed stream, a cut stream, and one blocking response each pass it. Capture a trace the same way as there: sed -n 's/^data: //p' stream.txt > trace.jsonl.
What this does not establish:
- Anything about your server. The endpoint is a teaching reference with the dedup policy, retention, and sweep chosen by this guide; your implementation needs its own equivalent business record before these assertions mean anything.
- Exactly-once for external side effects. The ledger is one process’s own file, appended and flushed but not fsynced; work shared across processes or external systems needs transactional storage or business-side deduplication, which is precisely what the protocol leaves open.
- Production-grade behavior: sends are serialized by one process-wide lock (a production server scopes the mutex to the idempotency key), each accepted task runs on one unbounded daemon thread (a production server uses a bounded executor), events are not replayed from an arbitrary offset, and push notifications and CancelTask are not implemented.
5. Observability: make stalls visible
- Stall alerts: the count of tasks still
WORKINGpast the business P99 duration without events; on trigger, include the task-age distribution and the latest event; - Terminal distribution: COMPLETED / FAILED / CANCELED / REJECTED ratios, FAILED clustered by reason;
- Duplicate rate: the share of messages hitting the idempotency key — it reflects both network quality and client retry policy;
- Recovery metrics: the share of reconnects that retrieve a terminal state, and the number of tasks finalized (rather than left hanging) after a restart.
Field naming follows the logging conventions in section 4 of the production readiness guide ; task IDs are for correlation only, never for authorization.
Related pages
- Task runtime guide : state rules, event streams, and the trace validator
- Task object page
: the state machine and
contextIdsemantics - Message object page
:
messageId,referenceTaskIds, and message ownership rules - A2A Error Codes : the JSON-RPC codes the reference endpoint returns
- Push receiver hardening : duplicate delivery on the notification side
Version and verification notes
Protocol statements are checked against the pinned A2A v1.0.1 specification
and the matching a2a.proto
: Send Message MAY be idempotent and implementations may use messageId to detect duplicates (section 3.3.1); messages to terminal tasks return UnsupportedOperationError (section 3.1.1); a subscription’s first event is the current Task and terminal tasks refuse SubscribeToTask (section 3.1.6); SendMessage wraps its payload in result.task while GetTask returns the Task itself (proto, oneof payload). Where the two disagree, the proto wins (section 1.4).
The reference endpoint and harness in section 4 were run to completion on 2026-10-31 (macOS, Python 3.14, curl 8.7, jq 1.8, loopback only): 40 assertions passed, including real connection cuts before the first event (abortive RST close) and mid-task (curl timeout), a mid-task subscription whose first event is the task at subscription time, and a refused subscription on a terminal task; both negative controls were detected, and the captured streams and blocking response pass the Task runtime guide’s trace validator. The same cut probes were also run against a deliberately regressed variant (business inside the stream handler, shallow-copy subscription): the pre-first-event cut left the accepted task WORKING with zero executions in 6 of 6 trials, and the regressed subscription reported WORKING together with the finished artifact about 3.3 s late — which is why the fixed assertions are the ones printed above. That is runtime evidence for this reference implementation on one machine — not for your stack, not exactly-once for external side effects, and not production behavior. Deduplication windows, persistence, stall thresholds, and event deduplication IDs remain implementation policies. The harness was rerun on 2026-11-14 with Python 3.12.16 on macOS 12.7.6 (loopback): 40 passed, 0 failed, with the same counts and states.
Sources: A2A data structures , Life of a task .