Revenue anomaly investigation swarm

An upstream anomaly detector flags a suspicious revenue window and pings this workflow. A coordinator agent investigates, optionally delegates to an analyst sub-agent for deeper root-cause work, and drafts a finance-grade summary artifact. The finance director reviews. On approval, the artifact is marked published in Concord and visible to the finance team in-app. Nothing is emailed or pushed externally.

Two supported ingress paths, same command type:

• external_webhook — from the anomaly detector. Payload carries a provider event_id.
• api_request — manual kickoff by a finance analyst via internal UI. Payload carries a UI-generated action_id.

Idempotency

Ingress-level idempotency_key template:

idempotency_key = "revenue_investigation:{ingress}:{event_id_or_action_id}"

Payload schema (key fields)

Implicit context: workspace_id, tenant_id, trace_id.

One command type, parents the entire swarm.

The standard status machine applies. compensated is not reachable: no external writes, so no compensation chain exists.

Composed at validate time and before each tool call:

• permission — only members of finance_investigators or the detector service account can trigger.
• cost — hard cap max_cost_units = 30 for the SwarmRun.
• data_safety — SQL tool runs under a row-level-secured view; raw PII columns are denied at the warehouse.
• connector_scope — Snowflake reader role; SELECT only, no INSERT/UPDATE/DELETE.
• agent_risk — max_steps = 20 per AgentRun, spawn_depth ≤ 2, max_agents = 2.
• memory_consent — every memory write is a candidate; require_approval from the requester before commit.
• approval_requirement — artifact transition to published requires finance director sign-off.
• external_sharing — set to deny for all egress connectors; the only "sharing" is in-app artifact visibility.
• rate_limit — bucket on connector_calls queue; cap SQL calls per minute per tenant.

Async, agentic swarm. The coordinator runs on agent_runs; the optional analyst runs on swarm_children. SQL goes through connector_calls.

Execution modes used

Read-only investigation. No CoreEffect rows are emitted to domain_effects. The artifact is an Artifact, not an effect.

Reads

The coordinator's retrieve_memory tool searches prior incidents at scope = project (the finance-investigations project). Hit shape: incident summary, date range, root cause, resolution.

Writes

On a confirmed root cause, the coordinator proposes a memory candidate:

The artifact also references a query_result sub-artifact per SQL run, for traceability.

One approval. Triggered by the approval_requirement policy when the artifact transitions from validated to published.

Review packet

• requested_action: publish revenue investigation artifact to finance team view
• requester: command's requested_by
• reason: anomaly score and flagged window
• affected_data: artifact draft preview + linked query_result snapshots
• triggering_policy: external_sharing + approval_requirement
• expected_outcome: artifact visible in-app to finance_team group
• risk_level: medium (financial reporting)
• expiration: 72 hours

Approver: finance director (role-resolved, not a hardcoded user).

On rejection: artifact stays validated, command transitions to succeeded (the investigation ran), and the rejection reason becomes an audit event. No alternate publish path.

On expiration: artifact stays validated; a follow-up approval_callback may re-request after edits.

SwarmRun

Coordinator AgentRun

Analyst sub-AgentRun (optional)

Allowed tools

Forbidden tools

Allowed connectors

All written to domain_events with appropriate purpose.

purpose = audit

purpose = agent_step

Per agent step: step_index, tool_name, latency_ms, token_usage, cost_units. Carries agent_run_id for the coordinator and analyst separately.

purpose = event

Mode: graceful. Transitions: running → cancelling → cancelled.

Each agent step's prelude checks the cancellation flag. In-flight SQL is allowed to finish (read-only, bounded); the next step is skipped and the swarm exits.

propagate_cancellation = true: cancelling the parent command cancels the coordinator AgentRun, which cancels the analyst sub-AgentRun if active.

Not chosen: compensate_then_stop. There are no external writes to undo.

Skipped — not applicable. Read-only investigation: zero CoreEffect rows, no external mutations, no inverse operations to declare. The compensation graph validator passes trivially (empty manifest).

Adapter

DBOS. Required capabilities: DURABLE_WORKFLOWS, DURABLE_STEPS, QUEUES, SIGNALS, SUBWORKFLOWS. SAGA_COMPENSATION_NATIVE not needed (no effects).

Queues

RetryPolicy

Constraints

• Latency: background (minutes acceptable; p95 target 5 min end-to-end excluding approval wait).
• Cost: max_cost_units = 30 per SwarmRun.
• Compliance: audit retention 7 years (SOX). PII never leaves the warehouse — row-level security at the view layer.
• Multi-tenant: hard isolation by workspace_id on every read.

Unit

• Payload validation: required fields, range ≤ 31 days, valid metric_scope.
• Idempotency key composition for both ingress types.
• Coordinator decision function: when to spawn analyst (pure over typed dataclasses).
• Review packet builder produces all required fields.

Integration

• Postgres + DBOS adapter: SwarmRun row, coordinator AgentRun row, optional analyst row, status transitions.
• Snowflake reader role: SELECT passes, INSERT/UPDATE/DELETE denied at connector.
• Memory candidate write + consent commit flow.

Workflow (end-to-end)

• Happy path: webhook → investigation → no analyst → artifact drafted → approved → published.
• Branch: deeper analysis triggered → analyst spawned → join via coordinator_synthesis → artifact reflects analyst findings.
• Approval rejected: artifact remains validated, command succeeded, audit event recorded.
• Approval expired at 72h: artifact remains validated, approval marked expired.
• Cancellation mid-flight: graceful exit, no orphaned state.

Safety

• Policy denial: non-finance_investigators trigger → policy_denied, no SwarmRun row.
• Forbidden tool call: coordinator attempts send_email → blocked at policy, audit event, agent step recorded.
• Subset rule: analyst attempts a tool the coordinator lacks → blocked, agent_risk policy denial.
• Memory consent: candidate without approval never commits.
• External sharing: artifact transition without approval → blocked.
• Cost ceiling: SwarmRun exceeds max_cost_units = 30 → terminates with agent_failed.

Boundary discipline

• concord_boundary_check.py rejects dbos / temporalio imports outside the runtime adapter file.

• Detector reliability. If the upstream detector floods the webhook (e.g., recurring false positives across many windows), the rate_limit bucket needs to be sized — current cap is TBD per-tenant per-hour. Wire to alerting if rejection rate climbs.
• Analyst spawn heuristic. The coordinator's "need deeper analysis?" decision is currently model-driven. Worth defining a small rubric (confidence threshold, metric variance, multi-dimension flag) and treating the heuristic as a policy input — measurable and overrideable.
• Approver fallback. If the finance director is unavailable and the 72h expiry hits, the artifact stalls. Open: define a delegation path (deputy approver) or alert-and-extend pattern. For now, expiry is terminal.
• Memory pollution. Project-scope memory could accumulate low-quality candidates if consent is granted reflexively. Consider periodic memory review as a separate workflow.
• Multi-investigation overlap. Two overlapping date ranges triggered separately produce two artifacts. Should one supersede the other? Currently independent; parent_artifact_id linkage is manual.
• Versioning across coordinator iterations. If the coordinator agent's prompt or tool surface changes meaningfully, agent_run rows should reflect a config_version. Adapter capability WORKFLOW_VERSIONING is not in DBOS today — flag for Temporal if it becomes load-bearing.
• Anomaly score absence. Manual trigger has anomaly_score = null. The coordinator's prompt should handle this gracefully — confirm in evals.