Operate and recover / Investigate and recover
中文

Investigate and recover

Find the evidence, stop the harm, and restore the affected flow and records.

Updated 5 Nov 2026View as markdown
On this page

1. Establish what is happening

Scenario: users report that OTP codes stopped arriving this morning.

Bad prompt
OTP is down. Restart the SMS worker and retry the failed sends.
Better prompt
Investigate [symptom and affected flow] in [environment] between [start] and [end, timezone]. Known identifiers: [app version, IDs or unknown].
Using approved read-only access, follow this attempt through jobs, external calls and errors, and compare it with the deployed code and recent changes.
Return a timeline with evidence references, supported hypotheses, missing evidence or access, and the next useful check.
Do not send messages, replay work, bypass checks, restart services or change production data or settings.

Why: a restart wipes the evidence, and a retry can send paid messages twice. Find where the flow stops, read-only, before anyone acts.

  • Start from the release record and failed success check in Release and close out, or from the reported symptom when no release is involved.
  • Name one incident owner, who decides what happens next. Bring in the PM and QA; the PM handles the client side under the project’s incident process.
  • Over major holidays, the on-call engineer investigates the logs together with the PM and QA. The project’s runbook names the rota.
  • Have the agent cite the matching records, not paste whole logs. Link events by operation, request or job ID; a nearby timestamp does not prove two events belong together.
  • Compare with the deployed artifact, not the current branch, and note timezone conversions before ordering events.
  • Preserve logs and records before restarting or changing configuration, unless that would delay stopping active harm. Redact secrets and personal data.
  • Every finding cites its evidence: a log line, record or operation ID. An explanation without evidence is a hypothesis, however confident the agent sounds.
ExampleOTP investigation note · attempt A-17

A user reports that the code never arrived. Scope: production, app 4.12.0, 09:40–09:45 UTC.

Evidence Observation What it supports
09:41:02 · client diagnostic C-17 The user tapped “Send code” and a bot check started. The flow started; the check’s result is unknown.
Backend log search Q-17 No match for A-17. Nothing yet: sampling, missing IDs or the wrong log source could explain it.
09:41:03 · error event E-17 A client exception in token handling after the bot-check callback. The app failed; the bot check itself may not have.
Deployed app source matching E-17 The exception happens before the backend request is made. This attempt never reached the backend.

Finding: the attempt stopped in the app, so there is nothing to look up at the SMS provider yet. Next check: reproduce the callback in app 4.12.0 with a synthetic account, and confirm no other path reaches the backend.

2. Stop the harm

Prompt
For [incident] in [environment], propose the smallest reversible action that stops [harm].
Read the account, environment and target from the active configuration and report them. Give the exact command or setting, its expected effect, the legitimate work it delays and how to reverse it.
Do not run it, and do not change production data or settings.
  • Choose the smallest reversible action that stops it: pause the affected job, switch the flag off, or use the kill switch for value or paid sends. Say what legitimate work it delays.
  • Present the exact target, action, expected effect and how to reverse it. Production and destructive actions need a person’s explicit go-ahead; approval to read logs is not approval to write.
  • Recheck the account, environment and target immediately before running anything.
  • A release rollback follows its recovery plan. It stops new damage and undoes nothing already done.
  • For a leaked secret, follow Access, secrets and data: revoke it first, then check its use.

3. Restore the flow and reconcile affected work

Prompt
For [incident] between [start] and [end, timezone], list the affected user flow, pending or failed work, changed data and external effects, comparing our records with the external service's where they exist.
Propose recovery with exact targets, expected effects and verification, and list unresolved items with owners.
Use read access only. Do not replay work, change records or make calls that create effects.
  • Restore and test the affected user flow, then reconcile pending and failed work. A working screen does not prove queued work completed.
  • Check external effects such as sent messages and created orders; a rollback does not undo them.
  • When an outcome is unknown, look it up under the original operation identity before retrying, and leave it pending with an owner until it is known. See Jobs and external calls.
  • Correct value with new, linked ledger entries, never by editing history.
  • For a database restore, use the service runbook. Know the recovery point and the data it loses, restore to a separate instance first, keep the damaged state, and reconcile external operations before reopening writes.
  • Recovery restores the affected user flow and reconciles affected work. Check pending and failed operations, changed data and external effects against the available records, including provider evidence where relevant. Every unresolved item has an owner and a next check.
Failure example · illustrative

Retry the failed ones

An export worker writes files, then crashes before recording completion. An agent requeues every pending export under new job IDs, so customers get duplicate files and repeated notifications. Checking the original operations and the existing files would have shown which work was already done.

4. Close the incident

Prompt
Draft the incident record for [incident]: timeline, cause, actions, evidence references, and open operations with owners.
On a branch from [base branch], write a regression test that fails on the original failure, and show it failing against [commit before the fix].
List the instructions or docs that misled people or agents, and the evidence we could not find.
Do not push or open a PR yet.
  • Confirm the harmful activity has stopped and the backlog has cleared. Follow one recovered operation end to end.
  • Write a short record where the project keeps them: timeline, cause, actions, evidence, and open operations with owners.
  • Turn the failure into a regression check and show that it fails on the original failure. Correct any instruction that misled people or agents.
  • Ship the permanent fix as an urgent fix when production can’t wait for the next release.
  • Ticket any evidence you could not find. A missing operation ID or log line is the next incident’s blind spot.
  • Every incident leaves a regression check that fails on the original failure. The permanent fix goes through normal review; urgency does not skip it.