Jobs and external calls

Ask the agent to trace jobs and delivery

Review [Jira key, PR URL or diff] for queued work, retries and external effects.
For a Jira ticket, read its description, acceptance criteria and comments through the connected Jira MCP or authenticated acli; ask for pasted details if access is unavailable.
Trace each operation from its work record through the queue and external calls to completion. Check every external call for an explicit timeout, a surrounding database transaction, its retries and the provider's rate limit. Test duplicate attempts, crashes and timeouts with synthetic data and test services, and include claim and ledger checks when value moves.
Report each duplicate or lost-work risk with evidence.
Report which of this page's tests exist in the project's test suite. Add the missing ones that can be automated, and list the rest.
Do not resend, refund or change production records.

Preserve work and recognize repeated attempts

• Each operation has a stable identity, and a repeated delivery returns the existing result. A queue attempt ID changes on retry, so it does not identify the business operation. Reject a reused identity with different input, and stop two workers applying the same effect.
• Commit the work record with the data change that needs it, then dispatch it, so a crash in between cannot lose the work.
• Job arguments stay readable by the previous version. During a rolling deploy old workers pick up new jobs, and after a rollback new jobs run on old code. Ship a new job or argument one release before you queue it.
• After an external timeout, the outcome is unknown: look up the result under the original reference. If the service cannot tell you or deduplicate, leave it unresolved for reconciliation instead of creating a new operation.
• Cap attempts and backoff, and send exhausted work to a visible failed state with an owner.

Call external services safely

• Every external call has an explicit timeout and runs outside a database transaction. A client library’s default timeout can be minutes or none, and a call inside a transaction holds its locks and can’t be rolled back with it.
• Set connect and read timeouts separately, sized from the provider’s normal latency: for example 1–2 seconds to connect and 2–5 seconds to read for a call a user waits on.
• Record the intent and commit, then call or queue the call. In Rails, use after_commit, not after_save. Rails.
• Retry only requests that are safe to repeat: reads, or writes sent with the provider’s idempotency key. Back off with jitter, and on 429 or 503 wait as long as Retry-After says.
• Stay under each provider’s rate limit on purpose: throttle calls across all workers and servers, with a shared limiter or a queue with fixed concurrency. A per-process limit multiplies with every worker you add.
• Run bulk sends, syncs and backfills on their own paced queue, so they can’t use up the quota a sign-in or checkout needs.
• When a provider is down, degrade that feature, not the app: switch it off or queue the work, show a pending state and alert. One provider’s retries mustn’t fill the queue everything else uses.
• Treat webhooks as repeatable and out of order. After verifying them, answer quickly and do the work in a job, since a slow answer makes the provider send again. Record the event ID and ignore repeats, and never let an older event overwrite a newer state.
• Use sandbox credentials and test accounts outside production; production keys never reach staging.
• Log the provider’s request ID and outcome with your operation ID, not the payload. It’s how you settle an unknown result later.
• Make a test double of the provider slow, then failing, then answering 429. Slow calls time out within their limit; failures and 429s are retried only as allowed; no worker hangs, nothing is sent twice, and the user sees a pending state.
• Run a bulk job while users make requests. User-facing calls stay within their latency target, and the provider never receives more than its rate limit.

Bound batch work and fan-out

• Cap result size, batch size, concurrency and runtime using the project’s measured limits. Pagination alone does not bound the work behind a query.
• Queue long exports and stream their data so one request cannot exhaust memory. Scope the job and its file to the requesting user and tenant.
• Cap downstream calls per item and per job: a small input can fan out into many queries or external calls.
• Give expensive endpoints, such as search, exports and report generation, per-caller limits in the application, as SMS, OTP and paid endpoints does for paid sends. Otherwise one caller can exhaust them for everyone.

Test interrupted and repeated work

• Deliver the same job concurrently, then replay it after completion: one result, and no extra call, file or notification.
• Crash before dispatch, after the external service accepts, and before recording completion: the pending work is found and nothing is duplicated.
• Send delayed and duplicate callbacks: each maps to the original operation and cannot repeat its effect or overwrite a newer state.
• Exhaust retries: the failed record keeps the original identity, the error and the next action.
• Send the largest allowed export alongside normal traffic, and one over the limit: the first stays within memory and queue limits, the second is rejected.