Pilot QA runbook
Manual acceptance QA executed against a deployed AccountingKit before the first customer pilot.
Automated suites prove internal behavior against mocks and a local database. This runbook proves the
deployed system against a real Xero organization, a real customer webhook receiver, and a real
browser. Execute it in order; a phase depends on the artifacts produced by the phase before it.
Record every result in the defect log. A pilot starts only when every P0 check
passes.
| Symbol | Meaning |
|---|
| P0 | Blocks the pilot. No workaround is acceptable. |
| P1 | Blocks the pilot for the affected flow only. Needs a documented workaround. |
| P2 | Ship and fix during the observation window. |
Prepare these before touching the API. Every later phase references them by name.
| ID | Fixture | Requirement |
|---|
| F-01 | ORG_A | Console organization that owns the pilot. Two console users: one owner, one non-owner member. |
| F-02 | ORG_B | A second, unrelated console organization. Exists only to prove cross-organization isolation. |
| F-03 | XERO_DEMO | A Xero Demo Company or disposable staging organization. Never the customer's production ledger. |
| F-04 | XERO_SECOND | A second Xero organization authorized by the same Xero login. Proves tenant selection. |
| F-05 | KEY_TEST_A | acct_test_ key in ORG_A. |
| F-06 | KEY_LIVE_A | acct_live_ key in ORG_A. |
| F-07 | KEY_TEST_B | acct_test_ key in ORG_B. |
| F-08 | KEY_REVOKED | A key in ORG_A that is revoked immediately after creation. |
| F-09 | HOOK_GOOD | An HTTPS receiver that logs raw body plus headers, verifies the signature, and returns 200. |
| F-10 | HOOK_BAD | An HTTPS receiver that always returns 500. Used to force delivery failure and replay. |
| F-11 | Bank account | A Xero bank or clearing account in XERO_DEMO, with its Xero AccountID recorded. |
| F-12 | XERO_APP_A | Customer-owned Xero app for ORG_A; client credentials and webhook key are available. |
| F-13 | XERO_APP_B | A second app or fixture owned by ORG_B for provider-app ownership checks. |
HOOK_GOOD must log the exact bytes it received. Signature verification that re-serializes JSON
before hashing will pass locally and fail in production; the receiver is part of what is under test.
| ID | Pri | Check | Expected |
|---|
| G-01 | P0 | From a clean worktree: DATABASE_URL='postgresql://app:app@localhost:5433/app?sslmode=disable' pnpm verify:local-mvp | Exits 0. |
| G-02 | P0 | pnpm --filter @accountingkit/server db:migrate, db:rollback to the base, db:migrate again on a scratch database | All three succeed; final schema equals the first. |
| G-03 | P0 | Confirm the deployed image digest matches the commit under test | RELEASE_ID and IMAGE_DIGEST in the deployment match HEAD. |
| G-04 | P0 | Review the deployed variable set against deploy/cloudrun/env.yaml and the SERVICE_SECRETS manifest in deploy/cloudrun/config.env | No placeholder value survives; ACCOUNTING_ENCRYPTION_KEY is unique to this environment and is not the local one. |
| G-05 | P0 | ACCOUNTING_QUEUE_BACKEND=postgres, ACCOUNTING_EMBEDDED_WORKER_ENABLED=true, Cloud Run min/max instances match the declared release values | Confirmed in the deployment dashboard; provider budgets are sized for the configured maximum. |
| G-06 | P1 | CONNECT_ALLOWED_REDIRECT_ORIGINS lists only origins the pilot customer actually uses | No wildcard, no localhost in a live deployment. |
G-01 is the only step in this runbook that a green CI run substitutes for. Everything after it must
be executed against the deployed service.
| ID | Pri | Check | Expected |
|---|
| S-01 | P0 | API_ORIGIN=... WEB_ORIGIN=... bash scripts/verify-public-deployment.sh | Exits 0. |
| S-02 | P0 | API_ORIGIN=... API_KEY=$KEY_TEST_A PROVIDER_APP_ID=$XERO_APP_A_ID bash scripts/qa-pilot-smoke.sh | Exits 0; every assertion prints ok. |
| S-03 | P0 | GET /health and GET /ready over plain HTTP | Redirected or refused; never served over http:. |
| S-04 | P1 | Stop PostgreSQL, then GET /ready | 503 not_ready. GET /health may stay 200. |
| S-05 | P1 | Restore PostgreSQL, wait one poll interval, GET /ready | Returns to 200 ready without a restart. |
This phase is the one that must not be shortened. Every check here is a data-leak boundary.
| ID | Pri | Check | Expected |
|---|
| A-01 | P0 | GET /v1/connections with no Authorization | 401 UNAUTHORIZED. No data. |
| A-02 | P0 | Same, with Authorization: Bearer not-a-key | 401. Response body does not reveal whether the key existed. |
| A-03 | P0 | Same, with KEY_REVOKED | 401. |
| A-04 | P0 | With KEY_TEST_B, read a connection_id that belongs to ORG_A | 404, not 403. A cross-organization ID must not be confirmed. |
| A-05 | P0 | With KEY_LIVE_A, read a connection created under KEY_TEST_A | 404. Environments are separate data planes inside one org. |
| A-06 | P0 | With KEY_TEST_B, POST /v1/connections/<ORG_A connection>/sync | 404. Nothing is queued; confirm no operation row appears. |
| A-07 | P0 | With KEY_TEST_B, GET /v1/operations/<ORG_A operation_id> | 404. |
| A-08 | P0 | With KEY_TEST_B, POST /v1/operations/<ORG_A operation_id>/replay | 404. No new attempt on the original operation. |
| A-09 | P0 | With KEY_TEST_B, POST /v1/webhook-deliveries/<ORG_A delivery_id>/replay | 404. |
| A-10 | P0 | Create a key in the console; read the list again | The secret is shown exactly once. The list shows prefix and last four only. |
| A-11 | P0 | Revoke a key in the console, then use it within one minute | 401 immediately. No cache window. |
| A-12 | P1 | Sign in as the ORG_A non-owner member | Whatever the role permits is consistent between the console UI and the API; a hidden button is not an authorization control. Attempt the key-creation call directly. |
| A-13 | P1 | Console session cookie replayed against /v1/* | 401. Browser sessions must not authenticate the machine API. |
| A-14 | P1 | KEY_TEST_A sent to a console endpoint under /accounting/* | 401. |
| A-15 | P1 | Cross-origin POST to a console endpoint from an unlisted origin | Blocked by CSRF/CORS. |
| ID | Pri | Check | Expected |
|---|
| C-00 | P0 | Register XERO_APP_A in the console and copy its generated URLs to Xero | Credentials are never returned; the callback and opaque app-specific webhook URL are copyable. |
| C-01 | P0 | POST /v1/connect-sessions with provider, external_account_id, redirect_uri, and provider_app_id | 201 with connect_url and expires_at; an app ID owned by ORG_B is rejected without revealing it. |
| C-02 | P0 | redirect_uri outside CONNECT_ALLOWED_REDIRECT_ORIGINS | Rejected at session creation, not after the customer authorizes Xero. |
| C-03 | P0 | Open connect_url in a clean browser profile; authorize XERO_DEMO | Returns to the redirect URI; connection becomes active. |
| C-04 | P0 | Authorize a Xero login with access to both XERO_DEMO and XERO_SECOND | Tenant selection is required; the connection stays non-active until one organization is chosen. |
| C-05 | P0 | GET /v1/connections | provider_tenant_id, provider_tenant_name, external_account_id, and scopes including offline_access are correct. |
| C-06 | P0 | Open the same connect_url a second time after completion | Cannot be reused to attach another tenant. |
| C-07 | P0 | Open a connect_url after expires_at | Refused with a clear message, not a stack trace. |
| C-08 | P0 | Connect XERO_DEMO twice through XERO_APP_A, then once through another app | The duplicate within one app is refused. The same tenant through another app is allowed and receives only that app's webhook events. |
| C-09 | P0 | Cancel at Xero's consent screen | Connection does not become active; the customer sees a recoverable state. |
| C-10 | P1 | Abandon mid-flow, then restart the same session | No orphaned pending connection blocks a retry. |
| C-11 | P1 | Complete the flow on a phone-width viewport | The hosted pages are usable; the pilot customer's end users may not be on a desktop. |
| C-12 | P0 | DELETE /v1/connections/:id, then inspect Xero Connected Apps and read a resource | The exact Xero app connection is removed before local credentials are purged; reads fail cleanly and reconnecting works afterwards. |
Run each read against XERO_DEMO and diff it against the Xero UI by eye. Cached and live reads are
different code paths; check both.
| ID | Pri | Check | Expected |
|---|
| R-01 | P0 | GET /v1/connections/:id/{contacts,invoices,quotes,payments,purchase_orders,accounts,tax_rates,branding_themes,organizations} | Each returns 200 with meta pagination. source is cache. |
| R-02 | P0 | Page through a list with page and limit past the final page | No duplicated or skipped records; the empty page is not an error. |
| R-03 | P0 | Live read by provider ID for an invoice, quote, contact, and purchase order | source is provider; totals, tax, status, and line items match Xero exactly. |
| R-04 | P0 | Compare a multi-line invoice with a discount, a non-default account code, and a tax rate | DiscountAmount, Xero AccountID, account code, and tax type all survive normalization. |
| R-05 | P0 | Filter by number, status, and external_reference | Exact matches only. A partial number must not return a different document. |
| R-06 | P0 | Read an invoice in a currency other than the organization's base currency | Currency and currency rate are correct; no silent conversion. |
| R-07 | P1 | Read a contact with two addresses, a phone, a tax number, and a contact person | Nothing is dropped. |
| R-08 | P1 | Read a resource ID that does not exist in Xero | 404 with the standard envelope. |
| R-09 | P1 | Read on a connection whose Xero authorization was revoked at Xero | A clear action_required style failure, not a 500. |
| R-10 | P2 | Time a cold cached list and a live read | Record the numbers. They become the pilot's latency baseline. |
| ID | Pri | Check | Expected |
|---|
| W-01 | P0 | Any write with no Idempotency-Key | 400 BAD_REQUEST. Nothing reaches Xero. |
| W-02 | P0 | Create a contact, then repeat the identical request with the same key | Same result, Idempotent-Replayed: true, exactly one contact in Xero. |
| W-03 | P0 | Reuse the same key with a different body | Rejected. The original result is not overwritten. |
| W-04 | P0 | Fire the same create twice concurrently with one key | Exactly one Xero record. One response may be a replay. |
| W-05 | P0 | Create a draft invoice with external_reference set to an order ID | Xero Reference carries the order ID; Xero's own sequential invoice number is used. |
| W-06 | P0 | PATCH an invoice's status only | Document number, reference, branding theme, and every line survive unchanged. |
| W-07 | P0 | PATCH a contact with a subset of fields | Account number, status, phones, addresses, tax number, and default currency are not erased. |
| W-08 | P0 | Draft a quote with Terms, then update only its status | QuoteNumber and Terms are preserved. |
| W-09 | P0 | Convert an accepted quote to an invoice | Invoice date, due date, AUTHORISED status, reference, branding theme, tax mode, and every line clone exactly. |
| W-10 | P0 | Record a payment against AUTHORISED invoice using the F-11 bank account | Xero shows the payment on the routed account; the operation reaches succeeded. |
| W-11 | P0 | Cancel a DRAFT invoice | cancellation_status: deleted. |
| W-12 | P0 | Cancel a SUBMITTED and an AUTHORISED invoice | Correct delete-versus-void behavior for each; verify in Xero. |
| W-13 | P0 | Cancel a PAID invoice | Refused with a clear error. Never silently voids a paid document. |
| W-14 | P0 | Cancel by stored Xero provider ID via /invoices/by-provider-id/:provider_id/cancel | Same outcome as cancelling by internal ID. |
| W-15 | P0 | Create a purchase order with delivery date, delivery address, attention-to, telephone | All provider-native fields land in Xero. |
| W-16 | P1 | Write with an invalid contact_id, a negative amount, an empty line_items, a bad date | 400 with a message naming the field. Nothing partially written. |
| W-17 | P1 | Write a line description at the 4000-character limit and one past it | Accepted and rejected respectively, at the boundary the contract states. |
| W-18 | P1 | Write to a read-only resource type (tax_rates, branding_themes, organizations) | Refused. |
| W-19 | P1 | Set READ_ONLY_MODE=true, retry one read and one write | Read succeeds; write returns 403 READ_ONLY. Restore the flag afterwards. |
| ID | Pri | Check | Expected |
|---|
| D-01 | P0 | invoice.download_pdf | application/pdf; opens; content matches the current Xero invoice. |
| D-02 | P0 | quote.download_pdf | Same. |
| D-03 | P0 | purchase_order.download_pdf | Same. |
| D-04 | P0 | Edit the invoice in Xero, download the PDF again | The new PDF reflects the edit. A stale cached render here is a pilot blocker. |
| D-05 | P0 | invoice.get_online_url | The URL opens Xero's hosted invoice view for the right document. |
| D-06 | P1 | organization.get_deep_link | Lands on the correct Xero organization. |
| D-07 | P1 | Provider action with neither resource_id nor provider_id | 400, not a 500. |
| D-08 | P1 | Provider action against a connection in another organization | 404. |
This is the flow the pilot customer depends on most and the one mocks prove least. Point the
connection's webhook endpoint at HOOK_GOOD first.
| ID | Pri | Check | Expected |
|---|
| H-01 | P0 | POST /v1/webhook-endpoints for HOOK_GOOD | Signing secret returned once and never again. |
| H-02 | P0 | Xero intent-to-receive against /webhooks/xero/<XERO_APP_A id> | Passes, and Check setup reports the webhook as verified. |
| H-03 | P0 | Record a payment in the Xero UI against an AUTHORISED invoice | HOOK_GOOD receives exactly one invoice.updated, one invoice.payment_changed, and one invoice.paid. |
| H-04 | P0 | Verify each delivery's AccountingKit-Signature against the exact raw body | Valid. t is within five minutes. |
| H-05 | P0 | Inspect the invoice.paid payload | Amount paid, amount due, status, both invoice IDs, provider revision, and document.refresh_required are present and correct. |
| H-06 | P0 | Re-send the identical Xero notification (replay from the Xero portal or by hand) | No duplicate customer event and no second accounting write. |
| H-07 | P0 | Post to /webhooks/xero/<XERO_APP_A id> with a wrong X-Xero-Signature | Rejected before parsing or persistence. |
| H-08 | P0 | Post a batch containing several events in one Xero delivery | Every event is processed; the response is fast enough for Xero's five-second deadline. |
| H-09 | P0 | Take the API down briefly, change an invoice in Xero, bring it back | The 15-minute reconciliation, or an explicit POST /connections/:id/sync {"scope":"invoices"}, recovers the missed transition with the same event contract. |
| H-10 | P0 | Force a sequence gap (skip a Xero event number) | The hole is recorded and reconciliation is queued only for active connections using XERO_APP_A. |
| H-11 | P0 | Partially pay an invoice, then pay the remainder | invoice.payment_changed on the partial payment; invoice.paid only on the transition to paid, and only once. |
| H-12 | P0 | After invoice.paid, download the PDF again | Shows the paid state. The pre-payment PDF is not reused. |
| H-13 | P1 | Void a payment in Xero | The resulting transition is delivered and is not silently swallowed. |
| H-14 | P1 | Register a second endpoint for a subset of event types | Only subscribed types arrive at it. |
| H-15 | P1 | DELETE a webhook endpoint | Deliveries stop; existing delivery history is retained. |
| H-16 | P0 | Repoint to HOOK_BAD, trigger an event | Retries are visible with attempts, response status, and next attempt; the delivery ends in a terminal state rather than retrying forever. |
| H-17 | P0 | GET /v1/webhook-deliveries?state=dead_letter | The failed delivery is listed. |
| H-18 | P0 | Repoint to HOOK_GOOD, then POST /v1/webhook-deliveries/:id/replay with a reason | Delivered. Original attempt history is intact. |
| H-19 | P0 | Replay with no reason or an empty one | 400. |
| H-20 | P1 | HOOK_GOOD responds slowly (about 10 seconds) | Timeout and retry behave as documented; no duplicate accounting side effect. |
| ID | Pri | Check | Expected |
|---|
| O-01 | P0 | Queue an automation; poll GET /v1/operations/:id | 202 on create, then a terminal state. input, attempts, and timestamps are populated. |
| O-02 | P0 | Restart the API while an operation is running | The lease expires and the operation resumes from its persisted steps; completed steps do not re-run against Xero. |
| O-03 | P0 | Force a Xero failure (invalid account code) and let retries exhaust | State reaches dead_letter with a readable terminal error. No partial document left in Xero. |
| O-04 | P0 | Fix the cause, then POST /v1/operations/:id/replay with a reason | Succeeds. Prior successful steps are not repeated. The audit log records reason, actor, previous state, and previous attempt count. |
| O-05 | P0 | Expire the Xero access token, then issue several concurrent requests | Xero's rotating refresh token is consumed exactly once; every request succeeds with the same new credentials. |
| O-06 | P0 | Revoke the Xero authorization at Xero, then call the API | The connection reports action_required; reconnecting restores service without duplicating the connection. |
| O-07 | P1 | Run each of the seven automation recipes once against XERO_DEMO | Each reaches succeeded and produces the expected Xero record. automation.succeeded is delivered. |
| O-08 | P1 | Force a recipe failure | automation.failed is delivered with a usable error. |
| O-09 | P1 | Queue many operations on one connection at once | The per-connection provider budget throttles rather than triggering Xero rate limits; watch for 429 in logs. |
| O-10 | P1 | Replay an operation that already succeeded | Refused or a no-op. It must not duplicate an accounting write. |
| ID | Pri | Check | Expected |
|---|
| U-01 | P0 | Sign in with Google, Microsoft, and an email magic link, then sign out | Every transactional email arrives and its links work against the deployed origin. |
| U-02 | P0 | Switch between the test and live environment in the console | Connections, keys, operations, and usage all change together. No live data under test. |
| U-03 | P0 | Create and revoke an API key | The secret is shown once with a copy affordance and an explicit warning. |
| U-04 | P0 | Register a Xero app, copy both setup URLs, and run Check setup | No credential is displayed again; OAuth, webhook, and scope status are accurate. |
| U-05 | P0 | Start a connect session, select the intended Xero app, and open the flow | The OAuth popup closes after success, accounting queries refresh, and the new connection workspace opens without a manual refresh. |
| U-06 | P0 | Trigger a sync and disconnect from the connections view | Both reflect true server state. |
| U-07 | P0 | Open the automations view with a failed and a dead-letter operation present | Both are visible and distinguishable. An operator can find a failure without database access. |
| U-08 | P1 | Invite a teammate and accept the invitation | Works end to end on the deployed origins. |
| U-09 | P1 | Load every view with zero data | Useful empty states, no spinners that never resolve. |
| U-10 | P1 | Search and filter in the connections and automations views | Correct results; no crash on punctuation or an empty query. |
| U-11 | P2 | Reload each view directly by URL, and use browser back | Deep links resolve; the persisted environment remains selected. |
| U-12 | P2 | Console at 1280px and at a phone width | No overflow or clipped controls. |
| U-13 | P0 | Start a session for an existing customer, authorize two Xero organizations, and choose the intended one | The customer ID is preserved, the organization choice is explicit, and no duplicate connection is created. |
| U-14 | P0 | Cancel the Xero consent popup, then open the same link again | The console reports cancellation, keeps the unconsumed link, and the second attempt can complete. |
| U-15 | P1 | Open a connection workspace in each status | Health gates, safe identifiers, relevant recovery action, workflow coverage, and recent operations reflect server state. |
| ID | Pri | Check | Expected |
|---|
| M-01 | P0 | Set the pilot contract with commercial:set-contract | The console usage view shows the correct included volume and currency. |
| M-02 | P0 | Perform a known count of billable calls, then re-read usage | Counts match. Idempotent replays are not double-counted. |
| M-03 | P0 | Generate a period statement with commercial:statement | Measured counts, allowances, overage, and total are correct and frozen. |
| M-04 | P1 | Re-run the statement for the same period | Identical output. The frozen result does not drift. |
| M-05 | P1 | Verify connection-limit enforcement at the contract's limit | Refused with a clear error, not a partial connection. |
| ID | Pri | Check | Expected |
|---|
| P-01 | P0 | Grep a full day of deployed logs for the Xero client secret, an API key, a refresh token, an authorization code, and a raw webhook body | Zero hits. This check is repeated after the first pilot week. |
| P-02 | P0 | Inspect the credentials table directly | Xero tokens are ciphertext at rest. |
| P-03 | P0 | Confirm a restorable database backup exists and restore it once into a scratch database | Restore succeeds. An untested backup is not a backup. |
| P-04 | P0 | Confirm alerting on queued-operation age, dead-letter count, Xero 401/429/5xx, OAuth refresh failure, and webhook delivery latency | Each alert has a destination a human actually reads. |
| P-05 | P0 | Correlate one request end to end using X-Request-ID | The header round-trips and appears in logs, the operation, and the audit record. |
| P-06 | P1 | Redeploy while traffic is flowing | No dropped operation; no duplicate provider write. |
| P-07 | P1 | Confirm the documented rollback for the deployed release | Written down, and the migration in this release is safe to roll back under it. |
| P-08 | P1 | Agree the pilot support path with the customer | Who they contact, expected response time, and how a dead letter is escalated. |
| P-09 | P2 | Re-run pnpm --filter @accountingkit/docs build and read the quickstart as the customer | A developer can reach a first successful call using only the published docs. |
- Every P0 check passes on the deployed service against a real Xero organization.
- Every P1 check either passes or has a written workaround the pilot customer has accepted.
- Phases 8 and 9 have been executed at least twice, on separate days, with a fresh connection.
docs/internal/pilot-cutover.md live acceptance gates 1 through 6 are satisfied by evidence from this
run, referenced by check ID.- The defect log has no open P0 or P1 entry.
| ID | Check | Pri | Observed | Expected | Request ID | Status |
|---|
| | | | | | |
Capture the request ID for every failure. A defect without one costs more to reproduce than it cost
to find.