Incremental Synchronization

Maintain one checkpoint per account, resource, and filter set. Deposits, withdrawals, and ledger records do not share cursors.

Processing loop

1. Start at after_id=0 or the saved cursor.
2. Fetch a bounded batch.
3. Upsert records by their stable resource ID.
4. Save next_cursor after item persistence, preferably in the same database transaction.
5. Continue while has_more is true.
6. Resume at the saved cursor on the next poll.

Cursor feeds discover newly created records. They do not emit an existing deposit or withdrawal again when its status changes. Persist non-final records in a work queue and poll each detail endpoint until terminal.

The ledger is append-only and can be synchronized entirely by cursor. If a filter changes, initialize a new checkpoint instead of reusing the old one.

Make consumers idempotent so replaying the last batch after a crash does not duplicate business effects.

Worker separation and observability

Use one discovery worker for each stream and a separate status worker for non-final records. Discovery advances monotonically; status workers retry individual records without blocking new records behind one slow transaction.

Record cursor lag, last successful poll, batch size, non-final queue age, detail-poll failures, and terminal transition count. Alert on a stale cursor or an old non-final record, not only on HTTP failures.

Perform initial backfill with the same idempotent pipeline used for ongoing polling. A separate one-off importer often produces different deduplication behavior.