Deeka's Devlog

Expense-Trackers

3 min read

Implement Android SMS ingestion and replaceable two-stage AI drafts

Summary

Refined the Android SMS import architecture, saved its Mermaid diagram and seven-phase delivery plan, and implemented manual ingestion plus the Phase 4 backend draft pipeline. Documentation was added in docs/research/android-sms-transaction-import.md, android-sms-ingestion-implementation.md, and android-sms-phase4.md. Android gained native opt-in/permission settings, manual Scan now, cached global sender filtering, a Room upload queue with atomic checkpoint commits, durable retries, cancellation fencing, encrypted device-token storage, and backup exclusions. Backend migration 0016 and SMS APIs added revocable scoped device tokens, consent versions, maintainer-managed senders, durable acceptance, account-scoped fingerprint uniqueness before processing, sync metadata, and seven-day raw-payload retention. Web Settings gained device-token creation/revocation. Migration 0017 and backend/internal/smspipeline added separately injectable Verifier and Extractor strategies, independent OpenRouter adapters reusing the existing key/model, strict completeness validation, leased database claims, stage-specific retries, stale-result/consent fencing, and atomic unique draft persistence. Owner-scoped job/draft listing and failed-stage retry endpoints were added; no approval endpoint or automatic ledger writes were introduced. SMS AI remained off by default, enabled through SMS_AI_ENABLED with an optional SMS_AI_MODEL override. Prompt/output boundaries, deterministic evidence checks, bounded provider requests, safe error codes, payload cleanup, and exclusion of SMS request bodies from crash capture were implemented. Go tests/vet, focused PostgreSQL SMS/provider/configuration race tests, migration round trips, 340 web tests, production web build, and Android unit/build/lint checks passed in the transcript. Phase 3 emulator instrumentation passed; it was not rerun for Phase 4 disclosure-only Android edits. Broad API regression passed with TestArchivedReserveSpendingCannotBeDeleted excluded after that failure was reproduced on unchanged baseline. Physical-phone SMS access and consented live-provider validation remained pending; tests used synthetic fixtures/stubs and no paid calls. Phase 5 review/edit/approval and final-save integration, Phase 6 daily/foreground automatic triggers, and historical-rescan UI remained unimplemented. The sender list started empty. Separately, Herdr Shift+Enter was configured to send Ctrl+J for newline entry. This logging pass reviewed transcript evidence and current changes without modifying project files; git diff –check passed.

Decisions

  • Used a global maintainer-managed sender allowlist cached on Android and rechecked by the backend; users controlled only opt-in. Empty configuration failed closed.
  • Used the local SQLite queue for crash recovery and backend unique user/fingerprint records for authoritative deduplication before AI. Checkpoints meant safely queued locally, not uploaded or processed. The agreed future scheduling design combined daily WorkManager scans with foreground catch-up, but only manual triggering was implemented.
  • Kept verification, extraction, orchestration, persistence, and deterministic policy separate behind provider-neutral Go interfaces. Reused OpenRouter configuration, not the existing natural-language transaction parser.
  • Required a strict yes/no completeness decision with explicit dated evidence, amount, counterparty, direction, and event type. Incomplete or ambiguous messages were rejected rather than sent to a needs-review queue; extraction retries did not rerun verification.
  • Mapped explicit card transactions, including debit cards under the chosen policy, to credit; other or unknown methods defaulted to cash. Settlement-like events became cash drafts with mandatory warnings, not settlement links. Sections remained optional AI suggestions; learned history matching was deferred.
  • Stored basic editable transaction fields in a separate source-linked draft table rather than cloning all ledger constraints. Only future explicit human approval would create ledger transactions.
  • Kept seven-day raw-payload expiration active even while AI processing was disabled; consent changes could discard in-flight results but could not recall requests already sent.

Learnings

  • Database uniqueness prevented duplicate drafts but could not prevent repeated provider billing after process death or lease expiry; stage claims and stale-token fencing were needed for safe recovery.
  • Existing Node/jsdom storage tests required NODE_OPTIONS=–no-experimental-webstorage on this installation. The unrelated reserve integration failure required baseline reproduction rather than being attributed to SMS work.
  • Migration regression checks exposed a rollback index/column dependency that was fixed before the successful round trip.
  • Workflow review identified Android scan/queue/consent coordination as the largest qualitative engineering effort and repeated builds/emulator runs as tooling overhead; reusable fake-source/backend tests and a unified verification script were proposed but not implemented.