Background Jobs

SeedTrust runs two separate background job systems: a Flask cron server (legacy, scheduled tasks) and a FastAPI Huey worker (modern, event-driven tasks). Both are required for the platform to function correctly.

Flask Cron Jobs

The Flask cron server runs scheduled jobs using the Crython library. Jobs are triggered on a time schedule and call internal API routes authenticated with INTERNAL_API_KEY.

All production jobs are disabled in development environments (DEV_ENV != "1" check). They will not run locally unless that env var is set.

Job Inventory

Job	Schedule	What It Does	Danger if It Fails
DR Auto-Approval	Daily, 6am EST	Auto-approves disbursement requests where the `auto_authorize` window has expired. Sends approval notifications to IPs, IP Reps, Case Managers.	HIGH — DRs stay pending indefinitely; funds not processed; parties not notified
Case Close Reminders	Daily, 6am EST	Sends close reminders to Escrow Specialists on their scheduled case close dates	MEDIUM — Closures are missed in notifications; manual follow-up needed
Account Balance Check	Weekly, Sunday 1am EST	Runs minimum account balance check across all active cases	MEDIUM — Balance alerts not triggered; compliance issues go undetected
Account Balance Reminders	Every 3 days, 1am EST	Sends minimum balance reminders to Case Managers and Admins	MEDIUM — Balance warnings not delivered; reactive rather than proactive
Report Generation	Every 1 minute	Polls for pending report requests (status=1), processes oldest first, marks in-progress (status=2), generates report, emails to user (status=3)	HIGH — Report queue backs up; users never receive requested reports
AWS Status Check	Every 5 minutes	Tests S3/CloudFront connectivity; posts status to internal dashboard API	MEDIUM — S3 outages invisible on status dashboard
User Off-Site Tracking	Every 30 minutes	Marks all users as off-site; clears payment viewing cache (`ViewingDR`, `ViewingPayment` tables)	LOW — Session presence data goes stale; viewing cache grows

Code: seedtrust_flask/seedtrust/app_cron.py

FastAPI Huey Worker

The Huey worker is a separate process that runs alongside FastAPI. It handles event-driven background tasks — things that are triggered by user actions but should not block the API response.

How to Start the Worker

uv run st run worker
uv run st run dev   # starts worker alongside all other services
uv run st run dev --no-worker   # most developers don't need this

Storage Backend

Production: Redis (HUEY_REDIS_URL env var) — confirmed deployed in production, enables priority queuing
Local / Fallback: SQLite at ~/.seedtrust/huey/huey.db

Registered Tasks

Task	What It Does	Risk if It Fails
`PROCESS_HUNTINGTON_BATCH`	Processes Huntington bank transfer batches (ACH/wire transfers to surrogates)	CRITICAL — Bank transfers stuck; funds not disbursed
`auto_heal_huntington_batches_periodic`	Runs every minute. Reconciles false-failure Huntington batch transactions — marks as completed if the underlying transfer succeeded despite a worker timeout	HIGH — Failed transfers incorrectly remain as failed; reconciliation breaks

Example and test tasks (EXAMPLE, EXAMPLE_ERROR, EXAMPLE_PROGRESS, EXAMPLE_WITH_ARGS) exist for development testing only.

Job Lifecycle

Every task run is tracked in the JobRun database table:

Task queued → PENDING
Task picked up by worker → STARTED (started timestamp set)
Task completes successfully → COMPLETED (result stored)
Task raises exception → ERROR (error logged to Errors table, PostHog notified)
Task timed out in queue → EXPIRED
Task manually cancelled → REVOKED

Failed tasks support exponential backoff retries: up to 2 retries with a 2-second initial delay (max 8 seconds).

Monitoring Jobs

Check job health via the JobRun table. Alerts to watch for:

resolution = ERROR count spiking
status = PENDING records accumulating (worker may be down)
EXPIRED tasks (queue is backed up)

All Huey errors are also captured and sent to PostHog for observability.

Code: seedtrustapi/src/seedtrust_huey_consumer.py, seedtrustapi/src/workers/

What Breaks if Jobs Are Down

Severity	Impact
CRITICAL	Huntington transfers stuck, DR auto-approvals stop, report queue backs up
HIGH	Huntington batch reconciliation breaks, case close reminders missed
MEDIUM	Balance alerts not sent, AWS status stale, account checks skip
LOW	Session presence stale, viewing cache grows over time

Gotchas for Developers

Flask cron jobs don’t run locally by default. They check DEV_ENV != "1" before executing. If you need to test a cron job locally, you must either set this env var or call the underlying job function directly.

The Huey worker must be running for Huntington payments to process. Without it, PROCESS_HUNTINGTON_BATCH tasks queue up in Redis/SQLite but never execute. Payments will appear stuck.

job_dr_approval is the Flask auto-approval path, not the FastAPI auto-approver. These are two separate systems: the Flask cron handles the auto_authorize date-based approval, while the FastAPI auto-approver module handles rule-based bulk approval. Both must be functional for the full auto-approval feature to work.

Never add new scheduled jobs to app_cron.py. It is 66KB of unstructured procedural code. New background tasks go in the Huey worker (seedtrustapi/src/workers/).

Open Questions

1. Flask cron monitoring There is currently no alerting if a Flask cron job fails or if the cron server crashes. If job_dr_approval or job_report_generation silently fails, the team will not be notified. Incident detection for cron failures currently depends on user reports or manual log review.