CAPABILITY / 09

Data pipelines & scheduling

In production

Data that arrives on time and jobs that say when they do not: scheduled ingestion, reconciliation and backups.

Updated 15 Nov 2026

01 / PROOF

The measured part

Weekly
events bundle, Sunday 21:30 UTC on StrikeHike
Source
10 min
in-process reconciliation with a soft lock on the Job Portal
Source
Gated
events bundle: quality gate, health check, rollback to the previous bundle
Source

02 / WHAT I DO

In practice

Schedule ingestion and builds, and make a failed run visible.
Keep jobs safe to retry with a soft lock.
Reconcile records with a lock and an independent backup path.
Use workflow schedulers and in-process timers where each fits.

03 / WHERE I'VE DONE IT

The systems behind it

StrikeHikeWeekly events bundle, gated and rolled back on a failed activation.StrikeHikeFRED macro ingestion runs on demand into Parquet.StrikeHikeThe weekly digest is sent manually after approval, with a dry run by default.RVM CAD Job PortalPayment reconciliation every 10 minutes, with an hourly CI backup.RVM CAD Job PortalRecordings and transcripts filed to Google Drive with metadata (my statement, 15 Nov 2026).BOVEN TECHNOLOGIESData pipelines and scheduling in my roles (self-reported).

04 / STACK

What it runs on

Python
FastAPI backend, ingestion and analytics
See the system
GitHub Actions
Test, build and deploy pipeline
See the system
Airflow (in my roles)In my roles
Data pipelines and scheduling in my roles
AWS ▸AWS Glue (in my roles)In my roles
AWS Glue in my roles (self-reported)
Azure ▸Azure Data Factory (in my roles)In my roles
Azure Data Factory in my roles (self-reported)
PostgreSQL
Shared relational store
See the system

05 / LIMITS

What this does not claim

No Airflow production cluster is linked yet; Airflow is from my roles.

06 / RELATED CASE STUDIES

All work

ProductionStrikeHike

Live at strikehike.com; this engine replaced the first version on 25 Oct 2026. No accuracy or usage figures are claimed.

StrikeHike: a macro and planetary research terminal, built and run solo

A live research site that sets the US macro record beside the planetary sky, with tested claims, a weekly roundup, a public ledger, upcoming dates, and recession and decline histories. Next.js and FastAPI. Research, not advice.

131 of 131 commits on its main branch are mine

Built and testedRVM CAD Job Portal

Merged with 616 of 616 tests passing at 48c3cab1 (23 Jun 2026). RVM approved describing the behaviour on 15 Nov 2026; production rollout state was not independently verified.

Interviews that schedule, record and file themselves

One interview, end to end in the Job Portal: a Google Meet space and invites, a consent gate before recording, a recording state machine, and Drive filing with metadata. Built in one commit, with 616 of 616 tests passing at that commit.

70 files in the feature commit

07 / QUESTIONS

Short answers

What runs on a schedule on StrikeHike?

The weekly events bundle runs Sunday 21:30 UTC. The digest is a manual approve-then-send workflow with a dry run by default; FRED ingestion runs on demand.

Why move reconciliation into the app?

The GitHub Actions cron drifted to gaps of two to three hours. The Job Portal now reconciles every 10 minutes in-process with a 60-second soft lock, and keeps the CI job as an hourly backup.

Need this on a real system?

Hiring for a role, or scoping a short engagement? The Hire page has the role families and contact; Consulting has the scoped offers.

For hiring managers Consulting