Founding AI/ML Engineer, MyStage Music Inc

Where a scraped record goes before anyone can search it

Highlights6

Pipeline re-architecture: as founding AI/ML engineer, rebuilt a 14-Cloud-Function event pipeline as a single checkpointed LangGraph agent-worker on Cloud Run Jobs.
Scraping at scale: built and operated a Playwright service on Compute Engine that indexes 70,000+ records a day from 650+ locations, rotating proxies and backing off when sites push back.
Entity resolution: Gemini-based dedupe before records reach Firestore and Algolia lifted downstream search dataset accuracy by 25%.
Monolith to subgraph: refactored a 1,600-LOC Cloud Run service into a callable subgraph, cutting service dependencies 64%.
Search endpoints: FastAPI and Cloud Functions endpoints fronting Algolia for sub-50ms search latency and Firestore for real-time sync.
Safe concurrency: atomic Firestore claim transactions and deterministic task IDs for retry idempotency, covered by 200+ pytest unit and integration tests.

All 9 points

Highlights in full

As founding AI/ML engineer, re-architected a 14-Cloud-Function event pipeline into a single LangGraph agent-worker on Cloud Run Jobs - sourcing, extraction, resolution, and image fan-out now run as one composable, checkpointed graph.

Graph architecture

Factored out a reusable build_scraping_subgraph() factory compiled without a Firestore checkpointer so parent graphs compose it without nested-checkpoint conflicts; added route_entry bridge letting webhook-sourced scrapes skip URL-fetch and enter at extraction.
Implemented interrupt()/resume pause-for-research contract: subgraph idempotently enqueues a domain-research task on missing domain_metadata/{tld}, calls interrupt(f"domain_research:{tld}"), resumes from checkpoint when external pipeline flips parent task to ready.
Designed pure-function routing predicates (route_after_sourcing/extraction/resolution) over typed state for deterministic flow; used Send-based parallel image fan-out with per-Send error isolation via operator.add-reduced state field.
Refactored a monolithic 1,600-LOC Cloud Run service into a callable subgraph with history preserved, a full import-path rewrite, and dependency relocation - cutting service dependencies 64%.

Concurrency and idempotency

Wrote atomic Firestore claim transactions with status == "ready" preconditions for multi-worker concurrency; used deterministic agent-task IDs with create() + AlreadyExists for true retry idempotency (set() would clobber in-flight tasks).
Extended canonical Performance entity with next_reprocess_time / last_reprocess_time / reprocess_count; built Cloud Scheduler trigger emitting process-reprocess-event-data tasks and most-stale URL selection over entity_sources ordered by last_successful_scrape_time ASC.

Observability

Wired Logfire distributed-trace context propagation: scheduler captures root ctx per URL into agent_tasks/{id}.ctx, runner attach_context() on claim and resume so worker spans nest under producer trace across pause/resume; added per-tick metric_counter instrumentation.

Testing

Covered routing predicates, fetcher nodes, mocked Firestore claim transactions, idempotent enqueue under retry, HITL resume payloads and image fan-out isolation with 200+ pytest unit and integration tests; pytest-asyncio auto mode with mock_async_db fixtures for Firestore-free runs.