AI extraction pipeline for a luxury hotel-offers platform
2023 to 2026
Lead engineer on the extraction system. I took the product from data discovery and vendor evaluation through cloud setup, then built the pipeline and its tooling.
The problem
Luxury hotels sell packages and experiences on their own websites, and those offers change constantly. Entering them by hand took weeks per cycle, and some offers expired before anyone could type them in.
What I built
- A Python and Celery pipeline that crawled 1,200+ hotel properties across 130+ domains and used the Gemini API to extract structured offers.
- Multi-stage extraction: content-quality gating, page-type classification, traversal from list pages into each offer's detail page, schema-constrained prompting, and staged validation against hallucinated output.
- A second Gemini stage that scored every offer on six quality metrics, against rubrics defined with product stakeholders, citing the evidence behind each score.
- The React and TypeScript operations platform where the team managed, cloned, and test-ran 1,950+ crawl configurations, tracked live crawl progress, and reviewed extracted offers against a markdown view of each source page.
- A re-architecture of the single-stage Django/Celery pipeline into fetch and process stages on separate workers.
- A rebuilt Django ingestion for the vendor feed of hotel property data, hundreds of thousands of PostgreSQL records, using bulk writes and removing N+1 queries from the main write paths. A Celery handover passed an overrunning sync's unprocessed IDs to the next run, so update bursts of 8,000+ properties (8x the vendor's stated ceiling) no longer stacked up parallel syncs that blocked other tasks.
- A written make-vs-buy analysis that moved crawling to two scraping vendors, plus a provider-neutral fetch layer over them, hosted on DigitalOcean and Cloudflare.
- offers extracted for the public launch in mid-2025
- 15,000+
- CPU after the fetch and process split, with both workers on cheaper tiers
- 100% → <50%
- below equivalent AWS pricing for the hosting (estimated)
- ~30%
- per property for the vendor sync after the ingestion rebuild
- >1 min → seconds
Built with Python, Django, Celery, PostgreSQL, Redis, Gemini API, React, TypeScript, Next.js, DigitalOcean, Cloudflare