Resilient collection pipelines that move records past modern anti-bot walls — Kasada, Akamai, hCaptcha — and turn them into clean, deduplicated, structured data a business can actually use.
The valuable data sits behind walls: phone numbers revealed only on click, Kasada checking the browser fingerprint before it serves anything, hard per-IP limits, and hCaptcha puzzles mid-flow. A plain script is caught in seconds — and even if it gets in, one IP can pull only so much before the site starts hiding data.
Clients needed hundreds to thousands of fresh records a day, from several sources at once, clean and without duplicates — not a handful scraped by hand.
A resilient collection pipeline: get in unseen, pull at scale, come out as clean structured data.
Plain Selenium gets flagged instantly. I drive a real-fingerprint browser through AdsPower (over its API) or a recompiled Chromedriver with fingerprint noise and rotating User-Agents — so Kasada and friends see an ordinary visitor, not automation.
Sites cap what one IP can take (e.g. ~70 phone reveals a day). A counter watches that limit and, just before it, spins up a fresh browser, IP and fingerprint and keeps going — so the site never starts hiding data.
When a puzzle appears, a custom solver reads the image and computes the exact pixel offset — ~90% accuracy, square and circular variants. Off-the-shelf services were too slow and pricey at volume; this later became my open-source ReadyCaptcha.
A two-pass crawl (cheap catalog sweep, then targeted detail), a normalization layer (clean prices, country codes, dates) and three-level deduplication — auto-merge, merge-keeping-unique, manual for the unsure. Records batch-write straight into the client’s store: Airtable or a production AWS database.
What I took awayAll cases →Two things I now carry into every scraping project: deduplication is first-class work, planned from day one — not a cleanup at the end; and the “invisible browser” recipe (real fingerprint + recompiled driver + proxy rotation) is my default for any site with modern protection.