Description
We're building a crawler/scraper that continuously collects listings-style data from many external sites and keeps it updated, storing it in a structured database that a programmatic SEO site is built on top of. You'd own both the crawling and the database structure. Two core goals: build it with AI, and make it run as autonomously as possible once live — minimal manual checks, minimal ongoing coding, fast to fix when it breaks. I want someone with hands-on experience who has actually built this. Be technical and concrete, and explain your reasoning — I'm comparing how different experts think. Please walk me through how you would solve the following: 1. AI in the build — central. Where exactly would you use LLMs (parsing, adapting to layout changes, classifying data) to make it self-healing and low-maintenance, and where would you avoid them as unreliable or too costly? 2. Reliability — keeping it from breaking every time a source changes layout. Self-healing selectors, schema detection, AI-assisted parsing. 3 Architecture — stack, frameworks, infrastructure for pulling structured data from many differently-built source sites. 4. Database — how you'd structure the stored data so it's clean, deduplicated, and ready for a programmatic SEO layer to build pages from. 5. Anti-bot — handling rate limits, IP blocks, CAPTCHAs, and JS-rendered content reliably and responsibly. 6. Data quality — accuracy, deduplication, and detecting when a source breaks or returns bad data. 7. Monitor