Fifty states, fifty different answers to the same question. One has a clean JSON API. One is an ASP.NET WebForms app from 2009 that only works if you post its own view state back at it. One is behind a WAF that will not talk to a data center IP. One will sell you the whole database on an FTP server for free if you know the password is in a PDF on their website.
There is no clever general solution to that. There is a per-source adapter, written after actually reading what the source does, and a schema at the end that all of them have to satisfy.
What we learned the expensive way
- Never wait on network idle. Half of these apps hold a connection open forever and the wait becomes a timeout that looks like a block.
- A prefix search returning zero results is not proof of zero records. Probe deeper before believing it, or you silently drop a state.
- Bounded pagination loops only. An unbounded one on a site that repeats its last page runs until the disk fills.
- Concurrency against a single-writer database is a lock storm, not throughput.