-
|
Running the scraper, and noticing that every time it runs, it gets all data from scratch again. I think there's some caching mechanisms, but when I run, it seems to always get all content. Is there a way to either:
|
Beta Was this translation helpful? Give feedback.
Replies: 2 comments
-
|
Most states don't offer any way to tell what items have been updated or sort by date, so there's no way to just scrape the new things. If you happen to be scraping federal data, you can add If you're running the full openstates stack with the database you can do a scrape, import it, then query the db by update date. If you're doing a scrape to local files ( Just scraping new items is something that seems like it should be an easy problem -- but states often don't flag metadata updates like title and summaries, or offer update-date based searching at all. Trying to solve it at the request level with something like cache-control headers or etags has stymied us in the past because many states' web servers don't respect the standards. |
Beta Was this translation helpful? Give feedback.
-
|
Aah makes sense. Thanks. I may try the pgdump instead then instead. I see a past discussion that mentions it updates every month. But does the current month update every few days? |
Beta Was this translation helpful? Give feedback.
Most states don't offer any way to tell what items have been updated or sort by date, so there's no way to just scrape the new things. If you happen to be scraping federal data, you can add
start=YYYY-mm-ddto your scrape command to filter by date.If you're running the full openstates stack with the database you can do a scrape, import it, then query the db by update date.
If you're doing a scrape to local files (
--scrapeoption) then I'd recommend saving an md5 hash of the files on your last scrape to something like redis and checking against that.Just scraping new items is something that seems like it should be an easy problem -- but states often don't flag metadata updates like title…