Bigger Data Downloads
Hello - I am a historian who is using FamilySearch to study 19th century African American communities in Appalachia. I have specific research criteria which returned just over 20,000 results. However, in order to download the search results, we are currently limited to 100 records at a time. After the 19th page of results, I received an error message that read "no results" and the 20th page of results would not load.
In the age of big data, will FamilySearch enable larger data downloads that can be exported to excel or other data software for faster processing by researchers? Does FamilySearch have guidance for how researchers can more effectively download state-level data?
Comments
-
You will have 2 problems here: throttling, and the overall limit on the number of exported entries per search (5000).
If you are searching Records, I would suggest you filter by collection (and, if you still can't get below 5000, filter by sex as well - but be careful to include those with sex=unspecified), and if you get throttled just move to another collection and retry the offending one where you left off, in a new browser session, later.
If you are searching the Family Tree, again you'll need to find a way of reducing the overall total on each search. I find waiting 2 seconds between pages removes any risk of throttling here.
2 -
FamilySearch has had to resort to throttling and other methods to reduce abuse of the records it preserves for the record holders. There are/have been parties who were working to download entire record sets to put them on another site.
I'm not suggesting your project is abuse, but we all have been limited when FS had to protect its holdings from those who were abusive.
2
