Source Box search is unusable -- it constantly refreshes results

RTorchia

Whenever I try to search the Source Box for anything, the search never completes. It'll show a page of results, then remove those for about three seconds, then flash them again, and the process repeats. This is regardless of the search and regardless of whether any of the additional filters are used.

It's probably important to note that after eight years here, I have 140,000+ items in my Source Box and no easy way to clean that out.

Paul W

The problem possibly does relate to there being such a large number of items in your Source Box. After eleven years using Family Tree I have 190. I was shocked to find I even have that many since - as I believe is the case for many other users - my main purpose in using this feature is as a temporary storage place for sources I am unable to attach to an individual at the time I discovered it. (Can't work out if they relate to one John Smith or another, source contains multiple names, etc.)

For me, I would find dealing with a Source Box like yours to be completely unmanageable and try to reduce the number of items, as well as being very sparing in its use in future.

Julia Szent-Györgyi

My source box is behaving normally. I have 479 items in it currently; like Paul, I use it primarily for (unindexed) documents that I haven't yet attached to everyone mentioned, so most of those items are things I simply haven't gotten around to cleaning up. (I think I've cleaned up all of the indexed stuff that it added the times when I didn't notice that Source Linker had decided to select that checkbox. I really have no clue what purpose it serves to add an indexed source to one's Source Box: it's only supposed to apply to one person.)

auntsue.genealogyfoundry

I have 295 items in the Home folder and 4968 in all folders. I keep them for easy reference to research I have done, and because sometimes the citations need updates and I want to be consistent in my formatting. For instance, the records from Vorarlberg, Austria, were first on microfilm rented from the FamilySearch Library, then online from the Vorarlberger Landesarchiv in Bregenz, and now at Matricula, Ancestry, and FamilySearch, with different metadata and URLs. I don't want to retrace twenty years of research to update those citations. No matter its creators' intended purpose, the Source Box is the most convenient long-term storage. My second choice is Excel spreadsheets, because I have not found genealogy software that cleanly transfers citations into the source fields at FamilySearch.

RTorchia

Well "add to Source Box" defaulted to On and was probably up to 50k or so before I even started using the Source Box, and even then it was innocuous because I only used it for sources I was adding to multiple people that day and it really didn't impact performance at all. Now that Search has been introduced and this apparently is a problem... well, the horses on that one have already set sail, or whatever that idiom is.

I'd be fine just having everything but the user-generated sources removed, and if that wasn't possible, I'd be fine having FS nuke the whole thing, but some past posts I've seen made it sound like that wasn't an option. So... they should just make it work.

DavidJohnson

Here is a link that will Explain what Sources do.

Remove sources from my source box • FamilySearch

Copy a source to my source box • FamilySearch

Create a new source in my source box • FamilySearch

Delete a folder from my source box • FamilySearch

Rename folders in my source box • FamilySearch

Organize my source box with folders • FamilySearch

Print source from Source Box • FamilySearch

How do I search my source box? • FamilySearch

Filter my source box • FamilySearch

Add a historical record to my source box • FamilySearch

What is my source box? • FamilySearch

I hope this help you out to

You can use the FamilySearch Solution Gallery hear is the URL Link Home (site.com)

RTorchia

Remove sources from my source box • FamilySearch

This is working... slowly, but it made realize a couple funny and horrifying things about some of these recent changes. (This is long... the important stuff is in the last paragraph.)

I didn't think manually emptying out my Source Box was an option because it used to be you could only do one page at a time, but they changed the design of the Box so that now it's all on one infinitely-scrolling page, and there's still a "check all" box at the top. But they didn't implement it like regular infinite scroll, where the content is loaded when needed, as you scroll. No, as soon as the Source Box is opened, it starts loading all the sources. All the sources. ALL of them. I watched as the scroll bar thumb kept shrinking as the page kept getting longer and longer. I don't think I've ever seen a single web page take up 5GB of memory and 20% of my CPU to load before, and it's mostly just text.

I get that the intent of the design was infinite scroll, but why would you preload everything? Why wouldn't you put a hard cap in place for like 500 or 1,000 sources -- if for no other reason than to prevent your servers from getting hammered every time somebody opens their Source Box? (If there is a limit already, it's too big for me to notice.)

Anyway, first I tried to preserve the sources I created by moving those to a folder before deleting everything else. I set the filter to show only user-created sources and let it work. After about 30 seconds, the site threw up the "Something went wrong" page and made me answer a CAPTCHA to continue. Just from letting the filter run. But the filter just wouldn't work -- it would just kind of choke and keep starting over. So I gave up on that idea and just decided to purge everything.

Here's the next problem: The Select All box at the top selects whatever the page has loaded so far, so not really "all". (No problem, I expected I'd have to delete a chunk at a time.) So I refresh the page and give it about ten seconds to load a couple hundred sources, then click the Select All box. (Since the list is constantly loading, the checkbox immediately turns from an X to – to indicate a partial selection.) Then click Remove, then the Yes button on the confirmation dialog.

And nothing happens.

The site is trying to delete a bunch of records while still continually loading more and update the list in the background and respond to the user interaction by removing the modal dialog and it just. can't. handle it. The dialog just stays up unless you click to close it. The deletion process is (usually) working in the background, but there's no indication to the user that anything is actually happening. The source list doesn't change, the individual source selection boxes don't toggle back to empty, and the Select All box seems like it stopped working. The only option is to refresh the page.

So I do that, and about one in every five times, I get hit with another CAPTCHA. The process is working, slowly. I've been at it for about three hours and I'm down to 65,000 sources remaining.

But the main realization I had was this: We're seeing a ton of people complain about getting Internal System Errors, CAPTCHAs, Error 15s, random logouts and other assorted functional failures. I think I know part of the reason why: the site introduced a new security system to prevent cyber attacks at basically the same time as added a bunch of new features -- like Source Box infinite scroll, Source Box filtering, and continuous updates for person pages that are open in the background tabs -- that trigger the exact kind of data spikes and abrupt surges in activity that the security system sees as red flags. The bot-like behavior Imperva is throwing hissy fits over is the site itself.

Julia Szent-Györgyi

I do wonder how many other users have super-inflated Source Boxes due to Source Linker deciding to select that checkbox.

I'm lucky: I've been using FS for most of a decade now, but it has only been in the past year or so that indexed sources have really entered the picture for three-fourths of both my and my spouse's relatives. (They're mostly Lutherans in Hungary. When the Catalog was last updated, two years ago, none of the Lutheran registers in Hungary had been indexed. Not a single one.) Therefore, I've been exploring all of the nooks and crannies of the source box feature since before most of its nooks and crannies existed, and I try to make a habit of keeping it at least semi-clean and organized. This means that every now and then, I need to delete a half-dozen source copies that Source Linker "helpfully" added, but that's well within the design parameters.

One thing I don't get: there is (or at least used to be) a limit on the number of folders one can create in one's Source Box. Perusing old discussions on the topic, I get the impression that we all basically assumed that if we could only have 50 folders, then there must also be a cap on the number of things in those folders. Now, granted, nobody ever came up with a number for that presumed cap, but given the behavior you describe with your overgrown box, it appears that the programmers shared our assumption. I find it rather shocking that we were all wrong.

RTorchia

Yeah, 3/4 of my branches were pretty source-heavy.

So my Source Box is finally empty. For anybody who runs into this situation, something I wish I had figured out earlier during this process: Open the Developer Tools on your browser (Ctrl-Shft-I) and view the Network panel. If the Console isn't also showing, press the Esc key to show it. It'll look like this:

The Console is the bottom section. These windows will let you track the progress of deletion while the UI plays possum. There's three stages to the process:

1. While the Source Box is loading the list of sources, a message like this will appear frequently in both panels:

GET https://www.familysearch.org/service/tree/links/folder/MM8G-QVC/sources/filter?readExternalData=true&resultsPerPage=50&pagesToSearch=25&startIndex=100

The number at the end is how many sources have been loaded so far. It will increment by 50 each message. I found that the number of sources you remove in one pass doesn't seem to affect how fast the server processes them, but the browser itself starts to gets bogged down if the source list gets too long. Around 15,000 is when I noticed the UI start to get really lethargic.

2. After you Select All, click Remove and Yes to confirm, a message like this will appear in the Console for each source being removed:

DELETE https://www.familysearch.org/service/tree/links/folder/sources/3BNC-MRG

So if you're removing 10,000 sources, you'll see 10,000 of these messages. This is your browser telling FamilySearch which sources to remove. The list actually gets sent relatively quickly.

3. Those messages are also added to the Network panel in a series of columns:

[Status] | DELETE | www.familysearch.org | 3BNC-MRG

Initially, that status column will be blank. When a green box with the number 204 appears, that's the server saying the deletion request for that source has been completed. This process takes much longer than it takes to load sources or to upload the list of deletion requests, and the whole time, the page itself isn't really doing anything to indicate the process is happening. Before I found this, I kept closing the browser window because I thought the process was frozen. When the complete list has been processed, the dialog will close.

I did find one definite bug. I'll describe it for the engineers below, but for everybody else, after a deletion pass completes, refresh the browser page before deleting another next batch.

The bug is that after a deletion pass completes, the Source Box resumes downloading the source list from both where it paused during the delete cycle and from zero. For example, say you let the source list load to startIndex=5000, select all and delete. When that finishes, the GET request resumes from startIndex=5050 and starts another series of requests from startIndex=50 -- it's two independent cycles and happens at twice the frequency as usual.

I also think there might be an issue with the source ID indexes in this this specific situation: if the Source Box has a lot of sources and doesn't finish loading the full list before running a deletion pass, so that populating the source list resumes after the deletion cycle completes, and then another block of sources is selected and another delete pass is run. In that situation, I'm worried that the source IDs that were purged on the first might not have been removed from the index used to generate the list used to populate the Source Box (i.e. the same IDs that were just deleted might get requested for download again), and also that when the second deletion pass is executed that some source IDs from the first pass might be sent again, either because they're still in some selection buffer, didn't get removed from the current active source box list, got downloaded to the source list again, or that the list of IDs sent to the server for deletion wasn't cleared after the first pass. This is all mainly a hunch -- some weird responses from the server I thought I saw but couldn't investigate because I'd already purged my source list.