Find-By-Name results contradicted by Profile results

LegacyUser

Adrian Bruce said: Using menu option FamilyTree / Find / Find By name, I searched for Name = Charles Charity, Birth year range 1823 - 1823

The results are on https://www.familysearch.org/tree/fin... and look like this:


The one I'm looking for is PID LYRC-NVT, the 2nd one down.

Notice please that 'NVT's burial details are:
Burial
29 September 1870
Stockport St Peter, Cheshire, England, United King...

Now go to that profile and these are the burial details:


And hit the edit link to reveal the Standard and Display values:

So on his profile, he's buried in St. Peter, Elworth, Cheshire, England, United Kingdom

The "People Found in the Shared Family Tree" value says that he was buried in Stockport, Cheshire, the profile says that he was buried in Elworth, Cheshire.

The values are supposed to match. Why don't they?

A possible clue? I edited this guy last night - the burial place (display and standard) originally said "Elworth, Cheshire, England, United Kingdom". I "decorated" the display name by adding "St. Peter" on the front. During this process, the suggested value in the drop-down list was "Stockport St Peter, Cheshire, England ... etc". I have no idea whether I ignored it totally or accidentally accepted it - if the latter, I clearly altered it immediately after to a display name of "St. Peter, Elworth, Cheshire, England, United Kingdom".

So "Stockport St Peter, Cheshire, England ... etc" is the best guess that the standardisation system has for "St. Peter, Elworth, Cheshire, England, United Kingdom". Does this tell us anything about how the failure occurred? No idea...

Nvpattt

If you have feedback, post them at the link below.

https://community.familysearch.org/en/categories/familysearch-community

LegacyUser

Jordi Kloosterboer said: That is weird! On another point, you might want to rename it to Saint Peter's Churchard, Elsworth, Chesire, England, United Kingdom to disambiguate it more. And then request to add that place on FamilySearch Places as a burial ground.

LegacyUser

Adrian Bruce said: That can come later - right now I'm leaving it alone in case someone wants to check it through.

LegacyUser

Jordi Kloosterboer said: I tested this with other values and it appears that when you type in a place that does not exactly match it's standard place name, then it will show the first option regardless if you type click on another standard here are some pictures of another example (I just made up the display name to demonstrate this idea on my test person):

LegacyUser

Jordi Kloosterboer said: The standard options to choose from are three: Apeldoorn (the town), Apeldoorn (the municipality), and the one I chose. It displays the first one (apeldoorn the town) when I search for similar people...

LegacyUser

Jordi Kloosterboer said: good point.

LegacyUser

Jordi Kloosterboer said: Either case, engineers should change it to show the actual display value instead of a standard place..

LegacyUser

Adrian Bruce said: OK - this is getting ridiculouser... Suppose I use menu option FamilyTree / Find / Find By ID instead of Find By Name and enter the PID in question for Charles. This is what I see:


Burial is
29 September 1870
Elworth, Cheshire, England, United Kingdom

This is correct (i.e. it matches the profile - though it's the standard value rather than the display). Since both bring up the "People Found in the Shared Family Tree", one would expect Find By Name and Find By Id to give the same answer for the same profile. Clearly they don't. One is right - the other isn't.

And I wonder if Find By Name is standardising on the fly?

LegacyUser

Jeff Wiseman said: Similar type of problem in the source linker:
https://getsatisfaction.com/familysea...

When you compare ANYTHING with a recorded conclusion in the FSFT, you need to see the CONLCUSION in the comparison. It is ALWAYS about the conclusions. Everything that we are doing here is about improving the conclusions in all of the records. This is the ultimate goal. That's what we are SUPPOSED to be focused on.

If FS wants to ALSO show what the standard value is that is being used to standardize a display name, that's fine, add it too. But the display name is more important, more detailed, and far more useful in the overall scheme of things we're doing here. Don't replace the actual useful name with the "almost there" location name.

Standard names might be, but are not always conclusions. The Display names are always the conclusions.

LegacyUser

Adrian Bruce said: And this is becoming a black-hole of confusion. I only just noticed that previous screen shot had a mother Elizabeth Charity, M6NJ-8WV. Where did that come from given that it's not on the profile and not in the find by name?

I've no idea and what's more - I went to that M6NJ-8WV profile (she is indeed married to a John Charity, but one with a different PID) and then came back, refreshing my Find By PID - and she's gone!

LegacyUser

Lundgren said: The Tree search results localize the places into the language of your browser.

In doing this they run the original strings through the place standardization code.

You can see that here:

https://www.familysearch.org/research...

It then takes the result of that standardization and localizes it to the language of your browser, and displays it.

If you take the burial place and run it though the place standardizer here:
https://www.familysearch.org/research...

You will see that it is standardizing it to Stockport St Peter, Cheshire, England, United Kingdom

This behavior (for now) is unique to the Tree search display. (It is also not a new behavior, but has been in place for many years.)

LegacyUser

Adrian Bruce said: Lundgren - thanks for taking the trouble to respond.

However - those actions are surely nonsensical (so I'll assume that you didn't design them!) Further, you say "This behavior (for now) is unique to the Tree search display" as if the behaviour might be extended elsewhere in future....

It's a nonsensical behaviour because the locations have already been standardized. There is no need to take the original strings (what I term the Display name) and run them through the place standardization code. Just take the standardized value from the profile, and localize it to the correct language using the standard-place-name data.

Yeah, if there is no Standard on the profile then it has to work as you describe.

It's also nonsensical because on-the-fly standardisation of the original, display value with no human intervention simply doesn't work. If it did, we wouldn't need to ask the human to make a choice. As it is, all it does is take the first on the list and that is often wrong due to spelling variants, etc.

Please tell me that at some point the routine will use the already standardised value from the profile and not corrupt it (visually) by putting the display value through the place standardization code.

(I'm not that happy about the attempt to show the standard value rather than the display but let's set that aside for today.)

LegacyUser

Adrian Bruce said: To give you a flavour of the issues with on-the-fly standardisation of the original, display value with no human intervention...

It turns out that the church I want is already standardized as "Saint Peter Church Elworth, Elworth, Cheshire, England, United Kingdom".

However, if I type "st. peter, elworth, cheshire, england, united kingdom", the first option that comes up is "Stockport St Peter, Cheshire, England, United Kingdom" (as you suggest).

Presumably "st. peter" is a better match for "Stockport St Peter" than it is for "Saint Peter Church" (highlighting just the presumably significant bits). Nobody can be expected to keep track of standards for name components at such a low level as "Saint" v. "St".... And yet it sunk this auto conversion.

LegacyUser

Lundgren said: The place text that is added by the user is run through standards on the way out to allow the system to convert the human readable text into a place representation that the computer understands.

That has to happen for the system to covert it into text that the computer does not understand in the form of a localized language.

If this place were in standards, then you would not have seen the difference. If the Tree website did not allow the user to put a place in that does not standardize, then you would not have seen this. However, it would also mean that you could not enter any place that does not exist in the standards places. (That isn't desirable either...)

That would also only work for data from the tree website, users can also import gedcomx and sync with third parties to add data.

The standardized data set is a constantly moving target as more places are added to the data. The place in this burial location could be a real location, (I don't know) if it is, once it is added to the standards, then it will work as expected.

I believe that it would be desirable for the user to be able tot see the original value as well as the standardized values. (That isn't being shown today in the UI.)

LegacyUser

Jordi Kloosterboer said: And FamilySearch could implement something like this: https://getsatisfaction.com/familysea...

LegacyUser

Tom Huber said: Someone could ask that the standards teams join in in this discussion. There may be something in the works that is impacting the strange behavior.

LegacyUser

Juli said: The process Lundgren is describing simply makes no sense. It's ignoring the selected standard and trying to start all over again with the display name? How is that ever going to work? Why are we even bothering to choose a standard?

You can't have your cake and eat it, too: either you display the human-entered placename, exactly as entered, or you translate the standardized name.

LegacyUser

Adrian Bruce said: Lundgren,
Something is awry somewhere - either the algorithm that you describe is pointlessly duplicating work that's already been done (and getting it wrong in this instance) or my precise understanding of what data is being taken from where, is wrong.

I'm going to continue this in a Reply just to stop everything being collapsed when Comments are folded up.

LegacyUser

Adrian Bruce said: Lundgren (continued from above),
Something is awry somewhere - either the algorithm that you describe is pointlessly duplicating work that's already been done (and inevitably getting it wrong in this instance) or my precise understanding of what data is being taken from where, is wrong.

I think that the situation can be summarised in the question:
Why does FamilyTree / Find / Find By ID get the displayed standard place-name right but FamilyTree / Find / Find By Name gets it wrong?

What I believe the Profile "record" holds for an event is:
- The original place-name text that I enter (what I call the Display place-name);
- The machine-readable key of the Standardised place-name (as agreed by me and the system) (that has to be there otherwise the Standard place-name wouldn't be fixed);
- And possibly the textual place-name of that Standardised place-name (together with its language?)

The textual place-name of that Standardised place-name wouldn't be there in a normalised, relational database - it'd be looked up each time it was needed. Since FamilyTree isn't such a database I guess it might cache the last used place-name and language. Not sure it matters.

Now, when I run a FamilyTree / Find / Find By Name, it finds and lists the "People Found in the Shared Family Tree" that satisfy the selection criteria. It displays the name and basic details of the Vital Events - specifically the date and place-names. I have assumed that the dates and place-names come from the Profiles for the people. You said "they run the original strings through the place standardization code ... ". But if (if) we are talking about the Profiles then this is a pointless, duplicate step because the machine-readable key of the Standardised place-name is already on the Profile and it only needs to be used as a look up (in conjunction with the current language) to get the Standard text to be displayed.

Presumably FamilyTree / Find / Find By ID goes to the Profile in order to get the right standard place-name, via the machine-readable key???

What I am wondering is whether
(a) FamilyTree / Find / Find By Name uses a set of index data that doesn't include the machine-readable key of the Standardised place-name but only the original place-name text (the Display place-name); and
(b) it doesn't go to the Profiles to pick up the machine-readable key but attempts to reproduce it. Which won't always work - as here.

Grateful for any help in identifying the issue.

PS - yes, I've ignored a few exceptions like how the system deals with Vital Events that have a Display Place-name but no machine-readable key for a Standard Place-name. But they should be exceptions, they shouldn't drive the whole algorithm.

PPS - Again, to repeat myself, I suggest that it is beneficial to ponder "Why does FamilyTree / Find / Find By ID get the displayed standard place-name right but FamilyTree / Find / Find By Name gets it wrong?"

LegacyUser

Gordon Collett said: It does sound like this method of creating the place name to display needs to be fixed. Since Lundgren states that this "is also not a new behavior, but has been in place for many years," it must be old code that has been overlooked.

Each historical time period for each place in the Places database has a unique ID number:

Elworth, Cheshire, England, Unknown to 1801 = 10612097

Elworth, Cheshire, England, United Kingdom 1801 to Today = 2975950

Stockport St Peter, Cheshire, England, Unknown to 1801 = 10616103

Stockport St Peter, Cheshire, England, United Kingdom 1801 to Today = 2975931

None of these have display names in any language but English.

I would have assumed that what is being stored as the actual data for the Standardized Value for the place name is the ID number. Then in order to convert the Standardized Value from one language to another, all the program needs to do is take that ID number, check the Places database to see if there is a display name in the browser's language and display that version as appropriate.

The old routine is really outdated if it is taking the user entered name, running it through the standardizing menu and always picking the first of potentially dozens of choices on the list even though someone has already clearly and unambiguously declared that that choice is wrong by assigning a different ID number as the correct choice.

LegacyUser

Adrian Bruce said: Just to orientate things - Gordon's ID number is what I have referred to as the "machine-readable key" above.

Incidentally, it's probably not particularly odd that it's been overlooked. I think many of us have the opinion that most people set the "Display" text to the Standard text and leave it at that. They won't see any issue.

Also, even if the "Display" text is different from the standard, it won't cause any issue if the first item on the drop-down of suggested items is the desired one. I suspect that is often the case.

The problems possibly arise when the decoration on the front partially matches another standard - for instance, would "Nantwich Road, Crewe, Cheshire, England, United Kingdom" match "Nantwich, Cheshire, England, United Kingdom" or "Crewe, Cheshire, England, United Kingdom" (the correct one)? (The place-name utility suggests Nantwich is first choice but that may not apply in FamilyTree.)

LegacyUser

Gordon Collett said: This is a consistent, easily reproducible error.

I purposely set the Standardized value for a birth place to the fourth choice in the drop down menu:



Find By Name shows neither the user entered display name nor the declared standardized value, but rather the first choice in the drop down menu:



Find By ID does show the correct, declared standardized value:



I would categorize this as a major bug that needs to be fixed.

My preference would be to have the actual place name, not the standardized version, listed but if it has to be the standardized version, at least let it be the correct one.

LegacyUser

Adrian Bruce said: Thanks for working out how to reproduce it, Gordon.

LegacyUser

Juli said: Lundgren implied above that this error exists because the system is trying to translate placenames. I'm not sure that's really a good idea, but if FS insists on it, it absolutely and definitely MUST apply only to the standard name. That is, as Gordon described, the system should check the places database for the place ID that the chosen standard points at and see if there's a name for the desired language. (Most times, there will only be a translation for the highest jurisdiction: "Balassagyarmat, Nógrád, Hungary" versus "Balassagyarmat, Nógrád, Ungarn". [The town was sometimes called Jahrmarkt in German, but I haven't entered that in the database.])

It really is a black-and-white choice: either it displays the display name, exactly as entered, or it displays a translation of the (human-chosen) standard. In my experience, the first choice in the drop-down is very often not even on the right continent.

LegacyUser

Lundgren said: Thank you all for your comments, this is not a simple problem and that is being exposed here.

One issue is this: The standardized place data set is constantly evolving as more places are added to the data. The place service is also actively being developed, like all software it has bugs being fixed and occasionally added and later corrected.

This means:

If a place like "North of Dot, Durango, Mars" is not in the standardized data when a user adds it as a birth place, it could be assigned place corresponding of Dorado, Mars with id of 55211,7.

If that same place "North of Dot, Durango, Mars" may later be added and assigned a place id of 33422, 271221,7.

If later on a person searches for "North of Dot, Durango, Mars" it would then use the place id of 33422,271221,7. It would not match the record that is assigned the standardized value of 55211,7.

Because of this, the search and hinting systems do not currently use the standardized place ids created when a person is entered. They use the text version of the places the user adds and standardize it with a static long-term supported version of the place standards. That way when "North of Dot, Durango, Mars" is added to the search data it will have the place ID as all of the other times it has been added before for the duration of the current search/match data.

When a user supplies a search place, it takes that text value and runs it though the exact same long term standards that the records (or tree person) was standardized with. That guarantees that the same string will always return the same place hierarchy. (North of Dot would always get 55211,7 when standardized, and when supplied as a search term. While this will get some possibly incorrect values, it will also get correct ones.)

Tree is only one part of the data that is made searchable. Additionally indexed records and uploaded genealogies are handled the same way. Each data set has it's own issues. It is not uncommon for non-places to be stored in places as well.

The long term support place data is updated at a fixed rate. All of the search and hinting data is also re-generated at a regular frequency.

When tree search results are standardized for localization "on the way out" the "active" place standards system to standardized the place provided by the user.

LegacyUser

Lundgren said: In some cases (and more in the future) the search back end will provide both the standardized and original values to the UI teams. The UI teams have to decide what they display in the limited resource of your monitor/phone to provide good information without overwhelming. (I would like a way to see the original text and the localized values.)

There are cases where displaying just the original values is too difficult for the user to read.

Assuming you can't read/write Japanese:

Imagine if all of our search results returned values to you in Katakana. We may have records for Japan with your ancestors, that we can search and show to you, but if you can't read the characters, it will be very difficult to sort through the mass amounts of data you can't read to find the bits that you have managed to learn.

If we localized the Katakana places and dates to the English equivalent, then you can narrow things down much more quickly. This doesn't eliminate your need to understand some places that cannot be localized, and it doesn't eliminate your need to read some of the original records, but it helps you get to those records much more quickly.

My ancestry is Swedish and English, so I don't face this, but people who can no longer read even the characters of their ancestors language do. This becomes very apparent if you change your language on the site to any non-Roman language.

LegacyUser

Lundgren said: Gordon, would you mind providing some links to this?

Thank you

LegacyUser

Lundgren said: Some of your assumptions above are correct, some are not, and some I do not know how it is implemented.

This much I can tell you:

Find by ID doesn't actually use the search system or any of the same APIs. It does a direct retrieval from the tree data.

The Find by Name/values does a search of data indexed from the tree data. The API that the find by name uses is exposed to third parties to allow them to search the familysearch tree. The search system does not go back to the tree data to provide the search results.

You can think of it a little bit like the old card catalog, enough information is written on that card to display meta data, but not the entire book, referencing the original book for every summary card is too slow, so you just get the data on the card in the summary, every time the book changes materially, the card is replaced with the critical bits.

The original place as well as the standardized place (from the "constantly changing") standard system are stored in the tree. I explained how and why the long term standard place ids are used in search in another comment in this thread already, so I won't repeat that here.

Hopefully that helps.

PPS answer: They come from a different place that use the data differently. For find by name/data uses it to allow for best retrieval with changing place standards.

LegacyUser

Adrian Bruce said: Thank you. I think....

"this is not a simple problem" - nor is it a single problem as it calls into question the whole issue of whether or not original / display place-names should ever deviate from the standards; plus whether the standard place-name teams can resource all the updates they should be making if the expectation is that all place-names should be on that file.

LegacyUser

Adrian Bruce said: Lundgren, you refer to volatility of the place-name data. The problem is that the design of FS has exacerbated that volatility by allowing us to add stuff to the front of the place-names.

If I'd just left the event untouched, it'd have a Display (original) and a Standard place-name of "Elworth, Cheshire, England, United Kingdom". Searches would be done with (the equvalent of) Elworth (assuming Elworth had been there a while). That sounds good to me.

As it is, if I use the full church name of "St. Peter, Elworth, Cheshire...." in a search, searches are effectively being done with (the equivalent of) "Stockport St Peter, Cheshire..." finding both the Elworth data and Stockport data.

I am now rethinking entirely whether I should be doing any decoration of place-names or whether I should confine myself to just using the standard place-names. This would be a total waste of programming effort put into allowing Display and Standard to alter. It will also increase the workload on the Standard Places team because now, to get precise place-names, I'm going to need Standard Places for everywhere. So every church, chapel, cemetery, hospital, maternity hospital, work-house, in all my towns is going to need its own standard place-name - otherwise I know that search results (and hints!!!!) will be mixed up.

Fellow users - I may be more pedantic than some - but can anyone persuade me that I need not confine myself to just using the standard place-names?

LegacyUser

Adrian Bruce said: "Some of your assumptions above are correct, some are not, and some I do not know how it is implemented". That's fine, Lundgren, I never thought I'd get it all right! (grin)

"Find by ID doesn't actually use the search system or any of the same APIs. It does a direct retrieval from the tree data. " - OK - that makes sense - I suspected it was that, on reflection, but needed to draw it out and make sure.

"The search system does not go back to the tree data to provide the search results". OK.... Interesting... It does at least explain things. Yes, your card catalog analogy is fine - it's roughly what I was searching towards.

Thanks