Reconciling Sources and Records

LegacyUser

Steven Ricks Hansen said: I just finished verifying some sources that were attached to a record because the indexed information matched some record information (Name, gender, birthplace), but it did not match ALL the information. This seems to be a common problem that brings up a good point – Family Search does not have an effective view for reconciling source information. This leaves too much to the memory of the user, and results in incomplete review before sources are attached.

Right now we have “Indexed Information”, which shows what was indexed from the source.
There is also a way to view the sources for an event, and tag the sources for that event, but there is not a good way to see if all the indexed information from a source matches all of the context of a record.

Users could make better decisions with all the pertinent information in front of them if they had what I am calling a “Reconcile Source” view. This view would allow users to see the similarities of the source information with the applicable record information to see if it all agrees. (See included picture.) Where the information does not agree, it would be a perfect place to add comments as to why. (These could be hidden until opened, and appear below each row.)

I believe a view like this would make decisions clearer and promote agreement as to what is right between sources and records.

Another benefit of this could be additional process control. If Family Search required all reconciling links to be broken before a record could be deleted, it may also just add enough rigor to the process to help prevent well-established lines from getting messed up by uninformed users.

LegacyUser

Jeff Wiseman said: Hi Steven,

I'm not quite sure what you are asking for as there is already a mechanism to do this in the Source Linker. After selecting "Review and Attach" to get you into the source linker with a given source you can select the "compare" function and you basically get all of that information. You can do the compare prior to attachment, or review the same data after the attachment such as in the following example.

LegacyUser

Adrian Bruce said: Is part of the problem perhaps that it isn't altogether self evident that Source Linker is the way to go? Getting from a person's profile in FamilyTree to their attached sources that you want to review. Fine.

Then you have to click "View Source" to access "Review Attachments". If you're of a pedantically logical frame of mind (like me) "View Source" isn't quite what you want - you think you want to "View Source and Attachments" - except there isn't such an option.

Even after you've clicked "Review Attachments", you have to know to click "Details".

I suspect this to be a training thing rather than requiring a UI change....

LegacyUser

Gordon Collett said: This function is somewhat available, at least for Vitals, within the editing boxes:



In this example, the first two sources are manually created. It would be nice if one could add to the manual source the exact piece of information in that source, as it appears in that source, that should display here.

In the third source, only christening information was extracted, not the birth information, so nothing displays.

In the fourth source, the incorrectly indexed source information shows and I can explain why it is wrong in the reason statement.

If we could add information from manually created sources, the above screen could look like this:

LegacyUser

Jeff Wiseman said: Gordon,

That is certainly a great way to view this information. The fact that the indexed data on the right is limited to only the data chosen for that vital does mean that you are partially blinded, but probably not such a big deal since you can expand the source right there.

I do have a much bigger problem with the schizophrenic "Reason" box. Is it the reason a change was made (i.e., the note that goes in the change log), or the logic for how the vital was derived from the sources (i.e., why the information is correct)? There is a DISTINCT difference between these two and how they would be handled in the system.

The handling of the data in the box is stored in the system as a note for why the change was made in the change log. It is NOT a formal note describing the derivation of the vital (which should have it's own change history).

So my issue is that because it is so volatile, positioned where it is, and has the misleading title, people will start using it when they should really be creating a "collaboration" NOTE on the vital so that if the derivation logic in that note is modified, it can have it's own separate "reason for change" note.

If notes documenting more complex derivations of vitals from sources were allowed to be tagged the same as the sources for those vitals, it would then be far easier in that vitals modification window and it would also remove the need to duplicate derivation logic that applies to multiple vitals in each vital.

When I make a change to a vital, I document why that CHANGE was made. This will replace any of the reasons why previous changes had been made, because those reasons will not apply to the change I just made! That means if somebody is using that field to document the logic behind how the vital has been derived from sources, their work will very likely disappear because the entire nature of that text field is temporary at best. Although I may leave some text there if I want it in MY reason for change, but I have no problem in deleting it all so that my name is not being attached to reasons for some previous change someone else made instead of just the reason for the change I just made.

It is schizophrenic because it has been set up to do two DIFFERENT functions that are in opposition with each other. You can't have both.

FS has taken a short cut by presenting a text field as though it should be used for two similar, but quite different functions. They also are diluting the correct and efficient use of the NOTEs feature by giving people alternate places to record the exact same information. Is it any wonder that people get confused as to how the system is "supposed" to work?

They need to fix this by adding the ability to tag notes from the person's NOTE list in exactly the same way that you can tag sources from the person's SOURCE list where they show up on the right hand side of that window where they can be expanded and examined identical to the sources. Then change the title on the change history reason box to the far more correct "Reason for this change".

LegacyUser

Juli said: The reason box is presented much more like a "reasoning" box than a "change log" box: whatever you write in it shows next to the conclusion -- the thing you changed it _to_ -- without any reference to or indication of its history. This means that any actual statement about a reason for change comes across as disconnected nonsense; it's only statements about reasoning -- how the conclusion derives from the evidence -- that show any sort of coherence.

LegacyUser

Juli said: This has come up before: one partial fix would be for the right-hand side of the edit box to scroll independently of the left-hand side. This would mean that instead of opening the source and needing to scroll past the useless parts, thereby losing the useful parts of the edit screen:



you could scroll to the useful part of the source citation while keeping the useful part of the edit screen in view, as well:

LegacyUser

Jeff Wiseman said: EXACTLY! It is presented on the web page as something it is not.

Where do those notes go to? The only place to see them all is in the change history log. In fact, if you were to remove the entire change history (a bad idea) function and database pieces that support it, those notes would all go away.

This in spite of the fact that they are being presented as though they were an attribute of the vital itself!

The relationship of those notes to the vitals are only indirect. Those notes are an attribute of a CHANGE EVENT and are specifically and narrowly related to that one specific event.

This encourages people to put useful derivation logic documentation where it will likely disappear some day.

Whenever I have a vital that I have had to do some documenting on to clearly show why a specific value was chosen (e.g., 6 censuses, and 6 different years of birth, etc.), I will create a "collaboration" note with the Title identifying the vital being discussed and I will put all the gory details there. I'll then only give a summary in the reason for change field and point to the note (since the more reasonable feature of being able to tag notes doesn't exist). For shorter notes I may copy the note into the reason for change field. However, If you make that reason for change field too immense, going though the change logs for a person becomes very difficult (even on a large screen you can wind up with only one or two changes per page).

LegacyUser

Adrian Bruce said: Further, when you attach a source to a profile, the "Reason" box defaults to the contents of that Reason-why-I've-Attached. Which is (correctly) another separate bit of text and serves to confuse people over what the "Reason" is there for.

What have we got in various places?
- Why have I attached this source to this profile? (i.e. how do I know that the Fred Bloggs in this source is the same Fred Bloggs as on the profile?);
- Why do I believe the values for this event are these? (e.g. birthplace is from 1851 census, birth-date is from baptism);
- Why have I changed the data from what was there before? (e.g. standardising the birth-place; or, removing data from an incorrectly attached source - the first of those is probably clearest - in my mind at least - in showing the distinction - at least, if I understand Jeff's distinction correctly);
- Background narrative (e.g. she was the only child to be baptised in the Methodist Church)

Feel free to add / subtract / dispute!

LegacyUser

joe martel said: The purpose of the Reason can be many things and there could be nuanced different purposes. 
But the key is the Reason is there for the you to explain to the next user why you believe your action is more correct than what was there previously. It is to convince the user why they shouldn't change it, delete it or restore it. The desire is to evolve the conclusions over time to become more and more quality. The changelog tracks the change, attribution (user and date), and the user's reason for believing they have made the conclusion more correct. Not perfect, and short of having a full genealogical proof standard that interwines person data, sources, family's, narrative it's a simple approach that hopefully most users can grasp.

LegacyUser

David Newton said: The purpose of the reason statement could indeed be many things.

However it is not implemented correctly to do what you describe. What you are describing is still a mixture of two functions. The changelog element of it and the reasoning part of it need to be completely separated.

LegacyUser

Adrian Bruce said: Indeed. I think that the issue is illustrated by the theoretical case where a carefully constructed reason such as "Date of birth from baptismal register and place from 1851 census" can be wiped out by the single word entry "Standardized". That entry is perfectly valid, sensible and even necessary, but it has eliminated something that needs to be retained. Saying that the full reason is somewhere in the Change Log may be correct but ithe full reason needs to be retained in a clearly visible form.

I fear that when I started in FSFT, I may have wiped some careful reasons from view simply because I couldn't believe that the system didn't retain them in a visible form.

LegacyUser

ATP said: Jeff Wiseman,

Hear! Hear!

Absolutely spot-on. I had wondered if you were not a systems engineer and it seems that you are or as close kin as can get.

Keep on speaking!

Maybe someone will finally get the picture of what went wrong and how to change that!

Thanks for the explanation and where to start to change this into a viable working easy to use system!

LegacyUser

Jeff Wiseman said: One significant challenge that I think FS have is that as the system has evolved, many times they have had the knowledge that something was needed to be added to the system's infrastructure, but it was not real clear to them what the details should be. So they put part of the solution into the system, hoping to evolve it as they go. Frequently, if the initial implementation was slightly askew, it could continue to grow in that same direction, and when the issue is finally discovered, it has become embedded too far, and far too late to be corrected in a cost effective way.

So I grumble about things a lot just hoping that some of these ideas on FS's roadmap are reconsidered as it is always cheaper and far more efficient when you get it right the first time. I once heard a VP of engineering make a statement to the company to the effect that "I know that you can't get a software product right the first time". What a shocker! That just isn't in my paradigm as I have been involved in several exceptions (on multi-year projects) to that "rule".

And you are right, my core experiences are in defining the essential requirements for systems so that design engineers have a correct structure to build to. If you have access to LinkedIn, you can find me at:
https://www.linkedin.com/in/jeffawiseman

LegacyUser

joe martel said: Love to hear a design that addresses your issues, and that is simple to understand by all (or most beginner) users - because that is the goal - one software that all users can use to build the tree of mankind. Feel free to post a Requirements doc, mockups, whatever to explain and have the community build a better approach. Today we have this basic logical model:
1. A conclusion (like birth event) that has a value, attribution and Reason.
2. A conclusion that can change (edit, delete) over time and can be seen over time - changelog
3. A conclusion value that can be restored

Start with that, and then we can drop in the other Reason application pieces (Attach/Detach source, Not-a-Match, Sources, Delete, Relationships...) I know there are incoherencies in the way the model is applied across the various teams/features - I don't get to dictate design, or implementation, or anything. But I really appreciate elegant designs, with the understanding that organizations, and technologies and development processes dilute the outcome.

LegacyUser

Juli said: Better logical model:
1. A conclusion has a current value, connections to citations and notes, and change events.
2. Citations and notes have current contents, connections to conclusions, and change events.
3. A change event has an attribution, date, reason, and before-and-after values.

Restoring a conclusion (or citation/note) is another change event; it puts the "before" value from a previous change event in as the current value or content.

A conclusion may be an entry in Vitals or Other, or it may be a relationship. Citations or notes connected to a relationship conclusion should appear on the relevant collection pages (currently labeled "Sources" and "Collaborate") for both persons involved.

LegacyUser

Steven Ricks Hansen said: I believe a UI change is in order. The Reconcile Source suggestion does not simply state that buried information can be uncovered. It says that all the pertinent information is in front of the user to help them make the decision they need to make. The idea of reconciling is a line-by-line comparison to help the user see what agrees and what doesn't. The format needs to be more compact to help them compare, and allow comments to be made where things may not seem to agree. The Reconcile View should appear every place where the Open Details link is available, because it is the information the user needs to know first. It immediately tells the user WHY the source is attached and relevant. Then a Review Attachments link (which is another useful view) should be accessible from there.

LegacyUser

joe martel said: Good stuff Juli. THere is a design that has a much more "verbose" genealogical proof statement. That's a future (far away) enhancement to Reason. Reason today is just a version 1, and low cognitive load approach to launch FamilyTree many years ago. 

I also want to make sure we aren't hijacking the thread here. Steven is actually posing a different aspect - that of comparing records/sources with the person/family conclusions. There was a design to address this and I should dig that up. I don't see any adoption of that anytime soon. 

LegacyUser

Tom Huber said: Person-Family conclusions can be a problem because they are often traditions that have developed over the years, decades, and even centuries. Any family-based conclusion needs to have the same criteria applied as any conclusion.

That doesn’t mean that all such conclusions should be discarded, but it does mean that such conclusions should always be vetted to make sure it is valid.

LegacyUser

Jeff Wiseman said: Joe, in order to avoid further hijacking of poor Stephen's topic, I will start another topic addressing this. However, Juli's response below is along the lines that was thinking.

LegacyUser

Jeff Wiseman said: That's exactly right. I need to create a graphic that shows this better, but I'll do that in a different thread. From an implementation standpoint, you last sentence could be slightly reworded as "Citations or notes connected to a relationship conclusion should appear on come from the relevant collection pages (currently labeled "Sources" and "Collaborate") for both persons involved"

LegacyUser

Jeff Wiseman said: Joe,

If NOTEs could be tagged to vitals in the same way that sources are, it would go a long way towards improving flexibility and visibility. Anyone that knows how to use and tag sources will instinctively recognize the benefits of using Notes.

However, using notes that describe relationship issues will have the same challenge as those on other relationships. But when you start talking multi family relationships, you are probably best off for the time being by just creating a memory document and then use it to create a source for attaching to all involved.

LegacyUser

Jeff Wiseman said: Tom,

I really don't have an issue with family traditions, legends,oral histories, etc.. They are all sources of varying reliability. They just need to be documented as a source or note and tagged to all pertinent conclusions. That way they can be scrutinized against all other sources and notes for final determination of vital values.

LegacyUser

Tom Huber said: True, Jeff.

My finding is that not all traditions are reliable. One in particular on my wife's line was a tall tale of a U.S. Western flavor.

She has a relative who stole a horse and according to the family tradition, was captured and sentenced to be hung. The day before he was to be hung, he father and one of his brothers broke him out of prison (Boise, Idaho) and spirited him to Canada where he resided in British Columbia for the rest of his life. Members of his family still live in that province.

The truth of the matter is, from the records we obtained at the Idaho State Archives in Boise is that:
* yes, he stole a horse.
* yes, he was caught
* no, he was not sentenced to be hung -- he was sentenced to seven years hard labor.
* no, his father and brother did not break him out of prison -- he was released after four years for good behavior.
* no, the family did not help him escape to Canada -- he immigrated first to Alberta and then settled in British Columbia.

My wife has documented this on his FamilySearch page, complete with photographs provided by the Idaho State Archives (at no cost to us! -- they were very helpful in chasing down the actual records).

Then there is the story my father told me about how my Grandfather (my mother's father) was shot and killed in Whitebird, Idaho. He said that after the dastardly deed, the good townfolk rose up and captured the person and hung him from "that tree" -- which he pointed out to me while visiting Whitebird.

Some quick research revealed that the man shot my grandfather and then walked over to the steps of the parsonage (not far from where my grandfather was killed) and then killed himself.

What caused the story that my father related to me was that at the same time, there was another criminal who was chased down by a posse at the time, captured and rather than track back across the Idaho - Salmon River country to where he was slated to be hung, took care of the situation out i the wilderness. A second cousin related the events to me and we figured that my father had mixed to two events into one event.

My grandfather's death and the events following are documented -- I need to make sure the newspaper accounts are included as part of his memories.

LegacyUser

ATP said: Jeff Wisman

Well, it does indeed look as I suspected that you have a solid background in systems engineering! Your language in various observations and comments in this forum had a familiar sound to me as an assistant some number of years ago to a systems engineer in the commercial aviation industry and I even once thought of becoming such.

What you said about VP of engineering! Have heard similar talk from uppermost echelon management. There's not enough money, we can't afford it, based on not getting right the first time. Once convinced that "we can't afford not to", the money was appropriated on the promise that the software project had to be accomplished right the first time! And, it was!

Keep on speaking!

LegacyUser

joe martel said: Started a new thread and would love more input about the Reason statement. https://getsatisfaction.com/familysea...

David Peterson

I believe that @LegacyUser hit a significant answer on the head when suggesting that NOTES (for every data item) would help resolve this issue. I have observed this for many years. The various GEDCOM standards and other genealogical infrastructures have, from the beginning, indicated that every "item of data" and "relationship assertion" need to have an attached "note structures". As far as I know, none of the family search iterations have chosen to implement this, much to my great dismay. As you note, the reason one updates or changes or has confidence is different from the implications of how one arrived at a potential assertion.

Furthermore, the reference to Memories is quite apropos. Notes need to be stand-alone entities that can be linked to any number of applicable data, just like Memories can be linked to any number of applicable Persons. Multiple notes only linked at the Person level and only linked to a single Person do not address the reality that many Notes that are trying to describe the logic of a less-than-usual situation will involve multiple items (like both Birth and Death) and multiple people. I long for the day that I can write a single, comprehensive note and link it to all the participating data and have it be visible from any linked point, without having to worry about keeping multiple notes updated and in sync!

Julia Szent-Györgyi

@David Peterson, good news: the feature you long for already exists. Sort of. FS calls them "sources" rather than "notes", but they have the capabilities you mention: they can be attached to as many profiles as needed, they can be tagged to every relevant conclusion within those profiles, and updates or corrections only need to be made on one instance. The key is to put an instance of the "source" in your Source Box, so that you can add other instances to the other profiles where it applies.