experiment on knowledge acquisition
DESCRIPTION
This is our experiment in crowd-sourcing knowledge acquisition for universal truth creation out of complicated textbook sentences. An integration of Project Halo's SMW and AURA efforts.TRANSCRIPT
AuraWikiSubject Matter Expert - User Experience
Review
Jesse Wang, Wil Smith
Oct 2012
Project Goals“Crowd Source UT Authoring”
Can Subject Matter Experts Author Useful Universal Truths?Can We Speed Up Encoding a Textbook with Input from Subject Matter Experts?Can We Create a UT Authoring Portal for Multiple Textbooks?Can Existing Social Networks Provide Subject Matter Experts Capable of UT Authoring?Could Gamification be Applied to An Existing Portal to Add Non-Domain Experts?
Meeting GoalsThis is an end user meeting and we are doing it to hear your comments on the current system.
The team wants to explain the authoring process, and demonstrate the current application state.
Before we continue development on the project we need to review how the application, in the unfinished state, performs with our targeted user base.
MID PROJECT FEEDBACK. We do these meetings when releasing production products to ensure the final application complies with customer needs and is not “the engineering best way”.
Review SchedulePhase Method Time required
Introduction & Training • Goals of Project Halo and Aura Wiki
• Training for UT Authoring• What is a UT?• What is a concept /
context?• How do I formulate UTs
• System demonstration & hands-on
0:45
Assignment: UT Authoring
• You work by yourself using our system
• Support is provided by us
1:30
Questionnaire & Discussion
• Online questionnaire• Group discussion
0:45
How do we author a universal truth?
Subject Matter Expert - User Experience Review
From Sentence to Universal TruthsWhat is a UT? How to write UTs for a sentenceWhat it a concept? How to choose concepts for a UT
What is a UT?There is no exact definition of “Universal Truth”, but a good UT…
expresses a single fact (which can be compound, but must be indivisible).
is as simple as possible, but not simpler (to quote Einstein).
is context-independent, i.e. meaningful on its own, without surrounding sentences or other context.
is universally true. For example, a UT could start with "All Plants..." but not with "Some plants...".
is unambiguous.
is precise.
How to Write UTs for a Sentence
For each relevant sentence, write one or more UTs
There is no one correct set of UTs for a given sentence. The source sentences are open to interpretation, and it is up to you to determine what the most appropriate UTs are
The interpretation of the source sentences, and the paraphrasing into UTs, should be guided by your knowledge of the subject matter (and by common sense)
The following methods are not exhaustive. When in doubt, refer back to the definition of a “good UT”.
How to Write UTs for a Sentence
In the simplest case, the sentence has only one UT, and it is exactly the same as the UT.
Sentence: Matter is made up of elements.
UT: Matter is made up of elements.
How to Write UTs for a Sentence
Sometimes part of a sentence is irrelevant, even though the sentence as a whole has been deemed relevant. In such cases the irrelevant part is simply dropped.
Sentence: Some symbols are derived from Latin or German; for instance, the symbol for sodium is Na.
UT: Sodium has the symbol Na.
How to Write UTs for a Sentence
Often, a sentence contains several facts. In this case, the sentence needs to be split into several UTs.
Sentence: Pure sodium is a metal, and pure chlorine is a poisonous gas.
UT: Pure sodium is a metal. UT: Pure chlorine is a poisonous gas.
How to Write UTs for a Sentence
When a reference like "this", "that", "these", "those", "which", etc occurs in a source sentence, they should be replaced with the referent (the word that "this","that", etc refers to).
Sentence: Organisms are composed of matter, which is defined as anything that takes up space and has mass
UT: Organisms are composed of matterUT: Matter has mass and volume
Here, "which" clearly refers to "Organisms", and "is defined as" is unnecessarily verbose, and thus simplified.
How to Write UTs for a Sentence
Sometimes the referent is in a previous sentence in the same paragraph.
Sentence: These selective channels affect the rate at which water moves osmotically across the membrane
UT: Aquaporins are selective channelsUT: Aquaporins affect the rate at which water moves osmotically across the membrane
How to Write UTs for a Sentence
There are many other words that are used as linguistic flourishes or to bind sentences in a paragraph together and give the text some "flow". Such words ("another" in the example below) should be eliminated, but can sometimes be used to extract or infer "incidental" information from a sentence.
Sentence: Water (H20), another compound, consists of the elements hydrogen (H) and oxygen (O) in a 2:1 ratio
UT: Water consists of hydrogen and oxygen in a 2:1 ratioUT: Water is a compoundUT: Hydrogen has symbol HUT: Oxygen has symbol O
Here, the first UT contains the "main" information of the sentence, and the last three are "incidental".
How to Write UTs for a Sentence
Ideally, a UT should not contain words like "sometimes", "usually", "most", "some", "generally", etc. There are a couple of ways to handle them. One way is to generalize the weaker statement into a universal statement, usually just by dropping the word in question.
Sentence: Branching generally enables plants to harvest sunlight for photosynthesis more effectively.
UT: Branching enables plants to harvest sunlight for photosynthesis more effectively.
How to Write UTs for a Sentence
Another method of eliminating "some" etc is to instantiate the sentence by providing a name for the case where the sentence is true.
Sentence: Some prokaryotes even contain a row of tiny magnetic particles that allow the cells to orient in Earth's magnetic field
UT: Magnetotactic bacteria contain a row of tiny magnetic particles that allow the cells to orient in Earth's magnetic field
Here we replaced "some prokaryotes" with "magnetotactic bacteria". We only do this when the new name occurs somewhere in the textbook.
How to Write UTs for a Sentence
"some", etc are allowed when they do not refer to the main concept of the sentence (see "Choosing Concepts for a UT", below). For example, this is a good UT:
UT: In vascular plants, some substances may use more than one route
The UT as a whole states a fact about all vascular plants (the main concept), and "some" refers to substances within all vascular plants. One way to remember this distinction is to keep in mind that Universal Truths are just that -- universally true statements, i.e. statements that are always true of some type of entity, event, etc. If it is possible, we can also use Instantiation (see previous slide) in this type of case.
How to Write UTs for a Sentence
When "for example", "such as", etc occurs in a sentence, encode the specific information in the example.
Sentence: Some plants have horizontally oriented leaves; others, such as grasses, have leaves that are vertically oriented.
UT: Grass has vertically oriented leaves
As discussed above, the "some" part of the sentence is problematic.
How to Write UTs for a Sentence
In other cases, the "for example" does not add any new information.
Paragraph: We symbolize atoms with the same abbreviation used for the element that is made up of those atoms. For example, the symbol C stands for both the element carbon and a single carbon atom.
Here, the second sentence does not add any information, since its informational content is subsumed by the general "rule" of the first sentence (assuming we have already encoded that carbon has the symbol C).
How to Write UTs for a Sentence
Conditional statements should usually be rephrased to a non-conditional form.
Sentence: If we place a cell in a solution that is hypotonic to the cell, water will enter the cell faster than it leaves
UT: A cell in a hypotonic solution will gain water faster than it loses it
Here, the condition has been changed into a context, "in a hypotonic solution". See also the section on "Contextual Universals" below.
Review: What is a UT?
Or rather a good UT…
expresses a single fact (which can be compound, but must be indivisible).
is as simple as possible
is context-independent, i.e. meaningful on its own, without surrounding sentences or other context.
is universally true. For example, a UT could start with "All Plants..." but not with "Some plants...".
is unambiguous.
is precise.
Summary: Methods of UT
AuthoringCopy, remove (partial, irrelevant info), splitting
Reference resolution (from context)
Incidental extraction e.g. “Water, another compound, …”
Specialization, generalization
InstantiationE.g. “some prokaryotes” “magnetotactic bacteria”
Example-encodingE.g. “some plants…” “grass”
Condition to context
What is a Concept?
A concept is a noun or a noun phrase that describes a type of entity or event, in singular form
You may need to create a new concept for a UT if an appropriate one does not already exist.
Good concept names: Cell, Eukaryotic Cell, Cellular Respiration
Bad concept names: Cells, The eukaryotic cell, A plant
Choosing Concepts for a UT
The main concept of a UT is the entity or event that the UT is about.
It is usually the grammatical subject of the UT
UT: All plant cells have a nucleusMain concept: Plant Cell
Note that "All plant cells" is not a good concept, because it is not in singular form.
UT: Photosynthesis produces oxygenMain concept: Photosynthesis
Choosing Concepts for a UT
Some UTs are true in a specific context. In such cases, also choose a contextual concept –contextual concept is OPTIONAL
The contextual concept usually follows a word such as "in", "during", "while", etc.
UT: The phospholipids in a membrane move laterallyMain concept: PhospholipidContextual concept: MembraneNote that "the" or "a" is never part of the concept.
UT: The oxygen produced during photosynthesis is used during cellular respirationMain concept: oxygenContextual concept: Photosynthesis
UT: Cellular respiration in eukaryotes happens in the mitochondriaMain Concept: Cellular RespirationContextual Concept: Eukaryote
Bring your Best, then Move On
We hope you got good understanding of what you’re supposed to do
Give it your best shot, it does NOT need to be perfect
Do not spend too much time on one sentence/UT, move on if it’s too hard
Feel free to work on sentences or UT others started
Feel free to ask us if you’re really confused or not sure
Aura wiki tutorial
Subject Matter Expert - User Experience Review
Our Solution: Aura Wiki
Contents:A few chapters of Campbell Biology
Without figures and using a simplified layout
PurposeUse the power of crowdsourcing and online collaboration
Let users „translate“ sentences from the textbook into UTs.
Sample wiki page for the textbook
Sample wiki page for a whole section
Sample wiki page for a sentence
Creating UTsThe translation of sentences to UTs is done on the sentence page.
The sentence page can display the paragraph surrounding the sentence (or even more context) if needed for understanding and formulating the UT.
Under the heading Universal Truths you can see all UTs that exist for this sentence and you can add yor own.
Sample UTs presented in the sentence page
Creating UTs – Add new UT
Creating UTs – Input Form
Input form for new UT. First two inputs
are required.
Creating UTs - Autocompletion
Autocompletion is automatically triggered and
suggests existing concepts
Subject Matter Expert
Assignment
Subject Matter Expert - User Experience Review
The AssignmentNow it is your turn
Go to your User page
There is a list of sentences assigned to you
Once you have added UTs they will be displayed there together with the sentences (after refreshing the page)
At the top of your user page, you will find links to help-texts, some sample UTs and the questionnaire for after the session.
The AssignmentThis link always brings you back
to your user page
Sentences (and UTs)
Help documents and
a list of example UTs
The AssignmentIMPORTANT
This evaluation is conducted in order to assess the system, not to assess your work as a SME. You are supporting the evaluation, but you are not a test subject!
Please work with the system at your own convenience. You do not need to achieve any quantitative or qualitative goal. (This is of course no excuse for being lazy )
You have pencils and paper available – if you encounter any problems or have ideas you think are worth mentioning, feel free to take a short note so you can bring it up during the discussion later
If there are questions please ask them anytime
Subject Matter Expert Review
Subject Matter Expert - User Experience Review
QuestionnaireGo to your user page
There is a link to the online questionnaire at the top
In the first question, select the username you were assigned today
Fill out the entire questionnaire. Don’t forget to submit at the end
Discussion – System
FunctionalityDid you have trouble following the UT authoring workflow? If so, where and why?
What changes would you apply to the existing functionality and/or workflows to increase productivity and ease of use?
What new functionality could you think of that might have a positive influence on your work?
Discussion – UT Authoring
Did you understand the notion of UT and concept?
Did the training sufficiently explain the meaning of UT and concept?
Do you feel the training you received in the beginning and the help materials on the site were sufficient for executing your task? If not, what would you improve? Which areas need better coverage?
Conclusion
Subject Matter Expert - User Experience Review
Thank you.Contact: Jesse Wang ([email protected])
William Smith ([email protected] )