Knowledge elicitation techniques

Elicitation of expertise

  • Time-consuming
  • Multiple forms
    • e.g. theoretical, how-to-do-it
  • Multiple experts
  • Heuristic nature
    • distinguish empirical from heuristic
  • Managing elicitation efficiently
    • knowledge about when to use particular techniques

Expert types

  • Academic
    • Regards domain as having a logical structure
    • Talks a lot
    • Emphasis on generalizations and laws
    • Feels a need to present a consistent “story”: teacher
    • Often remote from day-to-day problem solving
  • Practitioner
    • Heavily into day-to-day problem solving
    • Implicit understanding of the domain
    • Emphasis on practical problems and constraints
    • Many heuristics

Human limitations and biases

  • Limited memory capacity
  • Context may be required for knowledge recollection
  • Prior probabilities are typically under-valued
  • Limited deduction capabilities

Elicitation techniques

  • Interview
  • Self report / protocol analysis
  • Laddering
  • Concept sorting
  • Repertory grids


Interview: Session preparation

  • Establish goal of the session
  • Consider added value for expert
  • Describe for yourself a profile of the expert
  • List relevant questions
  • Write down opening and closing statement
  • Check recording equipment
    • audio recording is usually sufficient
  • Make sure expert is aware of session context: goal, duration, follow-up, et cetera

Interview: Start of the session

  • Introduce yourself (if required)
  • Clarify goal and expectations
  • Indicate how the results will be used
  • Ask permission for tape recording
  • Privacy issues
  • Check whether the expert has some questions left
  • Create as much as possible a mutual trust

Interview: During the session

  • Avoid suggestive questions
  • Clarify reason of question
  • Phrase questions in terms of probes
    • e.g, “why …”
  • Pay attention to non-verbal aspects
  • Be aware of personal biases
  • Give summaries at intermediate points

Interview: End of the session

  • Restate goal of the session
  • Ask for additional/qualifying
  • Indicate what will be the next steps
  • Make appointments for the next meetings
  • Process interview results ASAP.
  • Organize feedback round with expert
  • Distribute session results

Unstructured interview

  • No detailed agenda
  • Few constraints
  • Delivers diverse, incomplete data
  • Used in early stages: feasibility study, knowledge identification
  • Useful to establish a common basis with expert
    • s/he can talk freely

Structured interview

  • Knowledge engineer plans and directs the session
  • Takes form of provider-elicitor dialogue
  • Delivers more focused expertise data
  • Often used for “filling in the gaps” in the knowledge base
    • knowledge refinement phase
  • Also useful at end of knowledge identification or start of knowledge specification
  • Always create a transcript

Interview structure for domain-knowledge elicitation

  • Identify a particular sub-task
    • should be relatively small task, e.g. an inference
  • Ask expert to identify “rules” used in this task
  • Take each rule, and ask when it is useful and when not
  • Use fixed set of probes:
    • “Why would you do that?”
    • “How would you do that?”
    • “When would you do that?”
    • “What alternatives are there for this action?”
    • “What if …?”
    • “Can you tell me more about ..?”

Interview pitfalls

  • Experts can only produce what they can verbalize
  • Experts seek to justify actions in any way they can
    • “spurious justification”
  • Therefore: supplement with techniques that observe expertise “in action”
    • e.g. self report

    Self report

Self report

  • Expert performs a task while providing a running commentary
    • expert is “thinking aloud”
  • Session protocol is always transcribed
    • input for protocol analysis
  • Variations:
    • shadowing: one expert performs, a second expert gives a running commentary
    • retrospection: provide a commentary after the problem-solving session
  • Theoretical basis: cognitive psychology

Requirements for self-report session

  • Knowledge engineer must be sufficiently acquainted with the domain
  • Task selection is crucial
    • only a few problems can be tackled
    • selection typically guided by available scenario’s and templates
  • Expert should not feel embarrassed
    • consider need for training session

Analyzing the self-report protocol

  • Use a reference model as a coding scheme for text fragments
    • Task template
  • Look out for “when”-knowledge
    • Task-control knowledge
  • Annotations and mark-ups can be used for domain-knowledge acquisition
  • Consider need for tool support

Self report guidelines and pitfalls

  • Present problems in a realistic way
  • Transcribe sessions as soon as possible
  • Avoid long sessions (maximum = 20 minutes)
  • Presence of knowledge engineer is important
  • Be aware of scope limitations
  • Verbalization may hamper performance
  • Knowledge engineer may lack background knowledge to notice distinctions

Use of self reports

  • Knowledge specification stage
  • Validation of the selection of a particular reference model
  • Refining / customizing a task template for a specific application
  • If no adequate task template model is available: use for bottom-up reasoning model construction
    • but: time-consuming



  • Organizing entities in a hierarchy
  • Hierarchies are meant as pre-formal structures
  • Nodes can be of any type
    • class, process, relation, ….
  • Useful for the initial phases of domain-knowledge structuring
    • in particular knowledge identification
  • Can be done by expert
    • tool support

Example ladder

    Concept sorting

Concept sorting

  • Technique:
    • present expert with shuffled set of cards with concept names
    • expert is asked to sort cards in piles
  • Helps to find relations among a set of concepts
  • Useful in case of subtle dependencies
  • Simple to apply
  • Complementary to repertory grids
    • concept sort: nominal categories
    • repertory grid: ordinal categories

Card sort tool

    Repertory grids

Repertory grid

  • Based on personal construct theory (Kelly, 1955)
  • Subject: discriminate between triads of concepts
    • Mercury and Venus versus Jupiter
  • Subject is asked for discriminating feature
    • E.g. “planet size”
  • Re-iterate until no new features are found
  • Rate all concepts with respect to all features
  • Matrix is analyzed with cluster analysis
  • Result: suggestions for concept relations
  • Tool support is required

Example grid

When to use which technique?

  • Knowledge identification
    • Unstructured interview, laddering
  • Knowledge specification
    • Domain schema: concept sorting, repertory grid
    • Template selection: self report
    • Task & inference knowledge: self report
  • Knowledge refinement
    • Structured interview

Creator: OlliG


Licensed under the Creative Commons
Attribution ShareAlike CC-BY-SA license

This deck was created using SlideWiki.