semantic annotation framework part 2: dialogue acts iso/tc37/sc4 n442 rev00 harry bunt tilburg...

Semantic annotation framework Part 2: Dialogue acts

ISO/TC37/SC4 N442 rev00

Harry BuntTilburg University

ISO TC 37/SC 4 meeting Marrakech, May 25, 2008

Purpose and justification

Dialogue acts are widely used in studies of dialogue phenomena, in dialogue annotation, and in the design of dialogue systems.

Dialogue acts are particularly useful for:• describing functional and intentional aspects of the

dialogue utterance meaning;• the design of dialogue management systems.

Dialogue acts

Well-known examples of communicative functions (“core dialogue acts”):

• question• WH-question• YN-question• check/verification

• statement/inform• answer (WH-answer. YN-answer)• confirmation, disconfirmation• request• instruct• promise• acknowledgement• greeting

Purpose and justification (2)

Alternative dialogue act schemas: TRAINS, Map Task, Verbmobil, DAMSL, SWBD-DAMSL, COCONUT,... with different:

• underlying approach to dialogue modelling• definitions of basic concepts• level of granularity• and mutually inconsistent terminology

Particularly unsatisfactory:• Lack of solid foundations of definitions and multidimensionality• Lack of interoperability

ISO approach

Preparatory studies in TDG 3 in a joint effort with eContent project LIRICS.

Focus: How to best support the annotation of dialogues with dialogue act information in an empirically and theoretically well-founded way.

Outcome:1. Design of a preliminary set of data categories for

multidimensional dialogue act annotation, based on DAMSL, DIT++, and other schemas, tested for coverage and usability by annotators and endorsed by ISO TC37/SC4/TDG3.

2. Recommendation to set up an ISO project based on 1 as part the Semantic Annotation Framework project.

Summary

Main points of project outlined in ISO/TC37/SC4 N442 (rev00):

Aim to:1. Provide more solid foundations for multidimensionality

of DA tag sets2. Design consistent truly semantic definitions of core

dialogue acts3. Develop agreed definitions in the form of ISO 12620

data categories and enter in ISO registry4. Define annotation language with abstract syntax,

concrete XML-based syntax, and semantics compliant with LAF

Theoretical foundations of DA annotation concepts

• Information-state change approach to dialogue semantics: the meaning of an utterance in dialogue is the way in which the information state of a listener is changed by understanding the utterance (Bunt & Romary, LREC 2002).

• A dialogue act has two components for describing utterance meanings: • the information which the speakers makes available to the

addressee - the “semantic content” • the “communicative function”, capturing the way the speaker

intends an addressee to update his information state with the semantic content.

Multifunctionality

1. U: Can you tell me what time is the first train to the airport on Sunday?

2. S: On Sunday morning the first train to the airport is at 5.32

3. U: Thank you.

- expression of thanks

Multifunctionality



3. U: Thank you.

Multifunctionality



3. U: Thank you.

- expression of thanks

Multifunctionality



3. U: Thank you. - expression of thanks - positive feedback (about understanding and

acceptance)

Multifunctionality



3. U: Thank you. - expression of thanks - positive feedback (about understanding and acceptance)

- indication of dialogue closure

Multifunctionality


1. S: The first train to the airport on Sunday is at ... let me see... 5.32

- positive auto-feedback about perception and interpretation - WH-answer (to indirect WH-question)

Multidimensionality

Utterances have multiple functions ==> multiple annotation tags are required

(or syntactically and semantically (!) complex tags -- cf. studies by Popescu-Belis),

i.e. annotation must be multidimensional.

Multidimensional annotation

Usual informal notion of dimension:

Set of mutually exclusive tags

Not satisfactory... See problems in multidimensional annotation according to DAMSL (Bunt, LREC 2006)

Dimensions in Dialogue

Basic intuition: participants in a dialogue do multiple things simultaneously, such as:

• making progress in performing the activity (“task”) which motivates the dialogue;

• providing and eliciting communicative feedback;• take and assign turns;• monitor contact, attention, use of time,...• greet, thank, apologize, say goodbye,...

Dimensions in Dialogue

A dimension is an aspect of participating in a dialogue such that:

1. There is a class of dialogue acts for addressing this dimension (empirical foundation);

2. It can be addressed independently of other dimensions;

3. Within a dimension, an utterance has at most one communicative function.

Observed dimensions (TDG3/LIRICS)

• Performing a certain task or activity through or with support from the communication

• Monitoring the interaction

- providing and eliciting feedback

- editing one’s own or one’s partner’s speech

- managing the turn-taking

- managing the use of time

- managing contact and attention

- managing the opening and closing of (sub-)

dialogues and thematic progression• Dealing with social obligations: greeting,

thanking, apologizing,…

Dimensions for dialogue actsExamples:

dimension function example1. Auto-feedback OverallPositive Okay.2. Allo-feedback EvaluationElicitation Okay?3. Turn management TurnGiving Yes4. Time management Stalling Well, you know,..5. Contact man’t ContactChecking Hello?6. Own comm. man’t Self-correction I mean...7. Partner comm.man. Completion ... completion8. Topic management TopicShiftAnnounc. Something else.9. Dialogue structuring DA-announcement Question: 10. Social oblig. man’t Valediction Bye11. Task/domain OpenMeeting I open this meeting

Dimensions of dialogue actsExample: Inform

• The KL204 leaves at 12.30. Task/domain


• The KL204 leaves at 12.30. Task/domain• I see what you mean. Auto-feedback


• The KL204 leaves at 12.30. Task/domain• I see what you mean. Auto-feedback• We should first discuss the agenda. Topic management


• The KL204 leaves at 12.30. Task/domain• I see what you mean. Auto-feedback• We should first discuss the agenda. Topic management• I’m very grateful for you help. Social obligation management


• The KL204 leaves at 12.30. Task/domain• I see what you mean. Auto-feedback• We should first discuss the agenda. Topic management• I’m very grateful for you help. Social obligation management

==> Inform acts can be used in every dimension.

Dimensions for dialogue acts

A number of the most commonly used types of dialogue act, such as questions, answers, statements, requests, instructions, or offers,.. do not belong to any dimension: they are ‘general purpose functions’: they can be used in any dimension.

(DAMSL ‘dimensions’ like Info-request and Answer are clearly not proper dimensions.)

General-purpose functions

Applicable in any dimension are:

Information-seeking functions WH-question, YN-question, Alternatives-question, Check,..

Information-providing functionsInform, WH-Answer, YN-Answer, Confirmation, Disconfirmation, Agreement, Correction,..

Commissive functionsOffer, Promise, AcceptRequest,..

Directive functionsInstruct, Request, Suggest,..

Core dimensions and dialogue acts

Data categories from LIRICS:Set of 54 core dialogue act types

• 24 general-purpose functions• 30 dimension-specific functions spread over 10 dimensions

described in the form of ISO (12620) data categories.

Compare:• DAMSL: 12 dimensions, 30 functions • SWBD-DAMSL: 60 functions• DIT++: 11 dimensions, 95 functions

Validation of LIRICS data categories

Usability for human annotators

• Inter-annotator agreement measurements for English and Dutch;

• 2 trained annotators working on raw text/audio• Results: almost perfect agreement (Rietveld &

van Hout, 1993: kappa ≥ 0.80)•

Inter-annotator agreement scores

Function class English Dutch average

Information-seeking 0.96 0.98 0.97

Information-providing 0.98 0.99 0.98

Feedback 0.98 0.99 0.99

Interaction management

0.92 0.96 0.94

Social obligations management

0.94 0.94 0.94

Validation of LIRICS data categories

• Applicability also demonstrated for Italian (annotated test suite developed in Pisa).

• Application of to multi-party multimodal AMI dialogues (Petukhova & Bunt, IWCS-7); results comparing favourably with use of AMI or DAMSL annotation schemes.

• Machine learnability investigations are promising (Geertzen et al., SIGDIAL 2007).

Towards a dialogue act annotation language

DA tag components: < Dimension name, Function name >

Examples: < Activity, Confirm > < Feedback, CheckQuestion > < Turn Management,Turn Release > < Social Obligations Management, Apology >

Note: for dimension-specific functions, the dimension name is in fact redundant.

Design of dialogue annotation language: DiaML

Distinction in Linguistic Annotation Framework:• annotations: information structures independent of

representation format (“abstract syntax”)• representations: annotations cast in a certain format

(“concrete syntax”)

Design of dialogue annotation language: DiaML

Distinction in Linguistic Annotation Framework:• annotations: information structures independent of

representation format (“abstract syntax”)• representations: annnotations cast in a certain

format (“concrete syntax”)

In addition:• semantics, defined for abstract syntax

DiaML abstract syntax

Abstract Syntax: format-independent definition of information structures

For dialogue acts: pairs of stretches (possibly discontinuous) of dialogue behaviour and sets of dialogue act types (at most one function in each dimension)

Information to be expressed in DiaML:• speaker and addressee(s)• segments of dialogue behaviour• dimensions• communicative functions• optionally: functional dependencies

(e.g. an utterance is an Answer to which Question, or provides Feedback on which previous dialogue act)


Information to be expressed in DiaML:• speaker and addressee(s)• segment of dialogue behaviour• dimension• communicative function• optionally: functional dependency

Conceptual elements:• finite set of dialogue participants• finite, ordered set of segment begin/end indicators• finite set of dimensions• finite sets of domain-specific and general-purpose comm.

functions


Information to be expressed in DiaML:• speaker and addressee(s)• segment of dialogue behaviour• dimension• communicative function

Structure definitions:• A DiaML segment (‘markable’) is a finite sequence of pairs of

segment begin/end indicators, defining a stretch of source text• A DiaML tag is an n-tuple of pairs

<dimension, function> • A complete DiaML structure is a 4-tuple

<speaker, addressee, segment, DiaML-tag>

DiaML concrete syntax

<diaML id=‘d2’ speaker=`s’ addressee=‘a’ markable=‘m1’ commfunctions=‘cfs1’>

<sourceText id=‘m1’ =‘sb1’..’se1’blabla` ‘sb3’..se3’blabla>

<cfs id=‘cfs1’ taskFun=‘f1’ feedbackFun=‘f2’> <comfun id=‘f1’ function=‘anwer’ respTo=‘d1’> <comfun id=‘f2’ function=‘positiv’ respTo=‘d1’> </cfs></diaML>

Current status

Result of NWIP ballot?

If NWIP approved:

1. Project (“editorial”) group:• David Traum • Claudia Soria• Jae-Woong Choe• Andrei Popescu-Belis• Jan Alexandersson• Alex Chengyu Fang• Koiti Hasida (tbc)• .....

2. Time schedule and meetings: • Moscow, August 2008? (TC 37 annual meeting)• Pisa, October 2008, workshop• Tilburg, January 2009 (IWCS-9), workshop

Dimensions in DAMSL

FLF Dimensions:1. Statement2. Info-request3. Influencing-addressee-future-action4. Committing-speaker-future-action5. Conventional Opening or Closing6. Explicit-performative7. Exclamation8. Other

Dimensions in DAMSL (3)

Example:

1. A: I hope you’ll have a good time!2. B: Yeah, thanks. And you’ll be visiting friends in Italy.3. A: That’s right.

Dimensions in DAMSL

Definitions of some FLF Dimensions:

1. Statement: Speaker makes a claim about the world

2. Info-request: Speaker requests Addressee to provide information

3. Influencing-addressee-future-action4. Committing-speaker-future-action5. (.... 8)


BLF Dimensions:

1. Agreement

2. Understanding

3. Answer

4. Information-relation


Conclusion:

‘Question’ and ‘statement’ are mutually exclusive tags => having them in different dimensions is wrong


Definitions of some BLF Dimensions:

1. Agreement2. Understanding: Utterances concerning the

understanding between Speaker and Addressee

3. Answer: Speaker provides information requested by the Addressee

4. Information-relation


Can ‘question’ and ‘answer’ be alternatives in the same dimension?

Consider:1. S: Did you ask me something?2. U: Can I change the contrast?

- question




- question - answer




- question - answer

So question and answer can co-occur => question and answer cannot be in the same dimension


Example:

1. A: I hope you’ll have a good time!2. B: Yeah, thanks. And you’ll be visiting friends in Italy.3. A: That’s right.


Example:

1. A: I hope you’ll have a good time!

2. B: Yeah, thanks.

And you’ll be visiting friends in Italy.

Statement? Question?

3. A: That’s right.


Example:

1. A: I hope you’ll have a good time!

2. B: Yeah, thanks.

And you’ll be visiting friends in Italy.

Statement? Question?

DAMSL: - statement

- info-request

... but a speaker cannot at the same time state something and question its truth!


Conclusion:

‘Question’ and ‘statement’ are mutually exclusive tags => having them in different dimensions is wrong

Can they be alternatives in the same dimension?



1. S: Did you ask me something?2. U: Can I change the contrast?

- question about what the task doman

- answer about what the speaker said

=> The task domain and what the speaker said are

different ‘dimensions’; an utterance can be a question in

one dimension and an answer in another.


Example:

A: And what possibilities do you have on Tursday?B: Did you say Thursday?

- function in (DAMSL’s) Understanding dimension, but which?

- Signal-understanding? - Signal-non-understanding?

- Check (function in DAMSL’s Info- request dimension)

Multidimensional annotation scheme for dialogue acts

Two parts:

1. dimension-specific communicative functions for each dimension

2. general-purpose functions (hierarchically organized, reflecting degrees of specificity)

semantic annotation framework part 2: dialogue acts iso/tc37/sc4 n442 rev00 harry bunt tilburg...

Documents

dialogue annotation

dialogue utterance

dialogue semantics

dialogue act information

justification dialogue

design of dialogue systems

studies of dialogue

laf slide