Recorded by Keke Chen

Discourse understanding with script

1. Different levels of analysis we have talked

Lexicon analysis

syntactic analysis

sementic analysis

...

pragmatic analysis

2. A difficult problem in pragmatic analysis is reference problem.

An Example:

-- Bill have a red balloon

-- Jim wanted it

How do you know what "it" refers?

Reference problem is a kind of identity problem, which can be solved

from context. Analyzing context from other paragragh is harder than

from sentences

* reference from knowledge of parts

1)

object part knowledge

-- She opened the book she had just bought

-- The title page was torn.

2)

event part knoledge

-- John went on the business trip to Los Angels

-- He took an early flight

* causul knowledge

-- There was a big storm yesterday

-- The schools were closed today

* goal-plan knowlege

goal -- Sally wants a new car

plan -- She decided to rob a bank

3. Plan - recognition, with script approach

plan understanding --> plan generation

The hard problem is plan generation. Script is used

to describe plan.

Script is a kind of knowledge representation that is good

for activity analysis. Script is also a model of particular

sequence of actions.

Language scripts - large scale

Vision scripts - short

Question: how to decide an activity?

Script has 6 parts

1) actor -- roles

2) object -- props

3) pre-condition --

4) post-condition -- results

5) tracks

6) scenes

Example:

actors - You

objects - shirts , shorts, shoes, socks ...

post-cond - you dressed up

pre-cond - you wore underclothes

scenes:

Y MBUILD on( shirt, Y)

Y PTRANS shirt, hand

Y PTRANS shirt, body

Script is a kind of model based discourse understanding

Question: How are the scripts stored at multiple levels of resolution

* scripts are stored at level of details

with zoom in and zoom out to change the levels

* scripts are stored at level of generalization

Model: script sementic memory

/ / | \

fast food

/ | \

Wendy's Mac BK

/ / \ \ / \ /\

example 1, example 2, .... example n episodic memory

If level of generalization is changed, level of detail must be changed too.

However, if level of detail is changed, level of generalization may not

be changed.

Because of inability of capturing all actions, exceptions out of

the models must occur. Some mechanism should be established to

handle exceptions.

Video understanding

problem: frame generation