Quick Tutorial of Smatch Calculator        Smatch Score Interpretation and Web Interface Tutorial         Version 0.1   April 28, 2012

### Smatch Score Interpretation

Assume we have two AMRs:

(a0 / watch                               (b0 / watch
:ARG0 (a1 / boy)                             :ARG0 (b1 / girl)
:ARG1 (a2 / tv))                               :ARG1 (b2 / boy))

An AMR can be viewed as conjunction of logical propositions (triples). A triple can be represented by relation(variable, value). Value can be a constant value or a variable.

Smatch score computes the maximum match number of triples among all possible variable mappings, and gets the F-score, precision, and recall. A detailed description can be found here.

In this case, smatch calculator chooses the variable mapping below:

a0(watch)-b0(watch) a1(boy)-b1(girl) a2(tv)-b2(boy)

which leads to matched triples and unmatched triples.

AMR 1 triples:

instance (a0, watch) ^ instance (a1, boy) ^ instance (a2, tv) ^ ARG0 (a0, a1) ^ ARG1 (a0, a2)

AMR 2 triples:

instance(b0, watch) ^ instance (b1, girl) ^ instance (b2, boy) ^ ARG0 (b0, b1) ^ ARG1 (b0, b2)

Smatch score: F-score: 0.60 (Precision: 0.60 (3/5) Recall: 0.60 (3/5))

Note 1: Blank line cannot exist between lines of an AMR, e.g. between the line "(w / watch) and ":ARG0 (b /boy)".

Note 2: Variables may be renamed in the output message to avoid duplicates.

### Computing scores of multiple AMRs

To compute smatch score of multiple AMRs, the input AMRs should be separated by a blank line, both in files or in the textarea.

For example,
(g / girl

(w / watch
:ARG0 (b / boy))

Smatch score will compute a single weighted score based on every pairs of AMRs. Redundant AMRs will be automatically ignored. Sentences are weighted according to their number of triples.

### Error/Warning Messages after Pressing "Upload" or "Submit"

Error Message

•   One AMR is blank
•   Your AMR contains error, e.g. you use the same variable twice or you have mismatched parentheses.
•   Warning Message

•   Possible errors, e.g. you use an undefined AMR tag.