Quick Tutorial of Smatch Calculator     
  Smatch Score Interpretation and Web Interface Tutorial         Version 0.1   April 28, 2012

  Smatch Score Interpretation

  Assume we have two AMRs:

  (a0 / watch                               (b0 / watch
      :ARG0 (a1 / boy)                             :ARG0 (b1 / girl)
      :ARG1 (a2 / tv))                               :ARG1 (b2 / boy))

  An AMR can be viewed as conjunction of logical propositions (triples). A triple can be represented by relation(variable, value). Value can be a constant value or a variable.

  Smatch score computes the maximum match number of triples among all possible variable mappings, and gets the F-score, precision, and recall. A detailed description can be found here.

  In this case, smatch calculator chooses the variable mapping below:

  a0(watch)-b0(watch) a1(boy)-b1(girl) a2(tv)-b2(boy)

  which leads to matched triples and unmatched triples.

  AMR 1 triples:

  instance (a0, watch) ^ instance (a1, boy) ^ instance (a2, tv) ^ ARG0 (a0, a1) ^ ARG1 (a0, a2)

  AMR 2 triples:

  instance(b0, watch) ^ instance (b1, girl) ^ instance (b2, boy) ^ ARG0 (b0, b1) ^ ARG1 (b0, b2)

  Smatch score: F-score: 0.60 (Precision: 0.60 (3/5) Recall: 0.60 (3/5))

  Note 1: Blank line cannot exist between lines of an AMR, e.g. between the line "(w / watch) and ":ARG0 (b /boy)".

  Note 2: Variables may be renamed in the output message to avoid duplicates.

Computing scores of multiple AMRs

 To compute smatch score of multiple AMRs, the input AMRs should be separated by a blank line, both in files or in the textarea.

 For example,
(g / girl
    :mod (s / sad))

 (w / watch
    :ARG0 (b / boy))

 Smatch score will compute a single weighted score based on every pairs of AMRs. Redundant AMRs will be automatically ignored. Sentences are weighted according to their number of triples.

  Error/Warning Messages after Pressing "Upload" or "Submit"

  Error Message

  •   One AMR is blank
  •   Your AMR contains error, e.g. you use the same variable twice or you have mismatched parentheses.

    •   Warning Message

  •   Possible errors, e.g. you use an undefined AMR tag.