Simple finetuning of LLaMA 3.2 reaches SMATCH F1 0.804 on the AMR 3.0 test set, matching the APT+Silver parser and coming within 0.05 of the Graphene state of the art.
SEMA: an Extended Semantic Evaluation Metric for AMR
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Abstract Meaning Representation (AMR) is a recently designed semantic representation language intended to capture the meaning of a sentence, which may be represented as a single-rooted directed acyclic graph with labeled nodes and edges. The automatic evaluation of this structure plays an important role in the development of better systems, as well as for semantic annotation. Despite there is one available metric, smatch, it has some drawbacks. For instance, smatch creates a self-relation on the root of the graph, has weights for different error types, and does not take into account the dependence of the elements in the AMR structure. With these drawbacks, smatch masks several problems of the AMR parsers and distorts the evaluation of the AMRs. In view of this, in this paper, we introduce an extended metric to evaluate AMR parsers, which deals with the drawbacks of the smatch metric. Finally, we compare both metrics, using four well-known AMR parsers, and we argue that our metric is more refined, robust, fairer, and faster than smatch.
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Evaluation of Finetuned LLMs in AMR Parsing
Simple finetuning of LLaMA 3.2 reaches SMATCH F1 0.804 on the AMR 3.0 test set, matching the APT+Silver parser and coming within 0.05 of the Graphene state of the art.