Pith. sign in

REVIEW 3 major objections 4 minor 29 references

SemSketches-2021: experimenting with the machine processing of the pilot semantic sketches corpus

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper presents a 915-sketch pilot corpus of Russian verb meanings and reports that the best automatic system for matching contexts to sketches achieves only 0.277 accuracy, showing the task is feasible but unsolved.

desk verdict A genuinely useful but narrowly scoped resource paper: first open corpus of Russian semantic sketches and a small shared task, with honest admissions about unquantified corpus errors and a single-judgment gold standard. read the letter →

arxiv 2505.17704 v1 pith:CQ5XG6R4 submitted 2025-05-23 cs.CL

Maria Ponomareva , Maria Petrova , Julia Detkova , Oleg Serikov , Maria Yarova This is my paper
classification cs.CL
keywords semanticsketcheswordframesemanticsrolelabelingsensedisambiguationcorpuslexicographyRussianlanguagesharedtask
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SemSketches-2021 introduces the first open corpus of semantic sketches for Russian: 915 verb meanings, each represented by a lexicographic table of the verb's most frequent semantic dependencies classified by semantic role. The paper argues that because these sketches carry the full Compreno semantic markup—actants as well as circumstantial dependents like Locative and Time—they capture aspects of predicate identity that actant-only role labeling misses, for instance the Locative slot separating 'be situated' from other senses of 'be'. To test whether machines can use the resource, the authors organized the SemSketches-2021 Shared Task, in which teams had to match anonymized sketches to contexts; the best system reached 0.277 accuracy on a manual gold set, far above the baseline but far below human performance. The take-home claim is that semantic sketches are a viable, human-interpretable counterpart to distributional embeddings, that automatic processing of them is possible, and that the main obstacle is the polysemy the sketches are designed to expose.

What carries the argument

The central object is the semantic sketch: a table for one verb sense listing its most frequent semantic dependencies, each tagged with a semantic role (Agent, Object, Locative, Time, etc.) and filled with ranked word fillers, built by statistical aggregation over the ABBYY Compreno parser's full semantic annotation of the General Internet-Corpus of Russian. The carrying mechanism of the paper's experiment is the Shared Task: participants receive anonymized sketches (roles and fillers but no predicate) plus contexts with the target predicate highlighted, and the evaluation metric is accuracy of context-to-sketch assignment. The paper also relies on the distinction between actant and circumstantial roles to argue that the sketches' information content goes beyond standard semantic role labeling.

What would settle it

Build an independent gold standard in which two or more annotators, working only from the contexts, assign each context to one of the 915 sketches; if inter-annotator agreement is low (e.g., kappa below 0.6), the task itself is under-determined and the 0.277 system score overstates the gap to human performance. Alternatively, re-run the matching task using only circumstantial slots (Locative, Time, modifiers) as features; if accuracy does not exceed actant-only features on predicates like 'находиться', the paper's claim about circumstantial dependencies identifying predicates better than actants is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that a corpus of 915 semantic sketches for Russian verbs can be built automatically from full semantic parsing and can support a concrete, measurable processing task. The paper shows that matching a predicate's context to its sketch is easy for humans yet hard for current language-model-based systems: in the SemSketches-2021 Shared Task the winning system scored 0.277 accuracy on the Manual Dev set of 100 sketches and 4,347 contexts, with two other systems at 0.127 and 0.121 and a Masked-LM baseline at 0.0035. The paper further establishes that the sketches' circumstantial dependencies (Locative, Time, modifiers) are not noise: for predicates like 'находиться', the Locative slot is more diagnostic of verb sense than the actant slots, so a full semantic model outperforms actant-only semantic role labeling in principle.

Load-bearing premise

The whole resource and all reported accuracies depend on the Compreno parser's semantic role assignments being correct and on the authors' single-judgment manual gold standard being a fair ground truth; if the parser's roles or the gold mapping are wrong in systematic ways, the sketches' representativeness and the evaluation numbers would be undermined.

Editorial extensions

If this is right

  • If the sketches are representative, the 915-entry corpus gives Russian lexicography and NLP a ready benchmark for word-sense disambiguation and semantic role labeling that is interpretable by construction.
  • The demonstrated 0.277 ceiling on automatic matching implies that current language models do not yet recover the role-structured compatibility information the sketches encode, pointing to a concrete gap in lexical semantics.
  • Because circumstantial slots like Locative and Time carry disambiguating power for predicates such as position verbs, semantic role labeling systems that ignore adjuncts are leaving useful signal on the table.
  • Releasing the corpus with its Dev and Manual Dev splits and the evaluation script makes the task reproducible and reusable as a few-shot or unsupervised benchmark for Russian.
  • The authors' plan to embed the sketches in the General Internet-Corpus of Russian and to extend the model to English indicates the resource is intended to grow beyond the pilot scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension: re-run the same Shared Task setup on the released corpus with a system that explicitly encodes circumstantial slots (e.g., separates Locative from Agent) to see whether the 0.277 ceiling rises; the paper's own analysis predicts it would.
  • The sketches could serve as a natural 'probing' format for contextual embeddings, with each role slot acting as a structured target; this would operationalize the paper's closing suggestion and give a role-level, not just sentence-level, probe.
  • Because the Manual Dev gold standard relies on single-judgment annotation, the measured accuracies have an unknown labeling-noise floor; an inter-annotator study on a sample of the 100 Manual Dev sketches would calibrate whether 0.277 is a system limit or partly a labeling-noise limit.
  • If the corpus grows to English, the same matching task becomes a cross-lingual few-shot benchmark, testing whether role-structured sketches transfer across languages.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces SemSketches-2021, a pilot open corpus of 915 semantic sketches for Russian verbs, built with the ABBYY Compreno parser, and describes a shared task in which participants must match anonymized sketches to sentence contexts containing the corresponding predicates. The paper reports the design of the corpus, the shared task setup, a baseline, and the results of three participating systems, with the best system achieving 0.277 accuracy on the Manual Dev gold standard. The authors argue that the sketches are representative of verb compatibility and that the inclusion of circumstantial semantic dependencies (not only actants) can help identify verb meanings.

Significance. If the corpus-quality and gold-standard concerns are addressed, this is a potentially valuable open resource for semantic role labeling, word sense disambiguation, and corpus lexicography, and the public GitHub release with evaluation scripts is a concrete strength. The shared task provides a benchmark for sketch-to-context matching. However, the central claims of representativeness and of the usefulness of circumstantial roles are not yet quantitatively established, and the main evaluation numbers rest on a gold standard whose reliability is not documented.

major comments (3)
  1. [Sections 3.1 and 5] The paper does not provide a quantitative error rate for the full corpus; it explicitly states in Section 5 that the total number of mistakes in the whole corpus was not evaluated. Since Section 3.1 documents nontrivial systematic errors (e.g., the sketch for pисать 'to paint' containing 'to write letters' in the Object slot, and the sketch for выходить:TO_WALK with an incorrect Agent Metaphoric slot), the representativeness of the 915-sketch corpus is not established. This directly affects the interpretation of the shared task scores, because systems may be matching noisy sketches. The authors should report an error audit on a random sample of the full corpus, with breakdown by error type and, ideally, a comparison between the Manual Dev subset and the rest of the corpus.
  2. [Section 3.1 (Manual Dev) and Table 3] The gold standard for Manual Dev (100 sketches, 4347 contexts) appears to have been created by the authors without any reported inter-annotator agreement or detailed annotation procedure. The main evaluation result, best Manual Dev accuracy 0.277, is therefore a single-judgment estimate. The authors should describe how many annotators constructed the gold mapping and report agreement statistics (e.g., Cohen's kappa or percentage agreement) on a subset; without this, the comparative scores in Table 3 are difficult to assess.
  3. [Section 4] The claim that circumstantial dependencies (e.g., Locative) identify predicates better than actants is supported only by anecdotal examples and is not backed by any quantitative experiment in the paper. Since this is a key motivation for the semantic-sketch approach, the authors should either provide empirical evidence (e.g., an ablation in the shared task comparing actant-only vs. full sketches) or moderate the claim to reflect that it is a hypothesis.
minor comments (4)
  1. [Section 3.3] The random baseline accuracy is not reported, so the statement that the baseline 'overperforming the random baseline' is not quantified. Moreover, the baseline Manual Dev score of 0.0035 is below the uniform chance rate of 1/100 = 0.01 on a 100-sketch test set; please explain this surprising result or correct it.
  2. [Sections 3.4 and 3.5] The team name appears as 'good501' in Table 3 and Section 3.4 but as '501good' in Section 3.5; please normalize the spelling.
  3. [Section 3.3] The baseline description mentions top-N mask replacements and the formula uses MLM1000, but it is not stated how N=1000 was chosen or whether it is a free parameter; please clarify.
  4. [Section 4] The paper states that semantic sketches can help with SRL and WSD, but the reported shared task is a sketch-context matching task, not a direct evaluation of SRL or WSD; please clarify the connection or provide supporting results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: shared-task evaluation is measured against an independent manual gold standard; self-citations are background rather than load-bearing.

full rationale

The paper's central empirical result is the SemSketches Shared Task, where systems map anonymized sketches to context sentences. The gold standard (Manual Dev) was created by manually selecting 100 sketches and evaluating the corresponding contexts; it is not derived from the submitted systems or from the sketches by construction. The evaluation metric is standard accuracy against this held-out manual mapping. No parameter is fitted to the gold labels and then renamed as a prediction. The sketches themselves are produced by the external ABBYY Compreno parser, and the contexts are collected from non-overlapping sources, so the matching task is not self-referential. The paper's self-citations to prior work on the semantic sketch concept and the parser's semantic markup are background support for the resource's design, not the source of the reported accuracy numbers. The acknowledged limitations—that the error rate was checked only on the Manual Dev subset and that inter-annotator agreement is not reported—concern the reliability and representativeness of the corpus and gold standard, not circularity. There is no equation in the paper that reduces a derived result to its own input, no fitted input is called a prediction, and no uniqueness claim is imported from the authors' prior work to force a conclusion. Thus no circular step is exhibited, and the appropriate score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the reliability of the proprietary Compreno parser, the author-created gold standard, and the context collection design. There are no invented physical or mathematical entities. Two hand-chosen thresholds affect the corpus composition and the baseline performance, but they are not fitted to the evaluation data.

free parameters (2)
  • dependent node count threshold for sketch inclusion = 2000
    Sketches were included only if the number of dependent nodes exceeded 2000; this threshold is chosen by hand to balance quality and quantity (Section 3.1).
  • MLM top-N candidates in baseline = 1000
    The baseline computes sketch scores using the intersection of top-1000 MLM replacements (Section 3.3).
assumptions (3)
  • domain assumption The ABBYY Compreno parser provides correct semantic roles, semantic classes, and syntactic dependency parses for the Russian texts used.
    The entire corpus of semantic sketches is built on this parser's output (Sections 2 and 3.1). If the parser is systematically wrong, the sketches are wrong.
  • domain assumption The Manual Dev gold standard mapping between sketches and contexts is correct.
    The gold standard was manually created by the authors without inter-annotator agreement or external validation (Section 3.1). The reported system accuracies depend on this mapping.
  • domain assumption The contexts collected for each predicate meaning correctly represent that meaning and do not overlap with the corpus the sketches were built on.
    The task design assumes non-overlap and correct assignment of contexts to meanings (Section 3.1). Overlap or mislabeling would inflate or distort the measured accuracies.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SemSketches-2021: experimenting with the machine processing of the pilot semantic sketches corpus." pith.science (2026). https://pith.science/paper/CQ5XG6R4

@misc{pith2026250517704,
  author       = {Pith},
  title        = {Pith review of: SemSketches-2021: experimenting with the machine processing of the pilot semantic sketches corpus},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CQ5XG6R4}},
  note         = {Machine review of arXiv:2505.17704}
}
read the original abstract

The paper deals with elaborating different approaches to the machine processing of semantic sketches. It presents the pilot open corpus of semantic sketches. Different aspects of creating the sketches are discussed, as well as the tasks that the sketches can help to solve. Special attention is paid to the creation of the machine processing tools for the corpus. For this purpose, the SemSketches-2021 Shared Task was organized. The participants were given the anonymous sketches and a set of contexts containing the necessary predicates. During the Task, one had to assign the proper contexts to the corresponding sketches.

Figures

Figures reproduced from arXiv: 2505.17704 by the authors.

Figure 1
Figure 1. the sketch for the verb «страдать:SUFFERING_AND_TORMENT» (‘to suffer’). Here the elements of the sketch are given with their rough translations. The participants of the Shared Task got the same representations, but did not get the titles of the sketches. However, as the pictures above demonstrate, it does not seem difficult for a human to guess the proper predicates for the sketches, which allows us to regard the sk… view at source ↗
Figure 2
Figure 2. the sketch for the verb «готовить:TO_PREPARE_FOOD_SUBSTANCE» (‘to prepare food, to cook’). Here the elements of the sketch are given with their rough translations. 3 The SemSketches Shared Task To explore the semantic sketches as far as their quality and representativeness are concerned, we have created the pilot corpus of Russian semantic sketches and made it the basis for the SemSketches Shared Task. The problem w… view at source ↗
Figure 3
Figure 3. the semantic sketch for the verb «выходить:идти:TO_WALK» ‘to go out’). Here the elements of the sketch are given with their rough translations. ID dev.sent.rus.116 target наполнились start 46 end 57 context Когда доктор вошел, она вспыхнула, и глаза ее наполнились слезами [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 26 canonical work pages

  1. [1]

    Matching semantic sketches to predicates in context using the BERT model // Proc Dialogue, Russian International Conference on Computational Linguistics

    Aleksandrova Polina, Mokhova Anna, and Nikolaenkova Maria. Matching semantic sketches to predicates in context using the BERT model // Proc Dialogue, Russian International Conference on Computational Linguistics. — Moscow. — 2021

    work page 2021

  2. [2]

    Devlin Jacob, Chang Ming-Wei, Lee Kenton, and Toutanova Kristina. BERT: Pre-training of Deep Bidi- rectional Transformers for Language Understanding // Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, V olume 1 (Long and Short Papers). — Minneapolis, Minnesota...

    work page 2019

  3. [3]

    Big and diverse is beautiful: A large corpus of Russian to study linguistic variation // Proc 8th Web as Corpus Workshop (W AC-8)

    Piperski Alexander, Belikov Vladimir, Kopylov Nikolay, Selegey Vladimir, and Sharoff Serge. Big and diverse is beautiful: A large corpus of Russian to study linguistic variation // Proc 8th Web as Corpus Workshop (W AC-8). — 2013

    work page 2013

  4. [4]

    Corpus as language: from scalability to register variation // Proc Dialogue, Russian International Conference on Computational Linguistics

    Belikov Vladimir, Kopylov Nikolay, Piperski Alexander, Selegey Vladimir, and Sharoff Serge. Corpus as language: from scalability to register variation // Proc Dialogue, Russian International Conference on Computational Linguistics. — Bekasovo. — 2013

    work page 2013

  5. [5]

    Peters Matthew, Neumann Mark, Iyyer Mohit, Gardner Matt, Clark Christopher, Lee Kenton, and Zettlemoyer Luke. Deep Contextualized Word Representations // Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, V olume 1 (Long Papers). — New Orleans, Louisiana : Ass...

    work page 2018

  6. [6]

    Tan Zhixing, Wang Mingxuan, Xie Jun, Chen Yidong, and Shi Xiaodong. Deep Semantic Role Labeling With Self-Attention // Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (...

    work page 2018

  7. [7]

    Differential Semantic Sketches For Russian Internet-Corpora // Proc Dialogue, Russian International Conference on Com- putational Linguistics

    Detkova Julia, Novitskiy Valeriy, Petrova Maria, and Selegey Vladimir. Differential Semantic Sketches For Russian Internet-Corpora // Proc Dialogue, Russian International Conference on Com- putational Linguistics. — Moscow. — 2020

    work page 2020

  8. [8]

    Distributed Representations of Words and Phrases and their Compositionality // Neural and Information Processing System (NIPS)

    Mikolov Tomas, Sutskever Ilya, Chen Kai, Corrado Greg, and Dean Jeffrey. Distributed Representations of Words and Phrases and their Compositionality // Neural and Information Processing System (NIPS). — 2013. — Access mode: https://papers.nips.cc/paper/ 5021-distributed-representations-of-words-and-phrases-and-their-compositionality. pdf

    work page 2013

Show all 29 references
  1. [9]

    Enriching Word Vectors with Subword Information // Transactions of the Association for Computational Linguistics

    Bojanowski Piotr, Grave Edouard, Joulin Armand, and Mikolov Tomas. Enriching Word Vectors with Subword Information // Transactions of the Association for Computational Linguistics. —

  2. [10]

    A Synopsis of Linguistic Theory 1930-1955 // Studies in Linguistic Analysis

    Firth J. A Synopsis of Linguistic Theory 1930-1955 // Studies in Linguistic Analysis. — Philo- logical Society, Oxford, 1957. — reprinted in Palmer, F. (ed. 1968) Selected Papers of J. R. Firth, Longman, Harlow

    1930

  3. [11]

    Generalized Inference with Mul- tiple Semantic Role Labeling Systems // Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005)

    Koomen Peter, Punyakanok Vasin, Roth Dan, and Yih Wen-tau. Generalized Inference with Mul- tiple Semantic Role Labeling Systems // Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005). — Ann Arbor, Michigan : Association for Computa- tion...

    2005

  4. [12]

    Automatic labeling of semantic roles // Computational Linguis- tics

    Gildea Daniel and Jurafsky Daniel. Automatic labeling of semantic roles // Computational Linguis- tics. — 2002. — V ol. 28, no. 3. — P. 245–288

    2002

  5. [13]

    Fine-tuned Language Models for Text Classification // CoRR

    Howard Jeremy and Ruder Sebastian. Fine-tuned Language Models for Text Classification // CoRR. — 2018. — V ol. abs/1801.06146. —1801.06146

    2018 arXiv

  6. [14]

    Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language

    Kuratov Yuri and Arkhipov Mikhail. Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language. — 2019. — 1905.07213

    2019 arXiv

  7. [15]

    Unsupervised Semantic Role Induction with Graph Partitioning // Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing

    Lang Joel and Lapata Mirella. Unsupervised Semantic Role Induction with Graph Partitioning // Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. — Ed- inburgh, Scotland, UK. : Association for Computational Linguistics. — 2011. — July. — P. 1320–

    2011

  8. [16]

    Cheng Jianpeng, Reddy Siva, Saraswat Vijay, and Lapata Mirella. Learning Structured Natural Lan- guage Representations for Semantic Parsing // Proceedings of the 55th Annual Meeting of the Associ- ation for Computational Linguistics (V olume 1: Long Papers). — Vancouver, Canad...

    2017

  9. [17]

    Semantic role labeling

    Palmer Martha Stone. Semantic role labeling. Synthesis lectures on human language technologies ; #6. — San Rafael, Calif.] : Morgan & Claypool Publishers, 2010. — ISBN:9781598298314

    2010

  10. [18]

    The Compreno Semantic Model: The Universality Problem // International Journal of Lexicography

    Petrova M.A. The Compreno Semantic Model: The Universality Problem // International Journal of Lexicography. — 2013. — 12. — V ol. 27, no. 2. — P. 105–129. — https://academic.oup.com/ijl/article- pdf/27/2/105/2731792/ect038.pdf

    2013

  11. [19]

    Probing Pretrained Language Models for Lexical Semantics // ArXiv

    Vuli ´c Ivan, Ponti E., Litschko Robert, Glavas Goran, and Korhonen A. Probing Pretrained Language Models for Lexical Semantics // ArXiv. — 2020. — V ol. abs/2010.05731

    2020 arXiv

  12. [20]

    Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? — 2021

    Ravichander Abhilasha, Belinkov Yonatan, and Hovy Eduard. Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? — 2021. — 2005.00719

    2021 arXiv

  13. [21]

    Reimers Nils and Gurevych Iryna. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks // Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). —...

    2019

  14. [22]

    The Sketch Engine: ten years on // Lexicography

    Kilgarriff Adam, Baisa V ´ıt, Bu ˇsta Jan, Jakub ´ıˇcek Milo ˇs, Kov ´aˇr V ojtˇech, Michelfeit Jan, Rychl´y Pavel, and Suchomel V ´ıt. The Sketch Engine: ten years on // Lexicography. — 2014. — P. 7–36

    2014

  15. [23]

    Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe // Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies

    Straka Milan and Strakov ´a Jana. Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe // Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. — Vancouver, Canada : Association for Computational Linguistics. —

    2017

  16. [24]

    V ., Druzhkin K

    Anisimovich K. V ., Druzhkin K. Ju., Minlos F. R., Petrova M. A., Selegey V . P., and Zuev K. A. Syn- tactic and semantic parser based on ABBYY Compreno linguistic technologies // Proc Dialogue, Russian International Conference on Computational Linguistics. — Bekasovo. — 2012

    2012

  17. [25]

    Syntax for Semantic Role Labeling, To Be, Or Not To Be // Proceedings of the 56th Annual Meeting of the Association for Computational Linguis- tics (V olume 1: Long Papers)

    He Shexia, Li Zuchao, Zhao Hai, and Bai Hongxiao. Syntax for Semantic Role Labeling, To Be, Or Not To Be // Proceedings of the 56th Annual Meeting of the Association for Computational Linguis- tics (V olume 1: Long Papers). — Melbourne, Australia : Association for Computationa...

    2018

  18. [26]

    — August. — P. 88–99. — Access mode: http://www.aclweb.org/anthology/ K/K17/K17-3009.pdf

  19. [29]

    Conneau Alexis, Kruszewski German, Lample Guillaume, Barrault Lo ¨ıc, and Baroni Marco. What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties // Proceedings of the 56th Annual Meeting of the Association for Computational Linguistic...

    2018

  20. [1331]

    — Access mode: https://www.aclweb.org/anthology/D11-1122

  21. [2017]

    — V ol. 5. — P. 135–146

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.