Overview of the Sensemaking Task at the ELOQUENT 2025 Lab: LLMs as Teachers, Students and Evaluators

Ond\v{r}ej Bojar; Pavel \v{S}indel\'a\v{r}

REVIEW 3 major objections 6 minor 11 references

Overview of the Sensemaking Task at the ELOQUENT 2025 Lab: LLMs as Teachers, Students and Evaluators

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM-as-a-Judge systems rate garbled and mismatched question-answer pairs as acceptable, so they are unreliable scorers of answer quality.

desk verdict A transparent shared-task overview with a real over-leniency signal, but the headline overstates the evidence by calling adversarial scores 'acceptable' without per-item human labels or a defined threshold. read the letter →

arxiv 2507.12143 v1 pith:K3DYRUTU submitted 2025-07-16 cs.CL

Pavel Šindelář , Ondřej Bojar This is my paper

classification cs.CL

keywords largelanguagemodelsgenerativeLLM-as-a-Judgequestiongenerationansweringadversarialtestingautomaticevaluationtextcomprehension

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports the 2025 Sensemaking shared task, which tests whether large language models can 'make sense' of source material in three classroom-inspired roles: generating questions (Teacher), answering them from the material alone (Student), and scoring answers on a 0–100 scale (Evaluator). The central result is negative: in adversarial tests, systems using the LLM-as-a-Judge paradigm routinely rate garbled question-answer pairs and answers to mixed-up questions as acceptable, with mean scores of 33 to 67 on corrupted items versus 83 on intact ones. The paper argues that LLM-based evaluators are therefore not reliable automatic scorers of answer quality, and that the current enthusiasm for LLMs as judges needs a caveat. It also reports that question-generation quality is difficult to measure automatically and that student models answer passably but often do not restrict themselves to the supplied texts.

What carries the argument

The load-bearing mechanism is the adversarial test set: deliberately corrupted question-answer pairs created by swapping questions between sections, shuffling words in answers, replacing answers with random text, and pairing questions with answers that are topically similar but unrelated. These items are the yardstick that reveals whether an evaluator actually checks the match between question, answer, and source text. The paper also uses ROUGE-L Recall and a brief manual Likert questionnaire as reference standards for judging whether evaluator scores are wrong, and relative ranking prompts for the Teacher evaluation.

What would settle it

Re-run the adversarial tests from Section 6 with a new LLM-as-a-Judge system and a fixed corruption set: if word-shuffled answers, swapped questions, and random-text answers all receive mean scores below 10 while intact pairs score above 80 on a 0–100 scale, the reported failure would not hold for that system. The paper's claim would also be weakened if a third reference standard—multiple expert raters, say—showed that the corrupted items genuinely deserve mid-range scores.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the LLM-as-a-Judge paradigm—using a large language model to assign a numeric quality score to an answer—fails precisely where automatic evaluation is most needed. When evaluator systems were given answers unrelated to the question, word-shuffled answers, or questions and answers drawn from mismatched source sections, average scores fell only to the 33–67 range instead of near zero. Evaluator rankings then disagreed with both ROUGE-L Recall and a small manual review, leading the paper to conclude that Evaluator ratings are generally unreliable. The same experiments showed that teacher-generated reference answers were often poor and that student systems leaned on shallow word overlap rather than genuine comprehension of the input.

Load-bearing premise

The paper's broad conclusion that Evaluator ratings are generally unreliable rests on treating ROUGE-L Recall and one rater's brief manual Likert scores as correct reference standards for when an answer is wrong; if those references are themselves weak, the disagreement could be between two imperfect metrics rather than a clear failure of the LLM judges.

Editorial extensions

If this is right

Automatic evaluation pipelines that use an LLM as judge should include adversarial controls before trusting any numeric score.
Disagreement between two evaluator systems on the same item can serve as a signal that the item is problematic and its score should be treated with caution.
Question-generation quality remains hard to measure automatically; simple embedding-based coverage, diversity, and relevance metrics did not distinguish good from bad question sets.
Student models need stronger grounding enforcement, since they performed acceptably on average but did not reliably confine answers to the source text.
Adversarial inputs were the most reliable evaluation tool in this study, more trustworthy than either LLM judges or reference-based metrics alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the pattern of failures suggests LLM judges are scoring topical similarity rather than logical entailment, so a well-formed answer about the same domain earns credit even when it answers a different question.
The paper's agreement tables imply a testable extension: evaluator disagreement could be used as a cheap automatic detector of problematic items, since agreement drops on adversarial inputs.
A natural next experiment is to turn the adversarial protocol itself into a benchmark sanity check for any new LLM-as-a-Judge system before deployment.
Because the paper's reference standards are imperfect, the strongest part of the evidence is the face validity of the adversarial tests; a replication with independent expert raters would make the claim crisper.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a worthwhile shared-task overview with a genuine negative result—LLM-as-a-Judge systems are over-lenient on adversarial QA pairs—but the headline claim is a bit stronger than the evidence supports. The adversarial stress tests are new measurements in a multilingual, multi-material setting, and that matters. The paper is also unusually transparent about its own limitations.

What's good: the three-track setup (Teacher/Student/Evaluator) is clean, the adversarial construction is sensible, and the negative result is consistent across tables: scores drop from ~83 to ~59, ~33, ~67 under different corruptions, but stay far above zero. That supports 'over-lenient' even if not 'erroneously rated as acceptable.' The paper's own manual review, brief as it was, also aligns with ROUGE-L against the evaluators' rankings. The limitations section is honest and specific.

Soft spots: only two valid submissions per track; data and code are not publicly released (only on request); the GPT-4.1 family appears as both judge and baseline; ROUGE-L is a weak reference and the manual evaluation was per-answer brief. The stress-test note is fair: there are no per-item human labels on the adversarial items, and no pre-defined threshold for 'acceptable.' So the headline claim rests partly on the assumption that the adversarial items obviously deserve near-zero scores. In many cases that's true, but some adversarial items are topically related to the correct answer and could plausibly get partial credit. The paper acknowledges this in Section 6.2. The claim is directionally right but should be phrased as 'judges are systematically over-lenient' rather than 'judges rate garbage as acceptable.'

Who it's for: anyone building automated evaluation pipelines in education or fact-checking, and shared-task organizers. The Teacher evaluation metrics (relevance/coverage/diversity) are a modest but usable contribution. This deserves a serious referee; it is not a desk reject. I would send it out with a request to soften the abstract and, if feasible, add per-item human labels or at least a clearer operationalization of 'acceptable.'

Referee Report

3 major / 6 minor

Summary. The paper reports on the ELOQUENT 2025 Sensemaking shared task, a three-track evaluation of LLM-based systems acting as Teachers (generating questions from source material), Students (answering those questions), and Evaluators (scoring answers on 0–100). With only two valid submissions per track plus GPT-4.1-nano baselines, the authors evaluate question generation using embedding-based coverage/diversity measures and LLM ranking, evaluate answering using ROUGE-L Recall and brief manual Likert ratings, and evaluate judging using adversarial inputs (swapped questions, shuffled words, and cross-material mismatches). The headline finding is that Evaluator systems are unreliable: adversarial scores drop from about 83 to 59/33/67 under different corruptions but remain far above what the authors argue should be near zero, leading them to conclude that LLM-as-a-Judge systems erroneously rate garbled or mismatched question-answer pairs as acceptable.

Significance. If the central claim holds, the paper provides a concrete, reproducible demonstration that LLM-as-a-Judge evaluation is over-lenient and can be fooled by surface-level perturbations, with direct implications for using LLMs in educational assessment and fact-checking pipelines. The main strengths are the adversarial test methodology (Tables 13–15; Figure 2), the comparison of multiple evaluation strategies (embedding metrics, LLM ranking, ROUGE-L, manual Likert), and the transparency about instability and limitations, including the appendix's systematic prompt-order and model-variation stability checks. These are valuable contributions. However, the central claim is currently stronger than the evidence: the paper never obtains human judgments on the adversarial items themselves, and it never defines what score constitutes 'acceptable.' The measured drops support over-leniency, but the specific wording 'erroneously rate as acceptable' requires either an external ground-truth label per item or a pre-specified threshold. This gap is fixable but should determine the revision's focus.

major comments (3)

[Section 6.1–6.2 and the abstract] The claim that evaluators 'erroneously rate garbled question-answer pairs and answers to mixed-up questions as acceptable' is not directly established by Tables 13–15. The tables show mean adversarial scores of 59.44, 32.86, and 67.40 for Gemma-3-27b-it and 26.38, 13.33, and 49.38 for the baseline, but the paper provides no human judgments for these specific items and no predefined acceptability threshold. The assertions 'should fall close to 0' (Section 6.1) and 'almost always completely nonsensical' (Section 6.2) are offered as qualitative interpretation, not measured ground truth. Because Table 15's mismatched answers come from the same section and are often topically related, scores in the 30–60 range can reflect a partial-credit judgment ('partially useful') rather than a failure to notice the mismatch. Please either collect per-item human annotations on a sample of adversarial items and report the distribution of answers the authors consider 'acceptable,' or explicitly weaken the claim to 'notably over-lenient' and define the score threshold used for 'acceptable.'
[Section 5.3 and Sections 6.5/6.7] The broader conclusion that 'Evaluator ratings are generally unreliable' is partly supported by disagreement between Evaluator scores and ROUGE-L Recall and by a quick manual evaluation. Section 5.3 states 'the examination per answer was very brief,' and Section 6.5 admits 'the inadequacies of ROUGE-L Recall.' With a single evaluator and a known-imperfect metric, the observed disagreement could be between two imperfect measures rather than evidence that the Evaluator is wrong. This does not undermine the adversarial results, which have face validity, but it does weaken the generalization drawn in Section 6.7. I recommend either adding adjudicated human labels for a sample of Evaluator-rated non-adversarial items or explicitly delimiting the 'unreliable' conclusion to the adversarial settings.
[Section 6.2, final paragraph] The paper itself concedes that some Student systems correctly detect unanswerable questions and respond with a refusal or hedge, and that Evaluators then score such responses highly (Figure 2 shows the baseline receiving a score of 100 for such a response). This means the aggregate means in Table 15 conflate two different cases: (a) the Evaluator correctly rewards a Student's explicit denial that the question is answerable, and (b) the Evaluator fails to notice the mismatch and gives a high score to a genuinely off-topic answer. The claim in the abstract concerns case (b), but the current aggregate evidence cannot separate it from case (a). Please report the frequency of Student responses that explicitly flag unanswerability and provide a breakdown of adversarial scores for that subset, or otherwise acknowledge that the headline result aggregates over both behaviors.

minor comments (6)

[Section 4.1, Eq. (2)] The quantile thresholds (0.5 and 0.95) in the relevance metric are introduced without justification; a short sensitivity analysis or a reference justifying these values would help readers interpret the Teacher auto-evaluation.
[Section 5.2 / Table 10] The Pearson correlations of ROUGE-L Recall scores are computed on data that are heavily zero-inflated (e.g., Llama-3-8b has mean 02±11 in Table 6); rank correlations or a breakdown restricted to answerable items would be more informative.
[Section 6.6 / Table 21] The text says Gemma-3-27b-it 'performs on par' with the GPT-4.1-nano baseline, and indeed the overall accuracies are both 0.50, but the per-class accuracies differ substantially (e.g., class-1 accuracy 0.01 vs 0.10); please state the chance level (0.33) and note the high variance so readers can judge whether these differences matter.
[Appendix A] The stability tests vary the evaluation model and the order of entries in the prompt, but not the prompt wording itself; the claim in Section 7 that Teacher evaluation is 'significantly more stable' is therefore based on a single prompt template and should be phrased more cautiously.
[Tables 13–15] The differences between non-adversarial and adversarial scores are reported as means ± standard deviation, and the standard deviations are large (e.g., 82.90±27.49 vs 67.40±40.42 in Table 15); reporting confidence intervals or paired difference tests would make the reliability of the observed drops clearer.
[Throughout] There are several typos and spacing issues in the text, including 'whisper-turbomodel' (Section 3) and 'no anwers' (Table 9); a careful proofreading pass is recommended.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central LLM-as-a-Judge claim rests on adversarial face validity and external references (ROUGE-L from [3], brief manual Likert ratings); the one same-family judge chain (GPT-4.1-mini judging GPT-4.1-nano) is explicitly cross-checked with the independent Gemini-2.5-flash, and the self-citations are non-load-bearing provenance.

full rationale

The paper's derivation chain is substantially self-contained. The central conclusion—that Evaluator systems using the LLM-as-a-Judge paradigm erroneously rate garbled and mismatched pairs as acceptable—is supported by adversarial probes whose face validity does not depend on any quantity fitted inside the paper: Tables 13-15 report mean scores of 13-67 on corrupted items versus 65-83 on intact ones, and the judgment that such items deserve near-zero scores is an explicit design prior (Section 6.1: 'Since these answers are almost always completely unrelated to the questions, the average rating should fall close to 0'), not a quantity derived from the paper's own equations. A challenge to that prior is a validity or correctness concern, not a circular reduction. Second, the reference standards used to assess Evaluators are external: ROUGE-L Recall (cited to [3], an outside work), the brief manual Likert questionnaire (Figure 6), and cross-model stability checks. The paper itself concedes the weakness of these references—Section 5.3: 'The time was limited, so the examination per answer was very brief' and Section 6.5: 'it should also be noted that some Student answers to adversarial questions do achieve a high rating purely because of the inadequacies of ROUGE-L Recall'—so the skeptical concern about reference quality is already acknowledged in the text and limits the scope of the claim rather than making it self-confirming. Third, the one near-self evaluation chain, GPT-4.1-mini judging the GPT-4.1-nano Teacher baseline, is explicitly examined in Section A and probed with Gemini-2.5-flash, an independent model family; the rankings barely change, and a human manual revision (Table 5) partially anchors the result. The baseline's test-set troubleshooting is disclosed ('we also troubleshooted the baseline on the test set. This means it is artificially stronger when compared to the contestant systems'), so it is not a hidden fit renamed as a prediction. Finally, the self-citations ([1], [8], [11]) are shared-task and team-report provenance, not load-bearing premises, and no uniqueness theorem or ansatz is imported from prior author work. One reporting inconsistency exists—Section 6.7's prose claims the Evaluator ranking places the GPT baseline last while Table 23 shows it highest (73.04)—but this is a data-integrity issue, and the adversarial evidence for the headline claim stands independently of it. Verdict: no significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No fitted theoretical parameters are present. The ledger consists of measurement and task-design assumptions. The most load-bearing choices are the trust placed in ROUGE-L and the minimal manual evaluation, plus the assumption that adversarial corruptions should score near zero. The paper acknowledges most of these explicitly, which strengthens transparency but does not remove the burden.

free parameters (3)

Relevance quantile thresholds = 0.5 and 0.95
Used in Eq. (2) to threshold embedding similarities; chosen by hand without sensitivity analysis, and they directly shape the relevance, coverage and diversity scores in the Teacher track.
Text window size and overlap = not specified
Section 4.1 segments material into overlapping windows; the window size and stride are not reported but affect all three automatic Teacher metrics.
ROUGE-L class bins = [0,33), [33,66), [66,100]
Used in Section 6.5 to measure agreement between Evaluator scores and ROUGE-L classes; the choice sets the chance-level accuracy of 0.33 that the paper compares against.

assumptions (5)

domain assumption The ability to create questions that assess understanding of arbitrary material indicates understanding of the material.
Section 1 states this assumption explicitly; if question generation can be driven by shallow cues, the Teacher track does not measure comprehension.
domain assumption Adversarially corrupted inputs (shuffled words, swapped questions, unrelated answers) should receive scores close to zero.
Used as the reference standard in Sections 6.1 and 6.2; plausible but not independently validated.
domain assumption ROUGE-L Recall and single-evaluator manual Likert ratings are adequate proxies for answer quality.
Sections 5.1 to 5.3 rely on these measures; the paper itself acknowledges their limitations, making this a load-bearing measurement choice.
domain assumption The submitted systems are representative enough to draw conclusions about LLM-as-a-Judge behavior.
Only two valid submissions per track, as stated in Section 9; the small sample limits external generalization.
domain assumption Cosine similarity of sentence embeddings captures question-text relevance.
Section 4.1 builds all automatic Teacher metrics on this equivalence, which is a modeling assumption rather than an established fact.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Overview of the Sensemaking Task at the ELOQUENT 2025 Lab: LLMs as Teachers, Students and Evaluators." pith.science (2026). https://pith.science/paper/K3DYRUTU

@misc{pith2026250712143,
  author       = {Pith},
  title        = {Pith review of: Overview of the Sensemaking Task at the ELOQUENT 2025 Lab: LLMs as Teachers, Students and Evaluators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K3DYRUTU}},
  note         = {Machine review of arXiv:2507.12143}
}

read the original abstract

ELOQUENT is a set of shared tasks that aims to create easily testable high-level criteria for evaluating generative language models. Sensemaking is one such shared task. In Sensemaking, we try to assess how well generative models ``make sense out of a given text'' in three steps inspired by exams in a classroom setting: (1) Teacher systems should prepare a set of questions, (2) Student systems should answer these questions, and (3) Evaluator systems should score these answers, all adhering rather strictly to a given set of input materials. We report on the 2025 edition of Sensemaking, where we had 7 sources of test materials (fact-checking analyses of statements, textbooks, transcribed recordings of a lecture, and educational videos) spanning English, German, Ukrainian, and Czech languages. This year, 4 teams participated, providing us with 2 Teacher submissions, 2 Student submissions, and 2 Evaluator submissions. We added baselines for Teacher and Student using commercial large language model systems. We devised a fully automatic evaluation procedure, which we compare to a minimalistic manual evaluation. We were able to make some interesting observations. For the first task, the creation of questions, better evaluation strategies will still have to be devised because it is difficult to discern the quality of the various candidate question sets. In the second task, question answering, the LLMs examined overall perform acceptably, but restricting their answers to the given input texts remains problematic. In the third task, evaluation of question answers, our adversarial tests reveal that systems using the LLM-as-a-Judge paradigm erroneously rate both garbled question-answer pairs and answers to mixed-up questions as acceptable.

Figures

Figures reproduced from arXiv: 2507.12143 by the authors.

**Figure 1.** A sample test item (i.e., the input material and a question) derived from the Demagog database with answers and ratings by competing systems. We underlined several spans in the input material and in the question to highlight what the models should have ideally noticed in order to answer correctly. Only baseline GPT-4.1-nano managed this, so the automatic ratings by Gemma-3-27b are also mostly wrong [PITH_FULL_IMAGE… view at source ↗

**Figure 2.** A sample adversarial test item (i.e., the material, and a misleading, unrelated question) derived from the Demagog database with answers and ratings by competing systems. The baseline GPT-4.1-nano succeeded in handling the adversarial input and was also correctly rated as perfect by Gemma-3-27b-it. Other models produce unrelated outputs, and in the first case (llama-3-8b-Instruct finetuned), this also confuses the r… view at source ↗

**Figure 3.** The simple prompt used for the Teacher baseline using GPT-4.1-nano. The prompt is relatively long because we wanted to make sure it did not fail on any part of the dataset. Further prompt optimization was surely possible [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: The prompt used in the Teacher ranking. Some parts are repeated to stress their importance. This prompt was constructed in a manual iterative updates so it is possible that there are prompts easier to read that achieve a similar or even better results. formation rather…

**Figure 5.** Figure 5: The simple prompt used for the Student baseline using GPT-4.1-nano. 5.2. Correlations of ROUGE-L Recall scores between teams To verify our choice of the ROUGE-L Recall evaluation method, we tested whether expert-made questions could be reliably separated into easy and …

**Figure 6.** Figure 6: The manual evaluation questionnaire. As input to the Student task, we used three kinds of question-material combinations: 1. The materials are the combination of a statement, a short fact-check explanation, and the corresponding long fact-check explanation. The questi…

**Figure 7.** Figure 7: The simple prompt used for the Evaluator baseline using GPT-4.1-nano. 6.2. Adversarial questions using Student submissions Another simple way to check the validity of an Evaluator system is to test it on adversarial questions. These were created by shuffling words, rep…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

11 extracted references · 8 canonical work pages

[1]

Karlgren, K

J. Karlgren, K. Artemova, O. Bojar, M. I. Engels, V. Mikhailov, P. Šindelář, E. Velldal, L. Øvrelid, Overview of ELOQUENT 2025: shared tasks for evaluating generative language model quality, in: J. C. de Albornoz, J. Gonzalo, L. Plaza, A. G. S. de Herrera, J. Mothe, F. Piroi, P. Rosso, D. Spina, G. Faggioli, N. Ferro (Eds.), Experimental IR Meets Multilin...

work page 2025
[2]

Z. Wang, R. Baraniuk, MultiQG-TI: Towards question generation from multi-modal sources, in: E. Kochmar, J. Burstein, A. Horbach, R. Laarmann-Quante, N. Madnani, A. Tack, V. Yaneva, Z. Yuan, T. Zesch (Eds.), Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), Association for Computational Linguistics,...

work page doi:10.18653/v1/2023.bea-1 2023
[3]

Evaluation of Question Answering Systems: Complexity of judging a natural language

A. Farea, Z. Yang, K. Duong, N. Perera, F. Emmert-Streib, Evaluation of question answering systems: complexity of judging a natural language, arXiv preprint arXiv:2209.12617 (2022)

work page Pith review arXiv 2022
[4]

Kamalloo, N

E. Kamalloo, N. Dziri, C. L. A. Clarke, D. Rafiei, Evaluating open-domain question answering in the era of large language models, 2023. URL: https://arxiv.org/abs/2305.06984.arXiv:2305.06984

arXiv 2023
[5]

Rajpurkar, J

P. Rajpurkar, J. Zhang, K. Lopyrev, P. Liang, SQuAD: 100,000+ questions for machine comprehen- sion of text, in: J. Su, K. Duh, X. Carreras (Eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Austin, Texas, 2016, pp. 2383–2392. URL: https://aclanthology.org/D16-1264/. d...

doi:10.18653/v1/d16-1264 2016
[6]

Koehn, Neural machine translation, Cambridge University Press, 2020

P. Koehn, Neural machine translation, Cambridge University Press, 2020

work page 2020
[7]

Javorský, D

D. Javorský, D. Macháček, O. Bojar, Continuous rating as reliable human evaluation of simultaneous speech translation, in: P. Koehn, L. Barrault, O. Bojar, F. Bougares, R. Chatterjee, M. R. Costa-jussà, C. Federmann, M. Fishel, A. Fraser, M. Freitag, Y. Graham, R. Grundkiewicz, P. Guzman, B. Haddow, M. Huck, A. Jimeno Yepes, T. Kocmi, A. Martins, M. Moris...

work page 2022
[8]

Sajdoková, M

A. Sajdoková, M. Macek, O. Hlava, M. Štefanec, A. Kříž, J. Kučera, O. Bojar, Team LLMinds Submission for ELOQUENT Sensemaking Task, in: G. Faggioli, N. Ferro, P. Rosso, D. Spina (Eds.), Working Notes of CLEF 2025 – Conference and Labs of the Evaluation Forum, CEUR-WS, 2025

work page 2025

Show all 11 references

[9]

S. Oh, H. Go, H. Moon, Y. Lee, M. Jeong, H. S. Lee, S. Choi, Evaluation of question generation needs more references, 2023. URL: https://arxiv.org/abs/2305.16626.arXiv:2305.16626

2023 arXiv
[10]

Reimers, I

N. Reimers, I. Gurevych, Sentence-BERT: Sentence embeddings using Siamese BERT-networks, in: K. Inui, J. Jiang, V. Ng, X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural La...

2019 doi
[11]

least” and “most

K. Lutsai, M. Thér, J. Venc, O. Bojar, ELOQUENT Sensemaking task - LLMs in the Evaluator role, in: G. Faggioli, N. Ferro, P. Rosso, D. Spina (Eds.), Working Notes of CLEF 2025 – Conference and Labs of the Evaluation Forum, CEUR-WS, 2025. A. Examining the stability of the Teach...

2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Karlgren, K

J. Karlgren, K. Artemova, O. Bojar, M. I. Engels, V. Mikhailov, P. Šindelář, E. Velldal, L. Øvrelid, Overview of ELOQUENT 2025: shared tasks for evaluating generative language model quality, in: J. C. de Albornoz, J. Gonzalo, L. Plaza, A. G. S. de Herrera, J. Mothe, F. Piroi, P. Rosso, D. Spina, G. Faggioli, N. Ferro (Eds.), Experimental IR Meets Multilin...

work page 2025

[2] [2]

Z. Wang, R. Baraniuk, MultiQG-TI: Towards question generation from multi-modal sources, in: E. Kochmar, J. Burstein, A. Horbach, R. Laarmann-Quante, N. Madnani, A. Tack, V. Yaneva, Z. Yuan, T. Zesch (Eds.), Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), Association for Computational Linguistics,...

work page doi:10.18653/v1/2023.bea-1 2023

[3] [3]

Evaluation of Question Answering Systems: Complexity of judging a natural language

A. Farea, Z. Yang, K. Duong, N. Perera, F. Emmert-Streib, Evaluation of question answering systems: complexity of judging a natural language, arXiv preprint arXiv:2209.12617 (2022)

work page Pith review arXiv 2022

[4] [4]

Kamalloo, N

E. Kamalloo, N. Dziri, C. L. A. Clarke, D. Rafiei, Evaluating open-domain question answering in the era of large language models, 2023. URL: https://arxiv.org/abs/2305.06984.arXiv:2305.06984

arXiv 2023

[5] [5]

Rajpurkar, J

P. Rajpurkar, J. Zhang, K. Lopyrev, P. Liang, SQuAD: 100,000+ questions for machine comprehen- sion of text, in: J. Su, K. Duh, X. Carreras (Eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Austin, Texas, 2016, pp. 2383–2392. URL: https://aclanthology.org/D16-1264/. d...

doi:10.18653/v1/d16-1264 2016

[6] [6]

Koehn, Neural machine translation, Cambridge University Press, 2020

P. Koehn, Neural machine translation, Cambridge University Press, 2020

work page 2020

[7] [7]

Javorský, D

D. Javorský, D. Macháček, O. Bojar, Continuous rating as reliable human evaluation of simultaneous speech translation, in: P. Koehn, L. Barrault, O. Bojar, F. Bougares, R. Chatterjee, M. R. Costa-jussà, C. Federmann, M. Fishel, A. Fraser, M. Freitag, Y. Graham, R. Grundkiewicz, P. Guzman, B. Haddow, M. Huck, A. Jimeno Yepes, T. Kocmi, A. Martins, M. Moris...

work page 2022

[8] [8]

Sajdoková, M

A. Sajdoková, M. Macek, O. Hlava, M. Štefanec, A. Kříž, J. Kučera, O. Bojar, Team LLMinds Submission for ELOQUENT Sensemaking Task, in: G. Faggioli, N. Ferro, P. Rosso, D. Spina (Eds.), Working Notes of CLEF 2025 – Conference and Labs of the Evaluation Forum, CEUR-WS, 2025

work page 2025