Do Large Language Models Judge Error Severity Like Humans?

Diege Sun; Guanyi Chen; Tingting He; Xiaorong Cheng; Zhao Fan

REVIEW 4 major objections 6 minor 28 references

Do Large Language Models Judge Error Severity Like Humans?

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Most LLM judges under-penalize colour errors that humans mark as severe, while still penalizing gender errors.

desk verdict Useful controlled study of LLM severity judgments, but item-level confounds and a thin human baseline mean the headline colour-claim overreaches. read the letter →

arxiv 2506.05142 v2 pith:5I7AW4EX submitted 2025-06-05 cs.CL

Diege Sun , Guanyi Chen , Zhao Fan , Xiaorong Cheng , Tingting He This is my paper

classification cs.CL

keywords LLM-as-a-judgeerrorseverityimagedescriptionmultimodalevaluationhumanalignmentcolourperceptiongenderbiasNLG

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether LLMs used as automatic judges penalize the same errors in image descriptions that humans penalize, and whether giving them the image changes that. It reports that humans rank gender errors as most severe, followed by colour, age, and clothing type, and that seeing the image makes colour and type errors seem worse. Most of the tested LLMs, by contrast, rate colour errors as relatively minor while still rating gender errors as severe, so their severity profiles diverge from the human profile in a systematic way. The paper concludes that only Doubao reproduces the human ranking, while the text-only DeepSeek-V3 correlates most strongly with human scores overall, making it the best practical LLM judge among those tested.

What carries the argument

The controlled minimal-pair stimulus design: nine images, each paired with a reference description and four erroneous descriptions that differ by a single character, rated on a 0--100 magnitude-estimation slider under unimodal (text vs. text) and multimodal (image vs. text) conditions. Error-type rankings, linear mixed-effects models, and Pearson and Spearman correlations between human and LLM scores carry the comparison. This design is what lets the paper attribute score differences to the four error types and to the presence of visual context.

What would settle it

A replication with a larger, more diverse human sample (for example, 100+ participants, 50+ images, and multiple languages) that finds a different severity ranking, or no reliable colour-versus-gender gap, would remove the target that the LLM comparisons are measured against.

Watch

Extended reading notes

Core claim

The central discovery is that humans and LLMs disagree about which errors are severe: humans punish both gender and colour errors heavily, but for different reasons, while most tested LLMs punish gender errors but let colour errors pass. Across four error types (age, gender, clothing type, clothing colour) in Chinese image descriptions, human raters produced the severity ranking $\mathrm{GENDER} \prec \mathrm{COLOUR} \prec \mathrm{AGE} \prec \mathrm{TYPE}$ (where $\prec$ means 'judged more severe than'), and visual context significantly lowered human scores only for colour and type errors. GPT-4o, DeepSeek-V3, and DeepSeek-R1 all produced the ranking $\mathrm{GENDER} \prec \mathrm{AGE} \prec \mathrm{COLOUR} \prec \mathrm{TYPE}$, treating colour errors as less severe than age errors. The only model with a human-like ranking, Doubao, failed to separate error types as sharply as humans do. Because colour and gender errors are severe in humans through different mechanisms --- colour through perceptual and neural processing, gender through social norms --- the paper interprets the LLM pattern as evidence that the models have absorbed social norms but lack perceptual grounding for colour.

Load-bearing premise

The paper's central comparison assumes the human severity ratings are a stable, representative target, but the target is built from 9 images, 25 university students, one language, and one rating scale, and the paper's own human experiment failed to reproduce the earlier published severity ranking.

Editorial extensions

If this is right

If one deploys GPT-4o or the DeepSeek models as judges for captioning, colour errors will be systematically under-penalised relative to human judges, so automated evaluation will rank outputs with colour mistakes too highly.
Using a unimodal LLM as a judge for multimodal outputs may be both cheaper and better aligned with human ratings than feeding the image to a multimodal model, since unimodal DeepSeek-V3 correlated highest with human scores in both conditions.
A model can match the human ranking of error types (as Doubao did) and still be unsuitable as a judge if it treats most error types as statistically indistinguishable.
A reasoning-focused model such as DeepSeek-R1 did not align better with humans than its non-reasoning counterpart, so additional reasoning does not automatically improve severity judgments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The human baseline itself is probably unstable: with only 9 images, 25 university students, one language, and a failed replication of the earlier published ranking, the specific order $\mathrm{GENDER} \prec \mathrm{COLOUR} \prec \mathrm{AGE} \prec \mathrm{TYPE}$ may shift with participants or stimuli, which would move the alignment target for any LLM comparison.
A testable extension would vary the language and image set and check whether the LLM colour gap persists; if it does, the under-penalisation of colour likely reflects a general absence of perceptual grounding rather than a quirk of these prompts or models.
An untested extension is whether the same asymmetry appears for other perceptually grounded properties such as shape or spatial relations; the paper's neural-mechanism explanation predicts that text-only models will under-penalise a broader class of perceptually grounded errors, not just colour.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should read this paper if you care about LLM-as-judge evaluation. The headline finding is that LLMs systematically under-penalize colour errors relative to humans, and a text-only model (DeepSeek-V3) correlates best with human severity judgments. The statistics are done carefully, and the authors report a failed replication of the prior ranking, which is a good sign.

What's genuinely new: they take the van Miltenburg et al. (2020) framework, translate it into Chinese, add a unimodal condition, and test four LLMs. The result that multimodal models do not outperform unimodal DeepSeek-V3 is interesting and could be practically useful.

The biggest issue is the stimulus design. Each error type is instantiated by exactly one word-change per image, and the annotators deliberately made the changes 'substantial'. So 'colour error' is confounded with the particular colour words chosen, and 'type error' with the particular clothing nouns. The observed human ranking (gender < colour < age < type) could partly be an artifact of item salience, not error category. This also explains why the ranking differs from van Miltenburg — different items, different ordering. The paper doesn't address this.

Second, the modality comparison is confounded. In the multimodal condition participants see the image plus the erroneous description; in the unimodal condition they see a reference description plus the erroneous description. So the effect attributed to 'visual context' might actually be the effect of having the reference text. The paper acknowledges they changed the multimodal condition but doesn't treat this as a confound.

Third, the human baseline is small: 9 images, 25 in-lab students, one language. The paper generalizes to 'humans' as if this were a stable target. The failed H1 shows the baseline is sensitive to setting.

Fourth, the neural-mechanism and social-norm explanations are untested speculation. Fine as post-hoc interpretation, but not a result.

Finally, no stimuli or code are released, so the item-specific fragility cannot be examined.

This deserves a serious referee — the question is important and the execution is mostly rigorous. But the authors need to release the materials, address the reference confound, and either add multiple items per error type or weaken the categorical claims. I'd send it to review but expect major revision.

Referee Report

4 major / 6 minor

Summary. This paper compares human and LLM judgments of error severity in Chinese image descriptions, extending the van Miltenburg et al. (2020) framework to unimodal (text-only) and multimodal (text + image) settings. The human experiment with 25 in-lab participants yields a severity ranking GENDER < COLOUR < AGE < TYPE, which differs from the earlier COLOUR < GENDER < TYPE < AGE ranking, and a visual-context effect on colour and type errors. The LLM experiments with GPT-4o, Doubao, DeepSeek-V3, and DeepSeek-R1 show that most models assign relatively low severity to gender errors and relatively high severity to colour errors, unlike the human ranking; Doubao alone reproduces the human ranking, while DeepSeek-V3 has the highest overall correlation with human scores. The paper concludes that LLMs have internalised social norms for gender but lack the perceptual grounding for colour sensitivity.

Significance. If the headline divergence is valid, the result is practically important for LLM-as-a-judge evaluation of multimodal NLG and is a useful cross-linguistic extension of prior work. The authors use appropriate mixed-effects modelling with multiplicity corrections, transparently report the failure of H1, and evaluate multiple models with repeated sampling. However, the reported effects rest on a small stimulus set with one substitution per error type per image and on a modality manipulation that is confounded with the presence of a reference text; these issues bound the current support for the categorical and mechanistic claims.

major comments (4)

[§3.1, Eq. (2), Table 3] The central category-level conclusion is confounded with the specific lexical substitutions. Each of the 9 images contributes exactly one erroneous description per error type, created by altering one character and chosen to be 'substantial' (e.g., red→purple for colour, shirt→coat for type). The LMM in §4.1 treats Error Type as a fixed factor with no random effects for image or for the specific word pair, so 'colour error' is instantiated by 9 hand-picked word pairs and 'type error' by a different 9. Severity differences can therefore reflect item-level salience rather than a categorical colour/type distinction. The failed replication of H1 (Eq. 2 vs Eq. 1) with a different item set is exactly what this confound predicts. To support the categorical claim, the authors should use several substitution pairs per type per image and include item-level random effects (or a by-item analysis). The Limitations section does not acknowledge this concern.
[§3.1, Design; Tables 1-2] The effect of visual context is confounded with the presence of a reference description. In the multimodal condition participants saw the image and the erroneous description; in the unimodal condition they saw a reference description and the erroneous description. Consequently, the significant Error Type × Modality interaction for humans in Table 1 (and the colour/type interaction terms in Table 2) could be driven by the availability of a reference text in the unimodal condition rather than by the image itself. A full 2 × 2 design (with/without image, with/without reference) is needed to unconfound these factors; otherwise the claim that visual context specifically amplifies colour and type severity is not supported. This also affects the LLM-versus-human modality comparison in §4.2.
[§4.3, Table 4] The headline that DeepSeek-V3 'achieves the highest alignment with human judgments' is not statistically established. Pearson/Spearman correlations over 64 stimuli from 9 images are reported without confidence intervals or tests for differences between dependent correlations, and no human reliability estimate (e.g., split-half or ICC) is given to bound the achievable correlation. The effective sample is also smaller than 64 because stimuli are nested in 9 images. Moreover, only two multimodal models (GPT-4o and Doubao) were used, so the abstract's phrase 'outperforming even state-of-the-art multimodal models' overstates the evidence. Please add confidence intervals, formal comparisons between correlations, and human reliability, and temper the claim.
[§4.2, Conclusion] The explanation that humans judge colour errors as severe because of 'distinct neural mechanisms' and gender errors because of 'social norms' is not tested in this study; the dataset contains no process measure or manipulation that separates these accounts. Presenting this interpretation as the reason for LLM-human divergence in the abstract and conclusion goes beyond the data. The authors should either mark this as a speculative explanation or provide an independent test.

minor comments (6)

[§2] The hypothesis list contains two items labelled H3 and no H4, yet §4.2 refers to 'hypothesis H4'. Re-number the hypotheses and ensure all referenced labels are defined.
[Table 3] The legend is inconsistent: red and green both denote differences in the 'opposite direction' to humans, with green adding 'statistically insignificant'; clarify what each colour means or use a single notation.
[Table 2] In the Doubao row for 'Type×Unimodal', the t and p columns appear garbled ('7.917 .113 .113'); verify the reported values and formatting.
[§4, first paragraph] The text says 'Figure 3 reports the scores given by humans and 5 LLMs', but only four LLMs are evaluated; correct the count to 4.
[Abstract and §5] The phrase 'outperforming even state-of-the-art multimodal models' overstates the evidence, since only two multimodal models (GPT-4o and Doubao) were tested; rephrase to something like 'the two multimodal models tested'.
[§3.1] The authors report that the two annotators 'discussed finalising the descriptions', but no inter-annotator agreement or reliability measure is provided for the construction of the erroneous descriptions.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the only self-citation (van Miltenburg et al. 2020) is used as a tested hypothesis, not as an assumed result.

full rationale

The study is a fresh empirical comparison: it collects new human severity ratings from 25 participants on 64 stimuli and separately queries four LLMs with the same stimuli, then compares the two sets of outputs. No parameter is fitted to the human data and then called a prediction; the LLM scores are generated independently of the human ratings, and the correlations in Table 4 are computed from these independent outputs. The main extension builds on van Miltenburg et al. (2020), a paper co-authored by one of the present authors, but the load-bearing use is not circular: H1 explicitly predicts that humans will reproduce the 2020 ranking, and the paper reports that this hypothesis was rejected (Equation 2 vs Equation 1). Thus the prior result is treated as a falsifiable benchmark rather than as an input that forces the conclusion. The later mechanistic explanation (colour severity from neural mechanisms, gender severity from social norms) is speculative, but speculation is not circularity. The design concern that each error type is instantiated by one convenience word pair per image is a validity/external-validity issue, not a reduction of the derivation to its inputs; the paper's Limitations section flags only closed-source models, not this item confound, but that is a completeness concern rather than evidence of circularity. Overall, no equation or conclusion reduces by construction to its own inputs, so circularity is minimal.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

Empirical study with no derivation, so no fitted constants appear in the analysis; the only hand-chosen number is the effect size used for power. The claims rest on the validity of the rating task, the representativeness of the small human sample, the stability of the severity ranking, the interpretation of neural and social mechanisms, and the adequacy of three LLM runs, all itemized above. No invented entities are introduced.

free parameters (1)

Cohen's f effect size for power analysis = 0.23
Hand-chosen input to G*Power that set the target sample size (18, with 25 actually recruited); it does not enter the reported LMM results or any conclusion, so its impact is limited to design.

assumptions (5)

domain assumption Slider quality ratings (0-100) constitute a valid operationalization of error severity
Participants were told the descriptions came from an AI system and asked to rate quality; the paper treats low quality scores as high error severity (Section 3.1, Procedure).
domain assumption The 9-image, 25-participant Chinese sample represents human severity judgment
Generalization from a small in-lab student sample to 'human judgments'; the paper's own H1 failure shows the ranking is sensitive to experimental setting (Section 3.1, Section 4.1).
domain assumption Three LLM runs sufficiently sample stochastic model behavior
Each LLM was run 3 times per stimulus; run-to-run variance is not reported (Section 3.2, Procedure).
domain assumption Colour errors are severe for humans because colour is processed by distinct neural mechanisms (literature claim)
Invoked from Connell (2007) and Liu et al. (2024b) to explain the LLM-human divergence; the link from neural architecture to LLM behavior is not tested in this paper (Section 4.2).
standard math Standard LMM and multiple-comparison statistical machinery
LMM with Shapiro-Wilk justification, Tukey HSD and Holm-Bonferroni corrections (Section 4.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Do Large Language Models Judge Error Severity Like Humans?." pith.science (2026). https://pith.science/paper/5I7AW4EX

@misc{pith2026250605142,
  author       = {Pith},
  title        = {Pith review of: Do Large Language Models Judge Error Severity Like Humans?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5I7AW4EX}},
  note         = {Machine review of arXiv:2506.05142}
}

read the original abstract

Large Language Models (LLMs) are increasingly used as automated evaluators in natural language generation, yet it remains unclear whether they can accurately replicate human judgments of error severity. In this study, we systematically compare human and LLM assessments of image descriptions containing controlled semantic errors. We extend the experimental framework of van Miltenburg et al. (2020) to both unimodal (text-only) and multimodal (text + image) settings, evaluating four error types: age, gender, clothing type, and clothing colour. Our findings reveal that humans assign varying levels of severity to different error types, with visual context significantly amplifying perceived severity for colour and type errors. Notably, most LLMs assign low scores to gender errors but disproportionately high scores to colour errors, unlike humans, who judge both as highly severe but for different reasons. This suggests that these models may have internalised social norms influencing gender judgments but lack the perceptual grounding to emulate human sensitivity to colour, which is shaped by distinct neural mechanisms. Only one of the evaluated LLMs, Doubao, replicates the human-like ranking of error severity, but it fails to distinguish between error types as clearly as humans. Surprisingly, DeepSeek-V3, a unimodal LLM, achieves the highest alignment with human judgments across both unimodal and multimodal conditions, outperforming even state-of-the-art multimodal models.

Figures

Figures reproduced from arXiv: 2506.05142 by the authors.

**Figure 1.** Image 34419 from MSCOCO: A man wearing a yellow shirt on a tennis court plays tennis. 2024; Gao et al., 2025). Their use as “LLM-as-ajudge” has shown promise in areas such as summarisation and dialogue (Liu et al., 2023; Liusie et al., 2024), where traditional metrics, such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), often fall short of capturing semantic nuances and user preferences. Nonetheless, key … view at source ↗

**Figure 2.** Screenshots of a trial in (a) multimodal condi [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** The results of both our human experiment and LLM experiment. Each error bar indicates the standard [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: The box plot of the distribution of reaction [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 16 canonical work pages

[1]

Edward H Adelson. 1993. Perceptual organization and the judgment of brightness. Science, 262(5142):2042--2044

work page 1993
[2]

Ilaha Ashrafova. 2024. Language and gender: Exploring structures and bias in linguistic norms. Acta Globalis Humanitatis et Linguarum, 1(1):39--50

work page 2024
[3]

Simone Balloccu, Patr \'i cia Schmidtov \'a , Mateusz Lango, and Ondrej Dusek. 2024. https://aclanthology.org/2024.eacl-long.5/ Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLM s . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), ...

work page 2024
[4]

Ellen Gurman Bard, Dan Robertson, and Antonella Sorace. 1996. Magnitude estimation of linguistic acceptability. Language, pages 32--68

work page 1996
[5]

Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fern \'a ndez, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, and 1 others. 2024. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. arXiv preprint arXiv:2406.18403

arXiv 2024
[6]

Guanyi Chen, Kees van Deemter, and Chenghua Lin. 2019. https://doi.org/10.18653/v1/W19-8667 Generating quantified descriptions of abstract visual scenes . In Proceedings of the 12th International Conference on Natural Language Generation, pages 529--539, Tokyo, Japan. Association for Computational Linguistics

work page doi:10.18653/v1/w19-8667 2019
[7]

Louise Connell. 2007. Representing object colour in language comprehension. Cognition, 102(3):476--485

work page 2007
[8]

Mark J Fenske, Elissa Aminoff, Nurit Gronau, and Moshe Bar. 2006. Top-down facilitation of visual object recognition: object-based and context-based contributions. Progress in brain research, 155:3--21

work page 2006

Show all 28 references

[9]

Mingqi Gao, Xinyu Hu, Xunjian Yin, Jie Ruan, Xiao Pu, and Xiaojun Wan. 2025. Llm-based nlg evaluation: Current status and challenges. Computational Linguistics, pages 1--28

2025
[10]

Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594

2024 arXiv
[11]

Douglas Guilbeault, Sol \`e ne Delecourt, Tasker Hull, Bhargav Srinivasa Desikan, Mark Chu, and Ethan Nadler. 2024. Online images amplify gender bias. Nature, 626(8001):1049--1055

2024
[12]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

2025 arXiv
[13]

Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65--70

1979
[14]

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

2024 arXiv
[15]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

2004
[16]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer vision--ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, ...

2014
[17]

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

2024 arXiv
[18]

Donglin Liu, Lijuan Wang, and Ying Han. 2024 b . Mental simulation of colour properties during language comprehension: influence of context and comprehension stages. Cognitive Processing, 25(4):587--600

2024
[19]

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

2023 doi
[20]

Adian Liusie, Potsawee Manakul, and Mark Gales. 2024. https://aclanthology.org/2024.eacl-long.8/ LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models . In Proceedings of the 18th Conference of the European Chapter of the...

2024
[21]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

2002
[22]

Karin S Pilz and Hao Lou. 2022. Contextual and own-age effects in age perception. Experimental Brain Research, 240(9):2471--2480

2022
[23]

John W Tukey. 1949. Comparing individual means in the analysis of variance. Biometrics, pages 99--114

1949
[24]

Emiel van Miltenburg and Desmond Elliott. 2017. Room for improvement in automatic image description: an error analysis. arXiv preprint arXiv:1704.04198

2017 arXiv
[25]

Emiel van Miltenburg, Wei-Ting Lu, Emiel Krahmer, Albert Gatt, Guanyi Chen, Lin Li, and Kees van Deemter. 2020. https://doi.org/10.18653/v1/2020.inlg-1.45 Gradations of error severity in automatic image descriptions . In Proceedings of the 13th International Conference on Natu...

2020 doi
[26]

Ni Xuanfan and Li Piji. 2023. https://aclanthology.org/2023.ccl-2.4/ A systematic evaluation of large language models for natural language generation tasks . In Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 2: Frontier Forum), pages 4...

2023
[27]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[28]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Edward H Adelson. 1993. Perceptual organization and the judgment of brightness. Science, 262(5142):2042--2044

work page 1993

[2] [2]

Ilaha Ashrafova. 2024. Language and gender: Exploring structures and bias in linguistic norms. Acta Globalis Humanitatis et Linguarum, 1(1):39--50

work page 2024

[3] [3]

Simone Balloccu, Patr \'i cia Schmidtov \'a , Mateusz Lango, and Ondrej Dusek. 2024. https://aclanthology.org/2024.eacl-long.5/ Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLM s . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), ...

work page 2024

[4] [4]

Ellen Gurman Bard, Dan Robertson, and Antonella Sorace. 1996. Magnitude estimation of linguistic acceptability. Language, pages 32--68

work page 1996

[5] [5]

Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fern \'a ndez, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, and 1 others. 2024. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. arXiv preprint arXiv:2406.18403

arXiv 2024

[6] [6]

Guanyi Chen, Kees van Deemter, and Chenghua Lin. 2019. https://doi.org/10.18653/v1/W19-8667 Generating quantified descriptions of abstract visual scenes . In Proceedings of the 12th International Conference on Natural Language Generation, pages 529--539, Tokyo, Japan. Association for Computational Linguistics

work page doi:10.18653/v1/w19-8667 2019

[7] [7]

Louise Connell. 2007. Representing object colour in language comprehension. Cognition, 102(3):476--485

work page 2007

[8] [8]

Mark J Fenske, Elissa Aminoff, Nurit Gronau, and Moshe Bar. 2006. Top-down facilitation of visual object recognition: object-based and context-based contributions. Progress in brain research, 155:3--21

work page 2006