Pith. sign in

REVIEW 4 major objections 5 minor 18 references

Multimodal LLMs can approximate expert brushstroke scores but not their rank ordering.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Multimodal LLMs approximate expert absolute scores for calligraphy brushstrokes but show no significant rank correlation; RAG guidance trades accuracy for ranking.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection A new stroke-level calligraphy benchmark with an honest negative result, but the 'useful MAE' claim is under-anchored without baselines or inter-rater agreement. the 4 major comments →

arxiv 2607.16330 v1 pith:3N25LWPM submitted 2026-07-15 cs.CV

Local Brushstroke Quality Assessment via Vision-Language Feedback

Mio Mitamura , Hirokatsu Kataoka This is my paper
classification cs.CV
keywords Calligraphy assessmentMultimodal LLMBrushstroke qualityVision-language feedbackKendall rank correlationRetrieval-augmented generationAesthetic evaluationEducational feedback
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether multimodal LLMs can judge the quality of individual brushstrokes in calligraphy and explain their judgments in words. On 45 before-after pairs of calligraphic works, three models (GPT-4o, Claude Sonnet 4, Gemini 2.5 Flash) scored marked strokes on a five-point scale and were compared against the averaged scores of three expert calligraphers. The result is split: all models land within roughly one point of the experts on average (best MAE 0.885), but none produce statistically significant rank correlations (Kendall's tau), meaning the models cannot reliably order strokes from worst to best the way experts do. Vocabulary analysis shows each model collapses multi-dimensional judgment into a single dominant perspective, and injecting expert rules via retrieval (RAG) improves ranking but worsens absolute accuracy. If correct, this separates 'roughly right' absolute scoring from 'expert-like relative judgment' and sets a concrete benchmark for future work.

Core claim

Across 45 works and 135 stroke evaluations, GPT-4o, Claude Sonnet 4, and Gemini 2.5 Flash each achieve mean absolute errors below 1.0 against the averaged expert scores (GPT-4o 0.885, Claude 0.902), yet their Kendall's tau correlations with human ranks are not statistically significant (0.093, 0.053, -0.025; p>0.05). The models' rationales exhibit characteristic vocabulary biases: Claude over-uses dynamic-action terms, GPT-4o and Gemini shift to dynamic-impression language, and static vocabulary is nearly absent. Adding retrieval-augmented generation with expert checklists to Claude improves rank correlation (tau 0.053 to 0.119) but worsens MAE (0.902 to 1.067), a trade-off the authors inter

What carries the argument

The central instrument is a comparative local-stroke assessment protocol: before/after calligraphy images with annotated stroke regions, scored on a five-point ordinal scale (-2 to +2), with the mean of three expert calligraphers' ratings as ground truth. Performance is measured by MAE for absolute agreement and Kendall's tau for rank agreement. A retrieval-augmented variant of Claude supplies expert checklist items as external knowledge. The protocol is what allows the paper to separate absolute accuracy from rank ordering and to expose vocabulary-level evaluation biases.

Load-bearing premise

The mean of three experts' ordinal scores is treated as a precise, low-noise ground truth for local brushstroke quality; if the experts disagree substantially, or if averaging five-point ratings is not a meaningful target, then both MAE and Kendall-tau comparisons lose their stated meaning.

What would settle it

Compute inter-expert agreement (e.g., pairwise Kendall's tau or ICC) on the 135 stroke evaluations. If expert-expert tau is as low as the model-expert tau, the central null result would be explained by ground-truth noise; alternatively, a larger dataset (e.g., 200+ works) that preserves significant model-expert tau for the simple character while remaining null overall would confirm the complexity-confound claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If correct, multimodal LLMs can serve as approximate absolute scorers for local brushstroke quality in calligraphy, useful for triage or practice feedback, but cannot yet be trusted to rank or compare strokes the way experts do.
  • The split between MAE and tau implies that improving ranking requires something beyond prompting: none of the zero-shot models recovered expert ordering, and RAG only partially helped.
  • Hane (hook) strokes are the hardest to score (largest MAE), suggesting inference about multi-phase 3D brush movement from static images is a bottleneck.
  • The vocabulary bias finding predicts that feedback generated by these models will over-emphasize one evaluation dimension, so educational use should pair model output with human review.
  • The RAG trade-off implies that feeding expert rules as text can make models more consistent in relative comparisons but less aligned with holistic human absolute scores.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The null rank correlations may be as much a product of the small sample (45 works, 135 stroke judgments) and unmeasured inter-expert disagreement as of model failure; the paper reports no inter-expert agreement statistic, so the human ground truth itself might be noisy.
  • A natural testable extension is to compute pairwise expert Kendall tau and ICC; if expert-expert agreement is also low on this rubric, the right conclusion is not 'models can't rank' but 'the ranking task is underdetermined at this scale.'
  • The RAG result suggests that retrieving exemplars with scores may be more promising than retrieving rule text; a RAG variant that retrieves similar before/after image pairs with expert scores could be tested against the current checklist-based RAG.
  • The pattern of correlations across characters (significant only for the simplest 4-stroke character) hints that visual complexity confounds local assessment; a controlled dataset with isolated single-stroke images would test whether models can judge strokes absent whole-character context.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper evaluates whether three multimodal LLMs (GPT-4o, Claude Sonnet 4, Gemini 2.5 Flash) can assess local brushstroke quality in Japanese calligraphy by comparing before/after image pairs and producing a score on a five-point ordinal scale (-2 to +2) plus a natural language rationale. Scores are compared against the mean of three expert calligraphers. The authors report that GPT-4o achieves the best MAE (0.885), that no model achieves statistically significant overall Kendall rank correlation with human experts (all p>0.05), that vocabulary analysis reveals characteristic evaluative biases, and that a RAG-augmented Claude variant improves rank correlation while worsening absolute accuracy on a small held-out set. The paper interprets these results as showing that multimodal LLMs can approximate absolute expert scores locally but cannot reproduce expert rank ordering, and that text-based rule injection fails to induce human-like holistic evaluation.

Significance. If the central claim were established, this would be a useful empirical contribution at the intersection of vision-language models and fine-grained aesthetic/technical assessment. The paper has several genuine strengths: it tackles an underexplored task, uses a domain-expert ground-truth protocol, reports negative results honestly (all p>0.05 are stated), and provides a qualitative vocabulary analysis that distinguishes model-specific biases. The RAG comparison, despite its limitations, is a valuable negative result because it tests a popular hypothesis (that injecting expert rules improves evaluation). The framework is simple and reproducible in principle, and the small dataset is clearly disclosed. However, the main claims currently lack the statistical anchoring needed to support the distinction between 'useful absolute accuracy' and 'no rank correlation': no inter-expert agreement measure, no trivial baselines, no confidence intervals or power analysis, and no significance test for the RAG trade-off.

major comments (4)
  1. [Section 3.1, 'Ground Truth Labels'; Sections 3.2 and 4.1] The ground truth is the mean of three expert ordinal scores (-2..+2), but the paper reports no inter-expert agreement (ICC, pairwise Kendall tau, or raw disagreement rates). This is load-bearing for both halves of the central claim. If experts disagree substantially, (i) a model MAE of 0.885 against the averaged label may be misleadingly small or large depending on the noise structure, and (ii) the null overall Kendall tau could be induced by label noise or by averaging ordinal ratings rather than by a model limitation. Please report the distribution of expert scores, pairwise/inter-expert agreement, and per-expert MAE, and discuss how averaging ordinal labels affects the interpretation.
  2. [Section 3.2, Table 1; Section 4.1] The claim that all models achieve 'useful levels of absolute score accuracy' is not anchored. MAE=0.885 on a 5-point scale (range 4) needs a trivial baseline for comparison, such as always predicting 0, always predicting the majority score, or a random uniform predictor. Without such baselines and confidence intervals, the term 'useful' is unsupported. Please provide baseline MAE values and, if possible, a comparison with human-human MAE on the same task (e.g., one expert versus the mean of the other two).
  3. [Section 3.3, Fig. 6] The RAG comparison is based on n=30 held-out strokes, but no significance test or confidence interval is given for the change in MAE (0.902 to 1.067) or Kendall tau (0.053 to 0.119). Both changes could be sampling noise at this sample size. The paper also does not state explicitly whether the zero-shot Claude values in Fig. 6 were recomputed on the same 30 held-out strokes or taken from the full 45-work results in Table 1. Please recompute both conditions on the identical held-out set, report p-values or bootstrap CIs for the differences, and temper the 'important negative result' claim accordingly.
  4. [Section 3.2, 'Quantitative Evaluation'; Fig. 3] The overall Kendall tau results are all p>0.05, but the paper interprets this as evidence that models 'cannot reproduce' expert rank ordering. Nonsignificance is not evidence of absence, and with only 45 works (and possibly 135 strokes if strokes are treated independently, though this is not stated) the test may be underpowered. Please state the unit of analysis and n for each tau, report confidence intervals or effect sizes, and discuss power. Relatedly, the character-level significant results in Fig. 3 (Claude tau=0.248, p=0.025; GPT-4o tau=0.242, p=0.027) are selected from multiple comparisons (3 models x 3 characters); without correction or pre-specified hypotheses these are weak exploratory evidence for the 'structural complexity' conclusion.
minor comments (5)
  1. [Section 3.3] The text says 'MAE increased from 0.902 to 1.067, while Kendall’s tau improved from 0.053 to 0.119.' If the 0.902/0.053 values are from Table 1 (45 works), the comparison with RAG on 30 strokes is confounded. Even if the figure recomputes both on the held-out set, the text should say so explicitly.
  2. [Fig. 5 caption] Typo: 'V ocabulary' should be 'Vocabulary'.
  3. [Section 3.1, 'Evaluation Metrics'] The SD metric is introduced but its definition and interpretation are not fully clear. Is it the standard deviation across the three model runs? If so, how does it relate to the MAE claim? Please clarify.
  4. [General] No data availability statement is included. Given the dataset was collected from a school festival, a statement about availability, privacy, and ethics would strengthen reproducibility.
  5. [Section 4.2] The section titled 'Implications' lists two 'structural limitations' that are actually empirical findings. Consider renaming this subsection or distinguish findings from limitations more clearly.

Circularity Check

0 steps flagged

No significant circularity: empirical benchmark with independent ground truth and held-out RAG split.

full rationale

This paper is an empirical evaluation, not a derivation. The central claims—useful MAE but non-significant Kendall tau for LLM scores versus human experts—are based on comparing independently produced model outputs against a ground truth defined as the mean of three expert calligraphers' ordinal ratings. No model score, MAE, or tau value is defined in terms of a fitted parameter from that same ground truth, and no equation in the paper reduces a reported result to its own input by construction. The RAG condition uses 35 of 45 works to build the knowledge base and is evaluated only on the remaining 10 held-out works, so retrieval is not circular. The prompt rubric is drawn from the same expert criteria used for ground truth, but that is a study-design choice rather than a logical reduction: the models' outputs are not mathematically forced to match the expert mean by that shared rubric. No load-bearing self-citation is present; the authors' own prior work is not invoked to justify the method. Concerns about unvalidated mean-of-ordinal-scores ground truth and absence of inter-expert agreement are validity/correctness risks, not circularity, and do not meet the bar for flagging a circular step.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The paper is an empirical evaluation rather than a derivation, so the central claims rest on measurement assumptions: expert-mean ground truth, dataset representativeness, and local-region attention. No numeric parameters are fitted to the target scores; the RAG knowledge base is built from 35 training works with a 10-work held-out test, so retrieval is a data-assembly choice, not parameter fitting. No new entities are introduced.

axioms (3)
  • domain assumption The mean of three expert ordinal scores is a valid scalar ground truth for brushstroke quality.
    Section 3.1 defines ground truth as the mean expert score; no inter-rater reliability is reported, and averaging a 5-point ordinal scale as a cardinal value is assumed.
  • domain assumption 45 works and 135 marked stroke regions are sufficient to estimate rank correlation with human experts.
    Section 3.1 describes a school-festival dataset with only 3 kanji characters; no power analysis is given, and insignificant tau values may be due to low power.
  • domain assumption Visual markers cause models to evaluate only the indicated local stroke region.
    Section 2 assumes the model attends to the marked region; Section 4.2 concedes models cannot fully ignore overall visual complexity, so this premise is only partially true.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Local Brushstroke Quality Assessment via Vision-Language Feedback." pith.science (2026). https://pith.science/paper/3N25LWPM

@misc{pith2026260716330,
  author       = {Pith},
  title        = {Pith review of: Local Brushstroke Quality Assessment via Vision-Language Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3N25LWPM}},
  note         = {Machine review of arXiv:2607.16330}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This paper investigates whether multimodal LLMs can evaluate local brushstroke quality in calligraphy and generate educationally useful natural language feedback. We construct an evaluation framework in which three multimodal LLMs (GPT-4o, Claude Sonnet 4, and Gemini 2.5 Flash) assess before-after image pairs of calligraphic works using a five-point ordinal scale, and compare their outputs against scores assigned by three expert calligraphers. We additionally examine a Retrieval-Augmented Generation (RAG) variant of Claude as a preliminary condition. Results show that all models achieve useful levels of absolute score accuracy (MAE), with GPT-4o performing best (MAE = 0.885). However, none of the models produce statistically significant overall rank correlations with human experts (Kendall's tau). Vocabulary analysis of generated rationales reveals characteristic evaluative biases in each model, and RAG is shown to improve rank correlation while worsening absolute accuracy, constituting an important negative result for text-based rule injection.

Figures

Figures reproduced from arXiv: 2607.16330 by Hirokatsu Kataoka, Mio Mitamura.

Figure 1
Figure 1. Figure 1: Three multimodal LLMs evaluate the same local stroke [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the proposed evaluation pipeline. Before and After calligraphy images with stroke markers are input to zero-shot [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: MAE by stroke type. All models show lowest error for [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Vocabulary distribution by model. Human evaluators [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 8
Figure 8. Figure 8: Feedback differences (文, right harai). Zero-shot gener￾ates holistic aesthetic feedback, while RAG over-concentrates on dynamic-action vocabulary, losing the multi-dimensional balance that human experts naturally integrate. significant overall rank correlations. Vocabulary analy￾sis revealed characteristic evaluative biases in each model, consistent with known LLM evaluation biases [12, 13]. RAG [14] const… view at source ↗
Figure 7
Figure 7. Figure 7: Vocabulary: Claude Zero-shot vs. Claude RAG. [PITH_FULL_IMAGE:figures/full_fig_p005_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 4 linked inside Pith

  1. [1]

    Economic Census for Business Activity 2012

    Ministry of Internal Affairs and Communications and Min- istry of Economy, Trade and Industry, “Economic Census for Business Activity 2012.”https://www.e-stat.go. jp/dbview?sid=00030857102

    2012

  2. [2]

    Economic Census for Business Activity 2016

    Ministry of Internal Affairs and Communications and Min- istry of Economy, Trade and Industry, “Economic Census for Business Activity 2016.”https://www.e-stat.go. jp/dbview?sid=0003218723

    2016

  3. [3]

    Economic Census for Business Activity 2021

    Ministry of Internal Affairs and Communications and Min- istry of Economy, Trade and Industry, “Economic Census for Business Activity 2021.”https://www.e-stat.go. jp/dbview?sid=00040063232

    2021

  4. [4]

    Aesthetic Evalu- ation of Chinese Calligraphy Using TabNet: Interpretability and Novel Features for Improved Accuracy,

    S. Yoshida, N. Yatoh, and M. Muneyasu, “Aesthetic Evalu- ation of Chinese Calligraphy Using TabNet: Interpretability and Novel Features for Improved Accuracy,”IEICE Trans- actions on Fundamentals, vol. E108-A, no. 3, pp. 357–361,

  5. [5]

    Evaluating Brush Move- ments for Chinese Calligraphy: A Computer Vision Based Approach,

    P. Xu, L. Wang, Z. Guan, X. Zheng, X. Chen, Z. Tang, D. Fang, X. Gong, and Z. Wang, “Evaluating Brush Move- ments for Chinese Calligraphy: A Computer Vision Based Approach,”Proceedings of IJCAI, pp. 1050–1056, 2018. 1

    2018

  6. [6]

    Aesthetic Judgment in Calli- graphic Tracing: The Dominant Role of Dynamic Features,

    Q. Yuan, G. Yang, and R. Lyu, “Aesthetic Judgment in Calli- graphic Tracing: The Dominant Role of Dynamic Features,” Behavioral Sciences, vol. 15, no. 4, article 525, 2025. 1

    2025

  7. [7]

    Intelligent Evaluation Method of Cal- ligraphy Characters Based on Deep Stroke Extraction,

    M. Li and G. Ren, “Intelligent Evaluation Method of Cal- ligraphy Characters Based on Deep Stroke Extraction,”Ad- vances in Computer, Signals and Systems, vol. 7, pp. 99–106,

  8. [8]

    GPT-4o System Card,

    OpenAI, “GPT-4o System Card,” arXiv:2410.21276, 2024. 1, 2, 3, 5

    Pith/arXiv arXiv 2024

  9. [9]

    Gemini 2.5: Pushing the Fron- tier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities,

    Gemini Team Google, “Gemini 2.5: Pushing the Fron- tier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities,” arXiv:2507.06261, 2025. 1, 2, 3, 5

    Pith/arXiv arXiv 2025

  10. [10]

    System Card: Claude Opus 4 & Claude Son- net 4,

    Anthropic, “System Card: Claude Opus 4 & Claude Son- net 4,” 2025.https://www-cdn.anthropic.com/ 4263b940cabb546aa0e3283f35b686f4f3b2ff47. pdf1, 2, 3, 5

    2025

  11. [11]

    GPT- 4V(ision) as a Generalist Evaluator for Vision-Language Tasks,

    X. Zhang, Y . Lu, W. Wang, A. Yan, J. Yan, L. Qin, H. Wang, X. Yan, W. Y . Wang, and L. R. Petzold, “GPT- 4V(ision) as a Generalist Evaluator for Vision-Language Tasks,” arXiv:2311.01361, 2023. 2

    Pith/arXiv arXiv 2023

  12. [12]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,”Advances in Neural Infor- mation Processing Systems (NeurIPS), 2023. 2, 4, 5

    2023

  13. [13]

    Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge,

    J. Ye, Y . Wang, Y . Huang, D. Chen, Q. Zhang, N. Moniz, T. Gao, W. Geyer, C. Huang, P.-Y . Chen, N. V . Chawla, and X. Zhang, “Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge,” arXiv:2410.02736, 2024. 2, 4, 5

    Pith/arXiv arXiv 2024

  14. [14]

    Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W. Yih, T. Rockt ¨aschel, S. Riedel, and D. Kiela, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,”Advances in Neural Information Processing Systems (NeurIPS), 2020. 1, 2, 3, 4, 5

    2020

  15. [15]

    Cal- liReader: Contextualizing Chinese Calligraphy via an Embedding-Aligned Vision-Language Model,

    Y . Luo, J. Tang, C. Huang, F. Hao, and Z. Lian, “Cal- liReader: Contextualizing Chinese Calligraphy via an Embedding-Aligned Vision-Language Model,”Proceedings of the IEEE/CVF International Conference on Computer Vi- sion (ICCV), pp. 23030–23040, 2025. 2, 5

    2025

  16. [16]

    An Automated Essay Scoring Systems: A Systematic Literature Review,

    D. Ramesh and S. K. Sanampudi, “An Automated Essay Scoring Systems: A Systematic Literature Review,”Artifi- cial Intelligence Review, vol. 55, pp. 2495–2527, 2022. 1, 4, 5

    2022

  17. [17]

    A Survey on Deep Learning-Based Automated Essay Scoring and Feed- back Generation,

    H. Misgna, B.-W. On, I. Lee, and G. S. Choi, “A Survey on Deep Learning-Based Automated Essay Scoring and Feed- back Generation,”Artificial Intelligence Review, vol. 58, no. 2, article 36, 2024. 4

    2024

  18. [18]

    A Decade of Action Quality Assessment: Largest Systematic Survey of Trends, Challenges, and Future Directions,

    H. Yin, P. Parmar, D. Xu, Y . Zhang, T. Zheng, and W. Fu, “A Decade of Action Quality Assessment: Largest Systematic Survey of Trends, Challenges, and Future Directions,”Inter- national Journal of Computer Vision, vol. 134, no. 2, 2026. 1, 4, 5

    2026

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.