Pith. sign in

REVIEW 4 major objections 5 minor 16 references

A Finnish-language agricultural RAG system, grounded in local documents and refined through user feedback, shows clear gains in answer quality and user-rated reliability.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 05:24 UTC pith:R7G3K6WE

load-bearing objection A readable, honest case study of Finnish agricultural RAG, but the central 'clear gains' claim is undercut by an uncontrolled before/after design and a holistic rating scale. the 4 major comments →

arxiv 2602.02208 v1 pith:R7G3K6WE submitted 2026-02-02 cs.CL cs.AIcs.IRcs.SE

Towards AI Evaluation in Domain-Specific RAG Systems: The AgriHubi Case Study

Md. Toufique Hasan , Ayman Asad Khan , Mika Saari , Vaishnavi Bankhele , Pekka Abrahamsson This is my paper
classification cs.CL cs.AIcs.IRcs.SE
keywords retrieval-augmented generationFinnish languageagricultural decision supportPORO modelsuser evaluationlow-resource languageiterative system designLLM feedback loop
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that a retrieval-augmented generation (RAG) system built specifically for Finnish-language agriculture—AgriHubi—can move from prototype to usable decision-support tool through iterative, feedback-driven refinement. The authors claim that combining Finnish agricultural PDFs with open PORO-family language models and an explicit source-grounding mechanism improves answer completeness, linguistic accuracy, and perceived reliability. They support this with two user studies: between April and August 2025, the share of low user ratings (1–2) fell from 46% to 38%, and top ratings (5) rose from 3% to 21%. The paper argues that the largest gains came from retrieval and preprocessing improvements rather than model scaling alone.

Core claim

AgriHubi integrates Finnish agricultural documents with PORO family models in a RAG pipeline connecting document store, FAISS retriever, generative model, and a chat interface with built-in five-point ratings. Over eight iterations, it evolved from a Llama-based prototype to a Finnish-optimized platform. The paper's central discovery: iterative refinement guided by user feedback—especially retrieval, chunking, and answer-length changes—produced measurable gains. In the April round (PORO-34B, 67 ratings), 46% of answers were rated 1–2 and 3% rated 5; in August (PORO-2-70B, 47 ratings), low ratings dropped to 38% and top ratings rose to 21%. The authors attribute gains mainly to system-level r

What carries the argument

The central mechanism is the domain-adapted RAG pipeline: preprocessing Finnish agricultural PDFs with OCR, chunking with metadata, embedding via text-embedding-ada-002 into a FAISS index, retrieval of top-k chunks with L2-normalized cosine similarity, and prompt assembly through a language handler that detects Finnish, Swedish, and English. The PORO family models generate answers, and a SQLite database logs every query, retrieved passage, response, and rating—this feedback loop drives the eight iterations.

Load-bearing premise

The two user studies are comparable enough that the improvement in ratings is caused by the system changes rather than by differences in user groups, question difficulty, or rating behavior.

What would settle it

Run the August system and the April system on the same fixed set of Finnish agricultural questions with the same group of raters, switching only the model and pipeline; if the 46%→38% low-rating and 3%→21% top-rating shifts disappear, the reported gains are an artifact of the simultaneous changes.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If correct, domain-specific RAG systems for low-resource languages can be built and improved by combining local documents with open multilingual models, without large English-centric fine-tuning.
  • Iterative, feedback-driven refinement of retrieval and preprocessing, rather than model scaling alone, can turn a prototype into a usable decision-support tool within months.
  • Larger models within the PORO family give more accurate and complete answers but increase latency, and latency directly shapes user trust in deployed systems.
  • Structured user studies with logged ratings and qualitative comments provide a workable evaluation method for tracking system improvement over iterations.
  • Human review remains necessary for catching subtle terminology and context errors that automated metrics miss in low-resource settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the latency-trust link is causal, optimizing response time in shared-GPU deployments may raise perceived reliability even without further retrieval changes; a direct test would compare user ratings under artificially slowed versus full-speed responses.
  • The same feedback loop could transfer to other low-resource language domains with national documentation (e.g., legal or medical Finnish); running AgriHubi's eight-iteration process on a different language-document pair would test that transferability.
  • Because the paper changed model, retrieval, chunking, and answer length simultaneously, the individual contribution of each factor is unresolved; a factorial experiment would isolate them and could reveal which single change carries the largest share of the gain.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents AgriHubi, a retrieval-augmented generation (RAG) system for Finnish-language agricultural decision support, built on Finnish agricultural documents and open PORO-family language models. The system was developed over eight iterations and evaluated in two user studies (April 2025, 67 ratings; August 2025, 47 ratings). The authors report that low ratings (1–2) decreased from 46% to 38% and top ratings (5) increased from 3% to 21% (Table I), and they interpret this as clear gains in answer completeness, linguistic accuracy, and perceived reliability. The paper also discusses latency/quality trade-offs and provides qualitative observations from user feedback.

Significance. If the central claim were established, AgriHubi would be a useful case study for domain-specific RAG in a low-resource language: it combines an open-source Finnish-capable model family with localized documents, integrates a user-feedback loop, and reports real-world deployment lessons. The paper is transparent about many architectural choices and limitations, and the system description is sufficiently detailed to be reimplementable. However, the headline claim of 'clear gains' rests on an uncontrolled before/after comparison with small samples, no statistical testing, and no direct measurement of the named quality dimensions. As it stands, the paper is better characterized as a system description and observational report than as evidence that the deployed system improved in the measured qualities. These gaps are load-bearing for the paper's main conclusion.

major comments (4)
  1. [Section V-B / Table I] The central claim of improvement is underdetermined by the April-to-August comparison. Between the two rounds the generative model changed (PORO-34B to PORO-2-70B/8B, Section IV-C), retrieval strategy and chunking changed (Iterations 5 and 8), and maximum answer length increased from 700 to 2000 tokens (Section IV-C). The paper itself acknowledges that 'differences in question types between evaluation rounds may have introduced minor scoring bias' (Section V-C). With no control for question difficulty, user composition, or rating behavior, the observed shift in Table I cannot be attributed to system quality. I recommend either re-analyzing the data with a matched question set or paired user ratings, or substantially softening the causal language throughout.
  2. [Section V-A / Table I] The abstract and Section I claim gains specifically in 'answer completeness, linguistic accuracy, and perceived reliability,' but the only quantitative instrument reported is a single holistic 1–5 Likert rating. There is no rubric, no per-criterion scoring, and no independent evaluation of the answer text for completeness or linguistic accuracy. Therefore the named dimensions are not directly operationalized. The paper should either provide per-criterion evaluation data or restrict the claims to what the rating measures, i.e., overall user-perceived answer quality.
  3. [Section V-B / Section V-C] The quantitative evidence lacks inferential statistics. With 67 and 47 ratings, the changes in proportions (46% to 38% low; 3% to 21% top) may be within sampling variability. No significance tests, confidence intervals, or effect sizes are reported. The authors acknowledge in Section V-C that the sample sizes were 'insufficient for fine-grained statistical analysis,' but this admission appears only in Limitations; the Results section presents the differences as definitive. At minimum, report exact tests (e.g., Mann-Whitney U, chi-square, or bootstrapped proportion differences) and interpret the results accordingly.
  4. [Section V-D / Section VI] The qualitative feedback is summarized without a systematic protocol. The paper does not describe how many users provided written comments, how the comments were collected, or how themes like 'clearer and more reliable' were derived. No representative quotations or inter-rater reliability are given. In addition, users were not blinded to the system version, so expectancy effects could contribute to the more positive August feedback. The qualitative evidence should be presented as illustrative rather than as confirmation of improvement.
minor comments (5)
  1. [Abstract / Section V-B] The phrase 'clear gains' is stronger than the evidence supports; consider 'reported improvements' or 'observed increases' in the abstract and conclusion.
  2. [Section V-A] The number of participants is not stated; only the number of responses (67 and 47) is given. If multiple ratings per user are included, the effective sample size and potential within-user correlation should be addressed.
  3. [Section IV-C] The notation 'PORO-2-70B/8B' in the evaluation section is ambiguous; specify which model was used for each response, since Section V-A says 'in some cases PORO-2-8B' was used.
  4. [Section V-C] The statement that question-type differences 'may have introduced minor scoring bias' should be justified. If the bias is assumed minor, explain why; otherwise the assumption undermines the comparison.
  5. [General] Figures 1 and 2 are referenced but not visible in the manuscript text; ensure the submitted version includes them and that they are legible.

Circularity Check

0 steps flagged

No significant circularity: the improvement claim is an empirical before/after observation, not a derived or fitted result.

full rationale

The paper's central claim—that AgriHubi showed gains in answer completeness, linguistic accuracy, and perceived reliability—is based on two rounds of user Likert ratings (Table I), not on a derivation from a model or an equation. The April-to-August comparison is an observed empirical difference, and no fitted parameter is renamed as a prediction. The paper's own limitations, such as 'differences in question types between evaluation rounds may have introduced minor scoring bias' (Section V-C) and 'the lack of a consistent evaluation process across iterations made structured comparison difficult' (Section VI-D), are internal-validity threats, not circularity. The two self-citations [15] and [16] appear only in background motivation and are not load-bearing for the evaluation results; they do not define the outcome or force any conclusion. There is no self-definitional step, no uniqueness theorem imported from the authors' prior work, no ansatz smuggled in via citation, and no renaming of a known result. Thus there are no circular steps.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

No new physical or theoretical entities are introduced. The free parameters are hand-set system hyperparameters that the development process tuned; they are not derived from data or theory. The key axioms are domain assumptions about RAG efficacy and the validity of user ratings, both inherited from prior work or the evaluation design.

free parameters (4)
  • top_k (retrieval) = 5
    Number of retrieved chunks per query; chosen by hand and adjusted during iterations based on user feedback, affecting answer completeness and grounding.
  • distance_threshold (FAISS) = 2.0
    Similarity threshold for accepting retrieved chunks; set by hand and tuned during development, influencing retrieval precision and recall.
  • MAX_TOKENS (chunking) = 8192
    Token limit for text chunks in the embedding pipeline; chosen manually, impacts context window composition and retrieval quality.
  • Maximum answer length = 2000 tokens (increased from 700)
    Changed in Iteration 8 based on repeated user requests for more complete answers; directly affects perceived completeness.
axioms (3)
  • domain assumption RAG improves factual grounding over standalone generation
    Cited prior work [3], [4] is used to justify the architecture, but AgriHubi is not compared against a non-RAG baseline, so this premise is untested within the paper.
  • domain assumption User Likert ratings and qualitative comments are valid measures of answer completeness, linguistic accuracy, and reliability
    The evaluation relies entirely on subjective user ratings; there is no expert scoring or objective metric to validate these claims.
  • domain assumption The two evaluation rounds are comparable enough to attribute rating changes to system improvements
    Section V-A says question topics were 'kept similar' but not identical, and Section V-C admits differences in question types may bias scores. This comparability is load-bearing for the before/after claim.

pith-pipeline@v1.3.0-alltime-deepseek · 8862 in / 9024 out tokens · 78481 ms · 2026-08-03T05:24:16.999391+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Towards AI Evaluation in Domain-Specific RAG Systems: The AgriHubi Case Study." pith.science (2026). https://pith.science/paper/R7G3K6WE

@misc{pith2026260202208,
  author       = {Pith},
  title        = {Pith review of: Towards AI Evaluation in Domain-Specific RAG Systems: The AgriHubi Case Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R7G3K6WE}},
  note         = {Machine review of arXiv:2602.02208}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language models show promise for knowledge-intensive domains, yet their use in agriculture is constrained by weak grounding, English-centric training data, and limited real-world evaluation. These issues are amplified for low-resource languages, where high-quality domain documentation exists but remains difficult to access through general-purpose models. This paper presents AgriHubi, a domain-adapted retrieval-augmented generation (RAG) system for Finnish-language agricultural decision support. AgriHubi integrates Finnish agricultural documents with open PORO family models and combines explicit source grounding with user feedback to support iterative refinement. Developed over eight iterations and evaluated through two user studies, the system shows clear gains in answer completeness, linguistic accuracy, and perceived reliability. The results also reveal practical trade-offs between response quality and latency when deploying larger models. This study provides empirical guidance for designing and evaluating domain-specific RAG systems in low-resource language settings.

Figures

Figures reproduced from arXiv: 2602.02208 by Ayman Asad Khan, Md. Toufique Hasan, Mika Saari, Pekka Abrahamsson, Vaishnavi Bankhele.

Figure 1
Figure 1. Figure 1: AgriHubi RAG workflow from query embedding to retrieval, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: UML Class Diagram showing the main modules, data flows, and interactions within the AgriHubi system. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 5 linked inside Pith

  1. [1]

    Generative artificial intelligence for software engineering- a research agenda,

    A. Nguyen-Duc, B. Cabrero-Daniel, A. Przybylek, C. Arora, D. Khanna, T. Herda, U. Rafiq, J. Melegati, E. Guerra, K.-K. Kemell et al., “Generative artificial intelligence for software engineering- a research agenda,”Software: Practice and Experience, vol. 55, no. 11, pp. 1806–1843, 2025

    2025

  2. [2]

    Reason- ing on efficient knowledge paths: knowledge graph guides large language model for domain question answering,

    B. Jiang, Y . Wang, Y . Luo, D. He, P. Cheng, and L. Gao, “Reason- ing on efficient knowledge paths: knowledge graph guides large language model for domain question answering,” in2024 IEEE International Conference on Knowledge Graph (ICKG). IEEE, 2024, pp. 142–149

    2024

  3. [3]

    Agrollm: Connecting farmers and agricultural practices through large language models for enhanced knowledge transfer and prac- tical application,

    D. J. Samuel, I. Skarga-Bandurova, D. Sikolia, and M. Awais, “Agrollm: Connecting farmers and agricultural practices through large language models for enhanced knowledge transfer and prac- tical application,”arXiv preprint arXiv:2503.04788, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Agriregion: Region-aware retrieval for high- fidelity agricultural advice,

    M. Fanuel, M. N. Mahmoud, C. C. Marshal, V . Lakhotia, B. Dari, K. Roy, and S. Zhang, “Agriregion: Region-aware retrieval for high- fidelity agricultural advice,”arXiv preprint arXiv:2512.10114, 2025

    arXiv 2025

  5. [5]

    Llms beyond english: Scaling the multilingual capability of llms with cross-lingual feedback,

    W. Lai, M. Mesgar, and A. Fraser, “Llms beyond english: Scaling the multilingual capability of llms with cross-lingual feedback,” arXiv preprint arXiv:2406.01771, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Poro 34b and the blessing of multilinguality,

    R. Luukkonen, J. Burdge, E. Zosa, A. Talman, V . Komulainen, V . Hatanpää, P. Sarlin, and S. Pyysalo, “Poro 34b and the blessing of multilinguality,” inProceedings of the joint 25th nordic conference on computational linguistics and 11th baltic conference on human language technologies (nodalida/baltic-hlt 2025), 2025, pp. 367– 382

    2025

  7. [7]

    A survey of large language models for european languages,

    W. Ali and S. Pyysalo, “A survey of large language models for european languages,”arXiv preprint arXiv:2408.15040, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Foundation models in agriculture: A comprehensive review,

    S. Yin, Y . Xi, X. Zhang, C. Sun, and Q. Mao, “Foundation models in agriculture: A comprehensive review,”Agriculture, vol. 15, no. 8, p. 847, 2025

    2025

  9. [9]

    Knowledge assimilation: Implementing knowledge-guided agricul- tural large language model,

    J. Jiang, L. Yan, H. Liu, Z. Xia, H. Wang, Y . Yang, and Y . Guan, “Knowledge assimilation: Implementing knowledge-guided agricul- tural large language model,”Knowledge-based systems, vol. 314, p. 113197, 2025

    2025

  10. [10]

    Enhancing ai-driven farming advisory in kenya with efficient rag agents via quantized fine-tuned language models,

    T. L. Owiti and A. K. Kipkebut, “Enhancing ai-driven farming advisory in kenya with efficient rag agents via quantized fine-tuned language models,” inProceedings of the Sixth Workshop on African Natural Language Processing (AfricaNLP 2025), 2025, pp. 24–30

    2025

  11. [11]

    Beefbot: Harnessing advanced llm and rag techniques for pro- viding scientific and technology solutions to beef producers,

    Z. Zhang, C.-A. Wilson, R. Hay, Y . Everingham, and U. Naseem, “Beefbot: Harnessing advanced llm and rag techniques for pro- viding scientific and technology solutions to beef producers,” in Proceedings of the 31st International Conference on Computational Linguistics: System Demonstrations, 2025, pp. 54–62

    2025

  12. [12]

    Agrigpt: A large language model ecosystem for agriculture,

    B. Yang, Y . Zhang, L. Feng, Y . Chen, J. Zhang, X. Xu, N. Aierken, Y . Li, Y . Chen, G. Yanget al., “Agrigpt: A large language model ecosystem for agriculture,”arXiv preprint arXiv:2508.08632, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Shizishangpt: An agricultural large language model integrating tools and resources,

    S. Yang, Z. Liu, W. Mayer, N. Ding, Y . Wang, Y . Huang, P. Wu, W. Li, L. Li, H.-Y . Zhanget al., “Shizishangpt: An agricultural large language model integrating tools and resources,” inInternational Conference on Web Information Systems Engineering. Springer, 2024, pp. 284–298

    2024

  14. [14]

    A systematic literature review of retrieval-augmented generation: Techniques, metrics, and challenges,

    A. Brown, M. Roman, and B. Devereux, “A systematic literature review of retrieval-augmented generation: Techniques, metrics, and challenges,”arXiv preprint arXiv:2508.06401, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Engineering rag systems for real-world applications: Design, development, and evaluation,

    M. T. Hasan, M. Waseem, K.-K. Kemell, A. A. Khan, M. Saari, and P. Abrahamsson, “Engineering rag systems for real-world applications: Design, development, and evaluation,” inSoftware Engineering and Advanced Applications, D. Taibi and D. Smite, Eds. Cham: Springer Nature Switzerland, 2026, pp. 143–158

    2026

  16. [16]

    From PDF to Dataset: Semi-Automatic Extraction of Fine-Tuning Data,

    N. Suvitie, M. Saari, and P. Abrahamsson, “From PDF to Dataset: Semi-Automatic Extraction of Fine-Tuning Data,” in49th ICT and Electronics Convention – MIPRO 2026. IEEE, 2026, submitted

    2026