pith. sign in

arxiv: 2606.08272 · v1 · pith:KED5JMR3new · submitted 2026-06-06 · 💻 cs.CL · cs.AI

AgriGov: A Structured Multilingual Dataset Curation for Indian Government Schemes for Farmers

Mohsina Bilal , Gopakumar G This is my paper

Pith reviewed 2026-06-27 19:37 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords multilingual datasetagricultural schemesfarmer welfaremachine translationHindi Marathi alignmentgovernment policiesdomain adaptationparallel corpus
0
0 comments X

The pith

A schema-driven pipeline produced a trilingual dataset of 50 Indian farmer schemes with human-corrected translations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes a new resource called AgriGov that collects and structures information from 50 government schemes for farmers into consistent fields such as eligibility, documents required, and application steps. The data starts in English from official portals and is converted to Hindi and Marathi through a sequence of machine translation tools followed by human review. Additional parallel sentences from an existing corpus are added to reach roughly 8000 aligned pairs. The resulting collection is positioned for use in domain-specific machine translation, question answering, and information retrieval systems that target agricultural policy content. The central mechanism is a controlled pipeline that keeps track of sources and corrections to maintain fidelity to the original government material.

Core claim

AgriGov supplies approximately 2100 source segments from 50 schemes in English together with aligned Hindi and Marathi versions created by automated translation plus human post-editing, then expanded to about 8000 sentence pairs by augmentation with external parallel data; the construction follows a schema-driven, human-corrected alignment process that records provenance and supports reproducible use in farmer-facing language applications.

What carries the argument

The schema-driven, human-corrected multilingual alignment pipeline that structures scraped scheme data into fixed semantic fields and applies machine translation followed by human editing to produce domain-faithful parallel text.

If this is right

  • Fine-tuning of machine translation models becomes possible on a domain-specific agricultural corpus rather than general text.
  • Question-answering and retrieval systems can draw from structured, provenanced scheme information instead of unstructured web text.
  • Experiments on summarization and information retrieval for farmer welfare programs can be repeated using the same source material.
  • Retrieval-augmented tools for farmers can reference the aligned parallel sentences directly.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Integration into mobile apps could let farmers query scheme details in their preferred language without needing English literacy.
  • The same pipeline structure could be applied to other government domains such as health or education schemes to create similar aligned resources.
  • Measuring downstream task performance on real farmer queries would test whether the dataset's domain fidelity translates into measurable accuracy gains.

Load-bearing premise

Automated scraping from official sites plus machine translation and human editing together produce Hindi and Marathi versions that retain the exact meaning and specialized agricultural terms of the original English scheme documents.

What would settle it

Independent review by agricultural domain experts that identifies repeated mistranslations, omitted eligibility conditions, or altered meanings in the Hindi or Marathi portions of the dataset.

read the original abstract

AgriGov is a curated, trilingual (English-Hindi-Marathi) dataset designed to address the scarcity of domain-grounded multilingual resources for agricultural policies and farmer welfare schemes. Initially, we collected and structured data from 50 government schemes sourced from trusted portals using automated scraping techniques, organizing it into predefined semantic fields (e.g., title, eligibility, application process, documents, exclusions). Translations were performed using a pipeline combining Google Translate API, MarianMT, and human post-editing, resulting in a domain-specific Hindi-Marathi dataset comprising approximately 2100 source segments. To enhance coverage, we augmented this dataset with sentences from the Samanantar corpus, leading to approximately 8,000 sentence-aligned Hindi-Marathi parallel pairs. The dataset now offers robust resources for fine-tuning machine translation models in this domain. AgriGov is designed for applications in domain-adaptive machine translation, question answering, information retrieval, and summarization systems. Its key contribution is a schema-driven, human-corrected multilingual alignment pipeline that ensures domain fidelity, provides provenance, and supports reproducible experiments, enabling retrieval-augmented applications for farmer-facing tools.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript presents AgriGov, a trilingual (English-Hindi-Marathi) dataset covering 50 Indian government agricultural schemes for farmers. Data are collected via automated scraping from official portals and organized into a fixed schema of semantic fields (title, eligibility, application process, documents, exclusions). Translations combine Google Translate, MarianMT, and human post-editing to produce ~2100 domain-specific segments; the corpus is then augmented with Samanantar sentences to reach ~8000 aligned Hindi-Marathi pairs. The stated contribution is a schema-driven, human-corrected alignment pipeline that guarantees domain fidelity, supplies provenance, and enables reproducible experiments for downstream tasks including domain-adaptive MT, QA, IR, and summarization.

Significance. If the fidelity and quality claims are substantiated, the resource would fill a genuine gap in domain-grounded multilingual data for Indian agricultural policy, directly supporting farmer-facing retrieval and translation tools. The explicit schema, provenance tracking, and public release orientation are clear strengths for reproducibility.

major comments (2)
  1. [Abstract] Abstract (translation pipeline paragraph): the central claim that the MT-plus-human post-editing pipeline 'ensures domain fidelity' is unsupported by any quantitative metric (term-level accuracy, inter-annotator agreement, error rate, or before/after examples). Because the fidelity guarantee is the load-bearing element of the contribution, this omission prevents verification of the weakest assumption identified in the review.
  2. [Abstract] Abstract (human post-editing description): no information is supplied on editor qualifications, editing guidelines, volume of edits performed, or fraction of the ~2100 segments that were corrected. Without these details the reproducibility and reliability claims cannot be evaluated.
minor comments (1)
  1. [Abstract] The abstract reports both 'approximately 2100' and 'approximately 8,000' without stating whether these are exact counts or rounded; providing precise figures or a table of segment counts per language and source would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the two major comments point by point below, agreeing that additional details are needed to support the claims made in the abstract.

read point-by-point responses

  1. Referee: [Abstract] Abstract (translation pipeline paragraph): the central claim that the MT-plus-human post-editing pipeline 'ensures domain fidelity' is unsupported by any quantitative metric (term-level accuracy, inter-annotator agreement, error rate, or before/after examples). Because the fidelity guarantee is the load-bearing element of the contribution, this omission prevents verification of the weakest assumption identified in the review.

    Authors: We agree that the abstract asserts domain fidelity without quantitative backing. The manuscript describes the pipeline (Google Translate, MarianMT, human post-editing) but provides no metrics, examples, or agreement scores. We will revise the abstract to qualify the claim and add a methods subsection with any available evidence (e.g., sample corrections or error analysis) or explicitly note the absence of such metrics as a limitation. revision: yes

  2. Referee: [Abstract] Abstract (human post-editing description): no information is supplied on editor qualifications, editing guidelines, volume of edits performed, or fraction of the ~2100 segments that were corrected. Without these details the reproducibility and reliability claims cannot be evaluated.

    Authors: The manuscript does not include these details on the human post-editing step. We will revise both the abstract and the methods section to specify editor qualifications (native Hindi/Marathi speakers with agricultural policy familiarity), the editing guidelines followed, and available statistics on edit volume or fraction of segments revised, drawing from our internal process records. revision: yes

Circularity Check

0 steps flagged

No significant circularity in empirical dataset curation

full rationale

The paper describes an empirical process of scraping government scheme data from portals, structuring it into predefined semantic fields, applying MT (Google Translate, MarianMT) followed by human post-editing, and augmenting with the external Samanantar corpus to produce ~8000 aligned pairs. No mathematical derivations, equations, fitted parameters, or predictions are present that could reduce to inputs by construction. Claims about domain fidelity and reproducibility rest on the described pipeline rather than self-citations, uniqueness theorems, or renamed known results. This is a standard data-curation contribution with no load-bearing circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The work rests on the assumption that official portals contain complete and accurate scheme information and that the chosen MT tools plus human editing preserve domain meaning; no free parameters or new entities are introduced.

axioms (1)
  • domain assumption Official government portals provide complete and accurate information on the listed schemes
    Invoked when the authors state they collected data from trusted portals using automated scraping.

pith-pipeline@v0.9.1-grok · 5728 in / 1187 out tokens · 20556 ms · 2026-06-27T19:37:46.391920+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 6 canonical work pages · 1 internal anchor

  1. [1]

    Ramesh, G., Doddapaneni, S., Bheemaraj, A., al.: Samanantar: The largest pub- licly available parallel corpora collection for 11 indic languages. Trans. Assoc. Comput. Linguistics (2022) https://doi.org/10.1162/tacl a 00452

    work page internal anchor Pith review doi:10.1162/tacl 2022

  2. [2]

    Muril: Multilingual representations for indian languages,

    Khanuja, S., Bansal, D., Mehtani, S., al.: Muril: Multilingual representations for indian languages. arXiv (2021) https://doi.org/arXiv:2103.10730

    work page arXiv 2021

  3. [3]

    Language-agnostic

    Feng, F., Yang, Y., Cer, D., Arivazhagan, N., Wang, W.: Language-agnostic bert sentence embedding (labse). In: Proceedings of the ACL (2022). https://doi.org/ 10.18653/v1/2022.acl-long.62 14

    work page doi:10.18653/v1/2022.acl-long.62 2022

  4. [4]

    Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages.arXiv preprint arXiv:2305.16307, 2023

    Gala, J., Chitale, P.A., Raghavan, A.K., al.: Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages. arXiv (2023) https://doi.org/arXiv:2305.16307

    work page arXiv 2023

  5. [5]

    In: ACL Workshop/Papers (2018)

    Junczys-Dowmunt, M., al.: Marian: Fast neural machine translation in c++. In: ACL Workshop/Papers (2018)

    2018

  6. [6]

    In: ACL (2016)

    Sennrich, R., Haddow, B., Birch, A.: Improving neural machine translation models with monolingual data (back-translation). In: ACL (2016). https://doi.org/10. 18653/v1/P16-1009

    2016

  7. [7]

    In: EMNLP (2019)

    Wei, J., Zou, K.: Eda: Easy data augmentation techniques for boosting perfor- mance on text classification tasks. In: EMNLP (2019)

    2019

  8. [8]

    Min, S., Michael, J., Hajishirzi, H., and Zettlemoyer, L

    Rei, R., Stewart, C., Farinha, A.C., Lavie, A.: Comet: A neural framework for mt evaluation. In: EMNLP (2020). https://doi.org/10.18653/v1/2020.emnlp-main. 213

    work page doi:10.18653/v1/2020.emnlp-main 2020

  9. [9]

    https://www.myscheme.gov.in/

    Ministry of Electronics and Information Technology, Government of India: myScheme: Government Schemes Platform. https://www.myscheme.gov.in/. Accessed: Feb. 10, 2025

    2025

  10. [10]

    https://agricoop.gov.in/

    Ministry of Agriculture and Farmers Welfare, Government of India: Depart- ment of Agriculture and Farmers Welfare Initiatives. https://agricoop.gov.in/. Accessed: Feb. 10, 2025

    2025

  11. [11]

    https://www

    Wikimedia Foundation: Wikipedia: The Free Encyclopedia. https://www. wikipedia.org/. Accessed: Feb. 10, 2025

    2025

  12. [12]

    TACL (2020)

    Liu, Y., Gu, J., Goyal, N., al.: Multilingual denoising pre-training for neural machine translation (mbart). TACL (2020)

    2020

  13. [13]

    In: NAACL (2021)

    Xue, L., Constant, N., Roberts, A., al.: mt5: A massively multilingual pre-trained text-to-text transformer. In: NAACL (2021)

    2021

  14. [14]

    Sentence- BERT : Sentence Embeddings using S iamese BERT -Networks

    Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. In: EMNLP (2019). https://doi.org/10.18653/v1/D19-1410 15

    work page doi:10.18653/v1/d19-1410 2019