Generative AI for Analyzing Participatory Rural Appraisal Data: An Exploratory Case Study in Gender Research

Aswathi Padmavilochanan; Christopher Coley; Rao R. Bhavani; Srividya Sheshadri; Unnikrishnan Radhakrishnan

REVIEW 4 major objections 5 minor 18 references

Generative AI for Analyzing Participatory Rural Appraisal Data: An Exploratory Case Study in Gender Research

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper reports that three state-of-the-art multimodal LLMs—GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro—frequently misclassify, mistranslate, or hallucinate elements when analyzing hand-drawn Participatory Rural Appraisal artifacts…

desk verdict Well-intentioned exploratory study with concrete failure examples, but the evaluation is too thin to support the strong claim. read the letter →

arxiv 2502.00763 v1 pith:PCI4KCJ2 submitted 2025-02-02 cs.CY

Srividya Sheshadri , Unnikrishnan Radhakrishnan , Aswathi Padmavilochanan , Christopher Coley , Rao R. Bhavani This is my paper · ORCID

classification cs.CY

keywords participatoryruralappraisalgenerativeAIlargelanguagemodelsmultimodalanalysiswomen'sempowermenthand-drawnartifactshallucinationIndicscripts

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether generative AI can take over the labor-intensive reading of Participatory Rural Appraisal (PRA) data—hand-drawn charts, produced by groups of rural women, that describe what an ideal village would look like and which problems the community can control. The authors ran ten Ideal Village drawings and ten Circle of Control drawings from five Indian states through GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro, asking each model to list elements, classify them into AWESOME framework dimensions, and locate Circle of Control items spatially. Their central finding is that all three models still fail in ways that matter: they mistranslate Hindi and Malayalam, misclassify items such as a school as an economic livelihood, and fabricate content, including a detailed dowry assessment for something not in the drawing. The paper concludes that current GenAI can assist analysis but cannot replace human interpretation in gender and rural-development research.

What carries the argument

The load-bearing instrument is the Ideal Village activity, a participatory exercise in which small groups of women draw their vision of an ideal village on chart paper; its Circle of Control variant adds concentric rings marking what the community can change, can influence, and can only be concerned about. The AWESOME framework supplies the categorization scheme the prompts force the models to apply, with dimensions such as Environment, Economics & Livelihood, Education & Skill Development, Health, Social/Political/Cultural Element, and Safety & Security. The mechanism under test is prompt-driven multimodal classification: each model receives a photograph of a drawing plus a structured prompt asking for element-by-element extraction, category assignment, and (for Circle of Control) spatial location, and the paper compares the generated tables against human readings of the same images.

What would settle it

Re-run the same set of drawings with pre-registered, optimized prompts and a detailed extraction codebook, and compute per-element agreement between each model and two or more trained human analysts; if all three models exceed about 95% agreement with no fabrications and correct translation of the Hindi and Malayalam text, the paper's claim of significant current limitations would be refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the current generation of multimodal large language models cannot be trusted to convert unstructured PRA visual data into structured categories without substantial human validation. The models show partial skill at basic visual parsing, but performance degrades sharply with handwritten multilingual text, low-resolution images, and culturally specific content: a Circle of Control phrase that said 'gatars' ('gutters') was rendered without that word, a 'flower' was read where the drawing likely said 'tower,' and an invented 'dowry' element appeared with a detailed explanation and severity assessment. The pattern is consistent across all three models, with GPT-4o facing the highest visual-interpretation difficulty, GPT-4o and Claude both showing high misclassification, Gemini rated medium on misclassification while still stumbling on ambiguities, and all three rated high on the need for human oversight and dataset quality.

Load-bearing premise

The ratings assume that the specific prompts, model versions, temperature setting, and human comparisons used here are a fair and competent test of each model's capability; if the prompts were suboptimal, the observed failures would show prompt limits, not model ceilings.

Editorial extensions

If this is right

If the finding holds, automating PRA analysis requires a human-in-the-loop verification step; the paper recommends manual validation for reliable insights.
Multilingual support in current models is not adequate for handwritten Hindi and Malayalam content, so scaling PRA analysis across Indian states will require better OCR, translation, or script-specific training.
Image quality is a first-order factor: the paper reports that hallucinations increase when imagery is poor, meaning dataset documentation and capture protocols directly affect AI reliability.
The AWESOME framework's empowerment dimensions cannot yet be populated from images alone; model output on community-level factors and vulnerabilities was preliminary and constrained by data limits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A fair comparison between models probably needs pre-registered prompts, multiple independent human raters, and inter-rater agreement statistics; the paper's prompts went through undocumented iterations, so its ratings may partly reflect prompt quality rather than an inherent model ceiling.
The 'dowry' hallucination hints that models will import stereotyped assumptions about rural Indian gender relations into the artifact, making bias amplification a central risk for gender research on top of the accuracy problem.
A practical two-stage pipeline—AI proposes candidate elements, human analysts confirm or reject—would test whether the models' errors are concentrated enough to be corrected cheaply; the paper gestures toward human oversight but does not measure this.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my read. The paper is the first thing I've seen that tries multimodal LLMs on hand-drawn PRA artifacts with Hindi and Malayalam text. That alone is worth a look for anyone working on participatory development data. The authors show concrete examples—a 'flower' read as 'tower,' a missed 'gatars' ('gutters'), a circle-location mix-up—and those examples are credible. They are also honest in Section VI about missing field-note context and dataset quality. So the paper earns the point that current models struggle with this kind of data.

Where it falls short is the evidence behind the headline claim. Table I rates the three models High/Medium/Low across seven themes, but there is no scoring rubric, no counts of correct vs. incorrect elements, no inter-rater agreement, and no human baseline. The reader can't verify 'cannot reliably classify' because the paper never says what 'reliable' would mean or how the ratings were derived. The absence of a human comparison is the biggest hole: PRA drawings are ambiguous, and if two expert coders disagreed at a comparable rate, the 'misclassifications' wouldn't be a model-specific finding.

The temperature 1.0 with no repeated sampling is also a problem—a single stochastic output per image could drive the qualitative conclusions. And the prompts went through undocumented iterations, so some failures might be prompt artifacts rather than model limitations. These are fixable issues: share the full prompts, run multiple samples, quantify errors, and code a baseline with two human annotators.

I don't think the paper has a load-bearing flaw. The central direction is plausible and the examples are genuine. But as reported, it's an anecdote-rich exploratory study, not a supported negative result. For practitioners it's still useful: it justifies continued human coding and motivates investment in better multilingual models. For AI researchers, the concrete failure cases are worth cataloguing.

I'd send it to review rather than desk-reject, because the question is underserved and the examples are valuable. But I'd expect the reviewers to push for a real evaluation protocol. I'd bring it to our reading group as a case study in how to (and how not to) evaluate LLM applications in qualitative research.

Referee Report

4 major / 5 minor

Summary. The paper reports an exploratory case study in which three generative AI models (GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro) are prompted to analyze hand-drawn Participatory Rural Appraisal (PRA) artifacts from the "Ideal Village" activity, with the goal of identifying and classifying empowerment-related elements. The authors describe their prompt-development process, apply the prompts to 10 Ideal Village and 10 Circle of Control drawings, and qualitatively assess model outputs. They report that the models face recurring difficulties in visual interpretation, translation of Hindi and Malayalam text, consistent classification, and avoidance of hallucinations, and they conclude that current GenAI systems require substantial human oversight before they can be used to analyze such data. The paper also situates the work within the AWESOME framework and discusses implications for gender research and participatory AI.

Significance. If its central claim is established, the paper addresses a genuinely underexplored problem: applying multimodal LLMs to community-generated, multilingual, hand-drawn visual data from rural development research. The strengths of the paper are its focus on real-world field data, its use of three frontier models under a common prompt protocol, and its concrete examples of misclassification and hallucination. These examples are valuable for practitioners who might otherwise assume that current vision-language models can reliably process such artifacts. However, the evaluation is entirely qualitative. The paper provides no accuracy counts, no inter-rater reliability checks, no statistical tests, and no human-expert baseline, so the central negative claim about model capability is not verifiable as reported. The contribution is currently at the level of a well-illustrated experience report rather than a rigorous evaluation.

major comments (4)

[Section V, Table I] The central claim that the models exhibit "significant challenges" depends on the High/Medium/Low ratings in Table I, but no scoring rubric, rating criteria, or quantitative basis for these ratings is provided. It is not possible to determine how many elements were correctly identified per model, how errors were counted, or whether a "High" rating under "Language and Translation Challenges" means high difficulty, high frequency, or high severity. The table caption also conflates "difficulty" and "success," making the scale ambiguous. To support the central claim, the authors should report per-model counts of correctly identified elements, misclassifications, and hallucinations, ideally with a confusion matrix or error taxonomy.
[Section V] The paper asserts that model misclassifications represent a model-specific limitation, but it does not include any human-expert baseline. PRA drawings are inherently ambiguous, and two trained annotators may disagree on elements such as "flower" versus "tower" or on the exact boundary between "middle" and "inner" circles. Without a measure of human-expert agreement on the same artifacts, the reported errors cannot be attributed to the AI models rather than to the inherent ambiguity of the data. At minimum, the authors should have two or more human coders independently label the same drawings and report agreement rates.
[Section IV-A and Section V] The evaluation uses a single sample per image at temperature 1.0 with no repeated sampling. Because autoregressive LLM outputs are stochastic, a single run could produce unrepresentative errors, and the qualitative conclusions could change across runs. The paper states that preliminary testing showed minimal variation between temperature 0 and 1, but no data for that test are provided. The authors should either run multiple samples per image and report aggregate results or justify a deterministic decoding setting. They should also document the final prompt versions and the number of iterative revisions, since the current description makes it impossible to assess whether the observed failures reflect model limitations or prompt-design choices.
[Section VI] The limitations section acknowledges missing field notes and the need for a more systematic evaluation framework, but it does not address the absence of quantitative performance metrics, which is the most load-bearing gap for the paper's conclusion. The paper's claim that models "cannot reliably classify" empowerment-related elements requires a threshold or comparative metric; the current evidence is a set of anecdotal examples. The authors should either add quantitative accuracy measures and human baselines or explicitly reframe the paper as a qualitative exploration rather than an evaluation of model capability.

minor comments (5)

[Section V, Table I] The rating labels are inconsistent with the theme names: for example, "Human Oversight and Dataset Quality" is rated "High" for all models, but it is unclear whether high indicates a high degree of challenge, a high requirement, or high model success. The table should define the scale explicitly and use consistent directionality.
[Section IV-B and V] The dataset description gives image resolutions and formats but no information on how many villages, participants, or states produced each drawing, and no indication of how the 10 Ideal Village and 10 Circle of Control images were selected. Adding a table with per-image characteristics would improve reproducibility.
[Section V, Figures 1 and 2] The figures show model output tables but not the original drawings side by side, so readers cannot independently verify the claimed misclassifications. Including the original images or making them available in supplementary material would strengthen the paper.
[Section IV-A] The sentence "The prompt responses included step-by-step instructions and the PRA visual artifacts" is unclear; it should say that the prompts included instructions and the images. Also, the number of prompt iterations is not specified, despite the importance of this detail for reproducibility.
[Section VI] The limitation that field notes were absent is well taken, but the sentence "the models exhibited frequent errors and hallucinations" would benefit from a concrete definition of "frequent" and from examples of how error frequency was estimated.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the evaluation is empirical and the self-citations (AWESOME framework) do not force the reported model limitations.

full rationale

This paper is an exploratory empirical evaluation rather than a derivation. The central claim that GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro make misclassifications and hallucinations on hand-drawn PRA artifacts is supported by concrete examples (e.g., flower vs. tower, missed gatars, middle vs. inner) and by a comparative rating table, not by an equation or fitted model. The AWESOME framework [8] is cited from the authors' prior work and supplies the classification dimensions, but the model outputs are compared against those externally fixed categories; the categories are not estimated from model outputs, so no self-definitional reduction occurs. The absence of a formal scoring rubric, inter-rater agreement, or human baseline is a real validity/reproducibility limitation, but it is not circularity. Likewise, undocumented prompt iterations and temperature 1.0 are methodological concerns about whether the test is fair, not evidence that the conclusion is equivalent to its inputs. No parameter is fitted to a subset and then renamed as a prediction, and no uniqueness theorem or prior result is invoked to force the interpretation. The self-citations [8,9] are background framing and do not carry the load of the empirical finding. Therefore no circular step can be exhibited.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper's central claim depends mainly on two unmeasured choices: the prompt and protocol used to query models, and the human judgement used as ground truth. No equations, fitted constants, or newly postulated entities are involved.

free parameters (2)

temperature = 1.0
Chosen after undocumented preliminary testing; high temperature may inflate hallucination rates and thus affect the paper's main finding.
output_token_limit = 4095
Imposed cap on response length; could truncate or shape outputs and affect completeness comparisons.

assumptions (3)

domain assumption The AWESOME framework's pre-defined dimensions are the correct categories for classifying empowerment-relevant elements in Ideal Village drawings.
Section II and Section V define the evaluation dimensions from this framework; if the categories are wrong, model 'misclassification' assessments are meaningless.
domain assumption Expert human interpretation is an appropriate ground truth, even though no formal scoring protocol is described.
Section V says outputs were assessed against expert interpretation but provides no procedure or reliability data.
domain assumption The sample of 20 drawings from five Indian states is sufficient to draw general conclusions about LLM performance on PRA data.
Section IV-B describes the dataset; no sampling strategy or power analysis is given, so representativeness is assumed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Generative AI for Analyzing Participatory Rural Appraisal Data: An Exploratory Case Study in Gender Research." pith.science (2026). https://pith.science/paper/PCI4KCJ2

@misc{pith2026250200763,
  author       = {Pith},
  title        = {Pith review of: Generative AI for Analyzing Participatory Rural Appraisal Data: An Exploratory Case Study in Gender Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PCI4KCJ2}},
  note         = {Machine review of arXiv:2502.00763}
}

read the original abstract

This study explores the novel application of Generative Artificial Intelligence (GenAI) in analyzing unstructured visual data generated through Participatory Rural Appraisal (PRA), specifically focusing on women's empowerment research in rural communities. Using the "Ideal Village" PRA activity as a case study, we evaluate three state-of-the-art Large Language Models (LLMs) - GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro - in their ability to interpret hand-drawn artifacts containing multilingual content from various Indian states. Through comparative analysis, we assess the models' performance across critical dimensions including visual interpretation, language translation, and data classification. Our findings reveal significant challenges in AI's current capabilities to process such unstructured data, particularly in handling multilingual content, maintaining contextual accuracy, and avoiding hallucinations. While the models showed promise in basic visual interpretation, they struggled with nuanced cultural contexts and consistent classification of empowerment-related elements. This study contributes to both AI and gender research by highlighting the potential and limitations of AI in analyzing participatory research data, while emphasizing the need for human oversight and improved contextual understanding. Our findings suggest future directions for developing more inclusive AI models that can better serve community-based participatory research, particularly in gender studies and rural development contexts.

Figures

Figures reproduced from arXiv: 2502.00763 by the authors.

**Figure 2.** Sample output illustrating GPT-4o’s analysis of a Circle of Control [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 16 canonical work pages

[1]

Cornwall and G

A. Cornwall and G. Pratt, ”The use and abuse of participatory rural appraisal: Reflections from practice,” Agriculture and Human Values , vol. 28, pp. 263-272, 2011

work page 2011
[2]

J. L. Parpart, ”Rethinking participatory empowerment, gender and de- velopment: the PRA approach,” in Rethinking Empowerment, Routledge, 2003, pp. 165-181

work page 2003
[3]

Chambers, ”Participatory rural appraisal (PRA): Analysis of experi- ence,” World Development, vol

R. Chambers, ”Participatory rural appraisal (PRA): Analysis of experi- ence,” World Development, vol. 22, no. 9, pp. 1253-1268, 1994

work page 1994
[4]

The contribution of transformative learning theory to the practice of participatory research and extension: Theoretical reflections,

R. Percy, “The contribution of transformative learning theory to the practice of participatory research and extension: Theoretical reflections,” Agriculture and Human Values , vol. 22, pp. 127-136, 2005

work page 2005
[5]

L. W. Cong, D. Xie, and L. Zhang, ”Knowledge accumulation, privacy, and growth in a data economy,” Management Science, vol. 67, no. 10, pp. 6480-6492, 2021

work page 2021
[6]

Challenges and opportunities beyond structured data in analysis of electronic health records,

M. Tayefi, P. Ngo, T. Chomutare, H. Dalianis, E. Salvi, A. Budrionis, and F. Godtliebsen, “Challenges and opportunities beyond structured data in analysis of electronic health records,” Wiley Interdiscip. Rev. Comput. Stat., vol. 13, no. 6, e1549, 2021

work page 2021
[7]

Exploring AI-driven approaches for unstructured document analysis and future horizons,

S. V . Mahadevkar, S. Patil, K. Kotecha, L. W. Soong, and T. Choudhury, “Exploring AI-driven approaches for unstructured document analysis and future horizons,” Journal of Big Data , vol. 11, no. 1, p. 92, 2024

work page 2024
[8]

Vulnerability mapping: A conceptual framework towards a context-based approach to women’s empower- ment,

C. M. Gressel, T. Rashed, L. A. Maciuika, S. Sheshadri, C. Coley, S. Kongeseri, and R. R. Bhavani, “Vulnerability mapping: A conceptual framework towards a context-based approach to women’s empower- ment,” World Development Perspectives, vol. 20, p. 100245, 2020

work page 2020

Show all 18 references

[9]

Sheshadri, C

S. Sheshadri, C. Coley, S. Devanathan, and R. B. Rao, ”Towards synergistic women’s empowerment-transformative learning framework for TVET in rural India,” Journal of Vocational Education & Training , vol. 75, no. 2, pp. 255-277, 2023

2023
[10]

M. A. Zimmerman and J. H. Zahniser, ”Refinements of sphere-specific measures of perceived control: Development of a sociopolitical control scale,” Journal of Community Psychology , vol. 19, no. 2, pp. 189-204, 1991

1991
[11]

Mezirow, Transformative Dimensions of Adult Learning

J. Mezirow, Transformative Dimensions of Adult Learning . San Fran- cisco, CA: Jossey-Bass, 1991

1991
[12]

Emerging challenges in AI and the need for AI ethics education,

J. Borenstein and A. Howard, “Emerging challenges in AI and the need for AI ethics education,” AI and Ethics , vol. 1, pp. 61-65, 2021

2021
[13]

Hello Siri, how does the patriarchy influence you? — Understanding artificial intelligence and gender inequality,

D. Thakur, A. Brandusescu, and N. Nwakamma, “Hello Siri, how does the patriarchy influence you? — Understanding artificial intelligence and gender inequality,” in Taking Stock: Data and Evidence on Gender Equality in Digital Access, Skills, and Leadership , pp. 330-338, 2019

2019
[14]

Halevy, C

M. Halevy, C. Harris, A. Bruckman, D. Yang, and A. Howard, ”Mit- igating racial biases in toxic language detection with an equity-based ensemble framework,” inProc. 1st ACM Conf. Equity Access Algorithms, Mechanisms, Optimization, Oct. 2021, pp. 1-11

2021
[15]

K. E. Trinkley, R. An, A. M. Maw, R. E. Glasgow, and R. C. Brownson, ”Leveraging artificial intelligence to advance implementation science: Potential opportunities and cautions,” Implementation Science , vol. 19, no. 1, p. 17, 2024

2024
[16]

Power to the people? Opportunities and challenges for participatory AI,

A. Birhane, W. Isaac, V . Prabhakaran, M. Diaz, M. C. Elish, I. Gabriel, and S. Mohamed, “Power to the people? Opportunities and challenges for participatory AI,” in Proc. 2nd ACM Conf. Equity Access Algorithms, Mechanisms, Optimization, Oct. 2022, pp. 1-8

2022
[17]

A survey of large language models,

W.X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong, and Y . Du, “A survey of large language models,” arXiv preprint arXiv:2303.18223, 2023

2023 arXiv
[18]

Decoding the Diversity: A Review of the Indic AI Research Landscape,

K. J. S., V . Jain, S. Bhaduri, T. Roy, and A. Chadha, “Decoding the Diversity: A Review of the Indic AI Research Landscape,” arXiv preprint arXiv:2406.09559, 2024

2024 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

Cornwall and G

A. Cornwall and G. Pratt, ”The use and abuse of participatory rural appraisal: Reflections from practice,” Agriculture and Human Values , vol. 28, pp. 263-272, 2011

work page 2011

[2] [2]

J. L. Parpart, ”Rethinking participatory empowerment, gender and de- velopment: the PRA approach,” in Rethinking Empowerment, Routledge, 2003, pp. 165-181

work page 2003

[3] [3]

Chambers, ”Participatory rural appraisal (PRA): Analysis of experi- ence,” World Development, vol

R. Chambers, ”Participatory rural appraisal (PRA): Analysis of experi- ence,” World Development, vol. 22, no. 9, pp. 1253-1268, 1994

work page 1994

[4] [4]

The contribution of transformative learning theory to the practice of participatory research and extension: Theoretical reflections,

R. Percy, “The contribution of transformative learning theory to the practice of participatory research and extension: Theoretical reflections,” Agriculture and Human Values , vol. 22, pp. 127-136, 2005

work page 2005

[5] [5]

L. W. Cong, D. Xie, and L. Zhang, ”Knowledge accumulation, privacy, and growth in a data economy,” Management Science, vol. 67, no. 10, pp. 6480-6492, 2021

work page 2021

[6] [6]

Challenges and opportunities beyond structured data in analysis of electronic health records,

M. Tayefi, P. Ngo, T. Chomutare, H. Dalianis, E. Salvi, A. Budrionis, and F. Godtliebsen, “Challenges and opportunities beyond structured data in analysis of electronic health records,” Wiley Interdiscip. Rev. Comput. Stat., vol. 13, no. 6, e1549, 2021

work page 2021

[7] [7]

Exploring AI-driven approaches for unstructured document analysis and future horizons,

S. V . Mahadevkar, S. Patil, K. Kotecha, L. W. Soong, and T. Choudhury, “Exploring AI-driven approaches for unstructured document analysis and future horizons,” Journal of Big Data , vol. 11, no. 1, p. 92, 2024

work page 2024

[8] [8]

Vulnerability mapping: A conceptual framework towards a context-based approach to women’s empower- ment,

C. M. Gressel, T. Rashed, L. A. Maciuika, S. Sheshadri, C. Coley, S. Kongeseri, and R. R. Bhavani, “Vulnerability mapping: A conceptual framework towards a context-based approach to women’s empower- ment,” World Development Perspectives, vol. 20, p. 100245, 2020

work page 2020