Pith. sign in

REVIEW 4 major objections 5 minor 41 references

Lessons for GenAI Literacy From a Field Study of Human-GenAI Augmentation in the Workplace

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GenAI use in the workplace varies so widely by role that AI literacy education must be taught in differentiated levels rather than as a single uniform competency.

desk verdict A transparent, timely comparative field study of GenAI use in Indian workplaces whose central three-way contrast rests on one manager's secondhand account of his team, so the evidence is thinner than the conclusions imply. read the letter →

arxiv 2502.00567 v1 pith:VWPYD5OK submitted 2025-02-01 cs.CY cs.AI

Aditya Johri , Johannes Schleiss , Nupoor Ranade This is my paper · ORCID
classification cs.CYcs.AI
keywords generativeartificialintelligenceAIliteracyworkplacestudieshumanaugmentationengineeringeducationcomputingqualitativefieldstudyworkforcedevelopment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a field study of ten professionals across three Indian firms doing product development, software migration, and digital content creation, asking how generative AI (GenAI) augments their work. It finds a wide spectrum: some teams fine-tune models and integrate LLMs into client products, while others only use off-the-shelf tools to polish text and brainstorm. The paper argues that workers' GenAI knowledge and literacy vary just as widely, so AI literacy education should not be a single uniform competency but a set of differentiated levels matched to role and technical background. The result matters because it gives curriculum and faculty-development planners a concrete reason to stop treating 'AI literacy' as one skill and start designing tiered instruction.

What carries the argument

The argument is carried by a comparative case-study design that pairs a human-augmentation perspective (specifically augmented cognition) with a six-construct AI literacy framework -- Recognize, Know and Understand, Use and Apply, Evaluate, Create, Navigate Ethically. The framework does the load-bearing work of turning interview evidence into comparable literacy ratings across the three teams, and those ratings are what support the claim that literacy needs differ by role.

What would settle it

A large-scale survey measuring the six AI literacy constructs across hundreds of product developers, software engineers, and content creators; if within-role variation in literacy turns out to be as large as between-role variation, the paper's core claim that literacy needs differ by work function would not hold. Alternatively, a controlled curriculum experiment showing that a single uniform GenAI training produces equivalent work outcomes across all three roles would undercut the tiering recommendation.

Watch

Extended reading notes

Core claim

Using a human-augmentation lens focused on augmented cognition, the study compares how GenAI changes work across three functions and rates each team against six AI literacy constructs. The central discovery is that GenAI augments work along a spectrum that tracks the user's technical depth: product developers with machine-learning expertise build and customize GenAI into solutions, software engineers with mixed expertise use IDE-integrated copilots that let junior coders write unfamiliar languages, and content creators with low technical background use GenAI as a brainstorming and editing aid. Because the tool's role and the knowledge required to use it safely differ by function, the paper concludes that different levels of GenAI understanding need to be integrated into courses, with deeper machine-learning and algorithm training for builders, tool-integration competency for practitioners, and a lighter but still critical awareness of limitations and outcome interpretation for end users.

Load-bearing premise

The conclusions rest on interviews with about ten professionals at three companies--including just one person from the software-engineering site--being representative enough to support generalizable claims about how GenAI augments work and what students should learn.

Editorial extensions

If this is right

  • Curriculum designers should replace a single AI literacy requirement with tiered instruction matched to career paths, from model-building depth to tool-use awareness.
  • Students preparing for technical development roles still need machine learning and algorithms foundations, not just prompt skills.
  • Students entering software practice need competency with IDE-integrated GenAI and the judgment to verify generated code, rather than full model-building expertise.
  • Non-STEM students need enough understanding of how GenAI works to recognize its limitations and interpret its outputs, even if they never build or customize models.
  • Faculty development should treat GenAI both as content to teach and as a teaching practice, so instructors can model its use and redesign assessments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's three-case spectrum suggests a testable hypothesis: a larger survey measuring the same six literacy constructs across job families would find that within-role variance is smaller than between-role variance, which would strengthen the case for role-based tiering.
  • The findings imply that generic 'AI literacy' certifications may mislead employers, since a single score cannot capture the qualitatively different knowledge builders, integrators, and end users need.
  • A natural extension is to study whether the observed tiering holds for non-technical industries outside India, or whether national education systems and organizational maturity shift the boundaries between the tiers.
  • The authors' concern that novices could become dependent on GenAI without architectural understanding points to a longitudinal study of whether junior developers who start with copilots ever develop the mental models of senior architects.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper reports a qualitative field study of how generative AI (GenAI) is used by professionals in three firms in India, corresponding to three work functions: product development (Project Concept), software engineering (Project Code), and digital content creation (Project Content). Based on approximately six hours of interviews with ten participants, the authors describe GenAI use, learning practices, and implications for future workforce development. They apply an AI literacy framework from Almatrafi et al. to rate each project team on six constructs, concluding that GenAI use and knowledge vary widely across functions and arguing that AI literacy education should be differentiated by role and technical background.

Significance. If the findings held, the paper would contribute a useful early comparison of GenAI integration across different knowledge-work functions and would support the design of differentiated AI literacy curricula. The study is honest about its small scale and openly acknowledges the single-informant Code case in the Limitations section. Its use of an established AI literacy framework and its attention to human augmentation as an analytical lens are strengths. However, the evidentiary base is thin: ten participants across three sites, no direct quotes, and no detailed analytic tracing from transcripts to findings. The central comparative claim therefore remains suggestive rather than established, making the paper more suitable as a preliminary qualitative report than as a basis for strong curriculum prescriptions.

major comments (4)
  1. [§III-B, §IV-B, Table II] The Code case rests on a single participant, P6, a senior manager, yet the findings describe the junior developers' knowledge and learning as empirical fact: 'The two junior people knew the basics of programming but not necessarily the language they were developing the code in' and they 'learned about how to use the system largely through documentation.' These are secondhand reports, not direct participant accounts, and they are used to assign Project Code's literacy levels in Table II. This creates an asymmetry with the Concept and Content cases, where multiple participants were interviewed, and it makes the cross-case variation in literacy ratings potentially an artifact of who was interviewed. The authors should either present the Code findings explicitly as the manager's characterization of his team or, if the claim requires direct evidence, collect data from the junior developers.
  2. [§V, Table II] The literacy ratings in Table II are presented as conclusions but the methodology for assigning them is under-specified. The text says only that the authors 'rated each team on their level of GenAI literacy' and that the rating is 'relative to each other, i.e., since we did not use any objective measure we used informants’ responses to rate them in comparison to other teams in the sample.' There is no description of who performed the ratings, whether ratings were made independently, how the six constructs were operationalized, or how individual responses were aggregated to team-level judgments. Because the paper's central claim depends on comparing these ratings across the three functions, the absence of a transparent rating procedure is a load-bearing gap.
  3. [§III-C, §IV] The data-analysis section describes an iterative, interpretive process but does not provide enough detail to assess trustworthiness: there is no codebook, no description of initial coding categories, no information on how many authors coded the transcripts or how disagreements were resolved, and no audit trail. In addition, the findings contain no direct participant quotations, making it difficult for the reader to judge the connection between the raw data and the reported themes and literacy levels. The authors should include representative quotes and a more detailed analytic procedure, especially since the sample is small and the paper asks readers to accept conclusions about knowledge and learning across three distinct functions.
  4. [§V, §VI] The conclusion asserts 'a wide variation in both how GenAI is used and the knowledge workers in different, but related industries, possess about GenAI' and the discussion draws implications for differentiated student training. These generalizations exceed what a set of ten participants in three Indian firms can support, especially given the single-informant Code case. The Limitations section acknowledges the small sample, but the Discussion and Conclusion do not consistently hedge their claims. The authors should reframe the conclusions as provisional and hypothesis-generating, and should explicitly state the limits of transferability beyond the studied contexts.
minor comments (5)
  1. [§II-B] The sentence 'In software development, the integration of GenAI into development environments has led increased it’s use GenAI' is garbled and should be rewritten, for example: 'has led to increased use of GenAI.'
  2. [§III-B] The phrase 'This purpose sampling was done' should be 'This purposive sampling was done.'
  3. [§V after Table II] The third paragraph describing literacy levels seems to be about Project Content, but the text says 'Project Concept participants had a reasonable literacy level...' This appears to be a mislabeling that confuses the reader; it should say 'Project Content.'
  4. [§V] The sentence 'They were adapt at using them' should read 'They were adept at using them.'
  5. [Figures 1 and 2] Figures 1 and 2 are referenced but not described in the text; the authors should ensure each figure is understandable on its own or add appropriate captions and in-text explanations.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the variation finding is an inductive field-study observation, not a consequence of the authors' prior framework.

full rationale

The paper's central claims—wide variation in GenAI use and knowledge, and the need for differentiated literacy instruction—are descriptive interpretations of semi-structured interview data from three organizations. The paper contains no equations, fitted parameters, or formal derivation chain. The only self-citation that could be considered load-bearing is Almatrafi et al. [8], which supplies the six AI-literacy constructs used for Table II. That taxonomy is an interpretive lens, not the empirical result: the authors state the ratings are 'relative to each other' and grounded in informants' responses, and the variation finding rests on the interview content (e.g., fine-tuning at Concept, IDE-integrated Copilot at Code, off-the-shelf content tools at Content). Even if the framework were removed, the observed differences in technical practice and learning resources would stand. The Code-case limitation—data collected from a single manager, with secondhand descriptions of junior developers—is an evidentiary weakness, not circularity. Likewise, purposive sampling across a technical spectrum makes 'wide variation' unsurprising, but the finding is still empirical about the studied firms rather than a constructed tautology. The score reflects one minor self-citation that is not load-bearing to the core conclusion.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. It depends on standard qualitative assumptions about interview reliability, purposive sampling, and the transferability of an AI literacy framework to workplace contexts.

assumptions (3)
  • domain assumption Interview self-reports are a reliable window into actual workplace GenAI practices.
    The study relies entirely on interviews and does not include participant observation or digital traces, as acknowledged in the Limitations section.
  • domain assumption Thematic saturation was reached with about six hours of interviews across three sites.
    In Section III-C the authors state they 'believe we were able to reach enough saturation to present important findings' despite the small sample.
  • domain assumption The Almatrafi et al. AI literacy constructs (Recognize, Know and Understand, Use and Apply, Evaluate, Create, Navigate Ethically) are appropriate for rating workplace GenAI competence.
    Section II-C introduces the framework and Section V uses it to rate each team, but the paper provides no validation of the framework in a workplace setting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lessons for GenAI Literacy From a Field Study of Human-GenAI Augmentation in the Workplace." pith.science (2026). https://pith.science/paper/VWPYD5OK

@misc{pith2026250200567,
  author       = {Pith},
  title        = {Pith review of: Lessons for GenAI Literacy From a Field Study of Human-GenAI Augmentation in the Workplace},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VWPYD5OK}},
  note         = {Machine review of arXiv:2502.00567}
}
read the original abstract

Generative artificial intelligence (GenAI) is increasingly becoming a part of work practices across the technology industry and being used across a range of industries. This has necessitated the need to better understand how GenAI is being used by professionals in the field so that we can better prepare students for the workforce. An improved understanding of the use of GenAI in practice can help provide guidance on the design of GenAI literacy efforts including how to integrate it within courses and curriculum, what aspects of GenAI to teach, and even how to teach it. This paper presents a field study that compares the use of GenAI across three different functions - product development, software engineering, and digital content creation - to identify how GenAI is currently being used in the industry. This study takes a human augmentation approach with a focus on human cognition and addresses three research questions: how is GenAI augmenting work practices; what knowledge is important and how are workers learning; and what are the implications for training the future workforce. Findings show a wide variance in the use of GenAI and in the level of computing knowledge of users. In some industries GenAI is being used in a highly technical manner with deployment of fine-tuned models across domains. Whereas in others, only off-the-shelf applications are being used for generating content. This means that the need for what to know about GenAI varies, and so does the background knowledge needed to utilize it. For the purposes of teaching and learning, our findings indicated that different levels of GenAI understanding needs to be integrated into courses. From a faculty perspective, the work has implications for training faculty so that they are aware of the advances and how students are possibly, as early adopters, already using GenAI to augment their learning practices.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 41 canonical work pages

  1. [1]

    Engineers’ perspectives on the use of generative artificial intelligence tools in the workplace,

    J. Joskowicz and D. Slomovitz, “Engineers’ perspectives on the use of generative artificial intelligence tools in the workplace,” IEEE Engineer- ing Management Review, 2023

    work page 2023

  2. [2]

    2024 work trend index annual report,

    Microsoft, “2024 work trend index annual report,” 2024. [Online]. Available: https://www.microsoft.com/en-us/worklab/work- trend-index/ai-at-work-is-here-now-comes-the-hard-part

    work page 2024

  3. [3]

    Workplace learning ecology of software engineers and implications for teaching and learning,

    A. Johri, A. H. Collier, B. K. Jesiek, R. Korte, and S. C. Brozina, “Workplace learning ecology of software engineers and implications for teaching and learning,” in Proceedings of CSEE&T, 2024, pp. 1–5

    work page 2024

  4. [4]

    Fostering genai literacy in higher education for future workplace preparedness: A mixed -methods study,

    C. Fleischmann, M. K. Logemann, J. Heidewald, P. Cardon, J. Aritz, and S. Swartz, “Fostering genai literacy in higher education for future workplace preparedness: A mixed -methods study,” in ECIS 2024 Pro - ceedings, 2024

    work page 2024

  5. [5]

    Generative ai and the future of higher education: a threat to academic integrity or reforma - tion? evidence from multicultural perspectives,

    A. Yusuf, N. Pervin, and M. Roma´n -Gonza´lez, “Generative ai and the future of higher education: a threat to academic integrity or reforma - tion? evidence from multicultural perspectives,” International Journal of Educational Technology in Higher Education , vol. 21, no. 1, p. 21, 2024

    work page 2024

  6. [6]

    Using rhetorical strategies to design prompts: a human-in-the-loop approach to make ai useful,

    N. Ranade, M. Saravia, and A. Johri, “Using rhetorical strategies to design prompts: a human-in-the-loop approach to make ai useful,” AI & SOCIETY, pp. 1–22, 2024

    work page 2024

  7. [7]

    Bridging the gulf of envisioning: Cognitive challenges in prompt based interactions with llms,

    H. Subramonyam, R. Pea, C. Pondoc, M. Agrawala, and C. Seifert, “Bridging the gulf of envisioning: Cognitive challenges in prompt based interactions with llms,” in Proceedings of CHI, 2024, pp. 1–19

    work page 2024

  8. [8]

    A systematic review of ai literacy conceptualization, constructs, and implementation and assessment efforts (2019-2023),

    O. Almatrafi, A. Johri, and H. Lee, “A systematic review of ai literacy conceptualization, constructs, and implementation and assessment efforts (2019-2023),” Computers and Education Open, p. 100173, 2024

    work page 2019

Show all 41 references
  1. [9]

    Developing a holistic ai literacy assessment matrix–bridging generic, domain -specific, and ethical competencies,

    N. Knoth, M. Decker, M. C. Laupichler, M. Pinski, N. Buchholtz, K. Bata, and B. Schultz, “Developing a holistic ai literacy assessment matrix–bridging generic, domain -specific, and ethical competencies,” Computers and Education Open, vol. 6, p. 100177, 2024

    2024

  2. [10]

    Generative ai at work,

    E. Brynjolfsson, D. Li, and L. R. Raymond, “Generative ai at work,” National Bureau of Economic Research, Tech. Rep., 2023

    2023

  3. [11]

    Human augmentation: Past, present and future,

    R. Raisamo, I. Rakkolainen, P. Majaranta, K. Salminen, J. Rantala, and A. Farooq, “Human augmentation: Past, present and future,” Interna- tional Journal of Human-Computer Studies, vol. 131, pp. 131–143, 2019

    2019

  4. [12]

    Intelligence augmentation: Human factors in ai and future of work,

    S. Paul, L. Yuan, H. K. Jain, L. P. Robert Jr, J. Spohrer, and H. Lifshitz- Assaf, “Intelligence augmentation: Human factors in ai and future of work,” AIS Transactions on Human-Computer Interaction, vol. 14, no. 3, pp. 426–445, 2022

    2022

  5. [13]

    Human augmentation, not replacement: A research agenda for ai and robotics in the industry,

    S. De´gallier-Rochat, M. Kurpicz-Briki, N. Endrissat, and O. Yatsenko, “Human augmentation, not replacement: A research agenda for ai and robotics in the industry,” Frontiers in Robotics and AI, vol. 9, p. 997386, 2022

    2022

  6. [14]

    Artificial intelligence (ai) or intelligence augmentation (ia): what is the future?

    H. Hassani, E. S. Silva, S. Unger, M. TajMazinani, and S. Mac Feely, “Artificial intelligence (ai) or intelligence augmentation (ia): what is the future?” AI, vol. 1, no. 2, p. 8, 2020

    2020

  7. [15]

    Intelligence augmentation: upskilling humans to complement ai,

    C. Dede, A. Etemadi, and T. Forshaw, “Intelligence augmentation: upskilling humans to complement ai,” The Next Level Lab at the Harvard Graduate School of Education. President and Fellows of Harvard College: Cambridge, MA, 2021

    2021

  8. [16]

    Intelligence augmentation: Towards building human -machine symbiotic relationship,

    L. Zhou, S. Paul, H. Demirkan, L. Yuan, J. Spohrer, M. Zhou, and J. Basu, “Intelligence augmentation: Towards building human -machine symbiotic relationship,” AIS Transactions on Human-Computer Interac- tion, vol. 13, no. 2, pp. 243–264, 2021

    2021

  9. [17]

    Reconstructing engineering from practice,

    J. Trevelyan, “Reconstructing engineering from practice,” Engineering Studies, vol. 2, no. 3, pp. 175–195, 2010

    2010

  10. [18]

    Transitioning to engineering practice,

    ——, “Transitioning to engineering practice,” pp. 821–837, 2019

    2019

  11. [19]

    Professional engineering work,

    R. Stevens, A. Johri, and K. O’connor, “Professional engineering work,” Cambridge handbook of engineering education research , pp. 119 –137, 2014

    2014

  12. [20]

    What are workplace studies for?

    L. Plowman, Y. Rogers, and M. Ramage, “What are workplace studies for?” in Proceedings of ECSCW: 10 –14 September, 1995, Stockholm, Sweden. Springer, 1995, pp. 309–324

    1995

  13. [21]

    Hutchins, Cognition in the Wild

    E. Hutchins, Cognition in the Wild. MIT press, 1995

    1995

  14. [22]

    Lifelong and lifewide learning for the perpetual development of expertise in engineering,

    A. Johri, “Lifelong and lifewide learning for the perpetual development of expertise in engineering,” European Journal of Engineering Educa - tion, vol. 47, no. 1, pp. 70–84, 2022

    2022

  15. [23]

    Education in the era of generative artificial intelligence (ai): Understanding the potential benefits of chatgpt in promoting teaching and learning,

    D. Baidoo -Anu and L. O. Ansah, “Education in the era of generative artificial intelligence (ai): Understanding the potential benefits of chatgpt in promoting teaching and learning,” Journal of AI , vol. 7, no. 1, pp. 52–62, 2023

    2023

  16. [24]

    Introduction: Composing with generative ai,

    N. Ranade and D. Eyman, “Introduction: Composing with generative ai,” p. 102834, 2024

    2024

  17. [25]

    How far can we go with synthetic user experience research?

    J. Li, “How far can we go with synthetic user experience research?” Interactions, vol. 31, no. 3, p. 26 –29, may 2024. [Online]. Available: https://doi.org/10.1145/3653682

    2024 doi

  18. [26]

    Bridging the skills gap of workers in industry 4.0 by human performance augmentation tools: Challenges and roadmap,

    E. Ras, F. Wild, C. Stahl, and A. Baudet, “Bridging the skills gap of workers in industry 4.0 by human performance augmentation tools: Challenges and roadmap,” in Proceedings of PETRA, 2017, pp. 428–432

    2017

  19. [27]

    The future of work: Augmentation or stunting?

    M. Furendal and K. Jebari, “The future of work: Augmentation or stunting?” Philosophy & Technology, vol. 36, no. 2, p. 36, 2023

    2023

  20. [28]

    The metacognitive demands and opportunities of generative ai,

    L. Tankelevitch, V. Kewenig, A. Simkute, A. E. Scott, A. Sarkar, A. Sellen, and S. Rintel, “The metacognitive demands and opportunities of generative ai,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, 2024, pp. 1–24

    2024

  21. [29]

    Generative artificial in - telligence in higher education: Evidence from an analysis of institutional policies and guidelines,

    N. McDonald, A. Johri, A. Ali, and A. Hingle, “Generative artificial in - telligence in higher education: Evidence from an analysis of institutional policies and guidelines,” arXiv preprint arXiv:2402.01659, 2024

    2024 arXiv

  22. [30]

    P. Luff, J. Hindmarsh, and C. Heath, Workplace studies: Recovering work practice and informing system design. Cambridge university press, 2000

    2000

  23. [31]

    J. A. Maxwell et al. , Designing a qualitative study . The SAGE handbook of applied social research methods, 2008, vol. 2

    2008

  24. [32]

    Conducting interpretive research in engineering education using qualitative and ethnographic methods,

    A. Johri, “Conducting interpretive research in engineering education using qualitative and ethnographic methods,” Cambridge handbook of engineering education research, pp. 551–570, 2014

    2014

  25. [33]

    Using qualitative, interpretive, and case methods to study information technology,

    M. L. Markus and A. S. Lee, “Using qualitative, interpretive, and case methods to study information technology,” MIS quarterly , pp. 35 –38, 1999

    1999

  26. [34]

    J. P. Spradley, The ethnographic interview. Waveland Press, 2016

    2016

  27. [35]

    A. L. Strauss, Qualitative analysis for social scientists . Cambridge university press, 1987

    1987

  28. [36]

    Ironies of automation,

    L. Bainbridge, “Ironies of automation,” in Analysis, design and evalua - tion of man–machine systems. Elsevier, 1983, pp. 129–135

    1983

  29. [37]

    Ai course design planning framework: Developing domain -specific ai education courses,

    J. Schleiss, M. C. Laupichler, T. Raupach, and S. Stober, “Ai course design planning framework: Developing domain -specific ai education courses,” Education Sciences, vol. 13, no. 9, p. 954, 2023

    2023

  30. [38]

    Integrating ai education in disci - plinary engineering fields: Towards a system and change perspective,

    J. Schleiss, A. Johri, and S. Stober, “Integrating ai education in disci - plinary engineering fields: Towards a system and change perspective,” Proceedings of SEFI, 2024

    2024

  31. [39]

    A roles -based competency framework for in - tegrating artificial intelligence (ai) in engineering courses,

    J. Schleiss and A. Johri, “A roles -based competency framework for in - tegrating artificial intelligence (ai) in engineering courses,” Proceedings of SEFI, 2024

    2024

  32. [40]

    Augmented sociomateriality: implications of artificial intel - ligence for the field of learning technology,

    A. Johri, “Augmented sociomateriality: implications of artificial intel - ligence for the field of learning technology,” Research in Learning Technology, vol. 30, 2022

    2022

  33. [41]

    Misconceptions, pragmatism, and value tensions: Evaluating students’ understanding and perception of generative ai for education,

    A. Johri, A. Hingle, and J. Schleiss, “Misconceptions, pragmatism, and value tensions: Evaluating students’ understanding and perception of generative ai for education,” IEEE Frontiers in Education, 2024

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.