Pith. sign in

REVIEW 4 major objections 5 minor 22 references

Unimib Assistant: designing a student-friendly RAG-based chatbot for all their needs

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This pilot study reports that a RAG-based chatbot built with OpenAI's Custom GPTs was perceived by its six student testers as fast, friendly, and reliable when sources worked, but that inaccuracies, omitted document content, and broken…

desk verdict Honest, small-scale pilot of a RAG chatbot for university students; the qualitative findings are plausible but the headline frequency claims outrun the evidence. read the letter →

arxiv 2411.19554 v1 pith:QGZK7GJR submitted 2024-11-29 cs.HC cs.AIcs.CY

Chiara Antico , Stefano Giordano , Cansu Koyuturk , Dimitri Ognibene This is my paper · ORCID
classification cs.HCcs.AIcs.CY
keywords ChatGPTRetrieval-AugmentedGenerationstudent-friendlychatbotuserexperiencehuman-centeredcomputingusabilitytestingquestionansweringhallucination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This pilot study reports the design and first usability test of Unimib Assistant, a question-answering chatbot for students of the University of Milano-Bicocca that combines GPT-4 with retrieval-augmented generation (RAG) through OpenAI's Custom GPTs feature. The authors claim that the six students who evaluated it experienced the assistant as fast, easy to use, well structured, and trustworthy when it supplied working sources, but that occasional inaccurate answers, omissions of information present in the uploaded documents, and unclickable links significantly damaged satisfaction and trust. The work's purpose is to show that such an assistant can be configured without programming, grounded in a user-centered cycle of needfinding interviews, prototype refinement, and qualitative usability testing. A sympathetic reading takes the contribution to be a replicable design pattern plus a candid catalogue of where a RAG chatbot on this platform still fails, rather than a statistical demonstration of accuracy.

What carries the argument

The central object is Unimib Assistant itself, a custom GPT built on the OpenAI Custom GPTs feature, configured manually with a purpose-and-behavior prompt, a set of university-related links, and uploaded PDF and PowerPoint documents. The mechanism that carries the argument is retrieval-augmented generation (RAG): when a student asks a question, GPT-4 generates an answer using both the user's query and the information retrieved from those uploaded sources, which is what lets a generic LLM speak about specific university procedures such as thesis requirements, scholarships, and Erasmus calls. The evaluation mechanism is the user-centered design loop, with needfinding interviews to select content, a qualitative usability test using think-aloud interaction and a semi-structured interview, and a redesign phase, because it is this loop that produces the reported strengths, weaknesses, and fixes. The paper treats this prompt-plus-RAG-plus-feedback configuration as the reusable core that other universities could copy.

What would settle it

A concrete test is to run the redesigned assistant with a larger, department-balanced and international sample on fixed guided tasks, then independently check every answer against the uploaded documents for factual accuracy and completeness and check every returned link's HTTP status; if error rates remain high, the pilot's accuracy and trust problems are systematic, and if they approach zero, those problems were artifacts of the small sample and early prototype.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a GPT-based RAG chatbot can meet students' desire for quick, direct, conversational answers, and that its perceived reliability hinges on the availability of correct, clickable sources. In usability tests with six Unimib students, the chatbot's simple ChatGPT interface, friendly tone, and structured answers were judged positively, and users called it fast, easy to use, and reliable when it provided adequate sources. At the same time, the system did not always return fully accurate information, sometimes ignored relevant content that was present in the uploaded prompt and documents, and occasionally fabricated broken links; the authors state that these issues severely impacted the overall experience and undermined trustworthiness. The authors respond with a redesign that adds more links, merges uploaded documents, and changes the name and logo, and they argue the whole recipe is easy for other institutions to replicate.

Load-bearing premise

The load-bearing premise is that six needfinding interviews, five of them with Psychology master's students, plus six evaluation interviews, can represent the information needs and user experience of the whole Unimib student body, including international students.

Editorial extensions

If this is right

  • Other universities can build a similar assistant without custom software by keeping the base prompt structure and swapping in their own links and documents.
  • Users' trust in such a chatbot depends less on interface aesthetics than on the model's ability to give accurate answers and working source links; a broken link is treated by testers as a reliability failure.
  • Adding more documents and links and merging files can reduce, but evidently does not eliminate, hallucinations and fabricated links, so institutions should still plan for disclaimers, human support, or both.
  • The qualitative results are design guidance rather than generalizable evidence; the paper's own planned next step is a larger, multi-department usability test with guided tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would infer that the decisive design variable is not retrieval quality alone but what the model does when retrieval fails: the fabricated links suggest the assistant prefers to answer over admitting it lacks a source, so a retrieval-confidence threshold or an explicit 'I don't know' fallback would likely be the next effective intervention.
  • I would infer that the 'neglected relevant information' failures could be measured objectively by prompting the same questions repeatedly with and without specific source documents and comparing how often the model cites each uploaded fact; random variations would indicate retrieval instability rather than systematic misunderstanding.
  • A testable extension is to monitor live usage with logging of queries, returned links, and HTTP status codes, so broken-link incidence and omission rates can be tracked over redesigns instead of relying on recollection from six interviews.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents Unimib Assistant, a RAG-based chatbot built with OpenAI's Custom GPTs for University of Milano-Bicocca students. A needfinding phase with six semi-structured interviews informed the first prototype; a qualitative usability test with six other students evaluated it. The authors report that the chatbot was perceived as fast, ease-to-use, and friendly, but that it also produced inaccurate information, neglected relevant information present in uploaded documents, and sometimes generated broken links. A redesign phase addressed these issues by adding links, merging documents, and changing the chatbot's name and logo. The paper closes with a list of limitations and future plans, including a larger usability test and API integration.

Significance. The paper makes a practical contribution by showing how a low-cost, non-programmer-friendly RAG chatbot can be built and iteratively refined for a university context. Its strengths include the user-centered design process, the separation of needfinding and evaluation participant groups, and an honest, explicit list of system limitations. The qualitative findings align with known LLM reliability issues and could be useful guidance for similar pilots. However, the significance is limited by the lack of systematic analysis of the evaluation data: the central claims about the frequency and nature of the chatbot's errors are not backed by quantified evidence or replicable coding procedures. The paper would be more valuable if it either provided such evidence or carefully restricted its claims to anecdotal observations.

major comments (4)
  1. [Abstract, §3.4–§3.5, §3.6] The central negative findings—'not always 100% accurate,' 'often neglect to report relevant information,' and 'sometimes provided nonexistent or broken links'—are based on unstandardized observation of six free-form testing sessions. No query log, error taxonomy, counts, or inter-rater reliability check is reported. Because participants interacted freely with the system, they likely asked different questions, so the frequency words 'often' and 'sometimes' cannot be interpreted as properties of the system rather than of a few idiosyncratic queries. This measurement gap underlies the abstract's claims, the redesign rationale in §3.6, and contribution item 8 in §5.
  2. [§3.4] The statement that the authors 'closely monitored each query to ensure that GPT's responses were accurate and aligned with the information contained in the uploaded documents' presupposes a verification procedure that is not described. There is no operational definition of 'accurate' or 'aligned,' no indication of whether all queries or only a subset were monitored, and no report of how many responses were judged to contain errors. This is load-bearing because the paper's main negative conclusions stem from this monitoring.
  3. [§3.5] The results are presented as a bullet list without any participant-level detail, direct quotes, or counts. It is thus impossible to know whether each finding reflects one participant, a majority, or all six. For example, 'The chatbot was considered fast, easy to use and understand, and reliable when it provided adequate sources' is reported as a single result, but the reader cannot tell how many testers expressed this view. This undercuts the strength of both the positive and negative conclusions.
  4. [§4, Table 1] The limitations section correctly notes the small sample size but does not acknowledge that the measurement of the headline errors is also unstandardized. Table 1 lists 'Usability Testing / Small sample size affecting generalizability,' but omits the more fundamental issue that the frequency claims were not derived from a systematic analysis of recorded sessions or logs. The planned larger study should include a defined error taxonomy and inter-rater validation for the accuracy and source-link checks.
minor comments (5)
  1. [§3.1 and §3.4] The demographic information is reported for each participant group, but the recruitment is described only as 'university colleagues' invited through personal contacts; the paper does not discuss how this may have influenced participants' familiarity with the research team or their motivation.
  2. [§3.3] The prompt and links are shown as screenshots (Figures 1–2), but the text does not provide a verbatim transcript or structured summary. This makes it hard for other researchers to replicate the exact configuration or assess how the prompt may have influenced the observed behavior.
  3. [§3.5] The bullet list mixes researcher observations with participant reports without distinction. For example, 'The information provided by the chatbot was not always 100% accurate' appears to be a researcher judgment, while 'The friendly tone... was considered good' is a participant report; distinguishing these would improve clarity.
  4. [References] There are several formatting and minor textual issues: reference [4] contains 'Learning edidattica online' in the URL, reference [22] has an extra space in the URL, and the citation numbering in §1.1 is not sequential ([12], then [16], then [13], [14], [15], [5], [21], [22]).
  5. [Throughout] The term 'needfinding' is written inconsistently as 'need-finding,' 'need finding,' and 'needfinding'; please standardize.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation is an independent qualitative usability study and the self-citations are related-work context, not load-bearing derivations.

full rationale

This paper contains no mathematical derivation chain, fitted parameters, or predictive model whose outputs could reduce to its inputs. The design process is empirical and sequential: six needfinding interviews inform the choice of prompt content, links, and uploaded documents (Sections 3.1-3.3), and a disjoint group of six other students then evaluates the prototype in free interaction with think-aloud commentary and a semi-structured interview (Section 3.4). Since the evaluators are not the same participants whose needs shaped the system, and no parameter is fitted to the evaluation outcomes, the reported strengths and limitations are independent qualitative observations rather than constructed results. The self-citations in the related-work section (references 11, 16, 21, and 22, which include the last two authors) are used only to support background claims about chatbot prompt behavior, prompt-learning activities, and limitations of LLM chatbots in asking clarifying questions; they do not supply the paper's central empirical findings, so they are not load-bearing. Section 4's acknowledged limitations, such as the small sample size, the inability to view referenced documents, and the lack of guided verification tasks, are methodological validity concerns rather than circularity concerns. The statement that the chatbot 'would often neglect to report relevant information even if present in the materials uploaded and prompt given' is unquantified, but it is an observational claim about system behavior and is not defined in terms of the evaluation outcome nor forced by the input materials. No step identified in the paper reduces by construction to its own inputs, and the central contribution is qualitative feedback rather than a derived prediction. Score 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical constructs, parameters, or entities; the 'Unimib Assistant' is an assembled artifact from existing tools and documents.

assumptions (3)
  • domain assumption The six need-finding interviewees represent the needs of the broader Unimib student population.
    Used to select prompt content and uploaded documents (Sections 3.1-3.3); the sample is 5 Psychology, 1 Law.
  • domain assumption The uploaded university documents and links are accurate, current, and complete.
    The system's answers are only as good as its source files (Section 3.3); no verification of source accuracy is reported.
  • domain assumption Self-reported qualitative feedback from six usability testers is sufficient to assess reliability and UX.
    The main evaluation (Sections 3.4-3.5) is based on think-aloud and semi-structured interviews with no objective accuracy scoring.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unimib Assistant: designing a student-friendly RAG-based chatbot for all their needs." pith.science (2026). https://pith.science/paper/QGZK7GJR

@misc{pith2026241119554,
  author       = {Pith},
  title        = {Pith review of: Unimib Assistant: designing a student-friendly RAG-based chatbot for all their needs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QGZK7GJR}},
  note         = {Machine review of arXiv:2411.19554}
}
read the original abstract

Natural language processing skills of Large Language Models (LLMs) are unprecedented, having wide diffusion and application in different tasks. This pilot study focuses on specializing ChatGPT behavior through a Retrieval-Augmented Generation (RAG) system using the OpenAI custom GPTs feature. The purpose of our chatbot, called Unimib Assistant, is to provide information and solutions to the specific needs of University of Milano-Bicocca (Unimib) students through a question-answering approach. We provided the system with a prompt highlighting its specific purpose and behavior, as well as university-related documents and links obtained from an initial need-finding phase, interviewing six students. After a preliminary customization phase, a qualitative usability test was conducted with six other students to identify the strengths and weaknesses of the chatbot, with the goal of improving it in a subsequent redesign phase. While the chatbot was appreciated for its user-friendly experience, perceived general reliability, well-structured responses, and conversational tone, several significant technical and functional limitations emerged. In particular, the satisfaction and overall experience of the users was impaired by the system's inability to always provide fully accurate information. Moreover, it would often neglect to report relevant information even if present in the materials uploaded and prompt given. Furthermore, it sometimes generated unclickable links, undermining its trustworthiness, since providing the source of information was an important aspect for our users. Further in-depth studies and feedback from other users as well as implementation iterations are planned to refine our Unimib Assistant.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 17 canonical work pages

  1. [1]

    S., & Sun, L

    Zhou, C., Li, Q., Li, C., Wang, Y., Liu, Y., Wang, G., Zhang, K., Cheng, J., Yan, Q., He, L., Peng, H., Li, J., Jia, W., Liu, Z., Xie, P., Xiong, C., Pei, J., Yu, P. S., & Sun, L. (2023, February 18). A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT. arXiv:2302.09419 (Cornell University). https://doi.org/10.48550/arxi...

    arXiv doi:10.48550/arxiv.2302.09419 2023

  2. [2]

    (2024, May)

    Similarweb. (2024, May). chat.openai.com Traffic & Engagement Analysis. https://www.similarweb.com/website/chat.openai.com/#overview

    work page 2024

  3. [3]

    S., & Srinivasan, K

    Casheekar, A., Lahiri, A., Rath, K., Prabhakar, K. S., & Srinivasan, K. (2024). A contemporary review on chatbots, AI-powered virtual conversational agents, ChatGPT: Applications, open challenges and future research directions. Computer Science Review, 52, 100632

    work page 2024

  4. [4]

    & Fung, P

    Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., ... & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1-38

    work page 2023

  5. [5]

    Mansurova, A., Mansurova, A., & Nugumanova, A. (2024). QA -RAG: Exploring LLM Reliance on External Knowledge. Big Data and Cognitive Computing, 8(9), 115

    work page 2024

  6. [6]

    & Cui, B

    Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., ... & Cui, B. (2024). Retrieval -augmented generation for ai-generated content: A survey. arXiv preprint arXiv:2402.19473v2 (Cornell University). https://arxiv.org/html/2402.19473v2

    arXiv 2024

  7. [7]

    Yu, H., Gan, A., Zhang, K., Tong, S., Liu, Q., & Liu, Z. (2024). Evaluation of Retrieval-Augmented Generation: A Survey. arXiv:2405.07437 (Cornell University). https://doi.org/10.48550/arXiv.2405.07437

    arXiv doi:10.48550/arxiv.2405.07437 2024

  8. [8]

    (2024, March)

    Chen, J., Lin, H., Han, X., & Sun, L. (2024, March). Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, 38(16), 17754-17762

    work page 2024

Show all 22 references
  1. [9]

    Wei, J., Kim, S., Jung, H., & Kim, Y. H. (2024). Leveraging large language models to power chatbots for collecting user self-reported data. Proceedings of the ACM on Human-Computer Interaction, 8(CSCW1), 1-35

    2024

  2. [10]

    D., Wong, R

    Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023, April 19). Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3544548.3581388

    2023

  3. [11]

    & Ognibene, D

    Koyuturk, C., Yavari, M., Theophilou, E., Bursic, S., Donabauer, G., Telari, A., ... & Ognibene, D. (2023). Developing Effective Educational Chatbots with ChatGPT prompts: Insights from Preliminary Tests in a Case Study on Social Media Literacy. In 31st International Conferenc...

    2023

  4. [12]

    Gravel, J., D’Amours-Gravel, M., & Osmanlliu, E. (2023). Learning to fake it: limited responses and fabricated references provided by ChatGPT for medical questions. Mayo Clinic Proceedings: Digital Health, 1(3), 226-234

    2023

  5. [13]

    Baek, J., Chandrasekaran, N., Cucerzan, S., Herring, A., & Jauhar, S. K. (2024, May). Knowledge-augmented large language models for personalized contextual query suggestions. In Proceedings of the ACM on Web Conference 2024 (pp. 3355-3366)

    2024

  6. [14]

    S., Hippalgaonkar, K., & Ng, L

    Thway, M., Recatala -Gomez, J., Lim, F. S., Hippalgaonkar, K., & Ng, L. W. (2024). Battling Botpoop using GenAI for higher education: a study of a Retrieval Augmented Generation Chatbot's impact on learning. arXiv:2406.07796 (Cornell University). https://doi.org/10.48550/arXiv...

    2024 doi arXiv

  7. [15]

    Jacobs, S., & Jaschke, S. (2024). Leveraging Lecture Content for Improved Feedback: Explorations with GPT-4 and Retrieval Augmented Generation. arXiv:2405.06681 (Cornell University). https://doi.org/10.48550/arXiv.2405.06681

    2024 doi arXiv

  8. [16]

    & Ognibene, D

    Theophilou, E., Koyutürk, C., Yavari, M., Bursic, S., Donabauer, G., Telari, A., ... & Ognibene, D. (2023, November). Learning to prompt in the classroom to understand AI limits: a pilot study. In International Conference of the Italian Association for Art ificial Intelligence...

    2023

  9. [17]

    (September 2024)

    OpenAI. (September 2024). Retrieval Augmented Generation (RAG) and Semantic Search for GPTs. OpenAI Help Center. https://help.openai.com/en/articles/8868588-retrieval- augmented-generation-rag-and-semantic-search-for-gpts (accessed 22 September 2024)

    2024

  10. [18]

    (November 2023)

    OpenAI. (November 2023). Introducing GPTs. OpenAI. https://openai.com/index/introducing-gpts/ (accessed 22 September 2024)

    2023

  11. [19]

    Jain, M., Kumar, P., Kota, R., & Patel, S. N. (2018, June). Evaluating and informing the design of chatbots. In Proceedings of the 2018 Designing Interactive Systems Conference (pp. 895- 906)

    2018

  12. [20]

    Google I/O ‘24: Introducing Gemini model capabilities for Places API, 3D Maps in the Maps JavaScript API, and open-source React components

  13. [21]

    (2024, September)

    Patania, S., Masiero, E., Brini, L., Piskovskyi, V., Ognibene, D., Donabauer, G., & Kruschwitz, U. (2024, September). Large Language Models as an active Bayesian filter: information acquisition and integration. In Proceedings of the 28th Workshop on the Semantics and Pragmatic...

    2024

  14. [22]

    Martinenghi, A., Koyuturk, C., Amenta, S., Ruskov, M., Donabauer, G., Kru schwitz, U., Ognibene, D. (2024). VON NEUMIDAS: Enhanced Annotation Schema for Human -LLM Interactions Combining MIDAS with Von Neumann Inspired Semantics. Presented at the Proceedings of the 28th Worksh...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.