{"id":"9716b0e7-0fb4-44c3-bfd6-438ffa2ac706","arxiv_id":"2505.23239","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A framework using LLM agents to simulate developers at four experience levels and automatically evaluate open source platform usability from generated code.","lead":"This paper introduces OSS-UAgent, a framework that uses large language models to simulate developers of different skill levels and then scores how easy an open source software platform is to use. It is a demo paper for a PVLDB submission, presenting the architecture and a graphical interface rather than measured results.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No quantitative validation links OSS-UAgent's automated scores to human usability judgments; the compliance metric and evaluator tuning make the claimed effectiveness an unsupported, self-referential assertion.","rationale":"I read the paper as a system description with a demonstration, not a validated evaluation method. The strongest claim, repeated in the abstract and introduction, is that OSS-UAgent automates usability evaluation effectively, reducing cost and enabling large-scale assessment without extensive human participation. For that claim to hold, the framework's scores must be shown to correspond to real usability as experienced by human developers. The paper contains no such evidence: there is no experimental section, no quantitative evaluation, no human baseline, and no statistical analysis. The reader's weakest assumption identifies the same gap, and I agree that it is the load-bearing weakness. The self-referential nature of the evaluation makes the gap more severe: the Evaluator is tuned iteratively until it produces 'satisfactory' results on test code (Section 2.4), and compliance is measured as deviation from a standard reference code provided by the same automated pipeline. This design cannot, by itself, establish that higher scores mean better usability. I am not raising an outside-consensus concern; this is an internally verifiable gap between the central empirical claim and the presented evidence. A relatively small human-subject validation study would resolve the concern, which is why the verdict remains REJECT unless such evidence is added. The reader's assessment is fair and the proposed test is the natural next step.","tokens_in":5167,"tokens_out":2523,"duration_ms":29299,"concrete_test":"Run OSS-UAgent on a fixed set of 10 OSS platforms in the graph-analytics domain. For each platform, recruit human participants whose experience levels are measured by a screening questionnaire, have them perform the same benchmark tasks, and record task success, completion time, and standardized usability ratings (e.g., SUS or PSSUQ). Compute the correlation and agreement between OSS-UAgent's per-level compliance/correctness/readability scores and the human-derived usability ratings (e.g., Spearman rho with confidence intervals, or ICC). If the correlation is not significant or does not exceed a pre-registered threshold (e.g., rho > 0.5), the claim that OSS-UAgent measures usability is unsupported. Also check whether the four simulated experience levels produce monotonically increasing scores and whether that ordering matches human judgment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that OSS-UAgent 'significantly reduces evaluation costs and enhances scalability' by replacing human evaluators, but the paper provides no empirical evidence for this. Section 2.4 defines compliance as closeness to a standard reference implementation and describes an iterative tuning loop: 'we provide feedback based on the output results to optimize the Evaluator's instructions. We iterate this process until it can produce stable and satisfactory evaluation results.' This means the evaluator is adjusted until its outputs look good on test code, so the resulting scores may reflect prompt/LLM self-consistency rather than developer-facing usability. Section 3 then shows only a GUI demonstration on graph analytics platforms, with no quantitative results, no human participants, no inter-rater reliability, and no comparison to established usability instruments such as SUS, task success rates, or think-aloud studies. The framework's validity depends on two untested assumptions: (a) LLM-generated role-specific code reproduces the behavior of real developers at Junior/Intermediate/Senior/Expert levels, and (b) an LLM evaluator's compliance/correctness/readability scores are valid usability measures without human calibration. Neither assumption is supported anywhere in the manuscript. Without external validation against human performance, the abstract's effectiveness claim is ungrounded.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OSS-UAgent, a framework that uses LLM-powered agents to evaluate the usability of open source software. The framework builds a platform-specific knowledge base from GitHub data, simulates developers at four experience levels (Junior, Intermediate, Senior, Expert) via role-specific prompts, generates code through a Code Generator agent, and scores the generated code with an Evaluator agent on three metrics: compliance, correctness, and readability. A GUI demonstration on graph analytics platforms is presented, and the abstract claims that the framework significantly reduces evaluation costs and enhances scalability by automating the evaluation process. The manuscript, however, contains no quantitative results, no comparison with human evaluators, no statistical analysis, and no reliability assessment; the only evidence is a workflow description and a visual interface.","tokens_in":5363,"tokens_out":3417,"duration_ms":36424,"significance":"If the framework's scores were shown to correspond to actual developer-perceived usability, OSS-UAgent could offer a low-cost, scalable complement to human-based usability evaluation. The open-source release of the code and the modular design are commendable. However, the central claim of effectiveness is unsupported by the evidence in the manuscript. The lack of any validation against human judgments, combined with the circular evaluator-tuning procedure, means the paper currently provides no credible evidence that the framework measures usability rather than LLM self-consistency. The idea is worth pursuing, but the present manuscript does not establish it.","major_comments":[{"comment":"The abstract claims that OSS-UAgent 'significantly reduces evaluation costs and enhances scalability' and the demonstration 'highlights its effectiveness.' Section 3 reports only a GUI workflow on graph analytics platforms, with no quantitative measurements, no comparison against human evaluators, no established usability instruments (e.g., SUS, task success rates, think-aloud), and no inter-rater reliability or statistical tests. Without any empirical evidence linking the automated scores to human usability judgments, the central effectiveness claim is ungrounded.","section":"Abstract and Section 3"},{"comment":"The evaluator is tuned in a circular manner. The text states: 'we provide feedback based on the output results to optimize the Evaluator's instructions. We iterate this process until it can produce stable and satisfactory evaluation results.' This means the scoring criteria are adjusted until they produce outputs the authors deem satisfactory on test inputs, so the resulting scores may reflect the LLM's self-consistency or the authors' expectations rather than genuine developer-facing usability. The compliance metric is defined relative to a standard reference implementation, yet no independent ground truth or human-calibrated benchmark is used. This circularity undermines the validity of all reported scores.","section":"Section 2.4 and Figure 2 (item 9)"},{"comment":"The framework's validity depends on the untested assumption that LLM-generated role-specific prompts reproduce the behavior of real developers at Junior, Intermediate, Senior, and Expert levels. No evidence is provided that the code generated at each level matches the code quality, error patterns, or API-usage difficulties of actual developers at those experience levels. A concrete test is needed: compare the LLM-generated code at each level with code written by human developers of corresponding experience on the same tasks. Without such a comparison, the simulation is an unvalidated proxy and cannot ground usability claims.","section":"Section 2.2"}],"minor_comments":[{"comment":"In the Level 4 description, there is a typo: 'Prompts contain comprehensive details and and they expect high-quality' duplicates 'and.'","section":"Section 2.2"},{"comment":"The figures are difficult to interpret from the text; the numbered steps referenced in the prose (e.g., Figure 2 items 1–9) are not clearly legible in the manuscript, making it hard to follow the workflow. Please ensure high-resolution figures with readable labels.","section":"Figure 1 and Figure 2"},{"comment":"The PVLDB reference format lists 'PVLDB, 14(1): XXX-XXX, 2020,' which appears to be a stale placeholder; the current submission year and volume should be corrected.","section":"References and metadata"},{"comment":"The anonymization step is described as ensuring that the Evaluator judges 'general usability rather than prior familiarity,' but the knowledge base is built from the same platform's documentation and the evaluator is tuned on code from that platform. The anonymization may not prevent the LLM from inferring the platform from code patterns, so the fairness claim needs clarification or evidence.","section":"Section 2.1"}],"recommendation":"reject","confidential_remarks":"The paper is a system demonstration without the empirical validation required to support its claims. The evaluator-tuning loop is a particularly serious methodological issue because it makes the evaluation self-referential. Even with significant additional experiments, the manuscript would require a redesign of the validation approach. I recommend rejection in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read the OSS-UAgent paper. The short take: it's a cleanly described system for automating API usability evaluation with LLM agents, but it has zero validation. The abstract's claim that it \"significantly reduces evaluation costs\" is asserted, not demonstrated. What you get is a framework description plus a GUI walkthrough.\n\nWhat's actually new: the specific assembly of a Researcher agent that builds an anonymized vector knowledge base from a GitHub repo, Developer agents simulating four experience levels, and an Evaluator scoring compliance, correctness, and readability. The compliance metric—how closely generated code matches a standard reference—is a plausible proxy for API intuitiveness. Each component is known, but the integration is distinctive and could become a useful screening tool. The demonstration on graph analytics platforms shows the pipeline runs end-to-end technically.\n\nThe soft spots are major. There are no quantitative results, no comparison with human evaluators, no inter-rater reliability, no statistical tests. The only evidence is the line in Section 2.4 about iterating until the Evaluator \"can produce stable and satisfactory evaluation results.\" That tuning loop is exactly the problem: the evaluator is calibrated on the same class of LLM that generates the code, with no external ground truth. Scores could reflect LLM self-consistency rather than developer-facing usability. Two load-bearing assumptions go untested: that role-specific prompts make the LLM produce code representative of real developers at those skill levels, and that an LLM's compliance/correctness/readability scores are valid without human calibration. Neither is supported anywhere. The paper even acknowledges LLM hallucination, then treats it as a mirror of human error—that's an assertion, not evidence.\n\nThe references are fine, and the anonymization step is a thoughtful touch. But the central effectiveness claim is a claim without derivation. This is not a valid research result as presented; it's a prototype description.\n\nWho is this for? A reader curious about LLM-based evaluation architectures might skim the pipeline, but they shouldn't treat the scores as meaningful yet. The paper would deserve a serious referee only if reframed as an experience/demo report, not as a full research paper. As is, I'd desk-reject it for a research track. The authors have a plausible system; they need to validate it against human usability judgments on a few platforms before claiming cost reduction.\n\nMy advice: don't sink referee time into this until it has a validation study. Reject as is.","headline":"A well-described LLM-agent pipeline for API usability evaluation that ships no empirical validation, so the headline effectiveness claim is unsupported as written.","tokens_in":5900,"tokens_out":1817,"would_cite":false,"duration_ms":21165,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that usability evaluation of open source software can be fully automated by LLM agents that simulate developers and score the code they write.","keywords":["agent-based usability evaluation","LLM agents","open source software","API usability","code generation","compliance metric","developer simulation","graph analytics platforms"],"falsifier":"Run the framework on a set of open source platforms, then conduct a parallel human usability study on the same platforms (for example, measuring task completion and satisfaction). If the agent's compliance, correctness, and readability scores do not correlate with the human-perceived usability ratings, or if the simulated 'junior' code fails to reproduce the error patterns of real novices, the central claim fails.","tokens_in":4964,"feed_emoji":"🤖","tokens_out":9093,"duration_ms":78613,"temperature":0.7,"pith_summary":"OSS-UAgent is a framework that replaces human usability evaluators with a pipeline of LLM agents. It simulates developers at four experience levels (Junior, Intermediate, Senior, Expert), builds a vectorized knowledge base from the platform's documentation, and asks the simulated developer to write code for a task. The generated code is then scored on compliance, correctness, and readability against a standard reference implementation. The authors claim this automated pipeline drastically lowers evaluation cost and scales to large open source projects, and they demonstrate it on graph analytics platforms through a graphical interface.","feed_headline":"LLM agents replace human testers to score OSS usability","feed_subtitle":"Code written by simulated junior-to-expert developers is scored on compliance, correctness, and readability.","key_machinery":"The load-bearing mechanism is a multi-agent LLM pipeline with a newly introduced 'compliance' metric. The Researcher agent builds a vector database of platform documentation to ground code generation; the Developer agent encodes experience levels via hierarchical prompts (task-only, API hints, examples, pseudocode). The Evaluator compares generated code to a standard reference, scoring deviation in function usage as compliance, alongside correctness and readability. The pipeline's output—code quality per experience level—is presented as the usability measurement.","core_discovery":"The central claim is that code quality produced under a developer simulation is a valid proxy for platform usability, and that an LLM evaluator can score that code without human judges. The framework chains four agents: Researcher, Developer, Code Generator, and Evaluator. The Researcher anonymizes and vectorizes platform documents; the Developer produces role-specific prompts; the Code Generator writes code conditioned on retrieved knowledge; the Evaluator scores the code on compliance, correctness, and readability. The demonstration on graph analytics platforms shows the full pipeline generating per-experience-level reports.","pith_inferences":["A natural next step would be to compare the simulated code from each experience level against code written by real developers, to calibrate whether the prompts actually reproduce novice and expert behavior.","Because the evaluator is itself an LLM, the scores may shift with the choice of model and prompt wording; a multi-model replication would show how much of the result depends on the particular tool.","If code quality does track usability, this pipeline could be run continuously as a regression check for documentation and API changes, flagging when a change hurts usability."],"forward_implications":["Platform maintainers could obtain usability scores for their software by supplying a repository URL, with no human participants needed.","The automated pipeline reduces evaluation cost and makes large-scale evaluation of many OSS platforms feasible.","The compliance metric gives a concrete measure of how well an API guides developers to standard usage, catching hallucinated or misused functions.","Experience-level breakdowns (Junior to Expert) show where usability fails for less experienced developers, pointing to documentation or API improvements."],"supporting_citations":[{"why":"Supplies the premise that large language model judgments can substitute for human evaluations.","marker":"[2]"},{"why":"Defines human-centered API usability evaluation, the standard practice the framework automates.","marker":"[5]"},{"why":"Provides the empirical basis for correctness and readability as established API usability metrics.","marker":"[7]"},{"why":"Surveys existing API usability evaluation methods, establishing the landscape and metrics the framework builds on.","marker":"[8]"},{"why":"Exemplifies the human-participant peer review approach whose cost and scalability limitations motivate the framework.","marker":"[3]"}],"fun_headline_variants":["LLM agents simulate developers to automate OSS usability scoring","Agent framework uses LLMs to replace human usability testers","Automated usability evaluation for open source with LLM agents","Simulated developers and LLM judges automate OSS usability tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation rests on the assumption that an LLM prompted to role-play a Junior or Expert developer produces code and usability ratings representative of what real developers at those skill levels would produce, without any human calibration.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents simulate developers to automate OSS usability scoring","Agent framework uses LLMs to replace human usability testers","Automated usability evaluation for open source with LLM agents","Simulated developers and LLM judges automate OSS usability tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000292,"raw_usage":{"total_tokens":1627,"prompt_tokens":789,"completion_tokens":838,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":405,"completion_tokens_details":{"reasoning_tokens":770}},"tokens_in":405,"tokens_out":838,"duration_ms":7834,"temperature":1.0,"reasoning_tokens":770,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:48:42.861352+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the framework on a set of open source platforms, then conduct a parallel human usability study on the same platforms (for example, measuring task completion and satisfaction). If the agent's compliance, correctness, and readability scores do not correlate with the human-perceived usability ratings, or if the simulated 'junior' code fails to reproduce the error patterns of real novices, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines human-centered API usability evaluation, the standard practice the framework automates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the empirical basis for correctness and readability as established API usability metrics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Surveys existing API usability evaluation methods, establishing the landscape and metrics the framework builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Exemplifies the human-participant peer review approach whose cost and scalability limitations motivate the framework."}],"review_version":1}