{"id":"6e0e5d02-9536-4fc6-8db7-d1ba795b5189","arxiv_id":"2505.08842","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An LLM multi-agent system with a public leaderboard claims to uncover hidden security, licensing, dependency, and compliance risks in 20 open-source AI libraries, matching up to 88% of OpenSSF Scorecard checks.","lead":"This paper introduces LibVulnWatch, an agentic LLM system that evaluates open-source AI libraries across licensing, security, maintenance, dependencies, and regulatory compliance, publishing results on a public leaderboard. It reports matching up to 88% of OpenSSF Scorecard checks and surfacing up to 19 additional context-dependent risks per library.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline coverage and novelty metrics are computed by the same LLM pipeline under prompts that force high-risk findings, with no human-verified ground truth; the central claim of surfacing hidden vulnerabilities is not yet empirically secured.","rationale":"The reader's weakest assumption identifies the same load-bearing weakness: the Coverage and Novelty Yield metrics are produced by the same LLM pipeline that wrote the reports, with no human-verified ground truth. My reading of the prompt listings strengthens rather than displaces this concern. Listing 4 contains two explicit instructions that bias the system toward high-risk findings: every section must contain at least one HIGH risk item, and missing information is itself treated as HIGH risk. Appendix A.2.8 then instructs the same model to list what the model found that Scorecard did not mention, with no adjudication of whether those findings are true, material, or within Scorecard's intended scope. This creates a direct pathway for the headline 'Novelty Yield' to be inflated by the system's own prompt design. The abstract's claim about 'surfacing up to 19 additional risks' is therefore not supported by the evidence as presented, even though the system architecture and the decision to benchmark against OpenSSF Scorecard are reasonable. The JAX example in Appendix A.4 illustrates the problem concretely: Scorecard reports 18 existing vulnerabilities while the agent report states no recent CVEs, and the discrepancy is left unresolved while the model's own findings are listed as novel. Because the flaw is in the evaluation protocol rather than in the system's architecture, and because the paper provides enough detail for targeted remediation, CONDITIONAL remains the appropriate verdict. I would not move to ACCEPT or REJECT based on this concern alone, since a human-audited validation of the novel-finding inventory could settle the question directly.","tokens_in":24773,"tokens_out":3580,"duration_ms":37126,"concrete_test":"Select five Table 1 libraries, including LangChain (the 19-yield case), and have two independent security researchers, blinded to the paper's conclusions, classify every item listed under 'Issues Found Only by Model' in the corresponding Appendix A.4-style comparisons as a true positive, a false positive, or out-of-scope for OpenSSF Scorecard, using current NVD/OSV data and each library's repository as reference. Recompute Novelty Yield counting only human-validated true positives. If the validated yield is substantially lower than the reported yield, or if removing absence-based findings ('no SBOM,' 'no compliance documentation') cuts the yield by more than half, the headline claim of surfacing hidden vulnerabilities is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's empirical demonstration rests on two self-computed metrics, Baseline Alignment and Novelty Yield, defined in Section 3.5 and produced by the LLM-based comparison in Appendix A.2.8. The model that wrote the reports also judges which report findings were 'not explicitly surfaced by the Scorecard,' and no independent, human-verified ground truth is introduced. This is not merely an absence of curation: the drafting prompt in Listing 4 explicitly instructs the agent that 'the most realistic assessment MUST include at least ONE HIGH risk item' and that 'Absence of critical information itself can justify a high risk rating.' Many 'novel' findings are therefore absence-based judgments, such as 'no SBOM' or 'no GDPR documentation,' which OpenSSF Scorecard does not attempt to measure by design. Counting these as 'additional risks' conflates a difference in assessment scope with a demonstrated capability to discover hidden vulnerabilities. The one available external comparison is also unresolved: Appendix A.4 states that Scorecard reports 18 existing vulnerabilities for JAX while the generated report concludes there are no recent CVEs, and the paper does not reconcile this contradiction. If the self-assessment inflates novelty, then the central claim that LibVulnWatch surfaces up to 19 additional risks per library, including critical RCEs, reduces to prompt-induced reframing rather than validated discovery. Reproducibility is similarly unsupported: the paper reports a single LLM run per library with no seeds, temperatures, or released code, so the 'reproducible scores' claim cannot be checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LibVulnWatch, a graph-based multi-agent system built on gpt-4.1-mini that assesses open-source AI libraries across five risk domains (licensing, security, maintenance, dependencies, regulatory compliance) and publishes scores to a public leaderboard. The authors apply the system to 20 widely used libraries and benchmark it against the OpenSSF Scorecard, reporting up to 88% coverage of applicable Scorecard checks and a Novelty Yield of up to 19 additional risks per library, including RCEs, missing SBOMs, and regulatory gaps. The methodology consists of prompt-governed planning, iterative evidence retrieval, per-domain drafting with quality checks, synthesis, and an LLM-based comparison against Scorecard output. The paper argues that LLM-based contextual reasoning complements static scanners and enables deeper supply-chain risk assessment.","tokens_in":25078,"tokens_out":4322,"duration_ms":41400,"significance":"If the empirical claims were fully supported, the work would be a useful demonstration that agentic LLM pipelines can conduct continuous, multi-domain risk assessment of open-source AI libraries at low per-library cost, and the public leaderboard would be a practical community resource. The paper has genuine strengths: it anchors coverage against an external benchmark (OpenSSF Scorecard), provides detailed prompts and an example report in the appendix, and makes a concrete, falsifiable claim about additional findings. However, the central contribution--uncovering hidden vulnerabilities--rests on Novelty Yield, a metric computed by the same LLM pipeline that produced the reports, with no human-verified ground truth. The prompt design also explicitly forces at least one high-risk finding and permits absence-of-information to be rated as high risk, which inflates the novelty count. These issues are load-bearing because the abstract's headline numbers ('up to 19 additional risks per library, such as critical RCE vulnerabilities') are exactly the unvalidated self-assessments.","major_comments":[{"comment":"The two headline metrics, Baseline Alignment and Novelty Yield, are both produced by the same gpt-4.1-mini pipeline that generated the reports; the comparison prompt in Appendix A.2.8 asks the model to judge which of its own findings were 'not explicitly mentioned in Scorecard.' There is no human-verified ground truth or independent vulnerability-database confirmation for the Novelty Yield counts in Table 1. Because the claim of surfacing 'hidden vulnerabilities' is load-bearing, the authors must provide a validation protocol (e.g., manual triage by security researchers, cross-checking against NVD/GitHub advisories, precision/recall on known vulnerabilities) or explicitly reframe the metric as 'coverage expansion relative to Scorecard' rather than validated vulnerability discovery.","section":"3.5; A.2.8"},{"comment":"The drafting prompt mandates that 'the most realistic assessment MUST include at least ONE HIGH risk item' and states that 'Absence of critical information itself can justify a high risk rating.' Many findings counted in Novelty Yield (e.g., no SBOM, no GDPR documentation, no audit trail) are by design outside OpenSSF Scorecard's measurement scope. Counting these as 'additional risks' conflates a difference in assessment scope with a demonstrated capability to detect hidden vulnerabilities. The paper should separate scope-expansion findings from vulnerability findings and, for the latter, provide evidence of exploitability, affected versions, or CVE references.","section":"A.2.4 (Listing 4)"},{"comment":"The JAX baseline evaluation contains a direct, unresolved contradiction: the Scorecard reports 18 existing vulnerabilities, while the generated JAX report concludes there are no recent CVEs, and the evaluation notes this conflict without explaining it. This undermines the claim that the security assessment is evidence-based and suggests the system may be missing or misreading CVE data. The authors must reconcile this discrepancy, explain how CVE evidence is ingested and filtered, and quantify the system's recall on known vulnerabilities before the 'hidden vulnerability' claim can be accepted.","section":"A.4"},{"comment":"All experiments use a single LLM run at unspecified sampling settings, and no code or complete prompt set is released; Section 7 even says full prompt details are 'beyond the scope of this paper,' despite Appendix A.2 presenting several prompts. The abstract's 'reproducible' claim is therefore not currently supported. Please release the pipeline, all prompts, and run-to-run variance (e.g., multiple seeds with documented temperature settings), or weaken the reproducibility claim accordingly.","section":"3.2; 7"}],"minor_comments":[{"comment":"Table 2 contains a stray '13.0' entry under 'Core ML/DL Frameworks' and inconsistent row formatting for 'Text Generation Inference'; the table should be cleaned so category averages are clearly labeled.","section":"Table 2"},{"comment":"The comparison prompt uses 'Model Coverage' and 'Model Seeking' while the body uses 'Baseline Alignment' and 'Novelty Yield'; the terminology should be aligned throughout.","section":"A.2.8"},{"comment":"The MetaGPT reference is cited as 'Zhang and colleagues, 2024' with the note 'first names omitted for brevity,' which is not an acceptable citation format; supply the full author list and standard citation.","section":"References"},{"comment":"Section 6 contains cross-references to 'Section 5' for limitations awareness, but Section 5 is the Discussion and does not actually acknowledge the cited limitations; the cross-references should be corrected or removed.","section":"6; 7"},{"comment":"Listing 7 has a formatting typo: the closing tag `</ Format }` should be `</ Format >`; a similar issue appears in Listing 8.","section":"A.2.7"}],"recommendation":"major_revision","confidential_remarks":"The paper's fit with cs.CR is acceptable, but this is primarily a systems and empirical-measurement paper. The headline claim of discovering hidden vulnerabilities currently depends on unvalidated LLM self-assessment. I recommend that the editor require an artifact release (code, full prompts, and report data) and an independent human-validated sample of 'novel' findings before the paper can be accepted; without these, the empirical demonstration does not support the abstract's claims. There are no concerns about citation practices or novelty disclosure beyond the empirical validation issue noted in the major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take on arXiv:2505.08842. The system is real, the leaderboard is live, and the prompts are published—that's more than most 'agentic' papers give you. But the headline numbers (88% coverage, up to 19 hidden risks) are computed by the same LLM that wrote the reports, under prompts that force high-risk findings. So the 'hidden vulnerabilities' claim is not yet empirically supported.\n\nWhat is genuinely new: a graph-based multi-agent pipeline that assesses 20 AI libraries across five governance domains, with a public Hugging Face leaderboard. Using OpenSSF Scorecard as an external anchor is the right instinct. The per-library reports, like the JAX one in the appendix, show real contextual reasoning (licensing complexity, missing SBOM, regulatory gaps). That part is useful for supply-chain practitioners.\n\nThe soft spots are the metrics. Baseline Alignment and Novelty Yield come from an LLM comparing its own report against Scorecard, with no human-verified ground truth. The drafting prompt in Listing 4 explicitly mandates at least one high-risk item and says absence of information can justify high risk. So many 'novel' findings are 'no SBOM' or 'no GDPR documentation'—things Scorecard deliberately doesn't measure. Counting those as hidden vulnerabilities conflates scope with discovery. The JAX appendix makes it concrete: Scorecard reports 18 existing vulnerabilities, the agent report says no recent CVEs, and the paper never reconciles it. That's a direct hit on reliability. Also, one LLM run per library, no seeds or temperature, no released code. The 'reproducible scores' claim cannot be checked.\n\nNone of this is fatal to the system as a proposed tool. The architecture is plausible and the leaderboard is a useful community artifact. But the paper's central claim—that the system surfaces hidden vulnerabilities beyond static scanners—is not yet backed by the evidence. It needs human validation of a sample of findings, reconciliation of contradictions like the JAX CVE count, and artifact release. Those are fixable.\n\nWho should read this: people working on LLM-based supply-chain security and AI governance tooling. They'll get a concrete design and a public dataset of assessments, but should treat the numeric claims as provisional.\n\nFor peer review: I'd send it out. It's a real system with real artifacts, and the flaws are addressable in revision. If it were submitted to me, I'd ask for the validation study and the code.","headline":"A real agentic supply-chain assessment system with a live leaderboard, but the coverage and novelty metrics are self-assessed under force-high-risk prompts, so the central 'hidden vulnerabilities' claim is not yet supported.","tokens_in":25621,"tokens_out":2878,"would_cite":false,"duration_ms":27895,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agentic LLM scans catch up to 19 hidden risks per AI library.","keywords":["LLM agents","software supply chain","supply-chain security","OpenSSF Scorecard","vulnerability discovery","open-source AI libraries","agentic workflows","risk scoring"],"falsifier":"Take five of the twenty published reports, such as JAX, LangChain, and CrewAI, and have independent security and legal reviewers re-derive each claimed extra risk from the cited public sources; if the verified count of real, meaningful, non-duplicate findings is far below the reported Novelty Yield, the central claim is not supported.","tokens_in":24603,"feed_emoji":"🛡️","tokens_out":9730,"duration_ms":85156,"temperature":0.7,"pith_summary":"LibVulnWatch proposes replacing shallow, rule-based checks with a team of specialized language-model agents that read repositories, advisories, and documentation and file structured risk reports across five domains: licensing, security, maintenance, dependencies, and regulatory compliance. On 20 widely used AI libraries, the system says it matches up to 88% of the checks in the OpenSSF Scorecard while adding up to 19 findings per library that the static tool does not report, including remote-code-execution exposures, missing software bills of materials, and compliance gaps. If the claim holds, this is a low-cost path to continuous, evidence-cited supply-chain monitoring: each assessment runs for roughly $0.10, and every report is published on a public leaderboard.","feed_headline":"AI library scans by LLM agents find up to 19 hidden risks each","feed_subtitle":"On 20 open-source AI libraries, the system also covers up to 88% of OpenSSF Scorecard checks.","key_machinery":"The central object is a DAG-based multi-agent workflow: an Assessment Planner generates queries; per-domain agents loop through query generation, evidence retrieval, drafting, and quality checks; a synthesizer compiles the final report; and a benchmark-validation agent runs the OpenSSF Scorecard and compares outputs to compute alignment and novelty. The load-bearing controls are engineered prompts that force quantifiable metrics, inline citations, explicit high-risk ratings, and conservative treatment of missing information, plus a 1-5 rating scale whose average forms the Trust Score.","core_discovery":"The paper's central discovery is that a graph-orchestrated set of LLM agents can transform open-source risk assessment from a checklist exercise into a knowledge-synthesis task. Given a library name, the system plans searches, retrieves evidence in parallel per domain, drafts findings with mandatory citations, quality-checks and refines them, then synthesizes a report with a risk dashboard and a composite Trust Score. Benchmarking that report against the OpenSSF Scorecard on 20 libraries produces Baseline Alignment of 55-88% and a Novelty Yield of 4-19 unique findings per library, with contextual risks such as licensing entanglements, undocumented telemetry, and absent SBOMs that static analysis does not surface.","pith_inferences":["Because the same LLM pipeline both writes the reports and compares them against Scorecard, the headline coverage and novelty numbers are best read as upper bounds until independent human reviewers verify a sample.","The 'missing documentation counts as high risk' rule will systematically penalize libraries that are secure but under-documented; a human audit of low-scoring libraries would show how often this produces false alarms.","If the approach transfers, the same evidence-gathering pattern could assess libraries in other regulated ecosystems, such as health-data tooling or fintech SDKs, but the scoring rubrics would need to be re-anchored to each domain's regulations.","A testable extension: re-run the leaderboard after 6-12 months and check whether public scores push maintainers to add SBOMs or write security policies, making the tool an intervention rather than just a snapshot."],"forward_implications":["If the system is right, static scorecards and LLM agents are complementary: agents add context-dependent findings while missing some structured checks, so the strongest deployment runs both.","At roughly $0.10 per library, continuous re-assessment of the whole open-source AI ecosystem becomes affordable, turning point-in-time audits into monitoring.","A public, evidence-cited leaderboard gives downstream users a repeatable basis for comparing AI libraries and for tracking whether maintainers improve after findings are published.","The five-domain template extends risk assessment beyond security to licensing and regulatory readiness, which AI adopters increasingly need for procurement and compliance decisions."],"supporting_citations":[{"why":"Supplies the OpenSSF Scorecard baseline used to compute Baseline Alignment and Novelty Yield.","marker":"Zahan et al., 2023"},{"why":"Defines the LLM supply-chain vulnerability landscape the system targets.","marker":"Wang et al., 2025"},{"why":"Establishes the technical governance gap that motivates the assessment framework.","marker":"Reuel et al., 2025"},{"why":"Provides the LLM application framework used to build the assessment agents.","marker":"LangChain AI, 2025a"},{"why":"Provides the graph orchestration layer that structures the multi-agent workflow.","marker":"LangChain AI, 2025b"}],"fun_headline_variants":["Agent system uncovers up to 19 hidden risks in AI libraries","LLM agent network finds 19 risks per AI library in deep audit","Graph-orchestrated agents surface 19 hidden risks per library","AI library risk audit yields 4–19 new findings per library"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The coverage and novelty numbers are produced by the same LLM pipeline that wrote the reports, comparing its own output against Scorecard with no human-verified ground truth, so the claimed hidden-vulnerability advantage depends on that self-measure being honest.","fun_headline_variants_meta":{"raw":{"variants":["Agent system uncovers up to 19 hidden risks in AI libraries","LLM agent network finds 19 risks per AI library in deep audit","Graph-orchestrated agents surface 19 hidden risks per library","AI library risk audit yields 4–19 new findings per library"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000786,"raw_usage":{"total_tokens":3450,"prompt_tokens":906,"completion_tokens":2544,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":2469}},"tokens_in":522,"tokens_out":2544,"duration_ms":16267,"temperature":1.0,"reasoning_tokens":2469,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:52:16.206007+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take five of the twenty published reports, such as JAX, LangChain, and CrewAI, and have independent security and legal reviewers re-derive each claimed extra risk from the cited public sources; if the verified count of real, meaningful, non-duplicate findings is far below the reported Novelty Yield, the central claim is not supported.","supporting_citations":[],"review_version":1}