{"id":"64fc327a-e403-4d93-bb74-eed98796e5dc","arxiv_id":"2412.15759","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ASPIRE is an open-source web dashboard that lets IR researchers compare retrieval runs, drill into query-level results, and explore how query characteristics and relevance judgments relate to performance.","lead":"The paper presents ASPIRE, an open-source visual analytics dashboard for evaluating information retrieval experiments from TREC-style query, qrels, and run files. It supports per-query analysis, multi-run comparisons, query-characteristic interplay, and collection-based retrieval views, demonstrated on the TREC Clinical Trials collection.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Query Text-based Report and Collection-based Report are described as analyses, but the listed outputs (word clouds, similarity plots, document-ranking visualizations) do not by themselves establish that the claimed performance-interplay analysis exists.","rationale":"The paper is a short system description; the core assertion is that the tool exists and provides four views. The public repository and hosted demo give independent support for existence, and the use of the ir_measures library is a credible basis for standard metric correctness. However, the text's descriptions of the two more novel views (query characteristics-performance interplay and collection-based retrieval analysis) are under-specified: the outputs named are not sufficient to establish that a link between query characteristics and effectiveness is actually analyzed. A word cloud is descriptive, not analytical; a similarity plot can exist without any grounding in performance. Since the paper provides no detailed methodology, statistical test, or explanatory figure for these sections, the central claim's scope is not verifiable from the manuscript alone. This is precisely the kind of missing support the reviewing rule asks to flag. The proposed check—inspecting the code/demo and running a known dataset—would settle it. If the code does implement statistical association tests, the concern is resolved and the paper is acceptable as a demo; if not, the claim should be reduced or the manuscript revised. The reader's weakest_assumption also noted the absence of a user study and scalability evidence; I partially agree, but the more immediate falsifiable issue is the feature-level implementation gap. I therefore keep the verdict CONDITIONAL; no change from the reader's verdict is needed, though the manuscript should ideally state more precisely what each report computes.","tokens_in":4996,"tokens_out":4873,"duration_ms":42959,"concrete_test":"Run the public ASPIRE demo (or local repo) on a known dataset such as TREC Clinical Trials 2021, where query length is known to correlate with effectiveness. Open the Query Text-based Report and check whether it computes a numerical association (e.g., correlation or regression) between query characteristics and performance, or whether it only shows a word cloud and a similarity plot without a performance link. Then inspect the source code to identify the exact computation behind the Collection-based Report (which documents are considered relevant across queries) and reproduce a small example by hand. Finally, stress-test scalability by uploading 50+ runs and a large collection (e.g., MS MARCO dev set); if the app times out or crashes, the scalability claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ASPIRE provides four analytically substantive views, not merely charts. Section 2 describes the Query Text-based Report as 'examines the link between query characteristics (e.g., length) and effectiveness' yet lists only word clouds and 2D/3D query similarity visualizations as outputs; neither inherently performs a link analysis, and no statistical test or measure is named. Similarly, the Query Collection-based Report is said to 'analyze relevance judgments distribution, identify documents relevant across multiple queries and visualize document rankings,' but no method, definition, or output type is given. The paper supplies no screenshot legend, algorithm description, or evaluation that would let a reader verify these sections actually implement the advertised analysis. This is load-bearing because if those pages only display descriptive plots, one of the four advertised aspects collapses. A secondary unsupported claim is the scalability statement ('built to handle large collections... all participant submissions'), which lacks any benchmark. Because the tool is open source, these gaps are checkable; until then, the central claim is only partially supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ASPIRE, an open-source, web-based visual analytics tool for information retrieval experiments. ASPIRE accepts standard TREC-style query, qrels, and run files, and provides four analysis pages: an experiment performance report (with multi-run comparison and significance testing), a query-based report, a query text-based report (aimed at linking query characteristics to effectiveness), and a query collection-based report (aimed at analyzing relevance judgment distributions and document rankings). The paper describes the tool's architecture, its dependencies (streamlit, ir_measures, plotly, transformers), and demonstrates it on TREC Clinical Trials data. The central claim is that ASPIRE supports four key aspects of IR experiment evaluation through an extensive and user-friendly interface.","tokens_in":5168,"tokens_out":4520,"duration_ms":40733,"significance":"If the tool works as described, it addresses a genuine practical need: accessible, open-source, interactive analysis of IR evaluation results beyond tabular measures. The public GitHub repository and live demo are concrete, checkable assets, and the modular design (separate pages/sections) is a real strength for extensibility. The use of ir_measures and standard plotting libraries gives reasonable confidence in the basic measured values. However, the paper's evidence is currently stronger for the tool's existence and accessibility than for the analytical depth of its two 'advanced' reports, and the scalability claim is unsubstantiated. The contribution is therefore promising but not yet fully demonstrated in the manuscript.","major_comments":[{"comment":"The Query Text-based Report is described as 'examines the link between query characteristics (e.g., length) and effectiveness' and the Query Collection-based Report as 'analyzes relevance judgments distribution, identifying documents relevant across multiple queries and visualizing document rankings', yet the only listed outputs are word clouds, 2D/3D similarity visualizations, and document-ranking plots. No statistical test, correlation coefficient, grouping procedure, or other analytical operation is named, and no screenshot or figure legend is provided to show what these pages actually compute. If the pages only display descriptive plots, the advertised 'query characteristics-performance interplay' and 'collection-based retrieval analysis' aspects are not established. Please specify the exact analytical methods (e.g., per-query correlation of query length with NDCG, significance testing, overlap analysis) and include annotated screenshots or a worked example for at least one query and one collection-based view.","section":"Section 2"},{"comment":"The claim that ASPIRE 'is built to handle large collections and evaluate numerous runs simultaneously, accommodating, for example, all participant submissions in shared tasks' is unsupported by any benchmark, architecture detail, or stress test. Since shared-task organizers are a stated target user group, this scalability claim is load-bearing for part of the paper's value proposition. Please provide a simple scalability experiment (e.g., runtime and memory usage on a TREC collection with a realistic number of runs, or a comparison with a baseline like trec_eval on a large run set), or substantially qualify the claim to what has been tested.","section":"Section 2, final paragraph"}],"minor_comments":[{"comment":"Typo: 'e.g.documents' should read 'e.g., documents'.","section":"Section 1, Introduction"},{"comment":"Typo: 'Query T ext-based Report' contains an errant space; it should be 'Query Text-based Report'.","section":"Section 2"},{"comment":"The caption states that purple blocks represent 'plots and analysis' and yellow blocks 'show analysis', but the distinction is unclear; consider clarifying what each color denotes or removing the overlap.","section":"Figure 1"},{"comment":"The paper mentions 'two configuration web pages' but does not describe what these configuration pages do; a sentence on their purpose (e.g., file upload, measure selection, baseline settings) would help readers understand the workflow.","section":"Section 2, opening of the second paragraph"},{"comment":"The 'Use Cases' section mixes current functionality with future plans (e.g., 'plans to introduce a dedicated page'); consider clearly marking which features are implemented now and which are planned, to avoid overstating the current tool.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"The tool clearly exists and is accessible, which is a meaningful strength. The main issue is that the two analytically substantive reports are described only by their inputs and output types, not by the actual analysis performed; this is fixable within the manuscript's scope by providing method details and screenshots. The scalability claim also needs a minimal benchmark. I recommend major revision rather than rejection because the central claim is defensible and the required evidence is obtainable from the open-source repository."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ASPIRE is a genuinely available open-source web dashboard for IR evaluation, and the paper's main claim — that the tool exists and takes TREC-style files — checks out via the repo and live demo. That is the strong part, and it's not nothing. The code is the artifact, not just the abstract.\n\nThe soft spot is that the paper describes four 'analyses' but only one or two of them are concretely pinned down. The Experiment Performance Report and Query-based Report are standard and clear. The Query Text-based Report is supposed to examine the link between query characteristics and effectiveness, but the listed outputs are word clouds and 2D/3D similarity plots; no statistical test, no defined measure of 'interplay.' The Collection-based Report is even vaguer: it says it analyzes relevance judgment distribution and visualizes document rankings, but gives no method or output that a reader can verify from the text. That matters because the novelty claim rests partly on those two pages. The paper also asserts it can handle large collections and all participant submissions without any benchmark.\n\nNone of this is fatal. The tool is open source, so every claim is checkable by whoever bothers to run it. The authors also cite prior tools and position themselves accurately against them. The lack of a user study is a common weakness in tool papers, but it still leaves the 'user-friendly' claim unbacked.\n\nBottom line: this is a legitimate system description that deserves a rigorous peer review — not a desk reject. The reviewers should ask the authors to state precisely what each page computes, and ideally add a small correctness check against trec_eval or a scalability number. If the two vague reports turn out to just be word clouds and rank plots, that's still fine as long as the paper says so. The current text overpromises.","headline":"A real, open-source IR evaluation dashboard whose paper overpromises on two of its four analysis views, but it's checkable and deserves a proper review.","tokens_in":5662,"tokens_out":1990,"would_cite":false,"duration_ms":17555,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ASPIRE, an open-source web tool, claims that four interactive views—single-run, multi-run, query-level, and collection-level—are enough to move IR evaluation beyond score tables.","keywords":["information retrieval evaluation","visual analytics","interactive dashboard","TREC-style runs","query-level analysis","collection-based analysis","open-source tool","experiment comparison"],"falsifier":"Run ASPIRE on a public TREC collection with known results, compare every reported measure and significance outcome against trec_eval or ir_measures on the same runs, and look for mismatches; alternatively, have analysts complete a fixed set of comparison tasks with ASPIRE versus a table-based baseline and measure task accuracy and time. Any systematic discrepancy in the measures, or no measurable improvement in analysis quality, would undercut the claim that the tool provides trustworthy, useful in-depth evaluation.","tokens_in":4810,"feed_emoji":"📊","tokens_out":3893,"duration_ms":33042,"temperature":0.7,"pith_summary":"This paper introduces ASPIRE, an open-source web tool for analyzing information retrieval experiments beyond average scores. It claims that four complementary views—single-run evaluation, multi-run comparison, query-level breakdowns, and collection-based analysis of relevance judgments and rankings—cover the main analytical needs of IR researchers. ASPIRE accepts standard TREC-style query, qrels, and run files, computes standard measures, and produces interactive plots, statistical significance tests, and exportable reports. The authors demonstrate the tool on the TREC Clinical Trials collection and argue that making such analyses accessible and reproducible supports transparency in IR evaluation.","feed_headline":"Open-source ASPIRE turns TREC runs into four interactive views","feed_subtitle":"Compare runs, inspect per-query breakdowns, and explore collection-level patterns — all from standard TREC files.","key_machinery":"The carrying mechanism is the four-page modular architecture built with Streamlit: each page groups one analytical task, takes user-selected TREC files, and recomputes tables and plots on demand. Behind it, ir_measures computes retrieval effectiveness measures, statistics and statsmodels carry significance testing, plotly-express renders interactive plots, and the transformers library computes query similarity for the text-based report. The design makes each analysis a separate, extensible section, so new evaluation practices can be added without restructuring the tool.","core_discovery":"ASPIRE is presented as a working, modular visual analytics system that lets researchers upload standard TREC-style files and receive, in real time, four kinds of analysis: an experiment performance report with multi-run comparisons, statistical significance testing, and precision-recall curves; a query-based report with per-query effectiveness; a query text-based report linking query characteristics to performance with word clouds and transformer-based 2D/3D similarity visualizations; and a query collection-based report on relevance judgment distribution and document ranking behavior. The central claim is that this combination, delivered through an interactive web interface with adjustable parameters and downloadable outputs, meets a real community need for understanding why systems perform as they do, beyond reporting a table of scores.","pith_inferences":["A natural next step the paper does not take is to measure whether ASPIRE actually speeds up or improves analyst decisions, for example by comparing time-to-insight against a spreadsheet baseline.","The transformer-based query similarity map could be reused as a feature for query performance prediction, an area the paper lists as future work.","The collection-based view of documents relevant to multiple queries might support pooling and judgment-consistency checks in future shared tasks.","Since the tool is open source, an independent correctness pass against trec_eval on a variety of collections would be straightforward and would strengthen the tool's adoption."],"forward_implications":["Researchers can upload their own runs and get immediate, downloadable reports of standard measures, significance tests, and precision-recall curves.","Shared-task organizers can evaluate all participating runs at once rather than inspecting each system's output separately.","Readers of papers that publish run files can re-analyze published results interactively, supporting transparency and reproducibility.","Query-level and collection-level views connect observed effectiveness to query characteristics and relevance-judgment structure, which can guide targeted error analysis.","The modular Streamlit architecture allows future evaluation analyses to be added as new sections or pages."],"supporting_citations":[{"why":"Supplies the Cranfield offline evaluation paradigm that defines ASPIRE's input files and measures.","marker":"[8]"},{"why":"Grounds the argument that visual analytics can aid in-depth analysis of IR system behavior.","marker":"[10]"},{"why":"Describes RETRIEVAL, an earlier online evaluation platform that ASPIRE extends and contrasts with.","marker":"[14]"},{"why":"Provides the ir_measures library that ASPIRE uses to compute evaluation measures.","marker":"[17]"},{"why":"Supplies the TREC Clinical Trials 2021 runs used to showcase ASPIRE's interface.","marker":"[20]"},{"why":"Describes Vis-Trec, an existing open-source evaluation tool whose gaps ASPIRE aims to address.","marker":"[22]"}],"fun_headline_variants":["ASPIRE: four interactive views for TREC run analysis","Open-source ASPIRE visualizes IR experiments four ways","Explore TREC runs with ASPIRE's four analytic views","ASPIRE: beyond score tables - interactive IR analysis","Four interactive views to dissect TREC runs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes, without a user study or benchmark, that the analyses and visualizations ASPIRE produces are correct, reliable on large collections and many runs, and genuinely help researchers understand system performance better than tables of scores.","fun_headline_variants_meta":{"raw":{"variants":["ASPIRE: four interactive views for TREC run analysis","Open-source ASPIRE visualizes IR experiments four ways","Explore TREC runs with ASPIRE's four analytic views","ASPIRE: beyond score tables - interactive IR analysis","Four interactive views to dissect TREC runs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000569,"raw_usage":{"total_tokens":2635,"prompt_tokens":830,"completion_tokens":1805,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":1726}},"tokens_in":446,"tokens_out":1805,"duration_ms":11347,"temperature":1.0,"reasoning_tokens":1726,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:06:37.343921+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ASPIRE on a public TREC collection with known results, compare every reported measure and significance outcome against trec_eval or ir_measures on the same runs, and look for mismatches; alternatively, have analysts complete a fixed set of comparison tasks with ASPIRE versus a table-based baseline and measure task accuracy and time. Any systematic discrepancy in the measures, or no measurable improvement in analysis quality, would undercut the claim that the tool provides trustworthy, useful in-depth evaluation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Cranfield offline evaluation paradigm that defines ASPIRE's input files and measures."},{"cited_title":"Informa- tion Retrieval Evaluation in a Changing World: Lessons Learned from 20 Years of CLEF pp","cited_arxiv_id":null,"evidence_quote":"Grounds the argument that visual analytics can aid in-depth analysis of IR system behavior."},{"cited_title":"IEEE Trans","cited_arxiv_id":null,"evidence_quote":"Describes RETRIEVAL, an earlier online evaluation platform that ASPIRE extends and contrasts with."},{"cited_title":"In: European Conference on Information Retrieval","cited_arxiv_id":null,"evidence_quote":"Provides the ir_measures library that ASPIRE uses to compute evaluation measures."},{"cited_title":"UNIMIB at TREC 2021 Clinical Trials Track","cited_arxiv_id":"2207.13514","evidence_quote":"Supplies the TREC Clinical Trials 2021 runs used to showcase ASPIRE's interface."},{"cited_title":"In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval","cited_arxiv_id":null,"evidence_quote":"Describes Vis-Trec, an existing open-source evaluation tool whose gaps ASPIRE aims to address."}],"review_version":1}