{"id":"3ca1b943-1fb2-4578-8edb-8811d52a96fe","arxiv_id":"2606.01189","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Position paper proposing Model Science as a discipline to systematically analyze AI model behavior beyond benchmarks, drawing analogies from cognitive science, neuroscience, medicine, and agriculture.","lead":"The paper calls for shifting AI research from benchmark testing to a new systematic discipline called Model Science, organized around four perspectives: Verify, Explore, Steer, and Refine, plus shared infrastructure and single-model deep dives. A smart generalist might read it to understand how current leaderboards leave critical gaps in reliability and safety for deployed AI systems.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"The proposal assumes direct transfer of lessons from cognitive science, neuroscience, medicine and agriculture to AI without evidence that the analogies hold or require adaptation for computational systems.","rationale":"The reader's weakest assumption directly identifies the load-bearing gap in the argument's reliance on unadapted analogies. Full-text review would be needed to check for hidden operational details, but the abstract alone already exposes the missing evidence for transferability. This does not alter the UNVERDICTED status.","tokens_in":1738,"tokens_out":283,"duration_ms":12784,"concrete_test":"Locate the sections discussing the four precedents and the three foundations; extract any explicit adaptation steps or counter-examples for computational models (e.g., handling non-stationary weights or discrete token spaces). If none appear, the transfer claim remains unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the community is ready to consolidate into Model Science rests on precedents informing three foundations (four perspectives, shared infrastructure, single-instance analysis). The abstract invokes these fields to justify the move beyond benchmarks, yet supplies no mapping showing why single-case neuroscience methods would reveal LLM internals differently from population benchmarks, why medicine-style training would scale to model analysis, or how agriculture-style shared catalogues would overcome AI's rapid obsolescence. This leaves the readiness assertion dependent on an untested transferability assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper argues that the AI community should move beyond benchmark-driven research to establish a new systematic discipline termed 'Model Science.' It draws on precedents from cognitive science (complementary levels of analysis), neuroscience (value of single-case studies), medicine (specialized training alongside research), and agriculture (shared infrastructure for cumulative progress) to propose three foundations: (1) four functional perspectives—Verify, Explore, Steer, and Refine—for analyzing model behavior; (2) shared infrastructure including catalogues of datasets, models, and findings; and (3) deep analysis of individual model instances rather than only model families.","tokens_in":1841,"tokens_out":659,"duration_ms":18227,"significance":"If the proposed framework gains traction, it could help organize scattered model analysis efforts and address benchmark limitations such as failure to explain why models succeed or fail on tasks like hallucination detection. The paper correctly identifies that current leaderboards track performance gains but provide limited insight into internal mechanisms. However, the significance is limited by the absence of any concrete mappings, pilot implementations, or falsifiable predictions demonstrating that the cited field analogies can be adapted to computational models without substantial modification.","major_comments":[{"comment":"Abstract and the section outlining the three foundations: the central readiness claim—that precedents 'point the way forward' and that the community is 'now ready' to consolidate into Model Science—rests on an untested transferability assumption. No specific argument is given showing why single-case neuroscience methods would reveal LLM internals differently from population benchmarks, why medicine-style training would scale to model analysis, or how agriculture-style catalogues would overcome rapid model obsolescence.","section":"Abstract / Foundations section"},{"comment":"Discussion of the neuroscience precedent for single-instance analysis: the manuscript states that 'single cases can reveal what population studies miss' but supplies no mapping or example demonstrating how this would apply to trained neural networks, where population-level benchmarks are the dominant evaluation paradigm due to the statistical nature of learned parameters.","section":"Precedents from neuroscience / single-instance analysis paragraph"},{"comment":"Infrastructure foundation (catalogues of datasets, models, and findings): the proposal assumes such shared resources would enable cumulative progress, yet the text does not address or provide evidence against the risk that fast iteration cycles in AI would render catalogues obsolete faster than in agriculture, undermining the cumulative-knowledge goal.","section":"Infrastructure discussion"}],"minor_comments":[{"comment":"The four perspectives (Verify, Explore, Steer, Refine) are introduced at a high level; concrete operational definitions or example workflows for each would improve clarity.","section":"Four perspectives section"},{"comment":"The manuscript would benefit from additional citations to specific methodological papers in the referenced fields (e.g., single-case studies in neuroscience) to ground the analogies.","section":"Precedents paragraphs"}],"recommendation":"major_revision","confidential_remarks":"This is a position/proposal paper rather than an empirical or theoretical contribution. It may fit better as an invited perspective or in a venue focused on research practices than in a standard technical track."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on our position paper. We address each major comment below, clarifying the manuscript's scope as an argument for establishing Model Science rather than an empirical validation of the proposed analogies.","responses":[{"response":"We agree that the paper presents the precedents as suggestive rather than demonstrating specific transferability arguments or mappings. As a position paper, the intent is to outline why the community should pursue such consolidation, with the task of adapting and testing these ideas left to future work in the proposed discipline. We will revise the abstract and foundations section to explicitly frame the analogies as hypotheses to be investigated rather than established transfers.","revision_made":"yes","referee_comment":"[Abstract / Foundations section] Abstract and the section outlining the three foundations: the central readiness claim—that precedents 'point the way forward' and that the community is 'now ready' to consolidate into Model Science—rests on an untested transferability assumption. No specific argument is given showing why single-case neuroscience methods would reveal LLM internals differently from population benchmarks, why medicine-style training would scale to model analysis, or how agriculture-style catalogues would overcome rapid model obsolescence."},{"response":"The neuroscience reference is used to illustrate the potential value of single-instance analysis alongside population methods. We acknowledge the absence of a concrete mapping to neural networks. In revision, we will expand this paragraph with a short note on how methods such as circuit analysis on individual models could serve an analogous role to single-case studies, while recognizing that population benchmarks remain central due to the statistical nature of training.","revision_made":"partial","referee_comment":"[Precedents from neuroscience / single-instance analysis paragraph] Discussion of the neuroscience precedent for single-instance analysis: the manuscript states that 'single cases can reveal what population studies miss' but supplies no mapping or example demonstrating how this would apply to trained neural networks, where population-level benchmarks are the dominant evaluation paradigm due to the statistical nature of learned parameters."},{"response":"The manuscript does not discuss the risk of rapid obsolescence in AI relative to slower-moving fields like agriculture. This is a substantive concern that merits direct engagement. We will add a dedicated paragraph to the infrastructure section acknowledging this challenge and outlining potential mitigations, such as maintaining versioned catalogues focused on general principles and failure modes rather than transient model specifics.","revision_made":"yes","referee_comment":"[Infrastructure discussion] Infrastructure foundation (catalogues of datasets, models, and findings): the proposal assumes such shared resources would enable cumulative progress, yet the text does not address or provide evidence against the risk that fast iteration cycles in AI would render catalogues obsolete faster than in agriculture, undermining the cumulative-knowledge goal."}],"tokens_in":1522,"tokens_out":585,"duration_ms":17769,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that this is a position paper arguing the AI field should consolidate model analysis efforts into a new discipline called Model Science, organized around Verify, Explore, Steer, and Refine perspectives, plus shared catalogs and single-instance studies.\n\nThe paper does a solid job naming real benchmark weaknesses: they track performance gains but leave out why models succeed or fail on specific cases, including shortcuts and hallucinations. It pulls together threads from XAI work and gives them a functional framing that could reduce duplication. The single-case angle draws a fair parallel to how neuroscience sometimes spots patterns that population averages miss.\n\nThe soft spot is the direct transfer assumption. The argument invokes cognitive science, neuroscience, medicine, and agriculture to justify the move, yet supplies no mapping of how those practices would adapt to fast-changing computational models or any pilot showing they would surface new failure modes. Shared infrastructure sounds useful in principle, but the paper does not address AI-specific issues like rapid obsolescence or the cost of maintaining model catalogs.\n\nThis is aimed at researchers already active in interpretability who want a higher-level organizational lens. Readers seeking new methods, code, or empirical tests will not find them. It deserves a serious referee because the benchmark critique is grounded in observed practice and the proposed structure is coherent enough to generate useful discussion, even if the analogies require more work to hold.","headline":"Position paper proposes 'Model Science' with four perspectives but rests on untested analogies without adaptation details or supporting evidence.","tokens_in":2329,"tokens_out":342,"would_cite":false,"duration_ms":18731,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"AI research should consolidate scattered model analysis into a new discipline called Model Science built on four perspectives: Verify, Explore, Steer, and Refine.","keywords":["Model Science","AI model analysis","benchmark limitations","model verification","model exploration","single case studies","AI infrastructure","explainable AI"],"falsifier":"A sustained effort to build the proposed catalogues and apply the four perspectives that produces no new explanations of model failures beyond existing benchmarks would undermine the case for Model Science.","tokens_in":2652,"feed_emoji":"","tokens_out":718,"duration_ms":26222,"temperature":0.7,"pith_summary":"The paper claims that benchmarks have driven progress but cannot explain why models succeed or fail and routinely miss critical issues such as hallucinations and shortcuts. It proposes moving to Model Science by adapting lessons from cognitive science on multiple levels of analysis, neuroscience on single-case depth, medicine on paired training and research, and agriculture on shared infrastructure. The resulting discipline rests on three foundations: the four functional perspectives that address complementary questions about model behaviour, catalogues of datasets models and findings for cumulative knowledge, and detailed study of individual model instances. A sympathetic reader would care because current methods leave deployed systems that serve billions of users poorly understood. The proposal treats these elements as ready to be assembled into systematic practice.","feed_headline":"AI needs Model Science to explain why models succeed or fail","feed_subtitle":"Benchmarks track performance but miss hallucinations and shortcuts; four perspectives plus single-case analysis and shared catalogues are pr","key_machinery":"The four functional perspectives Verify, Explore, Steer, and Refine that together address complementary questions about model behaviour and form one of the three foundations for Model Science.","core_discovery":"We argue that the AI community is now ready to move beyond benchmarking and consolidate scattered efforts in model analysis into a systematic discipline, a direction we term Model Science. Precedents from cognitive science, neuroscience, medicine, and agriculture show that complex systems require complementary levels of analysis, single-case depth, specialised training alongside research, and shared infrastructure. These lessons support three foundations: consolidation around the four perspectives Verify, Explore, Steer, and Refine; catalogues of datasets, models, and findings; and deep analysis of individual model instances rather than only model families.","pith_inferences":["The four perspectives could provide a common language for integrating existing scattered tools for model inspection and control.","Infrastructure for Model Science might extend to regulatory requirements that demand evidence from Verify and Steer activities before large-scale deployment.","Single-instance analysis could become routine for high-stakes applications where aggregated metrics are known to overlook rare but severe failure modes."],"forward_implications":["Benchmarks will be supplemented by methods that identify why models succeed or fail rather than only measuring performance.","Shared catalogues of datasets, models, and findings will enable cumulative progress instead of repeated isolated studies.","Deep analysis of single model instances will reveal patterns that population-level studies across model families miss.","Specialised training in model analysis will develop in parallel with research practice.","Complementary levels of analysis will become standard for understanding complex model behaviours."],"fun_headline_variants":["Model Science: Verify Explore Steer Refine","Single Case Analysis Reveals Model Behaviour","Shared Catalogues for Model Science Progress","Beyond Benchmarks: Four Perspectives on AI"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"That precedents and practices from cognitive science, neuroscience, medicine, and agriculture can be transferred directly to create a viable new discipline for AI models.","fun_headline_variants_meta":{"raw":{"variants":["Model Science: Verify Explore Steer Refine","Single Case Analysis Reveals Model Behaviour","Shared Catalogues for Model Science Progress","Beyond Benchmarks: Four Perspectives on AI"]},"model":"grok-4.3","cost_usd":0.007571,"raw_usage":{"total_tokens":3501,"prompt_tokens":730,"num_sources_used":0,"completion_tokens":49,"cost_in_usd_ticks":75712000,"prompt_tokens_details":{"text_tokens":730,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2722,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":730,"tokens_out":49,"duration_ms":25130,"temperature":1.0,"reasoning_tokens":2722,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T17:22:23.643988+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A sustained effort to build the proposed catalogues and apply the four perspectives that produces no new explanations of model failures beyond existing benchmarks would undermine the case for Model Science.","supporting_citations":[],"review_version":1}