{"paper":{"title":"SUREON: A Benchmark and Vision-Language Model for Surgical Reasoning","arxiv_id":"2603.06570","primary_cat":"cs.CV","submitted_at":"2026-03-06T18:58:36+00:00","version_compiled":"v2"},"notice":"Every quoted sentence is the authors' text from the arXiv listing abstract. Labels and probabilities are the outputs of classifier-1.13.0 on the question bank abstract-p1-20260923, exactly as returned. No sentence in this record was written by a language model.","schema":"pith.compiled-record/v0","status":"compiled","abstract":{"source":"arXiv listing abstract v2","sentences":[{"i":0,"p":0.53,"id":"s0","own":false,"role":"background","text":"A capable surgical AI model should reason from visual evidence; yet, many existing surgical benchmarks can be partially solved without visual input, as models may exploit procedural priors or common anatomy-instrument interactions to infer answers from language alone.","hedge":{"p":0.63,"label":"asserted"},"own_p":0.07,"main_claim_p":0.04,"quantitative":false,"quantitative_p":0.06,"role_probabilities":{"gap":0.47,"method":0.0,"result":0.0,"background":0.53,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":1,"p":1.0,"id":"s1","own":true,"role":"contribution","text":"To address this, we introduce SUREON, a video question-answering (QA) dataset and benchmark designed to evaluate visually grounded surgical reasoning.","hedge":{"p":0.96,"label":"asserted"},"own_p":0.89,"main_claim_p":0.06,"quantitative":false,"quantitative_p":0.03,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":1.0,"interpretation":0.0}},{"i":2,"p":0.99,"id":"s2","own":true,"role":"method","text":"Our pipeline generates QA pairs from narrated surgical academic videos, refines incorrect options using multiple models to reduce linguistic biases, and uses trained adversarial guessers to filter out questions answerable without visual evidence.","hedge":{"p":0.98,"label":"asserted"},"own_p":0.96,"main_claim_p":0.04,"quantitative":false,"quantitative_p":0.03,"role_probabilities":{"gap":0.0,"method":0.99,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.01,"interpretation":0.0}},{"i":3,"p":0.68,"id":"s3","own":true,"role":"contribution","text":"The resulting dataset contains 76K training pairs covering 168 procedures and 10 specialties, alongside a benchmark of 340 questions across 10 categories verified by expert surgeons.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.96,"main_claim_p":0.05,"quantitative":true,"quantitative_p":0.99,"role_probabilities":{"gap":0.0,"method":0.14,"result":0.18,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.68,"interpretation":0.0}},{"i":4,"p":0.99,"id":"s4","own":true,"role":"result","text":"Benchmarking leading commercial and open-source models shows that surgical visual reasoning remains largely unsolved--no model exceeds 53% accuracy and even expert surgeons reach only 64.9%.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.94,"main_claim_p":0.32,"quantitative":true,"quantitative_p":0.99,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.99,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.01}},{"i":5,"p":0.84,"id":"s5","own":true,"role":"contribution","text":"We also introduce SUREON-VLM, a baseline trained on our curated data that avoids language shortcuts, outperforms existing open-source models on SUREON, and matches larger specialist models on multiple standard surgical benchmarks.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.96,"main_claim_p":0.21,"quantitative":true,"quantitative_p":0.83,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.16,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.84,"interpretation":0.0}},{"i":6,"p":0.96,"id":"s6","own":true,"role":"result","text":"Our training ablations show that distractor refinement and blind curation strengthen visual grounding, increasing the multimodal gain from 8.7 to 29.1 percentage points while reducing the training set from 176.6K to 76K filtered samples.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.97,"main_claim_p":0.09,"quantitative":true,"quantitative_p":0.99,"role_probabilities":{"gap":0.0,"method":0.01,"result":0.96,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.01,"interpretation":0.02}},{"i":7,"p":0.52,"id":"s7","own":true,"role":"interpretation","text":"SUREON thus offers both a verified testbed for surgical visual reasoning and training data that strengthens visual grounding.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.81,"main_claim_p":0.78,"quantitative":false,"quantitative_p":0.03,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.48,"interpretation":0.52}}]},"compiled":{"main_claim":{"p":0.89,"text":"SUREON thus offers both a verified testbed for surgical visual reasoning and training data that strengthens visual grounding.","location":"abstract s7","hedge_strength":{"p":1.0,"label":"asserted"}},"provenance":{"route":{"via":"TypeSafe API","model":"classifier-1.13.0","endpoint":"https://api.typesafe.ai/v1/systemone","provider":"TypeSafe"},"packing":"one request per paper","requests":1,"input_tokens":6503,"cost_usd_reported":0.0},"compiled_at":"2026-11-02","bank_version":"abstract-p1-20260923","model_version":"classifier-1.13.0","abstract_states":{"claims_first":0.1,"states_limitation":0.51,"human_participants":0.6,"controlled_comparison":0.73,"randomized_assignment":0.01,"states_quantitative_result":0.99,"compares_with_prior_results":0.97,"mentions_code_or_data_availability":0.11},"evidential_basis":{"p":0.98,"value":"observation or experiment","probabilities":{"argument":0.0,"literature":0.0,"proof_or_derivation":0.0,"analysis_of_existing_data":0.0,"observation_or_experiment":0.98,"simulation_or_computation":0.02}},"contribution_type":{"p":1.0,"value":"dataset or benchmark","probabilities":{"review":0.0,"method_or_tool":0.0,"empirical_study":0.0,"simulation_study":0.0,"theoretical_result":0.0,"dataset_or_benchmark":1.0,"position_or_commentary":0.0}},"contribution_statement":{"p":0.95,"text":"To address this, we introduce SUREON, a video question-answering (QA) dataset and benchmark designed to evaluate visually grounded surgical reasoning.","location":"abstract s1"}},"coverage":{"tier":"abstract","levels_run":["P abstract"],"not_examined":["full text","figures and tables","references and citing sentences","relations to other papers"],"sources_read":["arXiv listing abstract v2"],"levels_not_run":["S sections","T body sentences","C citations","R cross-paper relations"],"sentences_examined":8,"sentences_in_abstract":8},"measured_accuracy":{"url":"https://pith.science/accuracy","measured_on":"2026-10-02","method":"two independent model families labelled a fresh random sample blind; the figure is how often the published label matches their consensus","sentence_role_agreement":0.92,"main_claim_agreement":0.908,"typed_citation_agreement":0.933,"typed_citation_agreement_95_interval":[0.917,0.945],"review_comments_accurate":0.758,"review_comments_accurate_95_interval":[0.684,0.82],"review_caught_admitted_error_found_or_related":0.448,"review_comments_judged":{"judged_by":["gpt-5.6-luna","grok-4.7"],"judged_against":"Pith's own text copy of each paper: text extracted from the PDF, the first 90,000 characters","measured_on":"2026-10-04","accurate":113,"decided":149,"sampled_reviews":200,"split_by_lane":false,"by_lane":"https://pith.science/accuracy#by-lane"}}}