{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2024:PUPJYTWHNK4YH2XR4CRTMRVH3C","short_pith_number":"pith:PUPJYTWH","schema_version":"1.0","canonical_sha256":"7d1e9c4ec76ab983eaf1e0a33646a7d89d2b7a62aa4da2c1d77daa6c8dcc3784","source":{"kind":"arxiv","id":"2410.03859","version":1},"attestation_state":"computed","paper":{"title":"SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":["cs.AI","cs.SE"],"primary_cat":"cs.CL","authors_text":"Alex L. Zhang, Carlos E. Jimenez, Diyi Yang, Gabriel Synnaeve, John Yang, Joyce Yang, Karthik R. Narasimhan, Kilian Lieret, Niklas Muennighoff, Ofir Press, Ori Press, Sida I. Wang, Xindi Wu","submitted_at":"2024-10-04T18:48:58Z","abstract_excerpt":"Autonomous systems for software engineering are now capable of fixing bugs and developing features. These systems are commonly evaluated on SWE-bench (Jimenez et al., 2024a), which assesses their ability to solve software issues from GitHub repositories. However, SWE-bench uses only Python repositories, with problem statements presented predominantly as text and lacking visual elements such as images. This limited coverage motivates our inquiry into how existing systems might perform on unrepresented software engineering domains (e.g., front-end, game development, DevOps), which use different "},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2410.03859","kind":"arxiv","version":1},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CL","submitted_at":"2024-10-04T18:48:58Z","cross_cats_sorted":["cs.AI","cs.SE"],"title_canon_sha256":"c3a1f05bc7fa97f4192e215ba6e61eb07efba5e9b829ffb58cd70572a62fffde","abstract_canon_sha256":"617e889cee4a78d4fe27f50ef296a1a21fd29707e0340c191e2692c3bcc25081"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T09:16:25.656444Z","signature_b64":"euHUtSKQxdtobhcqpJyuGXYIozOZh6Qz0KReumeT4H8K1sDAGp3BWyMhXmMyFhzykd758NLN5vmWNvwwA+pTDw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"7d1e9c4ec76ab983eaf1e0a33646a7d89d2b7a62aa4da2c1d77daa6c8dcc3784","last_reissued_at":"2026-07-05T09:16:25.655929Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T09:16:25.655929Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":["cs.AI","cs.SE"],"primary_cat":"cs.CL","authors_text":"Alex L. Zhang, Carlos E. Jimenez, Diyi Yang, Gabriel Synnaeve, John Yang, Joyce Yang, Karthik R. Narasimhan, Kilian Lieret, Niklas Muennighoff, Ofir Press, Ori Press, Sida I. Wang, Xindi Wu","submitted_at":"2024-10-04T18:48:58Z","abstract_excerpt":"Autonomous systems for software engineering are now capable of fixing bugs and developing features. These systems are commonly evaluated on SWE-bench (Jimenez et al., 2024a), which assesses their ability to solve software issues from GitHub repositories. However, SWE-bench uses only Python repositories, with problem statements presented predominantly as text and lacking visual elements such as images. This limited coverage motivates our inquiry into how existing systems might perform on unrepresented software engineering domains (e.g., front-end, game development, DevOps), which use different "},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2410.03859","kind":"arxiv","version":1},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2410.03859/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2410.03859","created_at":"2026-07-05T09:16:25.655994+00:00"},{"alias_kind":"arxiv_version","alias_value":"2410.03859v1","created_at":"2026-07-05T09:16:25.655994+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2410.03859","created_at":"2026-07-05T09:16:25.655994+00:00"},{"alias_kind":"pith_short_12","alias_value":"PUPJYTWHNK4Y","created_at":"2026-07-05T09:16:25.655994+00:00"},{"alias_kind":"pith_short_16","alias_value":"PUPJYTWHNK4YH2XR","created_at":"2026-07-05T09:16:25.655994+00:00"},{"alias_kind":"pith_short_8","alias_value":"PUPJYTWH","created_at":"2026-07-05T09:16:25.655994+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":29,"internal_anchor_count":0,"sample":[{"citing_arxiv_id":"2606.25514","citing_title":"Unlocking Model Potentials Through Adaptive Multi-Agent Scaffolding for Efficient Issue Resolution","ref_index":33,"is_internal_anchor":false},{"citing_arxiv_id":"2606.19613","citing_title":"StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns","ref_index":61,"is_internal_anchor":false},{"citing_arxiv_id":"2606.17799","citing_title":"Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering","ref_index":50,"is_internal_anchor":false},{"citing_arxiv_id":"2606.17454","citing_title":"Dissecting model behavior through agent trajectories","ref_index":54,"is_internal_anchor":false},{"citing_arxiv_id":"2606.12191","citing_title":"Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application","ref_index":146,"is_internal_anchor":false},{"citing_arxiv_id":"2606.18284","citing_title":"Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontier","ref_index":38,"is_internal_anchor":false},{"citing_arxiv_id":"2606.11470","citing_title":"The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes","ref_index":282,"is_internal_anchor":false},{"citing_arxiv_id":"2607.00053","citing_title":"SWE-Router: Routing in Multi-turn Agentic Software Engineering Tasks","ref_index":20,"is_internal_anchor":false},{"citing_arxiv_id":"2606.05920","citing_title":"Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement","ref_index":48,"is_internal_anchor":false},{"citing_arxiv_id":"2606.03889","citing_title":"RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions","ref_index":24,"is_internal_anchor":false},{"citing_arxiv_id":"2605.30690","citing_title":"ElasticMem: Latent Memory as a Learnable Resource for LLM Agents","ref_index":49,"is_internal_anchor":false},{"citing_arxiv_id":"2606.00832","citing_title":"Momento: Evaluating Persistent Memory and Reasoning with Multi-Session Agentic Conversations","ref_index":7,"is_internal_anchor":false},{"citing_arxiv_id":"2606.00750","citing_title":"I-WebGenBench : Evaluating Interactivity in LLM-Generated Scientific Web Applications","ref_index":46,"is_internal_anchor":false},{"citing_arxiv_id":"2606.10106","citing_title":"What makes a harness a harness: necessary and sufficient conditions for an agent harness","ref_index":59,"is_internal_anchor":false},{"citing_arxiv_id":"2605.20520","citing_title":"Open-World Evaluations for Measuring Frontier AI Capabilities","ref_index":26,"is_internal_anchor":false},{"citing_arxiv_id":"2506.02387","citing_title":"VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments","ref_index":82,"is_internal_anchor":false},{"citing_arxiv_id":"2512.18470","citing_title":"SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios","ref_index":60,"is_internal_anchor":false},{"citing_arxiv_id":"2601.11848","citing_title":"Compass vs Railway Tracks: Unpacking User Mental Models for Communicating Long-Horizon Work to Humans vs. AI","ref_index":89,"is_internal_anchor":false},{"citing_arxiv_id":"2507.15003","citing_title":"The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering","ref_index":34,"is_internal_anchor":false},{"citing_arxiv_id":"2502.10517","citing_title":"KernelBench: Can LLMs Write Efficient GPU Kernels?","ref_index":44,"is_internal_anchor":false},{"citing_arxiv_id":"2502.18449","citing_title":"SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution","ref_index":61,"is_internal_anchor":false},{"citing_arxiv_id":"2605.14415","citing_title":"SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades","ref_index":58,"is_internal_anchor":false},{"citing_arxiv_id":"2507.21046","citing_title":"A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence","ref_index":109,"is_internal_anchor":false},{"citing_arxiv_id":"2605.09423","citing_title":"SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning","ref_index":94,"is_internal_anchor":false},{"citing_arxiv_id":"2509.16941","citing_title":"SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?","ref_index":17,"is_internal_anchor":false}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/PUPJYTWHNK4YH2XR4CRTMRVH3C","json":"https://pith.science/pith/PUPJYTWHNK4YH2XR4CRTMRVH3C.json","graph_json":"https://pith.science/api/pith-number/PUPJYTWHNK4YH2XR4CRTMRVH3C/graph.json","events_json":"https://pith.science/api/pith-number/PUPJYTWHNK4YH2XR4CRTMRVH3C/events.json","paper":"https://pith.science/paper/PUPJYTWH"},"agent_actions":{"view_html":"https://pith.science/pith/PUPJYTWHNK4YH2XR4CRTMRVH3C","download_json":"https://pith.science/pith/PUPJYTWHNK4YH2XR4CRTMRVH3C.json","view_paper":"https://pith.science/paper/PUPJYTWH","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2410.03859&json=true","fetch_graph":"https://pith.science/api/pith-number/PUPJYTWHNK4YH2XR4CRTMRVH3C/graph.json","fetch_events":"https://pith.science/api/pith-number/PUPJYTWHNK4YH2XR4CRTMRVH3C/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/PUPJYTWHNK4YH2XR4CRTMRVH3C/action/timestamp_anchor","attest_storage":"https://pith.science/pith/PUPJYTWHNK4YH2XR4CRTMRVH3C/action/storage_attestation","attest_author":"https://pith.science/pith/PUPJYTWHNK4YH2XR4CRTMRVH3C/action/author_attestation","sign_citation":"https://pith.science/pith/PUPJYTWHNK4YH2XR4CRTMRVH3C/action/citation_signature","submit_replication":"https://pith.science/pith/PUPJYTWHNK4YH2XR4CRTMRVH3C/action/replication_record"}},"created_at":"2026-07-05T09:16:25.655994+00:00","updated_at":"2026-07-05T09:16:25.655994+00:00"}