Strongest AI agent verifies only 27 of 43 code repositories
Passing 87 percent of individual proofs is not enough; whole-repo proofs stall without shared lemma libraries.
· “Vero: Can AI Agents Build Formally Verified Software Repositories?”
cs.SE
Software Engineering
Covers design tools, software metrics, testing and debugging, programming environments, etc. Roughly includes material in all of ACM Subject Classes D.2, except that D.2.4 (program verification) should probably have Logics in Computer Science as the primary subject area.
sort pith recommended most recent
1,646 vulnerability records show fixes span multiple patches
A new taxonomy of why patches fail and why current detection tools score below chance at spotting incomplete fixes.
Code-only flaky-test detectors fall to chance once shortcuts go
High F1 scores rested on near-duplicate labels and project overlap, not flakiness signal in the source.
· “How Far Are We from Detecting Flaky Tests? On the Limits of Code-Based Detection”
Runtime monitor holds AI agent GDPR violations to 12 percent or less
By checking every tool call against formal predicates for consent and data rules, the system works even when trace extraction contains 10 pe
Bidirectional analysis finds 118 unsafe flows in 87 MCP servers
Protocol-specific static analysis reveals recurring data-flow risks missed by existing tools across 15k real-world repositories.
· “Unsafe by Flow: Uncovering Bidirectional Data-Flow Risks in MCP Ecosystem”
Neuro-symbolic method detects threats in stripped industrial binaries
Lifting semantics into knowledge graphs improves CVE coverage and cuts false positives on real ICS hardware from multiple vendors.
Iterative checks boost LLM quantum solver success
Stronger models still produce numerically inaccurate outputs instead of crashes, limiting direct use for science.
· “Can LLMs Solve Science or Just Write Code? Evaluating Quantum Solver Generation”
Prefill signals from small LLMs locate root failures in agent traces
Two prefill passes on a 0.6B model rank failure sources in long multi-agent logs with no output tokens and seconds of latency.
· “MASPrism: Lightweight Failure Attribution for Multi-Agent Systems Using Prefill-Stage Signals”
Unrestricted autonomy breaks LLM test repair in enterprise UIs
Case study of hundreds of runs shows full freedom yields misleading fixes like weakened assertions, while constraints enable practical use.
Nygard's ADR template outperforms MADR in student usability test
Controlled experiment shows higher overall scores for concise documentation over structured detail capture in architectural decision records
SWE-TRACE lifts agent success rates while cutting tokens and latency
Shortest-path distillation plus rubric rewards used for both training and step-wise pruning let agents resolve more issues with less compute
LLM generates properties to catch Android app bugs
Identifying 1,210 functionalities and 25 unknown issues across 12 apps by synthesizing and refining specs from observed behaviors
· “From Exploration to Specification: LLM-Based Property Generation for Mobile App Testing”
Benchmark rates security agents at 97% true positive detection
SIR-Bench replays real incidents to test if agents discover new evidence or merely echo alerts.
· “SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents”
Bug references lift LLM accuracy on MPI errors to 77 percent
Adding references and structured prompts raises detection rates well above direct ChatGPT use
· “Improving MPI Error Detection and Repair with Large Language Models and Bug References”
Visual feedback raises GUI code success rate from 22% to 28%
VF-Coder lets models see rendered interfaces and simulate clicks to fix logic and layout errors text feedback misses.
· “Coding with Eyes: Visual Feedback Unlocks Reliable GUI Code Generating and Debugging”
LLM code patches show unstable non-functional quality
Literature, workshops, and real-issue tests reveal gaps between academic focus, practitioner priorities, and model outputs.
· “Quality Assurance of LLM-generated Code: Addressing Non-Functional Quality Characteristics”
Tool merging and retrieval lifts LLM tool accuracy by up to 38%
Reducing overlapping tools and picking only relevant ones for each query improves selection on standard benchmarks.
· “ToolScope: Enhancing LLM Agent Tool Use through Tool Merging and Context-Aware Filtering”
New benchmark tests AI agents on 1,865 enterprise software tasks
SWE-Bench Pro draws long-horizon problems from 41 repositories, including proprietary ones, to measure progress toward autonomous code work.
· “SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?”
Prompt reflection beats RL with 35x fewer rollouts
GEPA evolves stronger LLM prompts by reflecting on a handful of trajectories instead of many scalar rewards.
· “GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning”
Property checks raise LLM code fix rates over tests
A solver that verifies high-level properties and returns the smallest failing example lifts success by up to 13 percent.
· “Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback”
Models reach 13.8% on executable state changes in Scratch tests
Benchmark uses verified VM transitions to separate rule-following from copied persistent state.
· “ScratchWorld: Evaluating If World Models Compute Executable Consequences”
Refusal rate misranks LLMs on bio safety
Matched prompts show top risk discriminators often refuse fewer queries than high-refusal peers.
· “RefusalBench: Why Refusal Rate Misranks Frontier LLMs on Biological Research Prompts”
AI agents solve at most 39% of real version upgrade tasks
New benchmark of 115 multi-file changes from actual projects shows sharp drop from simpler bug-fix results.
· “RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades”
Type system refuses statistical claims a study can't afford
What an analysis read and how its data nests are recovered from the program, not the p-values.
· “Tacet: A Language and Type System for Automatic Statistical Validity Accounting”
Curated 10% of coding trajectories beats the full dataset
SWE-Prime's two-stage quality filter lifts resolution rates by up to 12.2% and 24.2% on the two SWE-Bench sets.
Free persona drift and audited execution need separate trust domains
A new pattern argues that any single-domain fix must rebuild typed-change gating, an approval point, and an audit anchor at higher cost.
Telemetry logs yield a replayable 532-row agent benchmark
A fixed pipeline turns building telemetry into agent tasks whose rebuild matches the release byte-for-byte.
Tool-level attacks become user-authorized in all six harnesses
Delegation, goals, and scheduled tasks relabel attacker content as user intent, defeating both the model and permission review.
· “When Context Gets Root: Privilege Escalation in LLM Harnesses”
This paper presents DeepRepro
State-aware iterative subplanning in DeepRepro outperforms static-planning coding agents on PaperBench Code-Dev paper-to-code reproduction…
· “DeepRepro: State-Aware Subplanning for Paper-to-Code Reproduction in Evolving Repositories”
Renaming identifiers sabotages code search: rankings drop 77 points
An attack built on a small open model transfers to closed-source embedders and LLMs.
· “Vulnerable Code Search: Transferable Attack for Code Language Models”
Lineage-first agent clears 80% MLE-bench tasks at 1/12 cost
Across 75 tasks it scores 60 medals, 49 gold, versus 55 and 34 for Claude Code, with auditable lineages.
· “Praxist: From Experimental Artifacts to Solution Lineages”
Best LLMs pass just 27% of repository unit tests
A five-language benchmark shows standalone test scores overstate real-world readiness.
AI that shows its work beats baselines in vulnerability scoring
EAVA trains a small LLM on reasoning traces and RL; analysts rated its evidence useful in most cases.
· “Answer Is Cheap, Show Me the Evidence! Augmenting Automated Vulnerability Assessment with Evidence”
Same Office file: one truth in Excel
Even specification-valid documents can show a benign value in Office while an extractor feeds a planted trap fact to the model.
· “Beyond the Editing Canvas: Evidence Divergence in OOXML-to-LLM Ingestion”
99% of internal Dockerfiles harbor security flaws
The fix already exists: most workload clusters contain a secure reference; copying it lifts scores 60.4%.
Keystroke logs flag stuck students before the first submit
Editing pauses and volume beat execution errors at the earliest stage, boosting AUROC to 0.674.
· “Predicting Struggling Students in CS1 Programming Using Keystroke-Level Editing Features”
Cloud docs alone can synthesize emulators that beat hand-built rivals
A neurosymbolic pipeline reads provider documentation, generates state-machine code, and aligns it against the real cloud.
Late Shopify apps outpaced early movers in 88% of categories
Seven-year panel shows platform governance, not entry timing, decides who grows and who silently fails.
UEFI patch-status tool: zero confident misclassifications on 20 pairs
Reports Vulnerable, Already Patched, or Uncertain with reviewable evidence for prepared firmware targets.
New UEFI bug-report fields flag architecture and regressions
A 92-issue study finds architecture and regression cues are often missing from free text, so EDK II's template would demand them.
· “Enhancing Bug Report Templates in the TianoCore UEFI Firmware Development Community”
Coordination cuts AI coders' task abandonment odds 13.7×
Two claim-coordinated agents on a shared CRDT workspace beat solo and naive-parallel runs at equal compute.
· “AgentRoom: Concurrent Multi-Agent Coding in a CRDT-Backed Shared Workspace”
Coding agents fail 94.6% of whole-repo migrations
A three-stage audit of 520 runs shows tests alone cannot tell a migration from a copy.
· “SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?”
A smarter harness lifts ARC-AGI-3 from 30% to 95.5%
Persistent code, recursive subagents, and saved skills let a fixed model turn extra compute into verified progress.
Agent that questions before searching recalls 1.9x more people
90% of its returned candidates are missed by all 20 query-driven baselines combined.
· “An Interactive Agent for Requirement-Driven Candidate Sourcing”
No surface cue flags an on-device model's confident errors
An audit shows confident-wrong and confident-correct outputs look identical; only multi-sample consistency catches the errors.
Middle-size agent splits lead
Two- and three-worker splits hit 0.830 on VAT cases vs 0.720 and 0.770 at the extremes; one bad record degrades every setup.
· “Right-Sizing LLM-Agent Decomposition in VAT Determination: A Pilot Controlled Sweep”
Fine-grained code moves across methods get automatable rules
Formal preconditions give 93.3–97.0% compile success; side-effect order stays the developer's call.
· “Formalizing and Automating Fine-Grained Move Refactorings Across Methods”
Split explore from write
Forcing a written diagnosis before test creation lifts SWT-Bench Verified results to 81.76% and cuts agent steps.
· “DPIAgent: Divide, Protocol, Isolate for Agentic Reproduction Test Generation”
One auditable design spec per web reference
A 19-section contract labels every design claim as measured, observed, inferred, or unknown so guesses stay visible.
In-car policies compile into checks that gate every draft move
Compiles policy text into rules, simulates the draft's effects, catching hallucinated IDs and missed side-effects.
UEFI firmware teams rank security first
A 13-person TianoCore study finds manual triage, patching, and supply-chain checks despite high awareness.
UEFI bug triage: 11 days to 7 minutes
An LLM pipeline handles validity, duplicates, priority, and assignment for EDK II, if maintainers trust its output.
· “TianoForge: An Automated Bug Triage Approach for the TianoCore UEFI Firmware Development Community”
LLMs now write FAIR metadata at every research stage
Metadata grows from design to provenance, and every generated crate still passes RO-Crate 1.1 validation.
· “LLMCrater: Lifecycle-Aware FAIR Metadata Generation using Large Language Models”
Two legacy science apps became reusable cloud workflows
An LLM guided by architecture reviews and dependency matrices kept each app's outputs identical to the originals.
Harness design shifts Modelica pass rates when the LLM is fixed
Pufibara passed 202 of 232 Modelica tasks versus 185 and 187 for Claude Code, using 76–82% fewer logical tokens.
Metric feedback loop cuts duplicate code in research notebooks
Six static-analysis metrics, fed back over five refinement rounds, drive measurable structural gains and expose quality trade-offs.
A five-viewpoint schema predicts integration mismatches before build
LLM-assisted reasoning on the same metadata almost triples detection F1 over static rule checks.
Agent finds correct Kubernetes root cause in all 10 test incidents
An MCP-grounded assistant names the right root cause every time, yet engineers doubt its fixes.
· “ARGUS: MCP-Grounded Root Cause Analysis for Kubernetes Incidents”
Appending failed calls backfires: repeat odds rise 2.8x
The verbatim call, not the error message, drives the damage; a runtime description instead removes 76% of it.
· “Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail”
Microservice timing is emergent: component guarantees won't compose
Local delays in service-based vehicles spread through shared resources and control loops that timing models overlook.
· “A Survey of Timing Variability in Microservice-Based Software-Defined Vehicles”
Automatic harness tuning lifts agent pass rates by up to 10 points
Patches prompts, tools, and control logic from failure traces, beating hand-tuned harnesses with far fewer rollouts.
· “AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces”
Line-level edits stop AI drafts from silently changing facts
A compile-and-evidence gate kept an untouched sentence intact in 192/192 tests where whole-slot rewrites changed it every time.
Fairness hazard analysis passes real-world hiring test
Two case studies show FHA surfaces recurring bias patterns that HR practitioners call realistic and mostly fixable.
Verilog reranking lifts Pass@1 by 11–15 points
Independent execution and reasoning signals, fused only at decision time, recover 60% of the oracle gap.
· “Execution-Anchored Hallucination Calibration Reranking for Verilog Code Generation”
Agentic AI coders can run unrestricted inside a four-safeguard sandbox
An untrusted coding agent is quarantined in a container, and every commit is human-gated before it reaches the repo.
· “Concepts for Securing Agentic AI Coding and the Terok Environment”
Standard-library imports inflate LLM hallucination rates by 9.4 points
Correcting for Python's built-in modules like os and math changes both hallucination numbers and defense rankings.
· “Evaluating Inference-Time Defenses Against Package Hallucination in LLM-Generated Code”
Invisible characters block AI code copy-paste up to 99.7% of the time
A system-prompt-only barrier disrupts unexamined transfer of generated code, but no single trick works across models.
· “Do Not Copy/Paste: Soft Barriers for Copying in AI-Assisted Programming”
Passing tests says little about code quality in LLM output
Across 340 C# solutions from four LLMs, correctness and quality correlate at just 0.075—Pass@k alone misleads.
No tool-calling model tops 0.60 under output noise
New stage-wise benchmark traces failures to their source and finds evidence handling is the real bottleneck.
· “ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents”
Function-level feedback beats whole-program fine-tuning on code
Automated per-function tests give code models local credit; hardest benchmarks gain up to 11.3% over KTO.
· “Function-Level Execution Feedback for Code Preference Optimization”
FPGAgent lifts LLM FPGA code correctness by 30.6 points
A three-agent loop verifies each kernel on real FPGA hardware, not just simulation, making LLM-written code deployable.
A model export shifted 12% of alerts while mean score barely moved
Paper argues serving-format conversion is a model change until decisions are re-checked at the deployed threshold.
Learned code-graph spectra lift clone-detection F1 from 0.37 to 0.67
Only the graph changes, and it degrades least when languages shift.
Tree-of-thought repair fixes 62% of real smart-contract bugs
Multi-branch reasoning plus static analysis beats linear chain-of-thought by 12 points on 50 real vulnerabilities
A rival trained on its own logs finds an RL agent's suboptimal moves
The agent's safety-testing history trains the rival; every state where it scores higher is flagged as a bug.
· “Learning from the Test: Self-Referential Differential Testing for Deep RL Agents”
Fail-stop guards beat open-ended LLM repair on real bugs
CodeMechanic turns a memory corruption into a controlled stop, with more correct patches at about a tenth of the token cost.
Editable 3-way diff halves time to fix commit boundaries
In an 8-participant study, median adjustment fell from 530 to 230 seconds with accuracy unchanged.
· “D-Diff: An Interactive Environment for Adjusting Commit Boundaries Based on an Editable 3-way Diff”
Router traces alone match text-consensus patch selection
Routing-only arbitration lifts SWE-bench resolved rate from 44.9% to 48.2%, matching consensus without reading patch text.
· “Disagree to Explore, Agree to Commit: Routing-Guided Test-Time Scaling for Software Agents”
AI makes views cheap
Ten-year software-visualization revisit: generation is solved, perception is the bottleneck.
· “Towards Actionable Visualization: Ten Years Later, What Generative AI Changes and What It Cannot”
Bounded DEVS models get simulation and model checking in one tool
A state exploration protocol reuses the simulation engine to build a full reachability graph, so no model conversion is needed.
· “Constraint-Driven Modeling Enabling Dual Model Checking and Simulation for Discrete Event Systems”
