REVIEW 14 cited by
Archon: An Architecture Search Framework for Inference-Time Techniques
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Inference-time techniques, such as repeated sampling or iterative revisions, are emerging as powerful ways to enhance large-language models (LLMs) at test time. However, best practices for developing systems that combine these techniques remain underdeveloped due to our limited understanding of the utility of each technique across models and tasks, the interactions between them, and the massive search space for combining them. To address these challenges, we introduce Archon, a modular and automated framework for optimizing the process of selecting and combining inference-time techniques and LLMs. Given a compute budget and a set of available LLMs, Archon explores a large design space to discover optimized configurations tailored to target benchmarks. It can design custom or general-purpose architectures that advance the Pareto frontier of accuracy vs. maximum token budget compared to top-performing baselines. Across instruction-following, reasoning, and coding tasks, we show that Archon can leverage additional inference compute budget to design systems that outperform frontier models such as OpenAI's o1, GPT-4o, and Claude 3.5 Sonnet by an average of 15.1%.
Forward citations
Cited by 14 Pith papers
-
AoA: Theorem Proving Agent over Abstract Syntax Tree of Redesigned Language
AoA proves theorems by editing a JSON-AST proof tree for the new Minilang language, reporting 2.9–6.9x fewer tokens and 2.3–4.7x lower API cost than Amazon's Isabelle agent with equal or better pass rates.
-
Training Language Models to Cooperate with Inference-Time Controllers
Multi-controller post-training with turn-level GRPO over reusable reasoning modules improves LLM transfer to held-out and shifted inference-time controllers versus single-controller RL.
-
A Workflow-Aware Serving Layer for Agentic Applications
A workflow-aware serving layer compiles per-node model-verifier-backend plans with an ILP and adapts only uncommitted work via pre-solved pressure rungs and residual re-solves.
-
Beyond Tier Labels: Role- and Deployment-Dependent Model Substitution in Multi-Call LLM Workflows
Model substitution value in multi-call LLM workflows is deployment-conditioned: the same medium-to-strong swap helps, hurts, or reverses depending on role, scale, pool, and task.
-
ADIAS: Automated Design of Interactive Agentic Systems
Persistently tracking repair issues instead of only candidate performance improves automated full-code AI agent design by 25.2% on average over five benchmarks.
-
Automated Discovery Has No Universally Superior Harness
No fixed discovery harness is reliably superior across 12 model–problem pairs, OpenEvolve-style recipes underperform simpler alternatives, and online pruning of weak partial runs improves budget-matched performance.
-
RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models
RoboMonkey shows that test-time sampling with Gaussian perturbation and a VLM-based action verifier improves the success rate of vision-language-action models on manipulation tasks.
-
Helpful Agent Meets Deceptive Judge: Understanding Vulnerabilities in Agentic Workflows
LLM agents frequently switch correct answers after one round of misleading feedback, and the new WAFER-QA benchmark measures this with web-backed critiques.
-
How Much Backtracking is Enough? Exploring the Interplay of SFT and RL in Enhancing LLM Reasoning
On three controlled tasks, the optimal number of backtracks in SFT warm-up data grows with task difficulty, while trajectory correctness has little effect on final RL performance.
-
Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks
Agents that accumulate their own successful trajectories as retrieval examples gain up to 20 points on ALFWorld, Wordcraft, and InterCode-SQL, with two curation methods pushing gains further.
-
COSMOS: Predictable and Cost-Effective Adaptation of LLMs
COSMOS predicts the performance and cost of QLoRA fine-tuning and retrieval-augmented in-context learning across LLMs, achieving 1.09% mean absolute error and 92.72% average cost reduction on eight benchmarks.
-
ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization
ScoreFlow uses a score-weighted variant of direct preference optimization to automatically generate and refine per-task LLM agent workflows, reporting an average 8.2% improvement over baselines on six benchmarks.
-
Introspection of Thought Helps AI Agents
INoT wraps prompts in XML-defined pseudo-code so an LLM simulates two debating agents internally, reporting better scores and lower tokens than seven baselines.
-
FlowReasoner: Reinforcing Query-Level Meta-Agents
FlowReasoner, a reasoning meta-agent trained by R1 distillation and reinforcement learning, generates a query-specific multi-agent workflow per query and improves code benchmark accuracy over fixed workflows and direc...
Discussion (0). Continue with ORCID to comment.