{"id":"2b5fb2d6-2706-4217-905a-53af329b9a4d","arxiv_id":"2412.09612","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Olympus is a trained MLLM router that delegates 20 vision tasks to specialist models and supports chain-of-action execution of up to five tasks per instruction.","lead":"Olympus trains a small multimodal language model to act as a task router, reading user requests and sending work to specialist models for image, video, and 3D tasks. It reports over 94 percent routing accuracy on 20 tasks and can chain up to five tasks from one instruction, offering a modular alternative to monolithic all-in-one models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OlympusBench is generated from the same GPT-4o prompt templates as OlympusInstruct; the 94.75% routing accuracy is an in-distribution score, and the 200-request human evaluation is the only out-of-distribution evidence.","rationale":"The method is technically plausible and the paper includes useful ablations (Tables 5-7, 15-17) showing that prompt components affect the human-eval success rate, which is a genuine attempt at OOD testing. However, the strongest numerical claims (94.75% routing accuracy, 91.82% chain precision) are measured on OlympusBench, which is generated from the same GPT-4o prompt templates as the training set, with the same prefixes, phrases, and complexity definitions. This makes in-distribution memorization a live alternative explanation for the reported performance. The 200-request human evaluation is too small and reports only success rate, not the routing metrics of Tables 2-3, so it cannot fully support the benchmark's external validity. The chain-of-action split additionally concatenates random independent tasks, so the 91.82% precision does not demonstrate sequential multi-task planning. A TF-IDF classifier baseline would decisively show whether the benchmark is saturated by surface lexical cues. Since the reader already conditioned acceptance on releasing artifacts and adding external evaluation, my stress test does not change the verdict; it sharpens the condition.","tokens_in":24636,"tokens_out":7970,"duration_ms":86361,"concrete_test":"Train a TF-IDF logistic-regression classifier on OlympusInstruct instruction texts (ignoring images) and evaluate on OlympusBench. If this text-only baseline reaches within 5 points of Olympus's 94.75% accuracy, the benchmark is dominated by template-specific lexical cues, and the routing numbers must be re-measured on out-of-template instructions (e.g., human-written requests or GPT-4o prompts with held-out prefixes/phrases) before claiming universal routing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sections 3.1 and 4.2 report 94.75% single-task accuracy and 91.82% chain precision on OlympusBench. But OlympusBench is produced with the same task-specific GPT-4o prompts as OlympusInstruct, including the same 7 prefixes, 12 phrases, 9 example pairs, and short/moderate/extended complexity levels (Figure 4, Appendix E). The 446K training examples let Olympus memorize template-specific lexical cues, so its high score reflects in-distribution classification rather than general routing. HuggingGPT, evaluated zero-shot, has not seen these templates, making the comparison a measure of template familiarity, not universal routing. The 200 human instructions (Table 4) are the only OOD test; the 86.5% success rate, below the 94.75% benchmark score and conflating routing with specialist success, does not resolve this gap. Additionally, chain-of-action evaluation concatenates random independent task samples (Appendix E), so 91.82% precision does not measure sequential dependency handling. The central claim of a universal router therefore rests on an in-distribution benchmark.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Olympus, a framework that turns a small multimodal large language model (MLLM) into a task router for computer vision. A controller MLLM, initialized from Mipha-3B, is fine-tuned on 446.3K GPT-4o-generated instructions (OlympusInstruct) to emit task-specific routing tokens, and at inference time these tokens invoke specialist models for image, video, and 3D tasks. The model is also trained and evaluated on chain-of-action settings, where a single user instruction triggers up to five consecutive routing steps. The authors report an average routing accuracy of 94.75% and chain precision of 91.82% on their own OlympusBench, a 86.5% success rate on 200 human-generated instructions, and comparable performance to Mipha-3B on 11 standard multimodal benchmarks.","tokens_in":24762,"tokens_out":6197,"duration_ms":64593,"significance":"If the routing results are robust, Olympus offers a practical modular alternative to all-in-one generative models: a 3B MLLM can coordinate specialized models for more than 20 vision tasks while retaining standard MLLM performance. The paper's strengths include the scale of the collected instruction data, the use of routing tokens to obtain chain-of-action behavior, the sanity check on standard multimodal benchmarks (Table 1), and the attempt to evaluate on human-written instructions. The main risk is that the central routing claim rests on a benchmark generated from the same GPT-4o prompt template family as the training data; the 200-request human study is the only explicit out-of-distribution evidence, and it conflates routing with downstream specialist success. The chain-of-action evaluation also concatenates independent random task samples rather than testing true sequential dependencies. These issues are load-bearing for the paper's central claims but appear addressable with additional evaluation, so a major revision is warranted.","major_comments":[{"comment":"The evaluation set OlympusBench is generated with the same task-specific GPT-4o prompt scheme as OlympusInstruct, including the same prefixes, phrases, example pairs, and short/moderate/extended complexity levels described in Figure 4 and Appendix E. Tables 2 and 3 therefore primarily measure how well Olympus imitates the template distribution used to create both training and test data; the comparison against prompted HuggingGPT reflects template familiarity as much as routing skill. This is load-bearing for the 'universal task router' claim. Please add an out-of-distribution evaluation (e.g., a benchmark produced by a different generation procedure, or real user requests) and report routing accuracy on that set, and provide per-task or stratified results on OlympusBench. If the evaluation prompts actually differed from the training prompts, the paper should state this explicitly and include the evaluation prompt texts.","section":"§3.1, §4.2, Appendix E"},{"comment":"The chain-of-action evaluation is constructed by selecting N random tasks and concatenating one sample from each task. Such chains contain no sequential dependencies: no later instruction refers to the output of an earlier routing step, and downstream failure cannot propagate. Consequently, the 91.82% precision and 0.18 edit distance in Table 3 measure the ability to emit several independent routing tokens in order, not the ability to plan and execute multi-step workflows. Please construct chain evaluations with genuine inter-step dependencies (e.g., 'then add books near the dog' after an image generation step) and report whether intermediate specialist outputs are actually consumed by later steps.","section":"§4.2, Appendix E (Chain-of-Action Samples)"},{"comment":"The human evaluation 'success rate' conflates routing quality with the fidelity of specialist models: a request can fail because the routed specialist cannot satisfy it even when the routing decision is correct. Since Table 4 is the main out-of-distribution evidence, the paper should also report routing-only accuracy on the 200 human instructions (e.g., comparing predicted routing tokens against gold task labels), together with confidence intervals or annotator agreement. The 200-sample size gives a margin of error of roughly ±4-5 percentage points at 95% confidence when the success rate is near 86.5%, so point estimates alone do not support fine-grained comparisons.","section":"Table 4, §4.2"},{"comment":"The headline '94.75% across 20 tasks' is reported only as a pooled aggregate. Table 10 shows large variation in evaluation sample sizes across tasks (e.g., 5,000 Image Generation samples versus 700 Image Deblurring samples), so a pooled accuracy can be dominated by high-frequency tasks and does not demonstrate uniform routing ability 'across 20 tasks.' Please report per-task accuracy and a confusion matrix, and clarify whether 94.75% is a pooled or macro-averaged number.","section":"Tables 2 and 10, §4.2"}],"minor_comments":[{"comment":"There are several typos and formatting slips that should be corrected: 'concreate' in Figure 4, 'autogressive' in §2.2, 'Image Denosing' in Figure 8, and 'MLLMs Traning' in Table 8.","section":"Throughout"},{"comment":"The paper says '20 tasks' but the routing token list in Figure 8 and the dataset table count controllable image/video generation conditions as separate token pairs; please clarify how tasks are counted (e.g., whether the six control conditions are grouped as one task or counted individually).","section":"§3.1 and Figure 8"},{"comment":"The statement 'we included prompts covering all task types supported by Olympus and excluded prompts for irrelevant tasks' should specify how the subset was chosen for HuggingGPT and whether exactly the same subset is used for Olympus; otherwise the comparison is not fully reproducible.","section":"§4.2"},{"comment":"The paper reports 200 human-generated instructions but gives no details on how they were collected, who the evaluators were, or the scoring rubric used for the success rate; these details are needed to interpret Table 4.","section":"§4.2 and Dataset Statistic"},{"comment":"The limitation section should also acknowledge the benchmark/training distribution overlap and the random-concatenation construction of chain-of-action samples, as these are the main threats to the routing and chaining claims; the current text only mentions GPT-4o quality and bias.","section":"Section 5 (Limitation)"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a useful system and dataset, but the evaluation design needs strengthening before the central claims can be accepted. I would ask the authors to include per-task routing results, an out-of-distribution evaluation beyond the 200-sample study, a routing-only human evaluation, and a genuinely sequential chain-of-action test. If the authors release OlympusInstruct and OlympusBench, the community can also re-verify the template-overlap issue directly; data release should be encouraged."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is a competent systems contribution: fine-tuning a 3B MLLM with task-specific routing tokens to delegate 20 vision tasks to external specialists is a natural, well-executed extension of HuggingGPT and Visual ChatGPT. Second, the headline numbers — 94.75% routing accuracy and 91.82% chain precision — come from OlympusBench, which is generated with the same GPT-4o prompt templates (same 7 prefixes, 12 phrases, 9 example pairs, same complexity levels) used to build OlympusInstruct. So those numbers measure in-distribution classification, not universal routing. The stress-test note is right, and the paper's own ablation in Appendix F confirms it: adding example pairs to the prompt template lifts success rate from 44.3% to 61.8%.\n\nWhat the paper does well: the dataset is a real asset. 446K training and 49.6K eval instructions across 20 tasks, with a clean routing-token scheme and chain-of-action support. The model preserves standard MLLM benchmark performance while gaining routing ability (Table 1), and the 200-request human evaluation is a genuine attempt at out-of-distribution testing, even if 86.5% is below the benchmark and conflates routing with specialist output quality. The writing is clear and the method is reproducible in principle.\n\nSoft spots, in proportion: the benchmark sharing the training distribution is the load-bearing weakness. Comparing against zero-shot HuggingGPT on those templates mostly measures template familiarity. Chain-of-action evaluation concatenates random independent task samples, so the 91.82% precision says little about handling true sequential dependencies. Missing error bars on routing metrics is minor; the margin is large. No code, data, or weights are released, which limits verification. The paper's framing that Olympus \"solves\" tasks overstates what a router alone does — final quality depends on the specialists.\n\nBottom line: these flaws don't sink the work, but they shrink the claim. As a demonstration that a small trained router can beat a frontier LLM zero-shot on its own distribution, it holds. As evidence of universal routing, it does not.\n\nI would send this to peer review. A good reviewer can push for an out-of-distribution benchmark, real sequential chain evaluation, and released artifacts. I'd bring it to reading group for the evaluation-methodology discussion, and I'd cite it if I worked on vision agents.","headline":"A competent and useful router paper whose headline numbers are in-distribution scores on a GPT-4o-generated benchmark, not evidence of universal routing.","tokens_in":25387,"tokens_out":1826,"would_cite":true,"duration_ms":20071,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Olympus claims a 3B MLLM, trained with task-specific routing tokens, can route 20 vision tasks with 94.75% accuracy and chain actions at 91.82% precision.","keywords":["multimodal large language models","task routing","computer vision","chain-of-action","routing tokens","instruction tuning","OlympusBench","specialist models"],"falsifier":"Collect a few thousand human-written requests for the same 20 tasks without using GPT-4o templates, run Olympus against HuggingGPT with GPT-4o, and compare routing accuracy and chain precision; if Olympus's lead shrinks or reverses on human prompts, the template-distribution dependence is confirmed. A second check is to build chain-of-action instructions whose task pairs never co-occur in OlympusInstruct and see whether planning precision stays near 91.82%.","tokens_in":24373,"feed_emoji":"🤖","tokens_out":6403,"duration_ms":54975,"temperature":0.7,"pith_summary":"Olympus claims that a small multimodal language model can be turned into a universal task router for computer vision by learning to emit task-specific routing tokens. On a single instruction it answers understanding questions itself and delegates generation, restoration, estimation, and 3D tasks to over 20 specialist modules; on a multi-part instruction it chains up to five such delegations. The paper reports 94.75% average routing accuracy on single tasks and 91.82% precision in chained actions on OlympusBench, beating HuggingGPT running on GPT-4o. These capabilities come at the cost of fine-tuning only a 3B model, with standard multimodal benchmark scores staying roughly at the level of the base model. If the result holds, it offers a modular path to broad vision-task coverage without training a heavy all-in-one generative model.","feed_headline":"3B model routes 20 vision tasks with 94.75% accuracy","feed_subtitle":"Fine-tuned vision-language router beats GPT-4o-based HuggingGPT at planning image, video, and 3D workflows.","key_machinery":"The load-bearing mechanism is a set of task-specific routing tokens, markup-like delimiters such as <image_gen>...</image_gen>, <pose_to_image>...</pose_to_image>, and <3D_gen_text>...</3D_gen_text>, learned as normal vocabulary tokens of the MLLM. They convert the problem of deciding which specialist to call, and what prompt to send it, into a next-token prediction objective, so the same cross-entropy loss used for instruction tuning also trains the planner. The supporting infrastructure is OlympusInstruct and OlympusBench, 446.3K and 49.6K GPT-4o-generated instruction-response pairs engineered with varied prefixes, phrases, and three complexity levels; the concise gold responses teach the model to strip a user request down to the essential prompt for the expert module.","core_discovery":"The central claim is that routing can be learned end-to-end as token prediction. Olympus writes every task delegation as a response wrapped in control tokens, for instance <image_edit>adding green trees</image_edit>, and trains the MLLM with ordinary next-token cross-entropy on 446.3K GPT-4o-generated instruction-response pairs spanning 20 tasks. At inference, the emitted token sequence both identifies the expert module and supplies the refined prompt that module should execute, and chained actions are just concatenated token spans processed in order. On the accompanying OlympusBench set, the 3B model reaches 94.75% routing accuracy and 95.80% precision in the single-task setting and 91.82% precision, 92.75% recall, and 0.18 edit distance in the chain-of-action setting, exceeding the HuggingGPT pipeline driven by GPT-4o on every reported metric. On 11 standard multimodal benchmarks, the same model remains comparable to its Mipha-3B base, losing ground on some tasks and gaining on others.","pith_inferences":["The headline numbers are likely optimistic for real-world use because OlympusBench shares GPT-4o's template distribution with the training data; a broader human-written test set would be needed to confirm the margin over HuggingGPT.","The routing-token mechanism is a general discrete-planning interface, so the same training recipe could route to non-vision tools or even physical actions, though the paper only demonstrates vision specialists.","A confidence-based fallback to a general LLM planner could mitigate out-of-template prompts, since the paper's own 200-instruction human evaluation shows some out-of-distribution transfer but with a small sample.","The reported per-task routing accuracy decline from 96.38% with 5 tasks to 94.75% with 20 suggests performance will keep degrading as the task catalog grows, which sets a practical limit on how many specialists a single small router can manage."],"forward_implications":["A 3B multimodal model can outperform a frontier LLM used as a prompt-engineered controller on routing accuracy, precision, recall, and F1 over 20 vision tasks, implying that specialized fine-tuning is a competitive way to build task planners.","The same framework transfers to different MLLM backbones at 3B, 7B, and 13B scale, and routing accuracy improves with model scale, suggesting the approach is not tied to one architecture.","Up to five specialist tasks can be planned within one instruction with 91.82% precision, which would let users express compound requests like generate, segment, then lift to 3D in a single sentence.","Because only the router is trained, new expert models can be swapped in without retraining the whole system, and the added training cost over base instruction tuning is about 23.6%.","Standard multimodal understanding is largely preserved while the model gains routing, so the router does not appear to sacrifice its original skills for planning ability."],"supporting_citations":[{"why":"Supplies the HuggingGPT controller baseline that Olympus compares against and whose prompt-engineering approach it replaces with trained routing.","marker":"[62]"},{"why":"Provides the Mipha-3B MLLM whose weights initialize Olympus and whose benchmark scores serve as the standard-understanding baseline.","marker":"[99]"},{"why":"Contributes the LLaVA visual instruction tuning recipe and the Mix665K dataset used alongside OlympusInstruct during fine-tuning.","marker":"[48]"},{"why":"Is the GPT-4o system used to generate OlympusInstruct and OlympusBench instruction-response pairs.","marker":"[29]"},{"why":"Supplies LLaVA-Instruct-158K, the general instruction data jointly used in training according to the appendix.","marker":"[45]"}],"fun_headline_variants":["Olympus routes 20 vision tasks with 94.75% accuracy","Universal vision router: 3B model delegates 20 tasks","Vision task router hits 94.75% accuracy, beats GPT-4o pipeline","One model to route them all: Olympus for vision tasks","Olympus: 3B model, 20 tasks, 94.75% routing accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4o-generated instructions in OlympusBench stand in for real user requests, because the evaluation set is built from the same templates, prefixes, and complexity levels as the training set.","fun_headline_variants_meta":{"raw":{"variants":["Olympus routes 20 vision tasks with 94.75% accuracy","Universal vision router: 3B model delegates 20 tasks","Vision task router hits 94.75% accuracy, beats GPT-4o pipeline","One model to route them all: Olympus for vision tasks","Olympus: 3B model, 20 tasks, 94.75% routing accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000725,"raw_usage":{"total_tokens":3231,"prompt_tokens":907,"completion_tokens":2324,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":2225}},"tokens_in":523,"tokens_out":2324,"duration_ms":16490,"temperature":1.0,"reasoning_tokens":2225,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:51:43.707108+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a few thousand human-written requests for the same 20 tasks without using GPT-4o templates, run Olympus against HuggingGPT with GPT-4o, and compare routing accuracy and chain precision; if Olympus's lead shrinks or reverses on human prompts, the template-distribution dependence is confirmed. A second check is to build chain-of-action instructions whose task pairs never co-occur in OlympusInstruct and see whether planning precision stays near 91.82%.","supporting_citations":[{"cited_title":"Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face","cited_arxiv_id":null,"evidence_quote":"Supplies the HuggingGPT controller baseline that Olympus compares against and whose prompt-engineering approach it replaces with trained routing."},{"cited_title":"Mipha: A comprehensive overhaul of multimodal assistant with small language models","cited_arxiv_id":null,"evidence_quote":"Provides the Mipha-3B MLLM whose weights initialize Olympus and whose benchmark scores serve as the standard-understanding baseline."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Contributes the LLaVA visual instruction tuning recipe and the Mix665K dataset used alongside OlympusInstruct during fine-tuning."},{"cited_title":"Improved baselines with visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Supplies LLaVA-Instruct-158K, the general instruction data jointly used in training according to the appendix."}],"review_version":1}