{"id":"8ee20351-737f-4ea9-a0ba-c7873164ffad","arxiv_id":"2504.17454","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A framework for per-query adaptive orchestration of modular GenIA systems is proposed, and a contextual-bandit instantiation is shown to outperform static GPTSwarm orchestration on question answering.","lead":"This perspective paper proposes that future modular generative information access systems should adaptively reconfigure their pipelines for each user query, and demonstrates a bandit-based proof-of-concept on a question-answering task. A smart generalist would read it to see a concrete roadmap and an early experiment showing that per-query module orchestration can beat static pipelines on an effectiveness-efficiency trade-off.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Time-included AQA results contradict the stated reward: for Context C, Eq. 3 makes OneR (r≈0.073) far better than IRCoT (r≈-1.62), yet Table 4 reports AQA(T) choosing an IRCoT-like pipeline (time ≈126 s). The claim that AQA optimizes the efficiency-effectiveness trade-off is unsupported as reported.","rationale":"After reading the paper in good faith, the central claim is that a contextual-bandit instantiation (AQA) with the time-included reward in Eq. 3 successfully adapts pipelines per query to balance effectiveness and efficiency, outperforming static GPTSwarm. The most load-bearing assumption is therefore that the reported AQA (T) results are consistent with the stated reward. They are not. For Context C, Table 3 gives OneR F1 0.146 at 6.4 s and IRCoT F1 0.458 at 184.85 s. Under Eq. 3 with β=0.5, OneR yields r≈0.073 and IRCoT yields r≈−1.62; any combination containing IRCoT is also strongly negative because the time penalty S/50 dwarfs the F1 term. The optimal time-included arm for Context C is therefore OneR, contradicting Table 4 where AQA (T) reports F1 0.523 and a time of 11.75 ln-ms ≈126 s, which can only come from an IRCoT-like pipeline. With roughly 1,167 training samples per context, LinUCB cannot plausibly fail to separate rewards 1.7 apart. This suggests a mismatch between the stated reward and the executed experiment, an error in the table, or a non-converged policy. Each possibility invalidates the specific claim in Section 5.4.1 and weakens the overall proof-of-concept. The reader's gold-label concern is valid for external validity, but the internal inconsistency is more fundamental because it undermines the experiment even under ideal label availability. The paper's conceptual framework may still be a useful perspective, so outright rejection is not recommended; the verdict remains conditional, but the condition must now include correcting and re-running the time-included experiment.","tokens_in":20359,"tokens_out":20641,"duration_ms":189577,"concrete_test":"Re-run the AQA (T) experiment using the public repository and log the arm selected for each Context-C test query. If the majority of Context-C selections are not the OneR-only arm (or another non-IRCoT arm with r≈0.07), Eq. 3 is not being optimized as stated. Additionally, print the LinUCB estimated reward for each arm after 3,500 timesteps on Context C: OneR must exceed IRCoT by about 1.7. If the estimates show IRCoT higher, the reward implementation differs from Eq. 3; recompute Table 4 accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.3 defines the time-included reward as r_t = β·P_t − (1−β)·T_t with β=0.5 and T_t = S_t·(1[1<S_t≤10]/10000 + 1[S_t>10]/50), where S_t is execution time in seconds. Using the training-set numbers in Table 3 for Context C, OneR has F1 0.146 and S=6.41 s, giving r≈0.5·0.146 − 0.5·(6.41/10000)≈0.0727; IRCoT has F1 0.458 and S=184.85 s, giving r≈0.5·0.458 − 0.5·(184.85/50)≈−1.62. Thus the optimal arm under Eq. 3 for Context C is OneR, not any IRCoT-containing graph. Nevertheless, Table 4 reports AQA (T) for Context C with F1=0.523 and time 11.75 (natural-log ms ≈ 126 s), matching the IRCoT pipeline and identical to AQA (NT). This is not a small discrepancy: after roughly 3,500/3≈1,167 Context-C samples, LinUCB should easily separate a reward difference of about 1.7. The reported result implies either the reward was implemented differently from Eq. 3 (e.g., a different denominator or β), the policy did not converge, or Table 4 is misreported. In all three cases, Section 5.4.1's conclusion that the time-based model 'successfully' optimizes the efficiency-effectiveness trade-off is not supported by the published data. This internal inconsistency is independent of the gold-label leakage concern and directly invalidates the experimental claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that future generative information access (GenIA) systems should be modular and adaptively orchestrated, constructing a new pipeline graph for each user query. It proposes a graph-based framework whose nodes are tasks, executors, and resources, and suggests planning and reinforcement-learning methods for construction. The framework is instantiated with a LinUCB contextual bandit on a QA task, using seven pre-computed pipeline graphs as arms, complexity labels as context, and a reward combining F1 with a latency penalty. Experiments compare this adaptive QA system (AQA) against GPTSwarm and report that AQA adapts to query complexity and achieves a better effectiveness-efficiency trade-off. The paper's central claim is that the instantiation 'successfully constructs system pipelines that are adapted to the characteristics of incoming questions and optimized for a given efficiency-effectiveness trade-off' (Section 5.4.1).","tokens_in":20791,"tokens_out":7427,"duration_ms":67951,"significance":"The perspective and framework are timely and useful: the module taxonomy and the formulation of pipeline construction as a contextual bandit problem are clear contributions, and the authors provide a public code repository, which strengthens reproducibility. However, the experimental demonstration currently does not support the central claim. The time-included reward specified in Eq. (3) is inconsistent with the reported behavior of AQA(T) for Context C, the context features are gold complexity labels unavailable at query time, the evaluation rests on a single run with no statistical significance testing, and the GPTSwarm baseline optimizes a different objective. These issues are substantial but fixable; the framework contribution is independent of the experimental validation.","major_comments":[{"comment":"The time-included results are internally inconsistent with the stated reward. For Context C, Table 3 gives OneR F1=0.146, S=6.41 s and IRCoT F1=0.458, S=184.85 s. Substituting into Eq. (3) with β=0.5 gives r(OneR) ≈ 0.5·0.146 − 0.5·(6.41/10000) ≈ 0.0727 and r(IRCoT) ≈ 0.5·0.458 − 0.5·(184.85/50) ≈ −1.62. The optimal arm under Eq. (3) for Context C is therefore OneR, yet Table 4 reports AQA(T) with F1=0.523 and time 11.75 (natural-log ms, ≈126 s), which matches an IRCoT-containing pipeline and is identical to AQA(NT). After roughly 1,167 Context-C samples, LinUCB should easily separate a reward gap of about 1.7. The reported result implies that the reward was implemented differently from Eq. (3), the policy did not converge, or the table is misreported; in any case, the conclusion in §5.4.1 that the time-based model 'successfully' optimizes the efficiency-effectiveness trade-off is not supported by the published data.","section":"§5.3 and Table 4 (Eq. 3)"},{"comment":"The contextual features are 'the complexity labels provided in the dataset,' i.e., the gold A/B/C labels described in §5.1, which were generated with Flan-T5-XL. Real systems do not have access to gold complexity labels at query time, so the demonstration of adaptivity presupposes an oracle signal. The paper should either use features computable from the query itself (e.g., text embeddings or a trained complexity classifier) or explicitly frame the results as an oracle-upper-bound proof-of-concept. Section 4.4 acknowledges that 'contextual features may not be sufficiently informative,' but it does not mention that the features used here are gold labels, which is a stronger and more specific concern. This directly affects the transferability of the claim that the system adapts to 'characteristics of incoming questions.'","section":"§5.3 (context features)"},{"comment":"The experimental evaluation consists of a single run of the LinUCB process for 3,500 timesteps, evaluated on a single test set of 51 questions, with no variance, confidence intervals, or significance testing. The differences in Table 4 (e.g., AQA(NT) vs AQA(T): F1 0.697 vs 0.687, time 9.99 vs 8.89) and the differences versus GPTSwarm could be within run-to-run noise. Because the paper's central conclusion is that AQA 'successfully' adapts and outperforms static orchestration, the authors should report multiple seeds or bootstrapped intervals and, ideally, significance tests for the test-set comparisons.","section":"§5.4 (experimental protocol)"},{"comment":"The GPTSwarm baseline is optimized for F1 only, without any time cost (the paper notes that GPTSwarm does not accommodate context or time cost), whereas AQA(T) is optimized for the combined reward of Eq. (3). The higher latency of GPTSwarm is therefore expected and does not by itself demonstrate the value of adaptivity. To support the claim that adaptive orchestration achieves a superior efficiency-effectiveness trade-off over static optimization, the static baseline should be optimized for the same objective (e.g., a static graph optimizer with a time-sensitive reward) or the comparison should be explicitly limited to an F1-only static baseline.","section":"§5.4.2 (baseline comparison)"}],"minor_comments":[{"comment":"Items (2) and (3) in the context-features list are both titled 'User profile features' and contain nearly identical text; they should be merged or renumbered.","section":"§4.3.2"},{"comment":"The legend is difficult to read in grayscale, and the color coding of the seven arms is not discernible in print; additionally, the legend label 'NOR' should be 'NoR' to match the text.","section":"Figure 2"},{"comment":"The caption should state that the 'Time' column reports natural-log-transformed milliseconds; this information appears only in the text.","section":"Table 4"},{"comment":"The definition of T_t is written with an awkward bracket structure; please rewrite it with explicit cases, e.g., T_t = S_t/10000 if 1 < S_t ≤ 10, and T_t = S_t/50 if S_t > 10.","section":"Eq. (3)"},{"comment":"Several references use '[n. d.]' instead of a year or venue (e.g., [9], [10], [55], [56]); these should be completed for the camera-ready version.","section":"References"},{"comment":"The text refers to 'dotted red lines' in Figure 2, but the figure appears to be in color only in the electronic version; please ensure the dashed/dotted lines are also distinguishable in grayscale.","section":"§5.4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is best viewed as a perspective paper with an illustrative proof-of-concept. The framework contribution is solid, but the experimental section requires substantial revision before publication. The internal inconsistency between Eq. (3) and Table 4 is the most serious issue and should be checked against the released code; if the reported numbers are correct, the reward used in the implementation must be described accurately. The gold-label context features and the single-run protocol are also important weaknesses that should be addressed, either by additional experiments or by explicitly downgrading the claims in Sections 5.4.1 and 5.4.2."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper's main contribution is a coherent taxonomy for modular generative IR systems — tasks, executors, resources — with a graph-based view of orchestration. That part is genuinely useful for anyone designing or surveying RAG-style pipelines. Second, the proof-of-concept experiment that is supposed to show the framework working has a serious internal inconsistency: the reported time-included AQA results do not match the reward function defined in Eq. (3).\n\nLet me give you the numbers. For Context C, Table 3 gives OneR F1 0.146 at 6.41 s and IRCoT F1 0.458 at 184.85 s. Plugging into Eq. (3) with beta=0.5 gives OneR r ≈ 0.073 and IRCoT r ≈ -1.62. So the best arm for Context C should be OneR. But Table 4 reports AQA(T) for Context C with time 11.75 in natural-log ms, which is about 126 s — clearly the IRCoT pipeline — and identical to the time-agnostic AQA(NT) result. After roughly 1,167 training samples per context, a LinUCB agent should have separated a reward gap of 1.7. Either the reward was implemented differently from Eq. (3), the policy never converged, or the table is wrong. In any case, Section 5.4.1's claim that AQA(T) 'successfully' balances effectiveness and efficiency is not supported by the published numbers.\n\nThere are additional softer problems. The context features are the dataset's gold complexity labels (A/B/C), which a live system wouldn't have at query time. The test set is 51 questions, single split, no error bars. The GPTSwarm baseline optimizes F1 only, so its higher latency is expected. The reward constants (beta, 1 s/10 s thresholds) are hand-set.\n\nWhere the paper does well: the framework itself is clearly presented, it situates itself honestly relative to Adaptive-RAG, MBA-RAG, and GPTSwarm, and it explicitly lists limitations of the RL/graph approach in Section 4.4. The roadmap is thought-provoking.\n\nWho should read it: people working on orchestration architectures for RAG or compound AI systems. The perspective is worth discussing even though the empirical result is not reliable. I'd send it to peer review because the framework merits engagement, but I'd require the authors to either fix the reward inconsistency, re-run the experiment, or remove the time-aware claim.","headline":"The framework and taxonomy are genuinely useful, but the time-aware AQA result contradicts the paper's own reward equation, so the experimental claim should not be trusted as reported.","tokens_in":21330,"tokens_out":3585,"would_cite":false,"duration_ms":31779,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that generative information access systems should reconfigure their processing pipeline per query, and demonstrates the idea with a contextual bandit that beats a static pipeline.","keywords":["adaptive orchestration","generative information access","contextual bandit","graph-based orchestration","question answering","efficiency-effectiveness trade-off","modular systems","LinUCB"],"falsifier":"Run LinUCB on the same seven graphs but with complexity labels predicted by a lightweight classifier (or with raw query text features only), and compare on the same test set; if per-query selection no longer beats the static GPTSwarm graph on F1 and latency, the paper's evidence for adaptive orchestration over static optimization collapses. Alternatively, collect user-judged complexity labels and show they disagree with the Flan-T5-XL labels, breaking the mapping the bandit learns.","tokens_in":20181,"feed_emoji":"🧩","tokens_out":6630,"duration_ms":56997,"temperature":0.7,"pith_summary":"This paper argues that generative information access systems should not execute a fixed pipeline for every query; instead, they should adaptively assemble a pipeline of tasks, executors, and information resources on the fly, optimizing for the trade-off between answer quality and cost. It proposes a graph-based framework in which pipeline designs are subgraphs over modules, and construction is framed as planning or reinforcement learning. To show the framework is practicable, the authors instantiate it for question answering: a LinUCB contextual bandit selects among seven pre-computed pipeline graphs using query complexity features, with a reward that balances F1 score against execution time. On a held-out test set, the adaptive system outperforms a static, optimized graph on both F1 and latency, supporting the paper's perspective that per-query adaptive orchestration is feasible and worth developing.","feed_headline":"Per-query pipeline choice beats static orchestration in QA","feed_subtitle":"A contextual bandit that routes each question to the right modules improves F1 and cuts latency over a fixed pipeline.","key_machinery":"The machinery is a graph-based representation of pipelines combined with a contextual bandit. Each pipeline is a directed acyclic graph; nodes are task modules (NoR, OneR, IRCoT, Aggregate), executors (Flan-T5-XL as LLM agent, BM25 as retriever, a majority-vote aggregator), and resources (Wikipedia and a multihop passage corpus), and edges encode both the flow of information and the assignment of executors and resources to tasks. The set of seven valid graphs forms the discrete action set of a LinUCB contextual bandit, with the query's complexity label as the context vector and a reward $r_t = \\beta P_t - (1-\\beta) T_t$ that makes the effectiveness-efficiency trade-off explicit. The static baseline, GPTSwarm, is optimized with REINFORCE on the same module set and converges to one fixed graph, which cannot specialize by query.","core_discovery":"The central claim is that the architecture of future generative information access systems will be dynamic and self-organizing: for each user input, the system should decide which tasks to run, which executors (agents or tools) should perform them, and which resources to consult, instead of applying a one-size-fits-all pipeline. The paper formalizes this as a directed acyclic graph whose nodes are module types and whose edges encode ordering and assignment, and it treats pipeline construction as an optimization problem with a composite objective over effectiveness and cost. In the proof-of-concept instantiation, LinUCB learns to map question complexity to one of seven valid graphs: simple questions are routed to direct generation (NoR), harder ones to single-step retrieval (OneR) or interleaved retrieval-chain-of-thought (IRCoT), and an aggregate node merges answers. With F1 and time as rewards, the adaptive system reaches 0.697 F1 at 9.99 ms (time-agnostic) or 0.687 F1 at 8.89 ms (time-based), while the static GPTSwarm-optimized graph reaches 0.502 F1 at 12.78 ms. The authors conclude that adaptive orchestration can construct pipelines adapted to incoming questions and optimized for a stated efficiency-effectiveness trade-off.","pith_inferences":["The paper's use of gold complexity labels as context is a favorable condition: in deployment, a complexity predictor trained without such labels would be needed, and the reported advantage over static orchestration may shrink if that predictor is noisy.","Because the action space is fixed to seven whole graphs, 'adaptivity' here is strategy selection rather than true on-the-fly construction; a more granular formulation with node-by-node additions could support pipelines that branch based on intermediate retrieval results, which the paper lists as an extension.","A direct test of the vision would replace the gold labels with a learned query-complexity classifier, or with bandit features derived purely from the query text, and check whether per-query selection still beats a static graph."],"forward_implications":["The same bandit-based selection generalizes to any modular system whose feasible pipeline graphs can be enumerated, making the framework applicable beyond question answering to retrieval, conversational search, and tool-use systems.","Adaptive orchestration can incorporate cost beyond latency, such as compute, API fees, and environmental impact, because any such cost can enter the composite reward.","Systems that adapt per query can integrate newly added modules over time: a new module only adds nodes and graphs to the action space, while the bandit re-learns their utility.","The demonstrated gap between adaptive and static orchestration (F1 0.697 vs 0.502, lower latency) implies that static pipeline optimization leaves substantial accuracy and efficiency on the table when query complexity varies.","Scaling to large module sets will require planning or general RL methods rather than exhaustive graph enumeration, as the paper notes."],"supporting_citations":[{"why":"Supplies the graph-based orchestration approach and the static baseline (GPTSwarm) that AQA must outperform.","marker":"[97]"},{"why":"Supplies the QA dataset with A/B/C complexity labels that serve as the bandit's context and define the task setup.","marker":"[28]"},{"why":"Provides the IRCoT procedure and the multihop passage corpus used in the graph's retrieval and reasoning nodes.","marker":"[70]"},{"why":"Supplies the LinUCB contextual bandit algorithm that performs the adaptive pipeline selection.","marker":"[42]"},{"why":"Provides the BM25 retrieval tool used as an executor module in the graph.","marker":"[61]"}],"fun_headline_variants":["Adaptive module routing beats fixed pipeline in QA","Per-query pipeline choice improves F1, cuts latency","Self-organizing IR outdoes static orchestration","Dynamic orchestration wins over static in GenIA","Contextual bandit picks best modules per query"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The demonstrated gains rely on the bandit receiving the dataset's gold complexity labels (A/B/C) as context, a signal real systems do not have at query time and that was produced by an LLM annotator rather than validated against user needs.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive module routing beats fixed pipeline in QA","Per-query pipeline choice improves F1, cuts latency","Self-organizing IR outdoes static orchestration","Dynamic orchestration wins over static in GenIA","Contextual bandit picks best modules per query"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1383,"prompt_tokens":1039,"completion_tokens":344,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":655,"completion_tokens_details":{"reasoning_tokens":271}},"tokens_in":655,"tokens_out":344,"duration_ms":3570,"temperature":1.0,"reasoning_tokens":271,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:39:46.742849+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LinUCB on the same seven graphs but with complexity labels predicted by a lightweight classifier (or with raw query text features only), and compare on the same test set; if per-query selection no longer beats the static GPTSwarm graph on F1 and latency, the paper's evidence for adaptive orchestration over static optimization collapses. Alternatively, collect user-judged complexity labels and show they disagree with the Flan-T5-XL labels, breaking the mapping the bandit learns.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the graph-based orchestration approach and the static baseline (GPTSwarm) that AQA must outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the IRCoT procedure and the multihop passage corpus used in the graph's retrieval and reasoning nodes."},{"cited_title":"Schapire","cited_arxiv_id":null,"evidence_quote":"Supplies the LinUCB contextual bandit algorithm that performs the adaptive pipeline selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the BM25 retrieval tool used as an executor module in the graph."}],"review_version":1}