{"id":"9052ef73-e85b-4328-ac84-356beb054bb7","arxiv_id":"2501.16634","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A declarative workflow system with an adaptive runtime can cut compound AI workflow completion time and energy use by auto-selecting resources and parallelism.","lead":"This paper proposes Murakkab, a system that lets developers describe AI workflows at a high level and lets a runtime pick models, hardware, and parallelism automatically. The prototype shows up to 3.4x faster completion and 4.5x better energy efficiency on one video-processing workflow.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported ~3.4x speedup conflates Murakkab's adaptive runtime with ordinary parallelism: Section 4's baseline deliberately disables intra-task parallelism, so a conventional parallel orchestrator may achieve the same gains.","rationale":"The reader's CONDITIONAL verdict is appropriate. I agree that LLM-based decomposition is an unproven weak point, but I see a more immediate load-bearing issue: the empirical evaluation does not isolate Murakkab's contribution. The baseline is constructed without parallelism, so the 3.4x speedup may simply reflect the difference between sequential and parallel execution, not the value of declarative fungibility or adaptive scheduling. This concern is checkable with a strong baseline, and the proposed test would either support or retire it. I do not regard this as grounds for rejection: the paper is a vision/position contribution with a preliminary prototype, and it openly labels the results as preliminary. However, the headline numbers should not be read as validating the architecture until the control exists. I also credit the paper for identifying real inefficiencies in current compound-AI stacks, discussing overheads, and being transparent about limitations (GPU-only energy measurement, single workflow, no artifacts). The energy claim is additionally constrained by the GPU-only measurement, which is relevant because the optimization moves work to CPUs; that too should be addressed, but the missing strong baseline is the primary issue.","tokens_in":9441,"tokens_out":8673,"duration_ms":90048,"concrete_test":"Re-run the Section 4 video-understanding workflow with a strong control: implement the same components (OpenCV frame extraction, Whisper STT, CLIP object detection, NVLM summarization) in a conventional DAG executor such as Prefect or Dask on the same two Azure VMs, applying exactly the parallelization and resource decisions Murakkab makes (STT across scenes in parallel, frame summarization on idle GPUs, Whisper on the same CPU configuration), and measure end-to-end time and full-system energy (CPU+GPU). If this control completes within ~10% of Murakkab's 77-83s and uses comparable energy, the reported gains are due to parallelism/resource choices rather than to the declarative adaptive runtime, and the paper's headline claim should be reframed. If the control is substantially slower or less efficient, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 compares Murakkab against a baseline that 'specifies a fixed execution without any intra-task parallelism or opportunity to utilize idle resources' (each scene and frame processed sequentially). Murakkab's three named optimizations are (a) running STT for multiple scenes in parallel, (b) parallelizing frame summarization on idle GPUs, and (c) moving Whisper from GPU to CPU. These are generic DAG-parallelism and resource-allocation choices available to any workflow engine; they do not exercise the paper's distinctive mechanisms (LLM-based job decomposition, fungible model/tool selection, or orchestrator-cluster-manager collaboration). Consequently, the headline speedup and the associated energy improvement are not attributable to the declarative/adaptive design: the comparison is between an intentionally sequential pipeline and a parallelized one, not between Murakkab and a strong conventional implementation of the same workflow. The paper also asserts identical output without reporting an output comparison, but the more load-bearing gap is the missing control for parallelization. Until such a control is run, the empirical claim that the Murakkab architecture itself improves efficiency is unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that current compound AI systems are inefficient due to tight coupling of application logic with execution configuration, a disconnect between orchestration and cluster management, and a perceived tradeoff between efficiency and quality. It proposes Murakkab, a prototype combining a declarative workflow programming model with an adaptive runtime that uses LLM-based job decomposition, task-to-agent mapping, model/tool selection, resource allocation, and workflow-aware cluster management. The evaluation on a video-understanding workflow reports ~3.4x speedup and ~4.5x higher energy efficiency compared with a sequential OmAgent-derived baseline, with the same output and accuracy asserted. The paper also discusses AI Workflows-as-a-Service (AIWaaS), quality control, proprietary models, and multi-cloud concerns.","tokens_in":9662,"tokens_out":7834,"duration_ms":75620,"significance":"The problem is timely and the architecture is plausible. The paper identifies real inefficiencies in compound AI deployments and proposes a concrete design, with an honest discussion of overheads and open problems. Strengths include the explicit treatment of resource fungibility, the connection to prior cluster-management work, and a preliminary end-to-end prototype. However, the empirical evidence is currently too thin: the evaluation covers one workflow, the baseline is deliberately sequential, no strong parallel baseline is included, output equality is asserted without measurement, and energy is measured only on the GPU. If the evaluation is strengthened, the system could make a useful contribution to the systems community; at present the headline efficiency claims are not fully supported.","major_comments":[{"comment":"The reported ~3.4x speedup does not distinguish Murakkab's adaptive runtime from ordinary parallelism. The baseline is described as 'a fixed execution without any intra-task parallelism or opportunity to utilize idle resources,' and the three optimizations attributed to Murakkab—parallel STT across scenes, parallel frame summarization on idle GPUs, and running Whisper on CPUs—are generic dataflow-parallelization and resource-allocation decisions available to any conventional workflow engine. A control that runs the same DAG with a standard parallel orchestrator (and with the same agent choices) is needed to attribute the speedup to the declarative/adaptive design. The current comparison only shows that a sequential pipeline is slower than a parallelized one.","section":"§4 (Baseline and Murakkab paragraphs)"},{"comment":"The statement that 'The execution output and accuracy are the same in all comparisons' is not supported by any reported comparison. The paper should define and report a quantitative measure of output equivalence (for example, exact or near-exact transcript match for STT, object detection precision/recall for CLIP, and final answer agreement for the summarization/LLM stages) and show these values for each configuration. Without this, the quality-preservation claim that underpins the fungibility argument is unverified.","section":"§4 (first paragraph)"},{"comment":"The energy-efficiency claim is based on an incomplete measurement. The paper measures only GPU energy, but the headline 4.5x improvement comes from running STT on 64 CPU cores instead of a GPU; excluding CPU energy from that configuration will trivially favor the CPU configuration. The authors should report full node or system energy, or at least include CPU power in the measurement. In addition, Listing 2 specifies MIN_COST, while Table 2 reports energy, not monetary cost; the paper should clarify whether the constraint is cost or energy and report the corresponding metric.","section":"Table 2"},{"comment":"The evaluation does not exercise the automatic job-decomposition mechanism. Listing 2 supplies the sub-tasks explicitly, and the reported optimizations follow from that explicit task structure. The LLM-based decomposition from a natural-language job description into a DAG, which is a central component of Section 3.2, is therefore untested. The authors should either evaluate a run that starts from only the 'desc' and inputs, or explicitly scope the claims to user-provided task hints.","section":"§3.2 and Listing 2"},{"comment":"The reported times and energies appear to come from single runs; no repetition, error bars, or variance is reported. Given that the speedup and energy claims are central to the paper, at least a few repeated runs with the observed range (or a clear statement that these are single preliminary traces) should be provided.","section":"Figure 3 and Table 2"}],"minor_comments":[{"comment":"There are several typos and misspellings; examples include 'throuhgput' (Section 2), 'underprovisionig' (Section 2), 'Task understansing' (Section 3.3), and 'Worfklow' in the Section 3.2 heading. The paper needs a careful copyedit.","section":"§2 and §3.3"},{"comment":"Reference [8] is mislabeled: the title 'What is provisioned throughput?' does not match the Azure Spot VMs URL; please point to the correct documentation.","section":"References"},{"comment":"The statement that DAG-creation LLM queries take 'less than 1% of the execution time' is not backed by a measurement or a derivation; please report the measurement or label it as an estimate.","section":"§3.3"},{"comment":"The 'CPU vs GPU' row reports 'Lower/No Change' for quality, which is ambiguous. Please clarify the conditions under which moving a model to CPU changes quality (for example, numerical precision or latency effects) and whether this is a single selection.","section":"Table 1"},{"comment":"The baseline completion time is given as 283 s in the text and 285 s in Table 2; please reconcile.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a vision/prototype paper; in its current form the empirical claims are not yet supported at the level expected for a systems publication. I would not reject the paper outright because the architecture and discussion are useful and the evaluation can be strengthened, but the authors should either add the missing baseline control and measurements or reframe the contribution as a position paper with a feasibility study. Self-citation is not a concern here."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful vision paper for resource-efficient compound AI systems, and the authors are upfront that it is preliminary. The new bits are worth naming: a declarative workflow model that hides model and hardware choices, a runtime that does LLM-based decomposition and maps tasks to a library of agents, and explicit integration between workflow orchestration and cluster manager. That last piece, plus treating hardware and model choice as fungible, is what distinguishes it from earlier declarative AI workflow systems like [3,24]. The AIWaaS framing in Section 5 is a nice discussion contribution.\n\nThe paper also deserves credit for being honest about overheads and open problems. Table 1 is a clean summary of the levers. The prototyping is real: they ran a video-understanding workflow based on OmAgent, varied Whisper placement, parallelized scenes, and reported energy and time.\n\nNow the soft spots. The stress-test note is on target. The baseline is described as a fixed execution without any intra-task parallelism or opportunity to utilize idle resources, with each scene and frame processed sequentially. That means the 3.4x speedup mostly measures the difference between sequential processing and generic DAG parallelism, not the value of Murakkab's declarative and adaptive machinery. Any competent workflow engine that parallelizes frame summarization and STT across scenes would get much of the same gain. To attribute the gain to Murakkab, they need a strong conventional baseline: the same workflow as an imperative DAG with parallelism enabled. Without that control, the headline claim is unverified. The energy comparison has a related problem: only GPU energy is measured, which is fine if justified, but combined with the baseline issue it overstates the system's efficiency story.\n\nThe second load-bearing assumption is LLM-based job decomposition. If the orchestrator LLM produces a semantically wrong DAG, output quality changes. The evaluation workflow is simple, and the authors supply the subtasks in Listing 2, so automatic decomposition from natural language alone is not exercised. The claim that output and accuracy are the same is asserted, not quantified. Also, single run, no error bars, no public artifacts. These are fixable evaluation gaps, not fundamental flaws.\n\nOverall, the central architectural argument—decouple application logic from execution details and let a runtime exploit that flexibility—holds up. The paper's current evidence does not prove the architecture delivers the numbers. It is a solid position paper and a promising research direction.\n\nI would send it to peer review as a vision and systems paper, with the clear expectation that the authors strengthen the evaluation before acceptance: add a realistic parallel baseline, repeat runs, quantify output similarity, and release code and data. If that happens, it could be a useful cite. For now, I would treat the speedup and energy numbers as suggestive, not established.","headline":"Solid vision paper with an honest prototype, but the headline numbers compare Murakkab against a deliberately sequential baseline, so the claimed speedup mostly measures ordinary parallelism.","tokens_in":10201,"tokens_out":2176,"would_cite":true,"duration_ms":21898,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Murakkab claims declarative AI workflows can run 3.4x faster and use 4.5x less energy with unchanged output.","keywords":["compound AI systems","declarative workflows","adaptive runtime","resource efficiency","workflow orchestration","cluster management","energy efficiency","fungible workflows"],"falsifier":"Run Murakkab on an unseen multipart job given only its natural-language description, with no subtask hints, and compare the LLM-generated task graph and final output against a hand-authored reference graph and output; any task graph that omits, reorders, or mislabels a required step, or any output that differs in quality, would falsify the claim that declarative abstraction preserves result quality.","tokens_in":9271,"feed_emoji":"⚡","tokens_out":5051,"duration_ms":51212,"temperature":0.7,"pith_summary":"This paper argues that compound AI systems waste resources because application logic is fused with fixed model, tool, and hardware choices, and because workflow orchestration and cluster management do not talk to each other. It proposes Murakkab, a runtime that accepts a high-level declarative job description, generates a task graph, maps tasks to interchangeable agents, and picks hardware and execution strategies from measured efficiency and quality profiles. In a video-understanding workflow, this design completed the job in 77 to 83 seconds versus 283 seconds for the imperative baseline, and cut speech-to-text energy from 155 Wh to 34 Wh by running on CPUs instead of GPUs, with identical output. The authors frame this as a first validation of a broader vision in which AI workflows become fungible and resource-efficient without sacrificing result quality.","feed_headline":"Declarative AI workflows run 3.4x faster on 4.5x less energy","feed_subtitle":"A new runtime decouples app logic from model and hardware choices, then picks the cheapest path with the same result.","key_machinery":"The load-bearing mechanism is the pairing of a declarative workflow specification with an adaptive runtime scheduling loop. The orchestrator LLM (NVLM) decomposes the job into a task graph using a ReAct-style approach, maps tasks to agents through a library of model and tool schemas, and the runtime then selects among fungible implementations and hardware configurations using execution profiles that record cost, power, latency, and quality tradeoffs. A separate cluster manager exposes real-time resource availability, and the two layers exchange workflow DAGs and utilization statistics to rebalance resources across agents, enabling parallelism and harvestable-resource decisions that a static workflow cannot make.","core_discovery":"The paper's central claim is that a Compound AI System can be written declaratively — a natural-language job description, optional subtask hints, and a constraint such as minimum cost — and an adaptive runtime can then lower that description into a directed acyclic graph, map each task to an agent from a library of models and tools, and choose execution configurations such as CPU versus GPU, parallel subtask execution, and extra execution paths. By doing so, Murakkab achieves a roughly 3.4x speedup in workflow completion time and a roughly 4.5x improvement in energy efficiency over a fixed, imperative workflow implementation of the same video-understanding task, while the authors state that the execution output and accuracy are the same in all comparisons. The result is meant to show that efficiency and quality are not inherently in conflict once application logic is decoupled from low-level resource and implementation details.","pith_inferences":["Editorial inference: the evidence is one workflow with explicit subtask hints; if automatic decomposition of open-ended job descriptions proves unreliable, the equal-quality claim would need to be scoped to workflows whose task structure is known in advance.","Editorial inference: if this architecture scales, AI workflows could evolve like SQL — a declarative job survives model and hardware churn, and the optimizer absorbs the churn, making the workflow definition itself a durable asset rather than a code artifact to be rewritten.","Editorial inference: the runtime levers suggest a low-cost benchmarking methodology for compound AI systems — report workflows as constraint-optimization problems such as MIN_COST or MIN_LATENCY, and compare the runtime's chosen configuration against the full frontier of model and hardware assignments.","Editorial inference: the CPU-versus-GPU speech-to-text swap is a testable pattern; applying the same substitution to other transcription models and languages could generalize the energy savings, but latency and accuracy shifts would need to be measured per model."],"forward_implications":["Developers stop pinning models, tools, and hardware: Murakkab chooses them at runtime from an agent library and per-configuration profiles.","Workflows written declaratively survive model and tool updates without rewriting or reprompting.","Cluster managers gain visibility into the workflow DAG and can move resources between agents based on upcoming demand rather than current utilization.","The same output can be delivered on a continuum of cost, latency, and energy tradeoffs, selected by a constraint such as MIN_COST.","One job can be split into parallel subtasks (for example, per-scene speech-to-text) whenever idle resources exist, which is how the 283-second baseline became a 77-to-83-second run."],"supporting_citations":[{"why":"Defines compound AI systems and motivates the multi-component workflow scope of the paper.","marker":"[42]"},{"why":"Supplies the OmAgent video-understanding workflow that becomes the paper's baseline and target application.","marker":"[43]"},{"why":"Supplies NVLM, the orchestrator LLM that decomposes jobs and maps tasks to agents.","marker":"[13]"},{"why":"Supplies Whisper, the speech-to-text model whose resource configuration the paper varies between GPU and CPU.","marker":"[33]"},{"why":"Supplies CLIP, the object-detection model used in the video-understanding workflow.","marker":"[32]"},{"why":"Supplies the ReAct approach that guides the orchestrator's task-decomposition loop.","marker":"[41]"},{"why":"Provides the SQL query-optimization analogy that motivates the declarative programming model.","marker":"[10]"},{"why":"Provides the concept of application fungibility that underpins runtime model and hardware swapping.","marker":"[36]"}],"fun_headline_variants":["Declarative AI: 3.4x faster, 4.5x less energy","AI workflows: 3.4x speedup, 4.5x energy savings","Murakkab: 3.4x faster AI, 4.5x more energy-efficient","Decoupled AI logic: 3.4x faster, 4.5x less power"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an LLM can decompose a natural-language job description into a semantically correct task graph and map each task to a suitable agent; if that decomposition is wrong, the workflow's output changes and the paper's equal-quality claim collapses. The validation uses a simple workflow with subtasks spelled out explicitly, so this hardest part of the design is not yet stressed.","fun_headline_variants_meta":{"raw":{"variants":["Declarative AI: 3.4x faster, 4.5x less energy","AI workflows: 3.4x speedup, 4.5x energy savings","Murakkab: 3.4x faster AI, 4.5x more energy-efficient","Decoupled AI logic: 3.4x faster, 4.5x less power"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000317,"raw_usage":{"total_tokens":1775,"prompt_tokens":908,"completion_tokens":867,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":767}},"tokens_in":524,"tokens_out":867,"duration_ms":7719,"temperature":1.0,"reasoning_tokens":767,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T11:48:01.408414+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Murakkab on an unseen multipart job given only its natural-language description, with no subtask hints, and compare the LLM-generated task graph and final output against a hand-authored reference graph and output; any task graph that omits, reorders, or mislabels a required step, or any output that differs in quality, would falsify the claim that declarative abstraction preserves result quality.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies CLIP, the object-detection model used in the video-understanding workflow."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SQL query-optimization analogy that motivates the declarative programming model."},{"cited_title":"Aguilera, Adam Belay, Seo Jin Park, and Malte Schwarzkopf","cited_arxiv_id":null,"evidence_quote":"Provides the concept of application fungibility that underpins runtime model and hardware swapping."}],"review_version":1}