{"id":"07a3136f-ea0e-49a6-a144-f4a57e2e5d96","arxiv_id":"2501.18320","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-agent LLM system with graph-based retrieval scores higher than prompt-only baselines on ten SASP modeling tasks, though the evaluation may be biased because the library contains the answers.","lead":"The paper presents MAG-RAG, a system that uses several large language model agents and a graph-based library of solved problems to automatically turn sensor array signal processing descriptions into optimization models. On ten classical problems, human scorers gave MAG-RAG higher average scores than simpler LLM-based baselines, but the test design raises doubts about whether the advantage is genuine.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation confound: SPAMR is built from documents containing standard modeling approaches for the same ten issues used as queries, so MAG-RAG's retrieved context can contain the target formulation, making the comparison with corpus-free baselines unfair.","rationale":"The manuscript promises automated optimization modeling for SASP and supports it with scores on ten 'classical' problems. The reader's weakest assumption is exactly the one I would attack: the test set overlaps with the corpus. I verified from Section III-A that for each issue the authors selected documents containing standard modeling approaches, and from Section II-C that K is constructed by retrieving top-3 PT nodes from that corpus. That makes the evaluation a retrieval test with the answer in the context, not a clean test of optimization-modeling skill. The comparison with pure MA and pure LLM is unfair because those baselines cannot retrieve these documents. There are additional weaknesses (no statistical tests, no repeated runs, no ablation of graph structure, inconsistent per-cell results), but they are secondary; the corpus overlap alone invalidates the headline comparison. Therefore the central claim as stated cannot be accepted. The leave-one-out check proposed above is a minimal, decisive experiment: it keeps the same data and infrastructure, so if the advantage survives, the paper would need only supplementary reporting; if it vanishes, the current rejection is confirmed.","tokens_in":7145,"tokens_out":4922,"duration_ms":46289,"concrete_test":"Leave-one-out ablation: for each query Qi, remove from the corpus all documents that were selected specifically for Qi before building the graph and running Graph-RAG; repeat the evaluation for all ten queries and recompute the highest-score rate. If the MAG-RAG advantage over pure MA and pure LLM disappears or drops substantially, the original advantage is explained by retrieval of the target solution; if it persists, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MAG-RAG outperforms AOM benchmarks on ten SASP problems. The load-bearing condition for this comparison to measure modeling ability is that the retrieval corpus is not an answer key. Section III-A violates that condition: for each of the ten issues, the paper says it 'finely select[s] a number of documents containing standard modeling approaches' to construct SPAMR, and the same ten issues are used as evaluation queries. MAG-RAG's Graph-RAG then retrieves top-3 nodes from this corpus for the query (Section II-C), so for each Qi the retrieved context can contain the very standard modeling approach that the scoring rubric rewards. The pure MA and pure LLM baselines have no access to this corpus. Consequently, the observed score advantage (e.g., the 67% highest-score rate in Fig. 4A) is compatible with an information-retrieval effect rather than with better optimization modeling. This is a load-bearing confound because the main evidence for the proposed architecture is exactly this comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MAG-RAG, an automated optimization modeling (AOM) system for sensor array signal processing (SASP) problems that combines a multi-agent LLM pipeline with graph-based retrieval-augmented generation. A Terminology Agent normalizes user queries, a graph-structured knowledge base built from domain documents is searched via top-k node retrieval, and an Optimization Modeling Agent generates the final formulation. The system is evaluated on ten classical SASP problems with four base LLMs, using human expert scoring across five metrics. The writing is clear and the proposed architecture is concrete, but the evaluation is compromised by a train/test overlap: the knowledge corpus is built from documents containing standard modeling approaches for the very same ten problems that are later used as test queries, so the main empirical claim is not currently supported.","tokens_in":7310,"tokens_out":6983,"duration_ms":62653,"significance":"If the results were valid, this would be a useful contribution to domain-specific AOM: the paper addresses a genuine gap (LLMs' lack of SASP modeling knowledge), proposes a structured way to inject that knowledge through a graph-based knowledge base, and reports experiments across several LLMs with expert human scoring. The authors also release code and data, which aids reproducibility. However, the central empirical claim 'MAG-RAG outperforms several AOM benchmarks' is not established by the current experiments because the retrieval corpus is seeded with solutions to the test queries, and because per-base-LLM results are mixed. The idea is promising but the evidence as presented is not.","major_comments":[{"comment":"The evaluation is confounded by the overlap between the retrieval corpus and the test set. The dataset SPAMR is constructed by selecting, for each of the ten test issues, documents 'containing standard modeling approaches' for that same issue, and the same ten issues are then used as evaluation queries. MAG-RAG retrieves the top-3 relevant nodes from this corpus (Section II-C) and feeds them to the Modeling Agent, while the pure LLM and pure MA baselines do not access the corpus. Since the scoring metrics reward completeness, standardization, and correctness, the retrieved 'standard modeling approach' is effectively an answer key. The observed advantage in Table I and Fig. 4(A) is therefore compatible with an information-retrieval effect rather than an improvement in optimization modeling. To support the claim, the authors must use a corpus that does not contain solutions to the test queries, or hold out entire test problems, and repeat the comparison.","section":"Section III-A"},{"comment":"The aggregate claim of superiority is not reflected in the per-base-LLM comparisons. On Haiku, MAG-RAG scores above pure LLM on only 3/10 problems (Q3, Q4, Q6) and above pure MA on 5/10; on GPT-4 it beats pure LLM on 5/10 with one tie and beats pure MA on 5/10. Only Sonnet and GPT-3.5 show more consistent gains. No confidence intervals, error bars, or significance tests are reported, and the table contains ties and zero scores (e.g., HT-Q3 and G3.5D-Q3) that are not discussed. The current evidence does not justify the abstract's blanket statement that MAG-RAG outperforms several AOM benchmarks.","section":"Section III-B / Table I"},{"comment":"The experiments do not isolate the contribution of the graph structure. The only RAG-based system is MAG-RAG; the baselines are a pure LLM and a multi-agent system whose knowledge comes from a Knowledge Generation Agent rather than retrieval. There is no flat or vector-RAG baseline using the same knowledge corpus, nor a comparison with existing AOM systems such as ORLM or Optimus. Consequently, the paper does not show that graph-based retrieval is superior to simpler retrieval, and the claim of outperforming 'several AOM benchmarks' is not supported because only two self-defined baselines are included.","section":"Section II-C / Section III-B"},{"comment":"The evaluation relies on scores assigned by three human scientists, each scoring a different subset of problems (Q1-Q3, Q4-Q6, Q7-Q10), without reporting the rubric operationalization, per-metric scores, inter-annotator agreement, or any measure of variability. The five metrics are aggregated into a single 100-point score, and Fig. 4 further aggregates across base LLMs and metrics. This makes it impossible to assess whether the reported differences are reliable or reflect annotator-specific scoring preferences, which the authors themselves acknowledge in the discussion of Table I.","section":"Section III-A (Metrics)"},{"comment":"The retrieval hyperparameters epsilon in Eq. (5) and k=3 are fixed without sensitivity analysis, and retrieval quality itself is never evaluated (e.g., recall of relevant nodes or whether the retrieved nodes actually contain the target formulation). Since the entire mechanism rests on retrieval effectiveness, the paper should report retrieval diagnostics and vary epsilon and k before attributing gains to Graph-RAG.","section":"Section II-B / Section II-C"}],"minor_comments":[{"comment":"The denominator in the cosine similarity formula should be ||vi|| * ||vj||, not ||vi|| * ||vi||.","section":"Equation (5)"},{"comment":"The 'Knowledge Generation Agent AKG' used in the Pure MA baseline is never defined or described; its prompts and behavior should be specified for reproducibility.","section":"Section III-A (Comparison Methods)"},{"comment":"The notation is inconsistent: K denotes retrieved knowledge in Eq. (3), but the keyword attribute is also called K in Section II-B; please use distinct symbols.","section":"Section II-B"},{"comment":"The zero scores in Table I (HT-Q3, G3.5D-Q3) are unexplained; if the model failed to produce a valid output, this should be stated explicitly.","section":"Table I"},{"comment":"The text says that 'four out of the eight samples had reduced scores originating from Q7'; with 30 problem-LLM combinations, the choice of 'eight' needs clarification.","section":"Fig. 4(B)"},{"comment":"The abstract and title emphasize sensor array signal processing, but some test problems (e.g., TOA-based localization) are not array-processing problems; please justify their inclusion or adjust the framing.","section":"Title / Section III-A"}],"recommendation":"major_revision","confidential_remarks":"The main empirical claim is currently unsupported because of the overlap between the retrieval corpus and the test set. I would like the editor to weigh whether the authors can credibly redo the evaluation; if the revision does not include a disjoint retrieval/test split, a flat-RAG baseline, and some statistical analysis, the paper should not be accepted. The comparison set is also very small (two self-defined baselines), so 'several benchmarks' overstates the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper combines two existing ideas—multi-agent LLM workflows and Graph-RAG—and applies them to automated optimization modeling for sensor array signal processing. That specific integration is new, and the authors ship code, prompts, and an evaluation dataset. I appreciate the honest reporting: they show scores for four base LLMs across ten problems, and they admit cases where retrieval hurts (Q7). The four-layer graph schema (PT, SM, OF, OA) is a sensible domain adaptation, and Fig. 4 gives a clear picture of where gains appear.\n\nThe soft spot is the one the stress-test flags, and it is load-bearing. SPAMR is built by selecting, for each of the ten test problems, documents containing standard modeling approaches. The same ten problems are then used as queries. MAG-RAG retrieves top-3 nodes from that corpus, so the retrieved context can contain the target formulation that the scoring rubric rewards. The pure LLM and pure MA baselines have no access to that corpus. So the headline improvement (67% highest-score rate vs. 25% and 8%) is compatible with a retrieval effect rather than better modeling. The paper does not report held-out problems, repeated runs, error bars, or an ablation removing the graph structure. On Haiku, MAG-RAG loses to pure LLM on 7/10 tasks; on GPT-4, it loses on 4/10. That inconsistency is consistent with the confound.\n\nThe authors do discuss some limitations in the conclusion, but they don't acknowledge the corpus-answer-key problem. The math and citations look fine; no red flags there.\n\nWho is this for? Someone working on LLM-based modeling automation might read it as a design study, not as evidence of superiority. The paper deserves peer review because the architecture is plausible and the flaw is fixable: rebuild the corpus so it doesn't contain solutions for the test queries, add held-out problems, and run multiple seeds. But as it stands, I would not cite the performance claim.\n\nRecommendation: send to reviewers with a clear request to focus on evaluation independence. If the authors can rerun with a non-overlapping corpus, the result could be meaningful.","headline":"A plausible architecture undermined by an evaluation where the retrieval corpus doubles as an answer key for the same ten queries.","tokens_in":7896,"tokens_out":1583,"would_cite":false,"duration_ms":13962,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Graph-RAG agents win 67% of top SASP modeling scores","keywords":["automated optimization modeling","large language models","sensor array signal processing","retrieval-augmented generation","multi-agent systems","graph-based retrieval"],"falsifier":"Run MAG-RAG on SASP problems whose standard solution documents are excluded from the graph corpus, or measure the overlap between retrieved nodes and the reference solution for each query; if the top-score advantage disappears on excluded problems, the reported gains come from known-answer retrieval.","tokens_in":1437,"feed_emoji":"📡","tokens_out":1308,"duration_ms":61655,"temperature":0.7,"pith_summary":"This paper argues that automated optimization modeling for sensor array signal processing (SASP) fails with plain prompt engineering because LLMs lack domain-specific knowledge. It proposes MAG-RAG, a pipeline that combines a multi-agent decomposition of the modeling task with a graph-based retrieval system that stores and retrieves SASP optimization knowledge. On ten classical SASP problems, the authors report that MAG-RAG achieves the highest score on 67 percent of evaluated items, versus 25 percent for the multi-agent-only baseline and 8 percent for the plain LLM. The central claim is that adding structured graph retrieval yields more complete and correct optimization models than prompting or agent chains alone.","feed_headline":"Graph-RAG agents win 67% of top SASP modeling scores","feed_subtitle":"Graph-based retrieval added to LLM agents makes signal-processing models more complete and correct.","key_machinery":"The key machinery is a Graph-RAG database coupled with a three-agent pipeline. An Extraction Agent distills raw domain documents into five-part modeling knowledge blocks (terminological description, example information, system model, optimization formulation, optimization algorithm), which are stored as a four-layer graph with nodes typed by layer and keyword embeddings added. A Terminology Agent translates a user's natural-language query into a technical problem description, the retrieval step selects the top-k problem-type nodes by cosine similarity (with k=3), and an Optimization Modeling Agent generates the final formulation using the knowledge blocks connected to those nodes by single-document edges. The graph structure lets a retrieved problem-type node pull along the complete modeling chain from the same document, giving the generation agent a full worked example rather than an isolated snippet.","core_discovery":"The paper's central discovery is that structuring retrieved domain knowledge as a four-layer graph—with layers for problem type, system model, optimization formulation, and optimization algorithm—and retrieving from the problem-type layer lets an LLM agent produce better optimization modeling results for SASP problems than either a pure LLM or a multi-agent chain without retrieval. This is demonstrated through human-scored evaluations on ten classical signal processing problems, where the full MAG-RAG system wins the highest metric scores 67 percent of the time, compared to 25 percent for the multi-agent-only approach and 8 percent for the plain LLM. The authors also report that the gain is concentrated in the completeness and correctness metrics, while standardization, relevance, and readability stay roughly level across methods.","pith_inferences":["A fair generalization test would run MAG-RAG on SASP problems whose standard solution documents are excluded from the retrieval corpus; if the top-score share falls toward the baseline levels, the reported advantage likely reflects retrieval of near-answer documents rather than general modeling competence.","The graph-based retrieval design could be transferred to other engineering optimization domains—such as communications, radar, or power systems—by swapping the SASP corpus for domain documents and re-running the extraction-agent pipeline.","The authors observe that readability and contextual relevance sometimes decline when prior knowledge is inserted, pointing to a trade-off between injecting domain content and preserving the user's phrasing; a reranking step that re-weights query relevance could recover some of those scores.","Ablating Graph-RAG to a flat vector-store retrieval would isolate whether the four-layer graph structure itself, rather than retrieval in general, is responsible for the gains."],"forward_implications":["The reported 67 percent top-score share for MAG-RAG, against 25 percent for pure multi-agent and 8 percent for pure LLM, implies that retrieval of structured domain knowledge is the main driver of modeling improvement.","Completeness and correctness improve most with MAG-RAG, while standardization, relevance, and readability remain roughly similar across methods, suggesting the knowledge insertion primarily strengthens the optimization-modeling content.","The positive effect of prior knowledge appears across multiple base LLMs, with positive score changes significantly outnumbering negative ones when retrieved knowledge is added.","Because the retrieval step returns complete single-document modeling chains, the agent receives coherent worked examples, which is likely why completeness and correctness scores rise."],"supporting_citations":[{"why":"Introduces retrieval-augmented generation, the foundational technique the paper adapts for SASP knowledge retrieval.","marker":"[18]"},{"why":"Supplies the graph-structured RAG approach that the paper extends into a four-layer domain-knowledge graph.","marker":"[19]"},{"why":"Provides the multi-agent decomposition of complex optimization tasks that motivates the MA pipeline in MAG-RAG.","marker":"[13]"},{"why":"Represents a prior prompt-based automated optimization modeling approach that the paper positions against and seeks to improve.","marker":"[6]"},{"why":"Supplies the text-embedding model used to compute node and query embeddings for cosine-similarity retrieval.","marker":"[26]"}],"fun_headline_variants":["Graph-RAG boosts LLM modeling to 67% top scores","LLM agents with graph retrieval ace SASP modeling","MAG-RAG: Graph retrieval lifts LLM optimization modeling","67% win rate: Graph-RAG agents dominate SASP modeling","Graph-based retrieval sharpens LLM agents for signal processing"],"cache_read_input_tokens":9984,"weakest_assumption_plain":"The ten evaluation problems are the same problems whose standard modeling documents seed the retrieval corpus, so a top score may reflect retrieving near-answer material rather than general modeling ability.","fun_headline_variants_meta":{"raw":{"variants":["Graph-RAG boosts LLM modeling to 67% top scores","LLM agents with graph retrieval ace SASP modeling","MAG-RAG: Graph retrieval lifts LLM optimization modeling","67% win rate: Graph-RAG agents dominate SASP modeling","Graph-based retrieval sharpens LLM agents for signal processing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000406,"raw_usage":{"total_tokens":2071,"prompt_tokens":865,"completion_tokens":1206,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":1121}},"tokens_in":481,"tokens_out":1206,"duration_ms":8163,"temperature":1.0,"reasoning_tokens":1121,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T23:54:32.840180+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MAG-RAG on SASP problems whose standard solution documents are excluded from the graph corpus, or measure the overlap between retrieved nodes and the reference solution for each query; if the top-score advantage disappears on excluded problems, the reported gains come from known-answer retrieval.","supporting_citations":[{"cited_title":"Chain-of-experts: When llms meet complex operations research problems,","cited_arxiv_id":null,"evidence_quote":"Provides the multi-agent decomposition of complex optimization tasks that motivates the MA pipeline in MAG-RAG."},{"cited_title":"Retrieval augmented zero-shot text classification,","cited_arxiv_id":null,"evidence_quote":"Supplies the text-embedding model used to compute node and query embeddings for cosine-similarity retrieval."}],"review_version":1}