{"id":"ccba3656-bc29-4460-84c2-b0ed03b7b387","arxiv_id":"2508.18724","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A multi-agent retrieval system that filters sources by a bias classifier reports an 81.82% relative drop in bias rate, but the evaluation uses the same classifier as the filter.","lead":"The authors built a multi-agent system that filters which news documents an AI retrieval assistant uses, aiming to cut the number of biased sources before an answer is written. In tests over 112 queries, one configuration lowered a biased-label rate from about 49% to 9%, although the effect was measured with the same bias classifier that does the filtering.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation circularity: Dbias is both the selection filter and the outcome metric, so the claimed 81.82% bias reduction may only reflect filtering on the classifier's own labels.","rationale":"The reader's weakest assumption precisely identifies the load-bearing flaw: the evaluation uses Dbias labels both as the selection criterion (Section 3.3.2 and Equation 4) and as the outcome metric (Section 4.1, Figure 5). This is not a minor methodological quibble; it directly undermines the paper's headline quantitative claim. Filtering on a classifier's predicted labels will necessarily lower the measured positive rate on that same classifier, so the reported 81.82% reduction cannot be interpreted as evidence of reduced bias without an independent validation step. The paper does report using MBIC and BABE, which contain human annotations, but does not use those annotations as the outcome measure, so the relevant external check is available in principle and absent in practice. I agree with the reader that the appropriate disposition is REJECT or, equivalently, no change to the reader's REJECT verdict. The architecture itself is coherent and could plausibly reduce bias under a properly validated metric, but the central empirical claim is not supported as currently presented. I would not manufacture a different concern: the circularity is sufficient and is the most load-bearing issue.","tokens_in":10660,"tokens_out":5813,"duration_ms":70189,"concrete_test":"Hold out the human bias labels from MBIC/BABE and recompute the reported bias rates using those labels as the outcome metric, while keeping Dbias only for candidate selection. For each of the 112 queries, run the no-source-selection baseline and the zero-shot/few-shot modes, then score each final output with the human labels (or, if human labels are unavailable, with a second independently trained bias classifier not used in the pipeline). Compute the bias-rate reduction under this independent metric. If the zero-shot reduction relative to baseline is substantially smaller than 81.82% (or absent), the claimed bias mitigation is an artifact of filtering on Dbias's own predictions rather than a real reduction in bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim rests on a circular evaluation. Section 3.3.2 defines the bias detector as Dbias, which produces the binary label γ and confidence β. Equation 4 in Section 3.3.3 selects only candidates with γ=0 and β≥0.7 (with few-shot analogously conditioning on β, γ, ρ). The headline outcome in Section 4.1 and Figure 5 is the fraction of outputs \"labeled biased\" by that same bias classifier. Selecting documents the classifier labels unbiased mechanically lowers the classifier's positive rate; the drop from 49.11% to 8.93% for GPT-4o-mini is therefore expected even if the classifier is noisy or only weakly related to reader-relevant bias. Retry logic and relaxed final-attempt rules do not remove this problem, because both still use Dbias's assessments. No independent outcome measure is reported: MBIC/BABE human annotations are mentioned but not used to score final outputs, no second bias classifier is used, and no human evaluation is presented. A secondary confound is that the baseline retrieves one top-1 document while the agent retrieves a candidate set, so the comparison conflates candidate-pool size with bias-based selection. Without an external evaluation signal, the 81.82% reduction is not established as a genuine fairness improvement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Bias Mitigation Agent, a multi-agent framework for retrieval-augmented generation that aims to reduce bias in retrieved knowledge. A manager agent coordinates a knowledge agent (retrieval via ChromaDB), a bias detection agent (using the Dbias classifier to assign bias confidence β and binary label γ), a source selection agent (zero-shot rule-based filtering or few-shot example-conditioned selection), and a writer agent. Three modes are compared: no source selection (baseline, top-1 retrieval), zero-shot selection, and few-shot selection, across three GPT models on 112 curated queries from MBIC/BABE. The headline result is an 81.82% reduction in bias rate for GPT-4o-mini in zero-shot mode compared to baseline (8.93% vs. 49.11%), with smaller reductions reported for other model/mode combinations.","tokens_in":10953,"tokens_out":3464,"duration_ms":40210,"significance":"If the reported effectiveness were established, the paper would offer a practical, modular architecture for fairness-aware source selection in RAG/agentic systems, with useful engineering components: explicit state management, retry logic, and a comparison of zero-shot and few-shot selectors. The use of MBIC/BABE news datasets is appropriate for the problem domain. However, the central empirical claim is not currently supported because the evaluation metric is the same classifier used to select sources, and the baseline is confounded with candidate-pool size. The paper's potential is real, but the evidence as presented does not demonstrate a genuine fairness improvement.","major_comments":[{"comment":"The evaluation is circular. The Dbias classifier in §3.3.2 produces the binary label γ and confidence β used in Eq. (4) to select only candidates with γ=0 and β≥0.7 (and analogously for few-shot). The outcome metric in §4.1 and Fig. 5 is the fraction of outputs the same Dbias-style classifier labels biased. Filtering on a classifier's own labels mechanically lowers its positive rate, even if the classifier is noisy or unrelated to reader-relevant bias. The reported 81.82% reduction is therefore not evidence of genuine bias mitigation. An independent outcome measure is required: e.g., scoring final outputs with human annotations from MBIC/BABE, or a second, independently trained bias classifier, and reporting agreement/error rates. Without such an external signal, the central claim is unsubstantiated.","section":"§3.3.2, §3.3.3, Eq. (4), §4.1, Fig. 5"},{"comment":"The baseline and the proposed agent differ in two confounded ways. The baseline retrieves a single top-1 document by Eq. (3), while the agent retrieves a candidate set C and then selects from it in Eq. (4). Thus the comparison conflates candidate-pool size with bias-based selection. To isolate the effect of source selection, the baseline should be run on the same candidate pool (e.g., randomly selecting among top-k candidates, or using top-1 from the same retrieved set), and the results reported separately for first-attempt and final outputs.","section":"§3.3.1, §3.3.3, Eqs. (3)–(4)"},{"comment":"The retry/query-expansion mechanism is a second confound. When no candidate meets the selection criteria, the knowledge agent rewrites the query based on the rejection reason ρ and retrieves a new candidate set. The retry rate is high (70.54% for GPT-4o-mini zero-shot in Fig. 6), and the final relaxed selection still relies on Dbias scores. The paper does not report how much of the bias reduction is due to the selection rule versus query rewriting, nor does it analyze first-attempt vs. final-attempt outcomes. Such an analysis is needed to attribute the improvement to the source selection mechanism as claimed.","section":"§3.3.1, Fig. 6, §4.2"},{"comment":"The experimental report lacks statistical grounding. Results are based on 112 queries, but no confidence intervals, standard errors, or significance tests are given for the bias rates shown in Fig. 5. The headline 81.82% reduction is for one model/mode; other rows show smaller gains (e.g., GPT-4.1-mini zero-shot drops from 52.68% to 27.68%). The paper should provide uncertainty estimates and, ideally, paired comparisons across models and modes to support claims that the agent 'significantly outperforms' the baseline.","section":"§4.1, Fig. 5, Table 1"},{"comment":"There are internal inconsistencies in the reported relevance scores and in the text. Eq. (2) defines ρ ∈ [0,1], but Table 1 lists negative relevance values (e.g., -0.058). The text in §4.2 states that GPT-4o-mini zero-shot's 'beat average relevance score was 0.366... even better than the baseline,' yet Table 1 shows 4o-mini zero-shot relevance of 0.157 ± 0.078 versus 0.169 ± 0.092 for the baseline. Clarify the relevance metric, its range, and correct the comparative claims.","section":"§4.2, §4.3, Table 1"}],"minor_comments":[{"comment":"The phrase 'Now a days' in §1 is informal; should be 'Nowadays'.","section":"Abstract/Introduction"},{"comment":"The text says 'The beat average relevance score' — likely a typo for 'The best average relevance score.'","section":"§4.2"},{"comment":"The sentence 'it achieved a bias rate of 14.3%, demonstrating a substantial improvement over the baseline mode by 69.48%' is ambiguous: state the absolute rates and the relative reduction clearly.","section":"§4.3"},{"comment":"Figure 5 lacks error bars or uncertainty intervals; add them or indicate the per-query variance. Also define what 'bias rate' is precisely (fraction of outputs labeled biased by Dbias?) in the caption or text.","section":"§4.1, Fig. 5"},{"comment":"Eq. (5) defines f_few-shot as 'learned or example-conditioned scoring function implicitly encoded via prompt demonstrations,' but no details of the few-shot examples (size, content, selection) are given. Include them or a reference to a supplemental appendix for reproducibility.","section":"§3.3.3"},{"comment":"The paper does not include a limitations section. Given the evaluation issues raised above, a candid discussion of the reliance on Dbias and the lack of human evaluation is necessary.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The circular evaluation is the core issue; it is fixable but requires substantial new experiments (independent outcome metrics, matched baselines, statistical tests). The paper also lacks code/data release and reproducibility details for the few-shot mode. If the authors can supply an external validation of the bias reduction, the manuscript could be resubmitted; in its current form, the headline claim is not established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline number is not supported. The paper uses Dbias both to filter candidate sources (Equation 4) and to label the final outputs as biased or unbiased (Figure 5). Selecting documents the classifier calls unbiased mechanically lowers the classifier's positive rate. The 49% to 8.9% drop for GPT-4o-mini tells you the filter works as designed, not that the system reduces bias in any reader-relevant sense. The few-shot and other model rows show the same issue; there is no independent bias metric, no human evaluation, and the MBIC/BABE ground-truth labels are mentioned but never used to score outputs.\n\nTo be fair, the paper does have real content. The architecture is clearly described: a supervisor coordinating a knowledge agent, a bias detector, a source selector, and a writer, with zero-shot and few-shot selection modes. The full 3x3 table with relevance, confidence, latency, and retry rates is a useful empirical sketch, and the writing is straightforward. The engineering is coherent, and in a narrow sense the specific numbers are new, not present in the cited prior work.\n\nThe soft spots beyond circularity: the baseline retrieves a single top-1 document while the agent retrieves a candidate set, so the comparison conflates pool size with bias-based selection. The retry logic, query expansion, candidate count, few-shot example set, and relaxed final-attempt rules are underspecified, and all of them can move the reported numbers. There's also no diff against the authors' own earlier Bias-Aware Agent (reference [19]), so I can't tell what this paper adds. No code or data are released.\n\nWho is this for? Someone building an agentic RAG pipeline might find the system description a useful starting point, but not the evaluation. As a research claim, it fails. As a systems note, it's a passable skeleton.\n\nFor peer review, I'd send it out anyway: the topic is timely, the architecture is concrete, and the circularity is a fixable problem if the authors re-evaluate with an independent classifier or human annotations. But I would reject it in its current form and make that the central revision request.","headline":"The headline claim is not supported: the same classifier filters sources and scores the outcome, so the 81.82% bias reduction is largely circular.","tokens_in":11480,"tokens_out":2331,"would_cite":false,"duration_ms":26073,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A source-selection gate powered by a bias classifier cuts biased retrieval outputs by 81.82% over a relevance-only baseline while preserving relevance, in the paper's best configuration.","keywords":["Information Retrieval","Agents","Retrieval Augmented Generation","Large Language Models","Bias","Fairness","multi-agent system","source selection"],"falsifier":"Have independent human annotators label a held-out sample of the generated answers from baseline and zero-shot modes without knowing the mode; if the human-labeled bias-rate gap is far below 81.82%, the reported reduction is an artifact of Dbias. Also test for label instability by rewording the same query and checking whether the same document's Dbias label flips, which would show the retry loop can be gamed.","tokens_in":10514,"feed_emoji":"⚖️","tokens_out":6921,"duration_ms":76242,"temperature":0.7,"pith_summary":"The paper sets out to show that bias in knowledge retrieval can be cut at the point of source selection, before a language model writes an answer. It builds a supervisor-based multi-agent system in which a knowledge agent retrieves candidate documents, a bias-detection agent scores each candidate using Dbias, a source-selector agent passes only high-confidence unbiased documents to the writer, and the manager enforces retries when no candidate passes the gate. On 112 queries drawn from two annotated news datasets, the zero-shot selector with GPT-4o-mini lowered the bias rate from 49.11% to 8.93%, an 81.82% reduction relative to a naive relevance-only retrieval baseline, while average relevance rose. The authors treat this as evidence that agentic source selection can make retrieval fairer without sacrificing relevance.","feed_headline":"Agent filters biased sources, cutting retrieval bias by 81.82%","feed_subtitle":"A source-selecting agent cuts biased outputs by 81.82% without sacrificing relevance.","key_machinery":"The load-bearing object is the Source Selection Agent's threshold gate: only candidates with binary bias label γ=0 and detector confidence β≥0.7 are eligible, and among those the highest-relevance document wins. The Manager Agent powers retries by query expansion when no candidate passes, while few-shot mode replaces the raw threshold with example-conditioned scoring. This gate is the entire mechanism that separates the new workflow from ordinary relevance-based retrieval.","core_discovery":"The central claim is that a bias filter inserted between retrieval and generation reduces biased output by roughly 82% compared with a naive retrieval strategy, and that the reduction does not come at the cost of relevance. The mechanism is a threshold rule: accept only candidate documents with binary bias label γ=0 and detector confidence β≥0.7, then choose the most relevant of those; if none qualify, expand the query and retry. In few-shot mode, the same decision is guided by in-context examples rather than a hard threshold. The best configuration (zero-shot, GPT-4o-mini) achieved 8.93% biased outputs versus 49.11% for the baseline, and its average relevance of 0.366 exceeded the baseline'","pith_inferences":["The measured reduction may be inflated by measurement coupling: the selector filters on Dbias labels and the reported bias rate is computed from the same labels, so part of the gain could be detector-error filtering rather than reader-visible fairness.","The retry mechanism invites gaming: each retry expands the query, so the system could keep rephrasing until a document happens to be classified unbiased, selecting for clean-looking rather than genuinely balanced sources.","Choosing exactly one source means opposing viewpoints cannot be combined; an extension that lets the writer synthesize several complementary sources could improve balance without relaxing the bias gate.","The architecture is portable only where a trustworthy bias detector exists; with an unreliable detector, the gate would give false confidence in downstream fairness."],"forward_implications":["In the paper's best configuration, bias-gated selection lowered biased outputs from 49.11% to 8.93% while average relevance rose from 0.169 to 0.366.","The mechanism is model- and mode-dependent: GPT-4.1 and GPT-4.1-mini still improve, but by smaller margins, and few-shot is usually less aggressive than zero-shot on bias for GPT-4o-mini.","Because the writer is told to rely only on the selected source, the fairness gain comes from input choice rather than output rewriting, keeping the intervention transparent and modular.","Few-shot selection retries far less often than zero-shot, so a substantial fairness gain can be obtained at lower latency in that mode."],"supporting_citations":[{"why":"Supplies Dbias, the bias classifier that produces the γ and β scores used by the selection rule and by the reported bias-rate metric.","marker":"[16]"},{"why":"Supplies BABE, one of the two expert-annotated news corpora from which the 112 evaluation queries were curated.","marker":"[20]"},{"why":"Supplies MBIC, the second annotated news corpus used to source documents and queries for evaluation.","marker":"[21]"},{"why":"Establishes retrieval-augmented generation as the retrieval-plus-generation setting that the relevance-only baseline and the agent both operate in.","marker":"[13]"}],"fun_headline_variants":["Agent cuts biased sources, bias down 81.82%","Source-selecting agent reduces bias by 81.82%","Agent filters sources, cutting bias 81.82% without losing relevance","Agent picks unbiased docs, bias falls 81.82%","Bias mitigation agent cuts retrieval bias by 81.82%"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The claimed reduction depends on Dbias's labels being the right measure of bias, because the selector rejects anything Dbias flags and the final bias rate counts Dbias labels on the outputs.","fun_headline_variants_meta":{"raw":{"variants":["Agent cuts biased sources, bias down 81.82%","Source-selecting agent reduces bias by 81.82%","Agent filters sources, cutting bias 81.82% without losing relevance","Agent picks unbiased docs, bias falls 81.82%","Bias mitigation agent cuts retrieval bias by 81.82%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001168,"raw_usage":{"total_tokens":4635,"prompt_tokens":675,"completion_tokens":3960,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":419,"completion_tokens_details":{"reasoning_tokens":3871}},"tokens_in":419,"tokens_out":3960,"duration_ms":32294,"temperature":1.0,"reasoning_tokens":3871,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:16:19.681612+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent human annotators label a held-out sample of the generated answers from baseline and zero-shot modes without knowing the mode; if the human-labeled bias-rate gap is far below 81.82%, the reported reduction is an artifact of Dbias. Also test for label instability by rewording the same query and checking whether the same document's Dbias label flips, which would show the retry loop can be gamed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Dbias, the bias classifier that produces the γ and β scores used by the selection rule and by the reported bias-rate metric."},{"cited_title":"MBIC -- A Media Bias Annotation Dataset Including Annotator Characteristics","cited_arxiv_id":"2105.11910","evidence_quote":"Supplies MBIC, the second annotated news corpus used to source documents and queries for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes retrieval-augmented generation as the retrieval-plus-generation setting that the relevance-only baseline and the agent both operate in."}],"review_version":1}