{"id":"605ef3ba-6779-49ea-a355-c4fbbb4d6154","arxiv_id":"2608.02877","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"GoT-CD applies graph-of-thoughts reasoning to causal discovery and shows that structural fidelity does not guarantee that a path-specific fairness audit recovers the true sensitive-to-outcome pathway.","lead":"This paper introduces GoT-CD, a large-language-model method that treats causal discovery as reasoning over complete candidate edge sets, and it demonstrates that post-hoc fairness audits on discovered graphs can report a confident zero when the sensitive-attribute pathway was missed. A smart generalist should read it because it shows why structural accuracy alone is not enough to trust fairness conclusions derived from discovered causal graphs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported GoT-CD results are internally inconsistent across tables for the same locked run, so the empirical demonstration of the fairness-fragility claim is not yet reproducible.","rationale":"The reader's weakest assumption identifies the same core problem: the results rest on a single locked run without variance estimates, and the hand-specified scorer and order-dependent greedy projection are not isolated. My stress-test finds the issue is broader and more concrete than the reader stated: Table 1 and Table 3 disagree not only on Alzheimer's F1 (0.757 vs. 0.811) but also on Asia, COVID-R, and Sweden-T, and Section 6.1's k=3 ablation description (one path, PSE = 0.582) contradicts Table 5 (two paths, PSE = 1.031) for the same configuration. This means the central empirical demonstration—that discovery methods can miss the unfair pathway while GoT-CD recovers it—is not yet reproducible. The formal claim is not in question; it is nearly definitional. But the paper's value is the demonstration, and that value is currently undermined by the unavailability of artifacts and by the order-dependence of the cycle-breaking step, which is acknowledged in Section 3.2 but not analyzed in Section 6.4. Releasing logs and running multi-seed experiments would settle whether the reported graphs and fairness outcomes are stable. Until then, the conditional verdict stands; the authors should resolve the table inconsistencies, add variance estimates, and release the protocol artifacts.","tokens_in":14982,"tokens_out":9029,"duration_ms":84938,"concrete_test":"Release the full logged protocol (prompts, raw model responses, parsed edge lists, validity scores, and greedy-projection edge order) for the GoT-CD Alzheimer's runs behind Tables 1, 3, and 5 and recompute both tables from those logs; then run the same locked protocol across 10 seeds and report F1, PSE|·|, and binary unfair-path recovery with a confidence interval. If Tables 1 and 3 cannot be reconciled from the same logs, or if GoT-CD recovers the pathway in fewer than 8 of 10 seeds, the path-recovery claim is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The formal assertion is sound: path-specific effects are sums over directed paths in the supplied DAG, so a graph with no S→Y path yields PSE = 0 by construction. The paper's empirical contribution is showing that real discovery methods produce such graphs and that GoT-CD does not. That empirical support is currently unreliable. Under the stated locked protocol (k=3, seed 0), Table 1 gives GoT-CD F1 = 0.757 on Alzheimer's with 18 predicted edges, and Table 5 reports two S→Y paths with PSE|·| = 1.031; Table 3's k=3 row gives F1 = 0.811 for the same configuration, and Section 6.1 describes a k=3 ablation run recovering exactly one path with PSE|·| = 0.582. Similar discrepancies appear for Asia, COVID-R, and Sweden-T between Table 2 and Table 3. Because every run is single-seed, no code or data are provided, and the greedy projection's tie-breaking is admitted to be order-dependent, the reader cannot determine which graph is the actual output or whether the path-recovery result is a stable property of GoT-CD or a lucky draw. If the reported graphs are not reproducible, the paper's empirical demonstration—five of eight methods missing the path, GoT-CD recovering it—is not established, even though the formal 'confident zero' point is correct.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GoT-CD, an LLM-based causal discovery method that treats complete candidate edge sets as 'thoughts' in a Graph-of-Thoughts pipeline, scores them with a deterministic validity function, merges candidates under a hard union constraint, and enforces acyclicity by greedy projection. It reports structural benchmarks on five datasets at n=100 with gpt-4o-mini, claiming DAG-valid outputs everywhere and the best DAG-valid F1 among LLM baselines on Asia, Alzheimer's, and COVID-Respiratory. On an Alzheimer's benchmark with the designated unfair path Sex→Brain Volume→MOCA Score, the paper applies a post-hoc path-specific fairness audit and finds that five of eight discovered graphs contain no S→Y path and hence report PSE=0, while GoT-CD recovers the path and GES inflates path mass; it concludes that structural fidelity does not imply fairness fidelity and recommends reporting pathway recovery alongside fairness estimates.","tokens_in":15319,"tokens_out":8996,"duration_ms":78843,"significance":"The 'confident zero' failure mode is a real and practically important point: since path-specific effects are sums over directed paths in the supplied graph, a missing protected-attribute pathway produces an estimate of exactly zero rather than a flagged uncertainty. This observation is partly definitional, and the paper states it correctly in Section 6.3; the value added lies in the empirical demonstration that real discovery methods actually produce such graphs and in the proposed GoT-CD method. The method design—full-graph reasoning units, deterministic scoring, and a hard union constraint—is a reasonable contribution, and the paper is transparent about several limitations in Section 6.4 (single backbone, single sample size, simulated data). However, the central empirical claims currently rest on internally inconsistent, single-run results with no code or data, so the paper's main demonstration cannot yet be verified.","major_comments":[{"comment":"The paper's locked protocol (k=3, seed 0, n=100) gives different results in different tables for every dataset: Table 2 reports GoT-CD F1 = 0.750 on Asia, 0.757 on Alzheimer's, 0.688 on COVID-R, and 0.111 on Sweden-T, while Table 3's k=3 row reports 0.632, 0.811, 0.727, and 0.286, respectively; Table 5 reports two S→Y paths with PSE|·|=1.031 for GoT-CD on Alzheimer's, whereas Section 6.1 describes the k=3 setting as recovering exactly one path with PSE|·|=0.582. Because all runs are claimed to be locked, these discrepancies are not a matter of error bars; the reader cannot determine which graph is the actual output of the method. Since the paper's central empirical claim is that GoT-CD recovers the fairness-relevant pathway while five baselines do not, this contradiction undermines the main contribution and must be resolved with a consistent protocol, code, data, and preferably multiple seeds.","section":"Tables 2-3 and Section 6.1"},{"comment":"All reported results are single runs with no standard errors, no multiple seeds, and no code or data release. LLM generation at temperature 0.7 is stochastic, and a fixed client-side seed does not by itself guarantee reproducible API outputs across versions, so the observed path-recovery result could be a single lucky draw. The paper should report repeated runs (or at least error bars and full candidate logs) before claiming that GoT-CD stably recovers the unfair path or that five of eight methods reliably produce confident zeros.","section":"Section 4"},{"comment":"The acyclicity guarantee is achieved by a final greedy projection whose output is order-dependent: the paper itself states that the surviving edge in any proposed cycle is determined by position in the emitted list rather than by evidential support. No tie-breaking rule, random order, or sensitivity analysis is provided. Because the projected DAG is exactly the graph used for the fairness audit, this arbitrary ordering could determine whether the S→Y pathway survives, so the central comparison requires either a deterministic tie-breaking rule with justification or results aggregated over many orderings.","section":"Section 3.2"},{"comment":"The deterministic validity scorer has several hand-chosen parameters (undeclared-variable penalty -5, self-loop penalty -3, acyclicity bonus +3, cyclic penalty -4, and density bounds 0.5n to 2.5n), and no sensitivity analysis is reported except for the branch factor k. With only single runs, these arbitrary weights could dominate the behavior of the method, so the paper does not yet establish that full-graph reasoning topology, rather than the specific scorer and projection choices, is responsible for the reported performance.","section":"Section 3.2"}],"minor_comments":[{"comment":"Reference [2] is a verbatim duplicate of reference [1] but is cited as the source of the 'recent collection of applied causal graphs' and the Alzheimer's reference graph; this citation cannot be correct and must be fixed.","section":"References [1] and [2]"},{"comment":"The k=5 row on Sweden-Traffic is reported as undefined F1 (p+r=0), meaning GoT-CD outputs an empty graph at k=5; this deserves an explanation and a discussion of what it implies for the method's behavior.","section":"Section 4.3"},{"comment":"The GoT-CD-BFS hybrid is described only briefly; please specify how candidate variable sets are generated, scored, and merged so the ablation is reproducible.","section":"Section 3.3"},{"comment":"The wording that full-graph reasoning 'yields always-acyclic discovered graphs' overstates the mechanism, since acyclicity is imposed by post-processing rather than by the reasoning process itself; rephrase for accuracy.","section":"Abstract and Sections 1.5/6.1"},{"comment":"The paper would benefit from a data and code availability statement; currently no link or repository is mentioned.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central conceptual point is worth publishing after the empirical claims are made reproducible. I would not accept the paper in current form; the cross-table inconsistencies are the main blocker. I also note that the duplicate reference [1]/[2] suggests a missing citation for the benchmark source."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper's central warning is right. A discovery graph that omits the S→Y path makes a post-hoc path-specific fairness audit return a confident zero, not a flagged failure, and that is exactly what an auditor would see as genuine fairness. The formal point is definitional—PSE is a sum over directed paths in the supplied DAG—but the paper's contribution is showing that real discovery methods produce such graphs and that their structural F1 does not predict pathway survival. That is a useful caution for the clinical auditing and fairness-discovery communities.\n\nThe GoT-CD method is a legitimate variation on LLM-based discovery: the reasoning unit is the whole candidate edge set, the scorer is deterministic, and the hard union constraint prevents the model from inventing edges. The new evaluation axis—asking whether the specific path an audit depends on survived discovery—is a good idea and should be adopted more widely. The paper is also admirably honest in its limitations section: it names the order-dependent greedy projection, the single backbone, the single sample size, and the reconstructed Sweden-Traffic graph.\n\nNow the soft spot, and it is substantial. The numbers do not cohere. Table 1 gives GoT-CD an F1 of 0.757 on Alzheimer's; Table 3 gives 0.811 for the same k=3 configuration. Table 5 shows two S→Y paths with PSE|·|=1.031, while Section 6.1 describes the k=3 ablation run recovering exactly one path with PSE|·|=0.582. Similar mismatches appear for Asia, COVID-Respiratory, and Sweden-Traffic between Table 2 and Table 3. Every run is single-seed, no error bars are provided, no code or data are released, and the greedy projection's tie-breaking is order-dependent. A reader cannot tell which graph is the actual locked-protocol output, or whether the pathway recovery is a stable property of GoT-CD or a lucky draw. That undermines the paper's load-bearing empirical demonstration—five of eight methods missing the path, GoT-CD recovering it—even though the formal \"confident zero\" point stands.\n\nThis paper deserves a serious referee, but the referee should be instructed to check the tables and demand reconciliation. My recommendation is conditional accept: the authors must fix the Table 1/Table 3 inconsistency (and the other cross-table mismatches), report variance across repeated runs, and release the generation, parsing, and projection artifacts. If they do, the paper becomes a valuable caution. If not, the empirical core is not reproducible and the contribution reduces to a true but thin definitional observation.","headline":"The fairness-fragility warning is correct and the GoT-CD design is a genuine variation, but the reported numbers are internally inconsistent and need repair before the empirical claim is credible.","tokens_in":15813,"tokens_out":2076,"would_cite":false,"duration_ms":18646,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A discovered graph that misses the protected-attribute pathway produces a confident zero in a path-specific fairness audit, indistinguishable from genuine fairness.","keywords":["causal discovery","large language models","graph-of-thoughts","path-specific fairness","counterfactual fairness","DAG validity","Alzheimer's disease","clinical decision support"],"falsifier":"Run the locked Alzheimer's protocol ($n=100$, the same backbone, $k=3$, seed 0) ten times and record $F_1$ and path-specific effect per run; if any graph with no $S\\to Y$ path returns a nonzero or flagged estimate, the confident-zero mechanism fails, and if the two reported $F_1$ values for the same configuration (0.757 in Table 1 and 0.811 in Table 3) do not reproduce, the runs are not stable enough to support the ranking.","tokens_in":14823,"feed_emoji":"🧠","tokens_out":6588,"duration_ms":71106,"temperature":0.7,"pith_summary":"This paper claims that a causal graph discovered from data can look structurally accurate yet be badly wrong for the fairness question it is meant to answer. It introduces GoT-CD, a language-model-based discovery method whose reasoning unit is a complete candidate edge set, and shows that on five benchmarks GoT-CD returns valid DAGs with the best structural F1 among LLM methods on three of them. On an Alzheimer's benchmark with a known unfair pathway, however, five of eight discovered graphs—including structurally competitive ones—contain no path from the protected attribute to the outcome and therefore report a path-specific effect of exactly zero, indistinguishable from a genuinely fair result. The paper argues that chains of discovery-plus-audit can certify unfair models as fair, and that evaluation should score survival of the specific pathway the audit depends on, not aggregate structural agreement alone.","feed_headline":"A missing bias pathway looks exactly like a fair model","feed_subtitle":"GoT-CD recovers the unfair path on Alzheimer's; five rival graphs report a false zero.","key_machinery":"The load-bearing object is the candidate edge set as the unit of reasoning in a Graph-of-Thoughts pass: Generate produces k complete graphs in parallel, Score applies a deterministic validity function, KeepBestN retains the best candidates, Aggregate merges them under a hard union constraint that forbids edges no reasoning branch proposed, Improve refines the merged graph, and a final greedy projection guarantees a DAG. The fairness side rests on the path-specific effect: for each simple directed path π from protected attribute S to outcome Y, $PSE(\\pi)=\\prod_{l=0}^{L-1} B_{v_l,v_{l+1}}$, and $PSE_{|\\cdot|}=\\sum_\\pi |PSE(\\pi)|$ measures total path mass. The mechanism that produces the false-clean result is that a graph with no $S\\to Y$ path yields exactly zero path-specific effect by construction, the same number a genuinely fair model would produce.","core_discovery":"The central claim is that path-specific fairness audits are only as good as the graph they are handed, and aggregate structural scores do not reveal when the relevant sub-structure is missing. On the Alzheimer's benchmark, GoT-CD recovers the ground-truth unfair path Sex→Brain Volume→MOCA Score, giving a total linear effect of −0.580 against the benchmark's −0.572, while LLM-BFS, with a competitive F1 of 0.649, recovers no S→Y path at all and reports PSE equal to zero and zero discrete effects. Because this zero is produced by structural omission, it is a false-clean certificate rather than evidence of fairness. GES shows the complementary failure: it finds the path but inflates total path mass to 4.034 via eight spurious routes, versus a true value of 0.572. The paper concludes that discovery-to-fairness pipelines should report pathway recovery alongside the fairness estimate, and that discovery methods intended for such pipelines should be evaluated on the edges the downstream analysis depends on.","pith_inferences":["The confident-zero mechanism is not specific to fairness: any estimand defined over a designated path, such as a mediation effect or a policy-relevant causal pathway, will inherit a missing pathway as an exact zero rather than a flagged failure; the paper's Alzheimer's case is one instance of a general property.","A natural testable extension is to replace the deterministic validity scorer with a data-based score such as BIC or conditional-independence statistics while keeping the global thought structure; the paper's own Sweden-Traffic results suggest this would improve pathway recovery where the language-model prior alone fails.","The order-dependent greedy DAG projection could be made more stable by ordering candidate edges by scorer reliability or confidence rather than list position, reducing the arbitrariness of which edge in a proposed cycle survives.","Running the locked protocol across multiple seeds, temperatures, and backbones would test whether the divergence between structural and fairness fidelity is a property of these methods or of this particular protocol; the paper leaves that question open."],"forward_implications":["For any pipeline that chains causal discovery to path-specific fairness, reporting whether the $S\\to Y$ pathway survived discovery is a minimal safeguard; without it, an omitted pathway is indistinguishable from a genuinely fair result.","Ranking discovery methods by aggregate structural F1 can mislead fairness users: LLM-BFS scores 0.649 on Alzheimer's yet misses the only unfair path, while GoT-CD at 0.757 recovers it.","Always-acyclic output matters for auditability: GoT-CD returns valid DAGs on all five benchmarks, whereas PC is cyclic on all five and GES on four, forcing heuristic edge-breaking before path-specific estimands are well-defined.","Discovery methods fail in opposite directions: a missing pathway produces a confident zero, while spurious recovered pathways can inflate unfairness mass, as GES does with PSE equal to 4.034 against a true 0.572.","The gain of GoT-CD comes from full-graph reasoning rather than Graph-of-Thoughts machinery as such, since the GoT-within-BFS hybrid underperforms and misses the unfair pathway on Alzheimer's."],"supporting_citations":[{"why":"Supplies the Graph of Thoughts operations (Generate, Score, KeepBestN, Aggregate, Improve) that GoT-CD adapts to full-graph edge sets.","marker":"[4]"},{"why":"Defines the LLM-BFS traversal baseline and the precision/recall/F1 reporting protocol that the structural comparison follows.","marker":"[13]"},{"why":"Establishes LLM pairwise causal querying, the baseline whose edge-flooding failure GoT-CD is contrasted against.","marker":"[16]"},{"why":"Provides the path-specific counterfactual fairness framework used post-hoc to compute path-specific effects on discovered graphs.","marker":"[41]"},{"why":"Supplies the Alzheimer's Disease reference graph and linear data used as the fairness case study with the known unfair pathway.","marker":"[2]"},{"why":"Provides GES, the strongest classical baseline on Alzheimer's, which recovers the unfair path but inflates path mass.","marker":"[8]"},{"why":"Provides NOTEARS, the continuous-relaxation baseline that dominates the linear-Gaussian Sweden-Traffic benchmark.","marker":"[47]"},{"why":"Provides DAGMA-linear, another continuous-relaxation baseline included in the structural comparison.","marker":"[3]"}],"fun_headline_variants":["A missing bias path can fake a fair verdict","Fairness audits fail when graphs miss bias paths","Missing path = false clean bill of fairness","GoT-CD finds bias path; rivals report false zero","False zero fairness: when discovery loses bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single locked run with one language-model backbone, one temperature, and one random seed is representative of how the methods compare, and that the hand-written validity scorer and the order-dependent greedy DAG projection do not dominate the reported rankings.","fun_headline_variants_meta":{"raw":{"variants":["A missing bias path can fake a fair verdict","Fairness audits fail when graphs miss bias paths","Missing path = false clean bill of fairness","GoT-CD finds bias path; rivals report false zero","False zero fairness: when discovery loses bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000809,"raw_usage":{"total_tokens":3601,"prompt_tokens":1049,"completion_tokens":2552,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":2481}},"tokens_in":665,"tokens_out":2552,"duration_ms":17080,"temperature":1.0,"reasoning_tokens":2481,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:56:14.984318+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the locked Alzheimer's protocol ($n=100$, the same backbone, $k=3$, seed 0) ten times and record $F_1$ and path-specific effect per run; if any graph with no $S\\to Y$ path returns a nonzero or flagged estimate, the confident-zero mechanism fails, and if the two reported $F_1$ values for the same configuration (0.757 in Table 1 and 0.811 in Table 3) do not reproduce, the runs are not stable enough to support the ranking.","supporting_citations":[{"cited_title":"Causal reasoning and large language models: opening a new frontier for causality.Transactions on Machine Learning Research, TMLR, 2023","cited_arxiv_id":null,"evidence_quote":"Establishes LLM pairwise causal querying, the baseline whose edge-flooding failure GoT-CD is contrasted against."},{"cited_title":"PC-fairness: a unified framework for measuring causality-based fairness","cited_arxiv_id":null,"evidence_quote":"Provides the path-specific counterfactual fairness framework used post-hoc to compute path-specific effects on discovered graphs."},{"cited_title":"Causal modelling agents: causal graph discovery 15 through synergising metadata- and data-driven reasoning","cited_arxiv_id":null,"evidence_quote":"Supplies the Alzheimer's Disease reference graph and linear data used as the fairness case study with the known unfair pathway."},{"cited_title":"Optimal structure identification with greedy search.Journal of Machine Learning Research, 3(Nov):507–554, 2002","cited_arxiv_id":null,"evidence_quote":"Provides GES, the strongest classical baseline on Alzheimer's, which recovers the unfair path but inflates path mass."},{"cited_title":"DAGs with NO TEARS: continuous optimization for structure learning","cited_arxiv_id":null,"evidence_quote":"Provides NOTEARS, the continuous-relaxation baseline that dominates the linear-Gaussian Sweden-Traffic benchmark."},{"cited_title":"DAGMA: learning DAGs via M- matrices and a log-determinant acyclicity characterization","cited_arxiv_id":null,"evidence_quote":"Provides DAGMA-linear, another continuous-relaxation baseline included in the structural comparison."}],"review_version":1}