{"id":"5717eb52-fd26-48f5-afe9-9984786eca5f","arxiv_id":"2506.19459","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Multi-tag variable annotations, generated by LLMs, allow causal discovery algorithms to orient undirected edges by transferring direction statistics from already directed edges with the same tag pairs.","lead":"This paper proposes a causal discovery method that annotates variables with multiple semantic tags, then uses statistics from already directed edges to guess the direction of the remaining undirected edges. It tests the idea with tags generated by large language models on eleven benchmark datasets and reports that it often improves over plain PC and GES baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed improvement over type-based relations is not demonstrated: no Typed-GES baseline is run, and the paper's own Tagged-PC vs Typed-PC comparison shows no tag advantage.","rationale":"The reader identifies Assumption 1 (tag distribution consistency) as the weakest assumption. That is a legitimate concern, and the authors themselves flag it as uncheckable and potentially violated at scale. However, the empirical results are supposed to provide evidence that the assumption holds on the evaluated benchmarks; if the evaluation protocol is not valid, those results cannot serve as evidence. The more directly load-bearing gap for the paper's stated contribution is the missing Typed-GES control. The abstract claims improvement over type-based relations, but the experiments never compare typing and tagging on the same base algorithm. The only such comparison, on PC, shows no tag advantage; the favorable Tagged-GES result is confounded with the base algorithm. This is a concrete, fixable omission: adding Typed-GES to Table 1 would settle the novelty claim. There is also an optimistic-selection issue: the configuration and LLM are chosen using the test datasets, so the reported ranks for the selected configuration are not a fair comparison to the unselected baselines. Both issues are load-bearing, but the missing control is the single most decisive for the central claim. I therefore keep the reader's CONDITIONAL verdict: the paper's idea is plausible and the experiments are extensive, but the central comparison needs to be run before the claim is established.","tokens_in":30217,"tokens_out":13658,"duration_ms":140239,"concrete_test":"Run a Typed-GES baseline: feed the same LLM-assigned single types (e.g., Claude-3.5 types, majority aggregation) into the GES CPDAG using the same evaluation protocol, and compute average F1 and ranks with the same metrics. If Typed-GES performs comparably to or better than Tagged-GES, the claimed advantage of tags over types does not hold. As a secondary check, apply a paired statistical test (e.g., Wilcoxon signed-rank) between Tagged-PC and Typed-PC on per-dataset F1 to test whether tagging helps at all when the base algorithm is held fixed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty is that multi-tag statistics improve on the single-type approach of Brouillard et al. (2022). The Abstract claims tagging \"improves upon purely type-based relations,\" but Table 1 contains no Typed-GES baseline: typing is evaluated only on PC (Typed-PC Naive/Maj.), while tagging is evaluated on both PC and GES. This confounds the tag-vs-type comparison with the base algorithm. The one same-base comparison available, Tagged-PC vs Typed-PC, does not support the claim: average ranks in Table 1 are 2.61 for Tagged-PC versus 2.46 for Typed-PC (Maj.), and Table 4 shows Typed-PC often better on smaller datasets. The headline \"Tagged-GES consistently ranks best\" therefore cannot be attributed to tagging rather than to GES. Additionally, the best configuration (including the choice of LLM) is selected on the test data by average F1 over all datasets and seeds, so the reported ranks for the selected configuration are optimistically biased relative to the unselected PC/GES baselines. The missing Typed-GES control is the load-bearing gap: without it, the central claim of improving on type-based relations is unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a tag-based causal discovery method. Variables are annotated with multiple tags (e.g., by an LLM); from a CPDAG produced by a base algorithm (PC or GES), the method counts, for each ordered pair of tags, how often directed edges point in each direction (Eqs. (3)-(4)), and then orients each remaining undirected edge by the average of these tag-pair statistics, with abstention and optional Meek-rule application (Eq. (5), Algorithm 2). The paper reports experiments on 11 bnlearn datasets and LUCAS, over 5 LLMs and 400 configurations, claiming that the approach improves on PC, GES, and the single-type method of Brouillard et al. (2022), and that the extracted high-level tag relations correspond to common knowledge.","tokens_in":30503,"tokens_out":8971,"duration_ms":83798,"significance":"If the central claim were established, the contribution would be a practical, low-cost way to use LLM background knowledge to reduce the Markov equivalence class, generalizing type consistency to multiple tags and avoiding the brittleness of single-type assignments. The paper is commendable for releasing code, running an extensive configuration sweep, and including ablation studies on graph faults and noisy tags; the idea of mining tag-level causal relations from ground-truth graphs is also interesting. However, the central comparison against type-based methods is currently confounded with the base algorithm and with test-data model selection, so the magnitude of the claimed improvement is not yet demonstrated.","major_comments":[{"comment":"The paper's central claim that tagging improves upon purely type-based relations is not supported by the reported experiments because no Typed-GES baseline appears in Table 1. Typing is evaluated only on top of PC (Typed-PC Naive and Typed-PC Maj.), whereas tagging is evaluated on both PC and GES, so the tag-versus-type comparison is confounded with the base algorithm. The only same-base comparison available, Tagged-PC versus Typed-PC, actually favors Typed-PC on average ranks (2.61 versus 2.46 in Table 1), and Table 4 shows Typed-PC often better on smaller datasets. Without a Typed-GES condition, the headline that Tagged-GES 'consistently ranks best' cannot be attributed to tagging rather than to the stronger GES base.","section":"§4.1, Table 1"},{"comment":"The best configuration is selected on the test data: Section 4.1 states that the best-performing configuration is identified by the average F1 over all datasets and seeds, and the caption of Table 4 says that for each method the configuration (including the LLM) that performed best for itself was chosen. This means the tagged rows are test-data-optimized while the PC and GES baselines are not, and it also makes the Tagged-PC/Tagged-GES comparison depend on each variant's own selected settings. The ranks in Table 1 are therefore optimistically biased, and the claimed improvement over the baselines needs a fair protocol (e.g., selection on a validation split, or a fixed configuration chosen before seeing the test results).","section":"§4.1, Tables 2 and 4"},{"comment":"The method rests entirely on Assumption 1 (Tag Distribution Consistency), which the authors state 'can usually not be checked in practice' and which 'might be violated when scaling to larger systems.' Because the benchmark data come with ground-truth graphs, a direct check is feasible: the tag informative values estimated from the directed edges of the recovered CPDAG can be compared with those computed from the true directions of the edges that were undirected. I ask for such a diagnostic (or a synthetic experiment that controls the degree of distribution mismatch), since without it the empirical claims are limited to settings where the transfer assumption happens to hold.","section":"§3, Assumption 1"},{"comment":"The variance-reduction argument treats the tag informative values I^Y_m as independent draws from a common Beta distribution, but these values are computed from the same set of directed edges and are therefore not independent, so the Central Limit Theorem does not apply as stated. The argument also sets the edge-specific noise N_ij to zero even though Eq. (2) introduces it; consequently the conclusion in Sec. 3.1 that predictions become perfect as the number of tag pairs grows does not follow. The derivation should be presented as heuristic motivation, with the independence and noise assumptions made explicit.","section":"App. A, Eqs. (7)-(12)"}],"minor_comments":[{"comment":"In Eq. (9), the summation uses I^Y_n inside the sum over m; this should be I^Y_m (or the index renamed) to avoid confusion with the outer n.","section":"App. A, Eq. (9)"},{"comment":"Please clarify how counts are handled when both tags of a pair appear in the tag sets of both endpoints of an edge; in that case a single directed edge can increment both C_ab and C_ba, and one would need to explain why p-hat_ab + p-hat_ba can be treated as a probability.","section":"§3, Eqs. (3)-(4)"},{"comment":"In Algorithm 2 (and the helper in Algorithm 1), the variable BackwardProb is used before it is defined; line 42 sets BackwardProb←1−BackwardProb, which should presumably read BackwardProb←1−ForwardProb. Please correct the pseudocode.","section":"App. C, Algorithm 2"},{"comment":"The caption of Table 1 notes that SIDmin treats undirected and correctly directed edges identically, but the same caveat applies to Precision; the sentence in the caption is incomplete as written.","section":"§4.1 / Table 1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick read: the multi-tag extension is real, and the experimental sweep is serious, but the paper's central claim—that tagging beats typing—is not actually demonstrated. The stress-test note is right: there is no Typed-GES baseline in Table 1. The only same-base comparison is Tagged-PC vs Typed-PC, and there tags don't win; Typed-PC (Maj.) has a slightly better average rank. The headline 'Tagged-GES consistently ranks best' could just be GES being better than PC. With GES as the base, we don't know how much of the gain comes from tagging versus typing, because typing was only run on PC. That is a load-bearing gap for the abstract's claim.\n\nWhat is genuinely new: generalizing Brouillard et al.'s single-type consistency to multi-tag sets, with statistics averaged over tag pairs, and using LLMs to produce those tags. That is a clean extension, and the variance-reduction argument for why multiple tags help is reasonable. The experiments are extensive—11 datasets, 5 LLMs, ablations on graph faults and noisy tags—and the robustness results are credible. The paper also honestly flags Assumption 1 as uncheckable and potentially violated at scale; that is a real limitation, but it is acknowledged, and the failure-mode analysis gives some sense of when it might bite.\n\nThe biggest soft spot besides the missing baseline is the configuration selection on test data: the best hyperparameters and the best LLM are chosen by average F1 over all datasets and seeds, so the reported ranks for the selected configuration are optimistically biased. The ablation and robustness parts are less affected, but the headline numbers should be taken with that in mind.\n\nWho is this for: people working on incorporating background knowledge into causal discovery, and anyone interested in LLM-assisted priors. It is worth a careful referee, but the authors should be asked to run Typed-GES, or at least a same-base control, and to either use a validation split or report the optimistic bias honestly.\n\nMy recommendation: send to peer review, and hold the authors to fixing the evaluation protocol. The mechanism is worth having in the literature even if the current comparative claim does not survive intact.","headline":"A genuinely useful multi-tag extension of type-based causal discovery whose headline claim over typing is undercut by a missing same-base baseline and test-set selection.","tokens_in":31020,"tokens_out":2594,"would_cite":true,"duration_ms":24683,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tagging variables with several high-level concepts and counting how often each tag pair points one way on already-directed edges orients the remaining edges better than single-type priors or standard causal discovery.","keywords":["causal discovery","tag-based priors","CPDAG edge orientation","Markov equivalence","large language models","type consistency generalization","tag informative value","edge direction prediction"],"falsifier":"Build a two-region benchmark in the same style as the paper's: in region A, 80% of edges between a $t_a$-tagged and a $t_b$-tagged variable point $t_a\\to t_b$, while in region B the same tag pair points $t_b\\to t_a$ with 80%. Arrange the base CPDAG so the directed edges fall almost entirely in region A and the undirected edges almost entirely in region B, run the tag-statistic transfer, and measure the orientation accuracy in region B. Assumption 1 predicts the informative values converge; if region B accuracy lands at or below chance (0.5), the transfer step — not the counting — is what fails, and the assumption is violated in exactly the settings the method is meant for.","tokens_in":30038,"feed_emoji":"🏷️","tokens_out":16423,"duration_ms":142924,"temperature":0.7,"pith_summary":"Observational causal discovery can usually identify a causal graph only up to Markov equivalence, leaving many edges undirected. This paper claims that direction statistics over semantic tags can break that deadlock: after a base algorithm such as PC or GES produces a partially directed graph, counting how often each pair of high-level tags points one way on the already-directed edges yields a prior for orienting the edges that remain. The paper shows experimentally, on eleven benchmarks with tags supplied by language models, that this tag-based transfer improves over the base algorithms and over single-type approaches, with the GES-based variant ranking best on most metrics. It also argues that the same tag statistics can be read in reverse, mining abstract causal relations between concepts that match common knowledge. The reason to care is practical: this is a way to use cheap, common-sense annotation to pin down directions that pure data cannot decide, without requiring interventions.","feed_headline":"Multi-tag statistics direct causal edges better than single types","feed_subtitle":"The same high-level tags that explain one edge, averaged, orient the edges a base algorithm leaves undirected.","key_machinery":"The load-bearing object is the tag informative value, $I^Y(t_a,t_b)$, the empirical probability over edges $E_{Y,t_a,t_b}$ that an edge connecting a $t_a$-tagged variable to a $t_b$-tagged variable points from the $t_a$ side to the $t_b$ side. Direction evidence for a candidate edge $X_i\\to X_j$ is the average preference $Q(E_{ij})=\\frac{1}{n}\\sum_{(t_a,t_b)\\in t_i\\times t_j}\\hat{p}_{ab}\\,o(t_a,t_b)$, where $\\hat{p}_{ab}$ estimates the informative value from the count matrix $C_{ab}$ of directed edges, $o$ filters tag pairs with no observed evidence, and the decision boundary sits at $0.5$ with an abstention band of width $2\\epsilon$. The paper's supporting theory models the informative values as draws from a $\\beta$ distribution and shows, via the central limit theorem, that the variance of the averaged prediction, $\\mathrm{Var}[d'^Y_{ij}] = \\alpha\\beta/((\\alpha+\\beta)^2(\\alpha+\\beta+1)n)$, falls as the number $n=|t_i\\times t_j|$ of tag pairs grows, so more tags, or tags shared by more variables, sharpen the direction estimate. Around this core, the algorithm greedily picks the edge with the most decisive preference, enforces acyclicity, and applies Meek rules after each orientation, so each directed edge propagates further constraints before the next choice.","core_discovery":"The authors' central claim is that the causal direction of an undirected edge is predictable from the statistical relationship between the tags carried by its two endpoints, provided those statistics are collected from the directed edges of a completed partially directed acyclic graph (CPDAG), the equivalence-class summary that base algorithms like PC and GES return. Formally, they define a tag informative value $I^Y(t_a,t_b)$: the fraction of directed edges in the graph whose endpoints carry tags $t_a$ and $t_b$ that point from the $t_a$-variable to the $t_b$-variable. A still-undirected edge with tag sets $t_i$ and $t_j$ is then directed by averaging these values over all tag pairs $(t_a,t_b)\\in t_i\\times t_j$; if the average is above $0.5+\\epsilon$ the edge is set to $X_i\\to X_j$, if below $0.5-\\epsilon$ to $X_j\\to X_i$, and otherwise the method abstains. On eleven benchmark networks, with tags generated by several large language models, the procedure applied on top of GES consistently ranked best across all compared methods and metrics, the exceptions being metrics that reward leaving edges undirected, and the tag relations extracted from the ground-truth graphs turned out to be semantically coherent, such as 'Alarm Trigger' → 'Response' and 'Vehicle Attributes' → 'Safety Features'. The paper presents this as evidence that the assumption that directed and undirected edges share the same tag statistics, while often not directly checkable, holds in practice for homogeneous domains.","pith_inferences":["My read: the method is a transfer-learning prior in disguise, where the tag informative value is an empirical prior whose weight is set by how many directed edges support it, so the same averaging machinery could combine any sources of direction evidence, including additive-noise scores, interventional labels, or expert statements, into one preference score.","I read the abstention band around 0.5 as an uncalibrated confidence lever: with per-tag-pair thresholds, a practitioner could guarantee a minimum precision on the edges the method chooses to direct, turning the post-processing pass into a safe procedure for high-stakes settings.","Because the paper shows that tags produced by different language models perform nearly identically, I suspect the gain comes from the aggregation mechanism rather than the vocabulary; scrambling or randomising the tag sets would test this directly.","A two-region graph with opposite tag-direction statistics is the cleanest boundary probe: if the method's accuracy there falls to chance, it has located precisely where tag statistics stop transferring."],"forward_implications":["Applied on top of GES, the tagging procedure ranks first among all compared methods on average across every metric that rewards correct direction decisions, so users of score-based discovery can tighten their partially directed graphs without collecting new data.","Because the method improves both PC and GES outputs, it functions as a general post-processing layer for any algorithm that returns a CPDAG, requiring only tag annotations for the variables.","Directing edges with tag statistics demonstrably shrinks the Markov equivalence class of the initial CPDAG toward the ground truth, meaning background knowledge expressed as tags can break equivalence where interventional data would otherwise be required.","The variance analysis implies that the advantage of tags over single types grows with the number of tag pairs per edge, which the experiments confirm by showing larger gains on larger datasets, where single-type evidence is too sparse.","The tag relations mined from the ground-truth networks, such as 'Alarm Trigger' → 'Response' and 'Vehicle Attributes' → 'Safety Features', are highly homogeneous and match common knowledge, so the same procedure doubles as a way to extract abstract causal statements from a known graph."],"supporting_citations":[{"why":"Supplies the typing-consistency framework and the naive/majority baselines that the paper generalizes from single types to multiple tags.","marker":"[Brouillard et al., 2022]"},{"why":"Defines the PC algorithm, one of the two base CPDAG-producing methods the tagging procedure is applied to.","marker":"[Spirtes and Glymour, 1991]"},{"why":"Defines greedy equivalence search (GES), the base method whose tagged variant achieves the paper's best results.","marker":"[Chickering, 2002]"},{"why":"Supplies the Meek rules applied after each edge orientation to propagate further directions without new data.","marker":"[Meek, 1995]"},{"why":"Provides the bnlearn repository datasets (Asia, Alarm, Child, Insurance, and others) used in all experiments.","marker":"[Scutari, 2010]"},{"why":"Provides the Python port of the bnlearn datasets used for sampling the evaluation data.","marker":"[Taskesen, 2020]"},{"why":"Supplies the LUCAS lung-cancer benchmark included in the evaluation.","marker":"[Guyon et al., 2011]"},{"why":"Defines the Structural Intervention Distance (SID) used to judge how close the recovered graphs are to the ground truth.","marker":"[Peters and Bühlmann, 2015]"},{"why":"Defines the Structural Hamming Distance (SHD) used as the primary edge-error metric.","marker":"[Tsamardinos et al., 2006]"},{"why":"Motivates distilling tag annotations from meta-information such as variable names, the annotation strategy the paper follows with language models.","marker":"[Long et al., 2023]"}],"fun_headline_variants":["Tags on variables orient causal edges more accurately","Multi-tag clues sharpen causal direction discovery","Tag statistics beat single-type causal directioning","Causal edges get direction from tag comparisons","Multi-tag averages orient undirected causal edges"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's load-bearing premise is that undirected edges obey the same tag-direction statistics as the already-directed edges they are inferred from, a condition the paper concedes can usually not be checked in practice.","fun_headline_variants_meta":{"raw":{"variants":["Tags on variables orient causal edges more accurately","Multi-tag clues sharpen causal direction discovery","Tag statistics beat single-type causal directioning","Causal edges get direction from tag comparisons","Multi-tag averages orient undirected causal edges"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000681,"raw_usage":{"total_tokens":3127,"prompt_tokens":1016,"completion_tokens":2111,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":2054}},"tokens_in":632,"tokens_out":2111,"duration_ms":14917,"temperature":1.0,"reasoning_tokens":2054,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:32:50.918075+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a two-region benchmark in the same style as the paper's: in region A, 80% of edges between a $t_a$-tagged and a $t_b$-tagged variable point $t_a\\to t_b$, while in region B the same tag pair points $t_b\\to t_a$ with 80%. Arrange the base CPDAG so the directed edges fall almost entirely in region A and the undirected edges almost entirely in region B, run the tag-statistic transfer, and measure the orientation accuracy in region B. Assumption 1 predicts the informative values converge; if region B accuracy lands at or below chance (0.5), the transfer step — not the counting — is what fails, and the assumption is violated in exactly the settings the method is meant for.","supporting_citations":[],"review_version":2}