{"id":"9e5151cc-bf9e-4462-a154-54fe45c28bc9","arxiv_id":"2608.08146","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"SkillCDG represents long SKILL policies as constraint dependency graphs, retrieves dependency-closed rule contexts for compliance detection, and uses a fitted scale-complexity model to guide small-model distillation.","lead":"This paper introduces SkillCDG, a graph-based framework that converts long business policy documents into a two-layer constraint dependency graph, then retrieves prerequisite-complete rule contexts for compliance detection. It also fits an empirical scaling trend linking model size and policy-graph complexity, and uses predicted headroom to prioritize distillation samples.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SkillCDG's own Table 2 shows F1 below RawSkill by up to 15.5 points at Qwen3.5-27B and on most DeepSeek-V4-Pro/GPT-5.4 rows, while the paper only reports ACC gains; the 27B scaling validation in Sec. 4.3 is asserted without numeric predicted-vs-actual evidence.","rationale":"The paper has a coherent engineering story: two-layer CDG, dependency closure, token savings, expert validation of extracted rules, controlled CompliBench-Long transfer, and a matched OPD design. These are real partial supports, and I am not questioning the authors' integrity. The problem is that the central claim, as stated in the abstract and in the reader's strongest_claim, is broader than what Table 2 supports. SkillCDG's F1 advantage over RawSkill is large and consistent only for compact judges (Qwen3.5-4B/9B). At 27B the advantage reverses on all three enterprise datasets by a wide margin, and the same reversal appears for DeepSeek-V4-Pro and GPT-5.4 on most datasets. Section 4.2 deliberately reports only ACC for the 27B block, so a reader cannot infer the F1 drop. Since F1 is the metric singled out in the abstract and defined as 'balanced detection', this is not a cosmetic issue. If a model over-predicts the majority class, ACC can rise while F1 collapses; the paper would need to show that the 27B behavior is not exactly that. The RQ2 scaling analysis compounds the problem: Eq. 3 is fit on per-sample binary correctness, so a high-ACC/low-F1 27B result can appear to 'align' while the practically relevant violation-detection quality degrades. The leave-one-scale-out validation is mentioned but never reported numerically; without predicted-versus-observed 27B values and confidence intervals, the extrapolative validity and the Eq. 5 headroom scores used for OPD are unverified. I would keep the CONDITIONAL verdict but require these disclosures; the concerns do not suggest fabrication, only that the claims need to be restricted and substantiated.","tokens_in":17215,"tokens_out":8164,"duration_ms":72120,"concrete_test":"Rerun the Qwen3.5-27B rows of Table 2 with the same frozen CDG and judge prompt, saving confusion matrices for each enterprise dataset, and compute violation-class F1 and macro-F1 alongside ACC. If SkillCDG's F1 remains below RawSkill by the Table 2 margin (about 7.6-15.5 points) while ACC is higher, the broad 'outperforms on F1' claim is invalid at scale and all conclusions must be restricted to compact judges. If artifact-level reproduction instead shows the discrepancy was an error, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract-level claim that SkillCDG outperforms baselines on detection F1 is contradicted by the paper's own Table 2 at larger scales. For Qwen3.5-27B, SkillCDG F1 falls to 46.5/40.7/45.6 (Fulfillment/AfterSales/MerchantSupport) versus RawSkill 54.1/56.2/58.7, a 7.6-15.5 point deficit on every enterprise dataset; F1 is also below RawSkill on most DeepSeek-V4-Pro and GPT-5.4 rows. Section 4.2 reports only the ACC gains at 27B ('raises ACC by 13.2, 1.8, and 9.2 points') and never mentions the F1 collapse, so the central efficacy claim is scale-limited at best. RQ2's fitted model (Eq. 3) predicts binary correctness, not F1, and Section 4.3's assertion that 27B 'aligns with the fitted trend' is not accompanied by a leave-one-scale-out table, confidence intervals, or predicted-versus-observed values; Figure 3 cannot support that extrapolation by itself. Because RQ4's sample-selection score gi (Eq. 5) is built from the same unvalidated model, the 127.7%/166.7% F1-recovery figures inherit the same risk. The enterprise gold-label provenance is a real limitation, but the table-internal metric reversal is more immediately decisive.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SkillCDG, a framework that converts long SKILL policy documents into a two-layer constraint dependency graph (CDG), retrieves scenario-relevant atomic rules, expands prerequisite dependencies via transitive closure, and feeds the resulting compact context to an LLM judge. Across three enterprise datasets and two variants of CompliBench, the authors report accuracy and F1 improvements over RawSkill and LightRAG baselines, particularly for small models, along with token-consumption reductions. The paper further fits an empirical scaling model (Eq. 3) relating model parameter count and CDG-based complexity to per-instance correctness, uses the fitted model to estimate teacher–student headroom, and shows that selecting on-policy distillation samples by that headroom improves recovery of the teacher's F1 relative to uniform sampling.","tokens_in":17580,"tokens_out":4860,"duration_ms":44603,"significance":"If the empirical claims are upheld, SkillCDG would be a practical contribution to enterprise compliance auditing: it makes prerequisite dependencies explicit, enables source traceability, and reduces inference cost substantially (64.3% token reduction on Fulfillment). The scaling-guided sample-selection idea is a useful bridge between scaling-law analyses and distillation data curation. The paper ships algorithms (Algorithms 1 and 2) and a controlled ablation (Table 5) that isolate the value of dependency closure; these are strengths. However, the central detection-accuracy claim is weakened by the F1 collapse at Qwen3.5-27B in Table 2, and the scaling-law validation is not backed by numeric leave-one-scale-out evidence, so the significance cannot be fully assessed as submitted.","major_comments":[{"comment":"In the Qwen3.5-27B rows of Table 2, SkillCDG F1 is 46.5/40.7/45.6 on Fulfillment/AfterSales/MerchantSupport, which is 7.6–15.5 points below RawSkill F1 (54.1/56.2/58.7), while SkillCDG ACC is simultaneously higher than RawSkill ACC; Section 4.2 reports only the ACC gains ('raises ACC by 13.2, 1.8, and 9.2 points') and the abstract claims general F1 superiority, so the central efficacy claim is scale-limited at best. The large ACC–F1 divergence at 27B (e.g., 67.6 ACC vs 46.5 F1 on Fulfillment) is not discussed, and the paper should report, explain, and condition its claims on this pattern.","section":"Table 2, Section 4.2"},{"comment":"The leave-one-scale-out validation of Eq. 3 is asserted but no predicted-versus-observed values, confidence intervals, or a table are provided, so the extrapolation to the 27B checkpoint cannot be assessed. Moreover, Eq. 3 is fit on the same five datasets used for the main results, and the headroom score g_i in Eq. 5 is computed directly from that fitted model, so the RQ4 sample-selection experiment does not independently validate the scaling law. Please provide leave-one-scale-out numeric results with uncertainties and clarify the logical status of the RQ4 test.","section":"Section 4.3, Figure 3; Eq. 5 in Section 4.5"},{"comment":"The model-selection procedure (session-grouped five-fold cross-validation with the one-standard-error rule) and the comparison against the additive logistic and nested saturating models are described but no results of these comparisons are reported; the claim that the data favor the parsimonious form over a scale–complexity interaction is unsupported. Additionally, C0 is defined as the training-fold median, making the complexity normalization training-set-dependent, and the paper does not state how C0 is computed for an unseen test instance.","section":"Section 3.2, RQ2"},{"comment":"The gold-label provenance is described as pre-existing 'independent human assessment' in operational records, but there is no reporting of label-level inter-annotator agreement, instructions to annotators, or reconciliation with model-based assessments; since all downstream enterprise results depend on these labels, the absence of any label-quality validation is a load-bearing gap rather than a presentation issue.","section":"Appendix B.4"}],"minor_comments":[{"comment":"The sentence 'At 27B, it further raises ACC by 13.2, 1.8, and 9.2 points' omits the corresponding F1 changes, which are negative in all three datasets; please report both metrics consistently in the same passage.","section":"Section 4.2, Table 2"},{"comment":"The caption says 'three business datasets' but Section 4.3 says the scaling model is fit on all five datasets; please clarify which datasets are displayed and whether the shown curves come from the full fit.","section":"Figure 3"},{"comment":"Recovery values exceeding 100% (127.7% and 166.7%) are presented without discussion of the variance of the F1 estimates; confidence intervals or a repeated-trial analysis would help the reader judge whether the student truly surpasses the teacher.","section":"Section 4.5, Table 6"},{"comment":"The recovery measure is undefined when teacher and base F1 are equal; please state the assumption or handle the degenerate case explicitly.","section":"Eq. 6 in Section 4.5"},{"comment":"When the seed-rule set is empty, Algorithm 2 falls back to full-rule coverage; this behavior should be mentioned in the main-text pipeline description because it affects the strength of the 'filtering' claim for samples with no keyword match.","section":"Algorithm 2, Section 3.1.3"}],"recommendation":"major_revision","confidential_remarks":"The abstract and Section 4.2 materially overstate the F1 results: the 27B row of Table 2 directly contradicts the claim that SkillCDG outperforms baselines on F1, and the ACC–F1 gap within SkillCDG itself is left unexplained. The scaling analysis is also presented as validated when the only support is a fitted curve without numeric leave-one-scale-out predictions. The CDG construction and the ablation isolating dependency closure are solid, but the evidence as submitted is not sufficient for acceptance; the authors should condition claims on model scale and provide the missing validation numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: the two-layer constraint dependency graph with typed require edges and closure-based retrieval is a real contribution. It cleanly separates scenario routing from prerequisite completion, and the ablations show closure matters. The token savings (up to 64.3%) are concrete and credible. This is not another routine RAG tweak; it is a genuinely different way to structure policy documents for compliance auditing.\n\nThat said, the paper overstates its headline result. Table 2 shows F1 for SkillCDG below RawSkill on every enterprise dataset at Qwen3.5-27B (46.5 vs 54.1, 40.7 vs 56.2, 45.6 vs 58.7), and mostly below on DeepSeek-V4-Pro and GPT-5.4 as well. The abstract's claim of outperforming baselines in F1 is technically true only because of the 'up to' qualifier and the strong gains at 4B/9B. The RQ1 text mentions only ACC gains at 27B and never the F1 collapse. That is selective reporting, and it matters because the practical question is whether SkillCDG helps the larger judges you would actually deploy.\n\nThe scaling analysis has a circularity problem. Eq. 3 is fit to the same five datasets used for the main results, and the distillation score gi in Eq. 5 is computed from that fitted model. RQ4 therefore is not an independent test of the scaling law; it is a test of whether a model trained on its own fitted predictions helps, which is much weaker. The paper claims leave-one-scale-out validation for the 27B prediction but gives no predicted-versus-observed numbers or confidence intervals. For a paper whose third contribution is 'estimating and validating' a scaling trend, that is a serious omission.\n\nSmaller issues: gold labels come from operational records whose provenance is asserted but not audited; the human evaluation only covers LLM-extracted rule correctness, not the gold labels used for ACC/F1. No code or data is released, so nothing here is independently reproducible.\n\nWhat is genuinely good: the method itself is coherent, the case study is instructive, and the public-benchmark transfer results, while modest, are consistent. The framework is well specified in the algorithms. The central argument that structured access helps smaller models holds up; the overgeneralization to 'outperforming baselines' does not.\n\nFor peer review: yes, a serious referee should see this. The idea deserves attention, and the flaws are fixable in revision. But the authors need to report F1 across all scales, show the leave-one-scale-out numbers, and ideally release data or code. Without those, the strongest claims remain on internal data and selective metrics.","headline":"The two-layer CDG and closure pipeline are a genuinely useful idea with real token savings, but the paper's F1 claims are contradicted by its own Table 2 at larger scales, and the scaling model is fit to the same data it is used to explain.","tokens_in":18104,"tokens_out":2224,"would_cite":true,"duration_ms":24777,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-layer policy graph turns long SKILL auditing into dependency reasoning, lifting detection F1 by up to 12.8 points while cutting online token use by up to 64.3 percent.","keywords":["compliance detection","constraint dependency graph","SKILL documents","dependency closure","policy auditing","scaling law","on-policy distillation","retrieval augmentation"],"falsifier":"Run the pipeline on a held-out public corpus with independently audited violation labels and a manually verified constraint graph: the central claims would fail if dependency closure over require edges does not beat flat raw-document judging on dependency-heavy policies, or if Eq. 3, fitted on the 0.8B, 2B, 4B, and 9B checkpoints, does not predict the next checkpoint from the same model family.","tokens_in":17013,"feed_emoji":"🛡️","tokens_out":6635,"duration_ms":60949,"temperature":0.7,"pith_summary":"Long enterprise SKILL documents give agents operating rules, but auditing whether an agent followed them is hard because the governing workflow, its prerequisites, and its exceptions are scattered across thousands of tokens. This paper claims that the right representation fixes the problem: convert each SKILL into a two-layer constraint dependency graph, retrieve the relevant scenario and seed rules, then expand along mandatory \"require\" edges into a dependency closure before asking an LLM to judge compliance. Against raw-policy and LightRAG baselines, SkillCDG improves detection F1 by up to 12.8 percentage points while cutting average online token use by up to 64.3 percent. The paper also claims a complexity-aware scaling relationship in Eq. 3 that predicts a 27B model's performance from smaller checkpoints, and uses the predicted teacher–student headroom to allocate a fixed on-policy distillation budget, letting a 4B student beat or match a 9B teacher on two datasets.","feed_headline":"Policy graph lifts audit F1 by 12.8 points, cuts tokens 64%","feed_subtitle":"Routing to atomic rules and expanding prerequisites beats flat documents and generic retrieval for long-policy audits.","key_machinery":"The machinery is the constraint dependency graph (CDG): a two-layer directed graph whose upper layer indexes SKILL names and descriptions for scenario routing and whose lower layer stores atomic condition–action rules linked by \"require\" edges, meaning that judging one rule obliges also checking another rule as a prerequisite. The argument runs on three operations: block-wise LLM extraction of atomic rules and local edges, cross-block fusion to reconnect prerequisites across Markdown sections, and online transitive dependency closure from lexically matched seed rules. This closure is what supplies the judge with the full prerequisite set while keyword filtering removes irrelevant content; the same graph yields the complexity metric $C_i = I_{d,i} I_{c,i}$ (rule density times dependency coupling) used in the scaling model. The second machine is Eq. 3, a saturating logistic model of per-instance correctness as a function of parameter count, with a complexity-dependent gain term; it is fit on four checkpoints and used to rank samples by predicted teacher–student headroom for distillation.","core_discovery":"The central claim is that long-policy compliance detection should be treated as action-conditioned logical reasoning over a structured policy graph rather than as flat-text reading. The paper represents business policies as atomic condition–action rules linked by a uniform \"require\" semantics, where one rule's compliance status depends on evaluating another rule as a prerequisite, making prerequisite completion explicit instead of leaving it implicit inside the judge. Inference retrieves candidate SKILLs by scenario, retrieves seed rules lexically, computes the transitive closure over require edges, and renders only that closure to the judge, with source spans retained for traceability. The authors argue that this separation of policy access, dependency reasoning, and judgment is what accounts for the measured gains, and that the same graph's rule density and dependency coupling define an instance-difficulty metric that predicts how much larger models improve.","pith_inferences":["The \"require\"-edge semantics should transfer to any rule-based auditing domain where one obligation conditions another, such as regulatory compliance, contract review, or safety checklists; a public benchmark with dependency-rich policies would test this transfer.","The complexity metric $C_i$ could be used at policy-authoring time: authors could see when new passages add many prerequisite edges and restructure documents to reduce audit difficulty before deployment.","The scaling predictor, if stable across model families, could drive dynamic model routing, sending low-complexity samples to a small judge and high-complexity samples to a large one; the paper reports the needed components but does not assemble that cost-accuracy controller.","Because dependency closure makes the judgment basis explicit, SkillCDG verdicts come with an inspectable chain of rule identifiers and source spans, which could serve as a human-review artifact; the paper demonstrates this in a case study but does not turn it into a formal audit-trail metric."],"forward_implications":["Compact judges benefit most: on 4B and 9B models SkillCDG raises detection F1 by up to 12.8 and 10.8 points respectively, so structured policy access and larger model capacity solve complementary parts of the task.","Dependency closure is the load-bearing component: removing it costs up to 7.5 F1 points on Qwen3.5-27B, a larger drop than removing keyword filtering, which mainly controls context length.","Token savings grow with document length: the largest reduction is 64.3% on the longest dataset (Fulfillment), with 33.9% and 39.1% on the other two enterprise datasets.","The fitted scaling relation extrapolates to an unseen checkpoint: 27B results align with the trend fitted on the 0.8B, 2B, 4B, and 9B checkpoints, so CDG-derived complexity can predict where extra capacity pays off.","A fixed distillation budget goes further when samples are ranked by predicted teacher–student headroom: the 4B student recovers 127.7% and 166.7% of the F1 gap on two datasets, compared with 85.1% and 83.3% under uniform sampling."],"supporting_citations":[{"why":"Supplies LightRAG, the graph-retrieval baseline SkillCDG must beat, representing entity-relation retrieval without prerequisite semantics.","marker":"(Guo et al., 2024)"},{"why":"Supplies CompliBench, the public benchmark whose conversations and labels are reused in the transfer experiments and whose judge-style violation detection framing the paper extends to long policies.","marker":"(Yang et al., 2026)"},{"why":"Supplies MiniLLM, the on-policy distillation method the OPD training setup adapts for teacher-scored student rollouts.","marker":"(Gu et al., 2024)"},{"why":"Establishes on-policy distillation from self-generated mistakes, the basis for the fixed-budget distillation experiment.","marker":"(Agarwal et al., 2024)"},{"why":"Supplies the context-aware scaling-law formulation whose functional form Eq. 3 extends to policy-graph complexity.","marker":"(Montgomery et al., 2025)"},{"why":"Supplies lost-in-the-middle evidence that long flat contexts are unreliable, motivating the compact closure-based policy context.","marker":"(Liu et al., 2024)"},{"why":"Supplies the retrieval-augmented generation framework that motivates retrieval-augmented judging and the lineage for graph-based policy access.","marker":"(Lewis et al., 2020)"}],"fun_headline_variants":["Policy graph lifts audit F1 12.8 pts, cuts tokens 64%","Compliance as logic: graph closure beats flat text by 12.8 F1","Rule graph + closure: 12.8 F1 gain, 64% token savings","Long-policy audits: graph reasoning lifts F1 12.8, slashes tokens","Closure-grounded checks: +12.8 F1, -64% tokens on long policies"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gold labels come from operational records that already contain human assessments, and the constraint dependency graph is built by LLM extraction that is later checked by experts; if those labels or the extracted rule graphs are systematically wrong, every measured gain inherits the error.","fun_headline_variants_meta":{"raw":{"variants":["Policy graph lifts audit F1 12.8 pts, cuts tokens 64%","Compliance as logic: graph closure beats flat text by 12.8 F1","Rule graph + closure: 12.8 F1 gain, 64% token savings","Long-policy audits: graph reasoning lifts F1 12.8, slashes tokens","Closure-grounded checks: +12.8 F1, -64% tokens on long policies"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000527,"raw_usage":{"total_tokens":2559,"prompt_tokens":979,"completion_tokens":1580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":1464}},"tokens_in":595,"tokens_out":1580,"duration_ms":11709,"temperature":1.0,"reasoning_tokens":1464,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:19:35.367555+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline on a held-out public corpus with independently audited violation labels and a manually verified constraint graph: the central claims would fail if dependency closure over require edges does not beat flat raw-document judging on dependency-heavy policies, or if Eq. 3, fitted on the 0.8B, 2B, 4B, and 9B checkpoints, does not predict the next checkpoint from the same model family.","supporting_citations":[{"cited_title":"2024 , doi=","cited_arxiv_id":null,"evidence_quote":"Supplies LightRAG, the graph-retrieval baseline SkillCDG must beat, representing entity-relation retrieval without prerequisite semantics."},{"cited_title":"2024 , doi=","cited_arxiv_id":null,"evidence_quote":"Supplies lost-in-the-middle evidence that long flat contexts are unreliable, motivating the compact closure-based policy context."}],"review_version":1}