{"id":"42cc2bba-f754-4a27-a167-6e05edf1d3d1","arxiv_id":"2607.04319","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Bounding attention values into fuzzy memberships or Boolean set operations yields 44–62% crisp selective detectors and an end-to-end legible transformer at baseline quality.","lead":"The paper makes transformer attention legible by construction: a sigmoid (or Boolean set ops) on the value turns channels into readable feature detectors, then couples this to a legible feed-forward layer. The resulting 125M end-to-end model matches baseline quality while letting named logical units be read off a generated token.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Parity and SELECTIVE rates rest on single-seed, one-epoch 125M runs with fixed λ and a hand-tuned metric; that is the load-bearing soft spot for the “free legibility” claim.","rationale":"The reader already isolates the correct soft spot: single-seed, one-scale, one-epoch evaluation plus hand-chosen selectivity coefficients and the SELECTIVE metric. The architecture (bound value, free write) is cleanly ablated and internally consistent; the concern is statistical power, not an internal contradiction. Multi-seed re-runs of the three headline rows would settle whether the reported rates and parity are stable or seed artifacts. That check does not change the CONDITIONAL verdict; it is exactly what the reader already requires before the stronger claim can be treated as established. No stronger load-bearing flaw (e.g., definitional circularity or failed ablation) is present in the text.","tokens_in":26516,"tokens_out":564,"duration_ms":5418,"concrete_test":"Retrain the three headline configs (membership-100 both-pressures, Boolean-100 crispness-alone, E2E even-split both-pressures) with three independent seeds at the same 125M/one-epoch setting; recompute SELECTIVE% (Table 1 definition) and LAMBADA/BLiMP. If any seed drops SELECTIVE below ~35% or moves LAMBADA outside the baseline ±0.03 band, the parity/legibility percentages do not generalize even at this scale.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (44–62% SELECTIVE detectors, quality at parity, end-to-end named composition) is supported only by single-seed, one-epoch 125M numbers (Tables 1, 3, 4; §3.8, 4.4, 5.2). The paper itself flags this and the shallow-attention collapse under a legible FFN (§5.4, 6). SELECTIVE is defined by fixed thresholds (variance ≥ 0.003 and crispness > 0.5) and fixed λs=10−3, λc=3×10−3; small shifts in those knobs or seed can move both the percentage and the LAMBADA/BLiMP placement inside the parity band. Without multi-seed or multi-scale confirmation, the stronger framing “no quality cost / free legibility” remains provisional rather than established.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes making transformer attention legible by construction by bounding the value path: a membership head applies a sigmoid to the standard value so each channel is a [0,1] detector, and a Boolean head restructures the value into within-token fuzzy intersection and set-difference, both with free (untied) output projection. Two selectivity pressures (sparsity and crispness) are applied to the operands; the right combination is design-dependent. At 125M parameters, five specialized-attention designs yield 44–62% SELECTIVE value channels whose legibility rises with depth, at LAMBADA/BLiMP/ARC-Easy parity with a GELU baseline. Coupling Boolean attention to the companion paper’s legible FFN produces an end-to-end model at baseline quality whose named FFN units can be read off a generated token via direct logit attribution. An appendix characterizes a non-monotonic trainability boundary for mixed bounded/unbounded heads.","tokens_in":26854,"tokens_out":1115,"duration_ms":9687,"significance":"If the result holds under multi-seed and multi-scale checks, the paper would change the object of study in mechanistic interpretability: a language model whose dominant FFN and deep-attention computations are named fuzzy set/quantifier operations by construction, rather than recovered post hoc. The bounded-value / free-readout principle, the design-dependent pressure rule, and the concrete end-to-end reading of a generated token (superlative ∩ negation → “worst”) are concrete contributions. Strengths include parameter-neutral designs, an explicit failed vocabulary-tied control that motivates the free-write choice, and a checkpoint-level mechanism for the trainability boundary. The work is currently limited by single-seed, one-epoch, 125M evidence and author-defined legibility thresholds, so the “free legibility / no quality cost” framing remains provisional until those are tightened.","major_comments":[{"comment":"§3.8, Tables 3–4, §4.4–5.2: All quality and SELECTIVE claims rest on single-seed, one-epoch 125M runs. The paper itself notes that within-band LAMBADA/BLiMP differences are smaller than seed noise and that ordering inside the parity band is not trustworthy. The central “parity / free legibility” claim therefore needs at least a multi-seed report (or an explicit, quantified seed-variance bound) before it can be treated as established rather than provisional.","section":null},{"comment":"§4.1, Table 1: SELECTIVE is defined by fixed thresholds (variance ≥ 0.003, crispness > 0.5, rail 0.1) and fixed λ_s=10^{-3}, λ_c=3×10^{-3}. The 44–62% rates and the design-dependent pressure rule are load-bearing for the main claim; without a short sensitivity sweep over those knobs (or a human naming study of a random sample of SELECTIVE channels), the quantitative legibility numbers remain author-metric-dependent.","section":null},{"comment":"§5.4, Figure 3, §6: Once the legible FFN is coupled, shallow attention collapses to CONSTANT (end-to-end SELECTIVE only 30–35%). The end-to-end “legible-by-construction language model” claim is therefore accurate only for FFN throughout and attention at depth. The abstract and conclusion should state this division of labor as a measured limitation, not only as future work, so the scope of the claim matches the evidence.","section":null}],"minor_comments":[{"comment":"Figure 1 vs Figure 5: The baseline DLA figure is clear; the end-to-end figure would benefit from an explicit side-by-side residual (GELU) vs named-path attribution fraction so the “three-quarters legible” claim is visible without reading the caption.","section":null},{"comment":"§3.6: State the exact loss terms (λ_s ∑|v| and λ_c ∑v(1−v)) once in equation form so the pressures are not only described in prose.","section":null},{"comment":"Table 2 and Table 6: Max-activating token lists are useful; a short note on how many channels were inspected and what fraction were judged cleanly nameable would make the qualitative sample less anecdotal.","section":null},{"comment":"Appendix A: The membership-75 divergence mechanism is well documented; a one-sentence pointer in the main text (§3.5 or §4) would help readers who skip the appendix.","section":null},{"comment":"References: Companion paper is cited as arXiv:2606.31845; ensure the final version cross-links the two manuscripts clearly for readers who encounter only one.","section":null}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a natural second half of a companion FFN paper; the architectural idea is interesting and the failed vocabulary-tied control is honest. The main risk for the journal is overclaiming “free / at parity” from single-seed 125M evidence. If the authors deliver multi-seed numbers and tighten the abstract/conclusion on the shallow-attention collapse, the paper is a solid contribution; without that, it is better as a workshop/extended abstract. Fit is appropriate for a methods/interpretability venue that accepts architectural proposals with careful empirical caveats."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The load-bearing move is simple and well-argued: put a sigmoid (or explicit A∩B / A\\B) on the value, leave WO free, and add design-dependent sparsity/crispness pressures on the operands. That is new relative to the usual post-hoc OV work and to pattern-side bounded attention. The vocabulary-tied collapse is the right negative control; the pressure split (membership wants both, Boolean wants crispness alone) is empirically clean and not a global knob; and the depth-rising SELECTIVE curves directly push back on the “heads only crystallize on punctuation” folklore. Coupling Boolean attention to the companion FFN and reading “worst” off named superlative ∩ negation units is a concrete payoff, not just a claim.\n\nWhat it does well: parameter-neutral designs, honest trainability appendix (the non-monotonic 75% membership failure is useful), and clear separation of detection-side vs promote-side legibility. Citations sit in the right places (Elhage circuits, Geva, Belrose, SAEs, fuzzy logic). Math is elementary fuzzy product t-norms; no overclaim of formal guarantees.\n\nSoft spots are real but proportional. Everything is single-seed, one-epoch, 125M. Within-band LAMBADA/BLiMP moves are smaller than the seed noise the paper itself flags, so “parity / free legibility” is supported at this scale, not established. SELECTIVE (var≥0.003, crisp>0.5) and the fixed λs/λc are author-chosen; qualitative max-activating names are illustrative, not a systematic naming study. Shallow attention going constant once the FFN is legible is acknowledged, not hidden. No code release yet.\n\nThis is for people who care about by-construction interpretability and residual-stream design, not for someone hunting a new SOTA LM. The central architectural claim holds up at the reported scale; the stronger framing needs multi-seed and scale checks. I would send it to peer review—serious referees will tighten the stats and ask for code, but the idea and the ablations deserve that time. Engage if you work on mech-interp architecture; skim the appendix if you only care about training stability of mixed bounded heads.","headline":"Clean architectural idea—bound the value, free the write—with real ablations and a working end-to-end 125M model; parity and SELECTIVE rates are single-seed and provisional.","tokens_in":27441,"tokens_out":560,"would_cite":true,"duration_ms":6762,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A transformer can be built so its attention values and feed-forward units are named logical operations, readable without post-hoc tools, at baseline language-model quality.","keywords":["legible-by-construction","attention","transformers","fuzzy set operations","mechanistic interpretability","language models","Boolean attention","selectivity pressures"],"falsifier":"Train the same architectures for multiple seeds at 125M and at least one larger scale; if LAMBADA and BLiMP fall reliably below the conventional baseline, or if the fraction of selective value channels collapses well below the reported 44–62% range, the claim of free-cost by-construction legibility fails.","tokens_in":27357,"feed_emoji":"🔍","tokens_out":769,"duration_ms":18927,"temperature":0.7,"pith_summary":"Most knowledge of what happens inside a language model is recovered after training with external tools that are partial, contested, and expensive to keep current. This paper takes the other route: choose primitives that are already named operations before anyone interprets them. Building on a companion result that rebuilt feed-forward layers from fuzzy set operations, it makes attention legible by a deliberately small change—pass each head’s value through a sigmoid, or restructure it as an explicit within-token intersection and set-difference—so every value channel becomes a readable detector of whether a feature holds at a token. The load-bearing rule is to bound what a head detects and leave what it writes free; pinning the write to the vocabulary collapses the head into a context-free constant. Two selectivity pressures, one for sparse firing and one for decisive firing at the rails, then shape those bounded values into detectors, and which pressure a design wants is not universal. Across five designs at 125M parameters, 44–62% of value channels become crisp, contextually selective detectors whose legibility rises with depth, quality sits at parity with a conventional baseline, and coupling Boolean attention to the legible feed-forward layer yields an end-to-end model whose named units can be read off a token it generates.","feed_headline":"Half of attention channels become readable detectors","feed_subtitle":"Bound what heads detect and free what they write; an end-to-end model then names the units that produce each token.","key_machinery":"The bounded-value / free-readout principle: constrain the pre-projection value as V=σ(W_v x)∈[0,1] (membership head) or as the concatenation of A⊙B and A⊙(1−B) (Boolean head), leave the output projection free, and apply sparsity and/or crispness pressures on the operands so channels fire sparsely and at the rails rather than as diffuse gates or dead constants.","core_discovery":"Attention becomes legible by construction when a head’s value is bounded into fuzzy memberships—or restructured into an intersection and set-difference—while its output projection is left free, and when two design-dependent selectivity pressures turn those bounded values into crisp, contextually selective detectors. Coupled to a previously legible feed-forward layer, this produces an end-to-end language model at baseline quality whose dominant computations are named operations that can be read in the act of predicting a token.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["44-62% of attention value channels become crisp detectors","Bounding values turns attention heads into selective detectors","Free output write lets bounded attention form readable detectors","End-to-end model names the units that produce each token","Attention joins legible FFN for fully named transformer"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That quality parity and the reported fractions of selective detectors, measured on a single seed at 125 million parameters after one training epoch, will hold under seed variation and at larger scales.","fun_headline_variants_meta":{"raw":{"variants":["44-62% of attention value channels become crisp detectors","Bounding values turns attention heads into selective detectors","Free output write lets bounded attention form readable detectors","End-to-end model names the units that produce each token","Attention joins legible FFN for fully named transformer"]},"model":"grok-4.5","effort":"low","cost_usd":0.004254,"raw_usage":{"total_tokens":1345,"prompt_tokens":906,"num_sources_used":0,"completion_tokens":60,"cost_in_usd_ticks":42540000,"prompt_tokens_details":{"text_tokens":906,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":379,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":906,"tokens_out":60,"duration_ms":3977,"temperature":1.0,"reasoning_tokens":379,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T20:05:58.024793+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train the same architectures for multiple seeds at 125M and at least one larger scale; if LAMBADA and BLiMP fall reliably below the conventional baseline, or if the fraction of selective value channels collapses well below the reported 44–62% range, the claim of free-cost by-construction legibility fails.","supporting_citations":[],"review_version":1}