{"id":"3ab82d70-d24d-417e-87be-34f3c1cfdbda","arxiv_id":"2606.11207","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":10,"one_line_summary":"A modular, LLM-executed rule engine for e-commerce session analysis achieves 56.4% accuracy on a 39-session pilot while prioritizing auditability over predictive performance.","lead":"The paper builds a rule-based framework that decomposes e-commerce session data into 24 hand-coded behavioral 'elements' and uses an LLM to execute those rules for purchase-intent inference. It trades predictive accuracy (56.4% on a 39-session pilot) for auditability and deterministic reproducibility.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The σ=0 reproducibility claim applies to the deterministic engine, but all reported quantitative results come from the LLM engine, which explicitly lacks σ=0. The deterministic engine's predictive performance is never reported.","rationale":"The reader's CONDITIONAL verdict is appropriate. The reader identified the empirical insufficiency (n=39, 56.4% below baseline) and correctly noted the 'conservative abstention' reframing. However, the reader's weakest_assumption focused on the element library structure and hand-coded weights, which is a real but secondary concern. The more load-bearing issue is the disconnect between the σ=0 claim and the empirical validation: the paper's headline advantage (deterministic reproducibility) is claimed for an engine whose results are never reported, while all reported results come from the LLM engine that explicitly lacks this property. This does not push the verdict to REJECT because the paper is transparent about the LLM variability boundary (§6.3, Table 9) and does not claim σ=0 for LLM outputs. But it does mean the central claim — that the framework offers a σ=0 alternative trading 'marginal' predictive gains — is currently unsupported by evidence from the σ=0 component itself. The concrete test of running the deterministic engine on the same sample would settle whether the σ=0 engine is a viable inference path or merely an architectural artifact.","tokens_in":13744,"tokens_out":3474,"duration_ms":170096,"concrete_test":"Run the deterministic engine (with E8/E10 set to their σ=0 fallback outputs) on the same 50-session masked-field sample and report accuracy, precision, recall, F1, and the uncertain rate. Compare against Table 7's LLM engine results. If the deterministic engine's accuracy differs from 56.4% by more than ~10 percentage points, or if its uncertain rate changes substantially, the claim that the two engines maintain 'scoring consistency' (§4.6, Phase 3) is unsupported, and the σ=0 framing becomes decoupled from any empirical evidence of utility.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central differentiator is σ=0 reproducibility. The abstract states: 'All quantitative results reported herein are produced by this engine [LLM-Integrated Semantic Inference Engine].' The same abstract acknowledges that 'LLM-dependent results (E8, E10) are subject to controlled output variability.' In §4.6, Phase 2 has the LLM 'executing formulas step-by-step, evaluating constraint conditions, selecting applicable variant formulae' — meaning element signal values are computed by the LLM, not the deterministic engine. Phase 3 only applies redundancy caps and TieredPenaltyCalculator post-hoc; it does not recompute element signals deterministically. Therefore: (1) the σ=0 property, which is the paper's primary selling point over black-box methods, is never empirically validated with any accuracy/F1/precision/recall numbers; (2) the 56.4% accuracy result in Table 7 comes entirely from the non-deterministic component; (3) we have no evidence that the deterministic engine achieves even this modest performance. The paper's framing — 'trading marginal predictive gains for element-level transparency' — implies the σ=0 engine is a viable alternative, but no results from that engine are presented to support this trade-off. The reader correctly identified the poor empirical performance and the 'conservative abstention' reframing, but the more fundamental issue is that the σ=0 claim and the empirical validation operate on different engines.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper presents SemantiClean, a modular framework for e-commerce behavioral inference that decomposes session data into 24 hand-coded behavioral elements organized in a four-layer architecture, with anti-inflation mechanisms (redundancy group caps, tiered penalties, adaptive constraints). A two-phase LLM-integrated inference engine selects and executes elements on demand. The framework is evaluated on a masked-field purchase intent prediction task using the OSPI dataset (n=39 evaluated sessions after 11 exclusions). The paper's central differentiator is auditability and sigma=0 reproducibility, explicitly trading predictive accuracy for transparency.","tokens_in":14714,"tokens_out":1464,"duration_ms":251110,"significance":"The paper addresses a legitimate gap in e-commerce analytics: the tension between predictive accuracy and decision auditability. The JSON-driven element library architecture is a reasonable engineering contribution, and the explicit declaration of design constants (R1-R10) without ablation support is commendably honest. The three-phase LLM inference architecture with on-demand element loading is a practical design. However, the significance is substantially undermined by a critical disconnect between the paper's primary selling point (sigma=0 reproducibility of the deterministic engine) and the empirical validation (which uses only the non-deterministic LLM engine). The pilot study (n=39) is too small to support any empirical claim about the framework's utility.","major_comments":[{"comment":"Abstract and §4.6: The paper's central differentiator is sigma=0 reproducibility of the deterministic engine, but all quantitative results (Table 7) come from the LLM-Integrated Semantic Inference Engine, which the abstract acknowledges lacks sigma=0. The deterministic engine's predictive performance is never reported. The abstract states 'All quantitative results reported herein are produced by this engine [LLM engine]' while simultaneously claiming 'Deterministic engine outputs remain fully reproducible (sigma=0).' This is internally inconsistent: the property that distinguishes the framework from black-box methods is never empirically validated. The paper must either report accuracy/F1/precision/recall for the deterministic engine on the same masked-field task, or explicitly reframe the contribution to accurately reflect that the validated component is the LLM engine, not the sigma=0-","section":null},{"comment":"§5.4, Table 7: The pilot study reports 56.4% accuracy on n=39 sessions (Wilson 95% CI: [40.1%, 71.8%]), which is below the 84% majority baseline. The paper frames this as 'not an optimisation failure but a structural consequence of the framework's conservative abstention policy.' This reframing is not supported: the 56.4% is computed only on the 39 evaluated predictions (excluding 11 uncertain outputs), so the abstention policy has already been accounted for by exclusion. Among evaluated predictions, the system still underperforms the majority baseline, with 14 false positives against only 5 true positives (precision 26.3%). The claim that results 'validate the system's conservative abstention policy' is not substantiated by the data presented.","section":null},{"comment":"§6.4, R1-R10: The paper lists 10 categories of design constants (element weights, normalization ceilings, thresholds, etc.) and explicitly states these are 'without ablation experimental support.' These constants are load-bearing for every quantitative result: the element signal values, redundancy group caps, and penalty multipliers all flow into Eq. (10). Without any sensitivity analysis or ablation, it is impossible to determine whether the 24 elements and their fixed weights capture semantically distinct, non-redundant signals or whether the framework's behavior is an artifact of arbitrary parameter choices. At minimum, a sensitivity analysis on the most influential constants (e.g., E1 weights [0.40, 0.30, 0.30], RedundancyGroup caps, AdaptiveConstraintMode thresholds [5, 8]) should be reported to demonstrate that the framework's behavior is robust to reasonable parameter perturb.","section":null}],"minor_comments":[{"comment":"§4.5, Eq. (10): The note that 'negative weights contribute positively to their assigned labels in identical fashion to positive weights of the same magnitude' is surprising and potentially confusing. A brief explanation of why this design choice was made (rather than standard signed aggregation) would help readers understand the scoring semantics.","section":null},{"comment":"Table 5: Token budget estimates are labeled 'architectural estimates' but it is unclear whether these are measured from actual runs or theoretical calculations. Clarifying this would strengthen the reproducibility claims.","section":null},{"comment":"§4.1: The 10-second threshold for Fallback Rule 1 cites Gorman [11], which is described as an 'RStudio Capstone' project. The citation format and nature of this source should be verified for appropriateness.","section":null},{"comment":"Appendix C, Table 10: The complete 24-element inventory is buried in an appendix. Given that the element library is the paper's central artifact, consider moving at least a summary table of all 24 elements (with layer, redundancy group, and status) into the main text.","section":null},{"comment":"§5.5, Table 8: The confidence band analysis is interesting but based on only 39 predictions. The bimodal confidence pattern (20 of 39 predictions at confidence=1.00) may be an artifact of the small sample rather than a structural property. This should be acknowledged alongside the sample size limitation.","section":null},{"comment":"The paper cites several 2026 references (e.g., [8], [9], [14]) including an arXiv preprint and a Zenodo technical report by the same author. These should be verified for availability and properly versioned.","section":null},{"comment":"§4.3, Table 3: The boundary condition note for New Visitor loyalty (both conditions yield 0.55) is mentioned but the rationale is unclear. A brief explanation would help readers understand the lookup logic.","section":null}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the sigma=0 / LLM engine disconnect is valid and is the most serious issue: the paper's abstract and framing position sigma=0 reproducibility as the primary advantage over black-box methods, yet no quantitative results from the deterministic engine are presented. This is a load-bearing gap that must be addressed in revision. The pilot study size (n=39) is also a concern but is acknowledged by the authors. The paper is honest about its limitations (design constants without ablation, small sample, single model), which is a positive sign, but honesty about limitations does not substitute for addressing them. If the authors can report deterministic engine results on the same task and provide even a basic sensitivity analysis, the paper could become a solid systems contribution."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and substantive review. The referee identifies three major issues: (1) a disconnect between the sigma=0 reproducibility claim and the LLM-based empirical validation, (2) unsupported framing of the 56.4% accuracy result as validating the abstention policy, and (3) absence of sensitivity analysis for load-bearing design constants. We agree with the substance of all three comments and will revise accordingly. The revision will reframe the contribution to accurately reflect that the empirically validated component is the LLM engine, correct the abstention-policy framing, and add a targeted sensitivity analysis. One genuine limitation remains: the deterministic engine has not been run on the masked-field task and cannot be added within the revision timeframe due to the deterministic engine's dispatch table only covering core elements E01-E11, not the full 24-element library used in the pilot.","responses":[{"response":"The referee is correct that there is a genuine disconnect between the sigma=0 claim and the empirically validated component. We acknowledge this as an internal inconsistency in the current framing. We will reframe the contribution to accurately reflect that the validated component is the LLM-Integrated Semantic Inference Engine, not the sigma=0 deterministic engine. Specifically, we will: (1) revise the abstract to lead with the LLM engine as the empirically evaluated contribution and clearly state that sigma=0 reproducibility is a property of the deterministic engine that has not yet been empirically validated on the masked-field task; (2) add an explicit statement in §4.6 and §5 that the deterministic engine's predictive performance on the masked-field task is not reported in this work; (3) adjust the conclusion to avoid implying that the pilot results validate the sigma=0 property. We cannot commit to reporting deterministic engine accuracy on the same task within this revision because the deterministic engine's dispatch table currently covers only core elements E01-E11, while the pilot used the full 24-element library via the LLM engine's on-demand loading. Running the deterministic engine on the masked-field task would require extending its dispatch table to E12-E24, which is engineering work we plan to complete but cannot deliver in this revision cycle. We will state this limitation explicitly.","revision_made":"partial","referee_comment":"Abstract and §4.6: The paper's central differentiator is sigma=0 reproducibility of the deterministic engine, but all quantitative results come from the LLM engine. The deterministic engine's predictive performance is never reported. The abstract is internally inconsistent. The paper must either report accuracy for the deterministic engine or reframe the contribution."},{"response":"The referee is correct on both points. First, the abstention policy has indeed already been applied via the 11 exclusions, so the 56.4% accuracy among the remaining 39 predictions cannot be attributed to conservative abstention. The current framing conflates the abstention mechanism (which produced the 11 uncertain outputs) with the accuracy among evaluated predictions (which is where the system underperforms the majority baseline). Second, the claim that the results 'validate the system's conservative abstention policy' is not substantiated by the data: 14 false positives against 5 true positives indicates that among evaluated predictions, the system is not conservative but rather over-predicts purchase. We will revise §5.4 to: (1) remove the claim that the 56.4% result validates the abstention policy; (2) explicitly acknowledge that among evaluated predictions, the system over-predicts purchase intent (14 FP vs. 5 TP); (3) reframe the abstention discussion to apply only to the 11 excluded sessions, not to the accuracy metric; (4) note that the bimodal confidence analysis in §5.5 already provides a more honest characterization—the system functions as a no_purchase detector at high confidence (85% precision) but produces unreliable purchase predictions across all confidence bands. The referee's observation that precision is 26.3% is correct and we will ensure this is foregrounded rather than obscured.","revision_made":"yes","referee_comment":"§5.4, Table 7: The 56.4% accuracy on n=39 is below the 84% majority baseline. The paper frames this as a structural consequence of conservative abstention, but the abstention policy has already been accounted for by exclusion. Among evaluated predictions, the system underperforms the majority baseline with 14 false positives against 5 true positives (precision 26.3%). The claim that results validate the conservative abstention policy is not substantiated."},{"response":"The referee is correct that the design constants are load-bearing and that the current manuscript provides no sensitivity analysis. We will add a targeted sensitivity analysis on the most influential constants identified by the referee: E1 weights [0.40, 0.30, 0.30], RedundancyGroup caps {0.35, 0.30, 0.30, 0.20}, and AdaptiveConstraintMode thresholds [5, 8]. The analysis will perturb each parameter by reasonable amounts (e.g., +/-20% for weights, alternative cap configurations, shifted thresholds) and report the effect on accuracy, precision, recall, and F1 on the same 39-session pilot. This will be a limited analysis given n=39, but it will address the core concern of whether the framework's behavior is an artifact of specific parameter choices or is robust to reasonable perturbation. We will also add an explicit caveat that the small sample size limits the statistical power of the sensitivity analysis and that systematic ablation on a larger sample (n>=500 as noted in §6.2) is needed for definitive conclusions. We acknowledge that without this analysis, the claim that the 24 elements capture semantically distinct signals is unsupported, and we will soften that claim accordingly.","revision_made":"yes","referee_comment":"§6.4, R1-R10: The paper lists 10 categories of design constants without ablation support. These constants are load-bearing for every quantitative result. Without sensitivity analysis, it is impossible to determine whether the 24 elements capture semantically distinct signals or whether behavior is an artifact of arbitrary parameter choices. At minimum, a sensitivity analysis on the most influential constants should be reported."}],"tokens_in":13791,"tokens_out":1358,"duration_ms":169639,"standing_objections":["The deterministic engine's predictive performance on the masked-field task cannot be reported in this revision because the deterministic dispatch table covers only E01-E11 while the pilot used the full 24-element library. We will state this as a limitation but cannot close the gap within the revision cycle."]},"desk_editor":{"model":"glm-5.2","letter":"The paper's main selling point — σ=0 reproducibility — is never empirically validated. All reported accuracy numbers come from the LLM engine, which the paper itself admits is non-deterministic. The stress-test note is correct on this: the abstract says all quantitative results come from the LLM-Integrated Semantic Inference Engine, while the deterministic engine's predictive performance is never reported. The σ=0 claim and the empirical validation operate on different engines, and the paper does not confront this gap clearly enough. That is the central problem, and it is load-bearing — the entire framing of 'trading accuracy for transparency' requires showing that the transparent (deterministic) engine actually works at some measurable level. We have no evidence of that. The reader's verdict of CONDITIONAL is fair, though I would lean slightly more negative given this structural mismatch. What the paper does well: the JSON-driven element library is a clean piece of engineering, and the three-phase LLM architecture (lightweight routing → deep analysis → Python aggregation) is a reasonable design for token-efficient structured inference. The anti-inflation mechanisms (redundancy group caps, tiered penalties) are thought through. The paper is also unusually honest about its own limitations — it openly declares all weights as design constants without ablation support and acknowledges the GRP_USER_IDENTITY imbalance. That candor is refreshing and worth crediting. On the empirical side, the pilot is too small to support any claim. n=39 with 56.4% accuracy against an 84% majority baseline is not a marginal gap — the system is worse than always predicting 'no purchase.' The paper frames this as 'conservative abstention,' but 14 false positives against 5 true positives is not conservative; it is poorly discriminating. The confidence-accuracy decoupling (confidence 1.0 for no-purchase predictions that are correct 85% of the time, but near-zero calibration for purchase predictions) confirms the scoring formula is not doing what a reader would expect. The reader correctly identified this. The 24 elements and their weights are hand-coded with no ablation, which the paper admits. That is acceptable for a system design paper but not for one claiming empirical validation. The gender inference target being non-functional is fine to disclose but further narrows the scope of what was actually tested. This is an honest engineering report that needs substantially more empirical work before its claims hold up. The deterministic engine needs to be evaluated on its own. The design constants need ablation. The sample size needs to be at least an order of magnitude larger. Without those, the paper is a system description with a pilot attached, not a validated contribution. I would send this to a serious referee who can evaluate the architecture on its design merits while requiring the empirical gaps to be addressed. The core idea — auditable rule-based behavioral inference with a shared element library — is not novel in concept but the specific implementation is new and could be useful as a reference architecture if the validation gap is closed.","headline":"The paper's main selling point — σ=0 reproducibility — is never empirically validated. All reported accuracy numbers come from the LLM engine, which the paper itself admits is non-deterministic.","tokens_in":14581,"tokens_out":696,"would_cite":false,"duration_ms":80483,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Audit-First E-Commerce Inference Trades Accuracy for Transparency","keywords":[],"falsifier":"Run an ablation where the design constants (element weights, normalisation ceilings, thresholds, redundancy group caps) are randomly perturbed within reasonable ranges; if predictive performance and confidence distributions remain essentially unchanged, the specific parameter values are not load-bearing and the element library's structure is decorative rather than functional. Conversely, if performance changes substantially, the hand-tuned constants are doing unacknowledged work that should be validated.","tokens_in":13984,"feed_emoji":"🔍","tokens_out":1135,"duration_ms":159020,"temperature":0.7,"pith_summary":"This paper presents SemantiClean, a framework that decomposes e-commerce session data into 24 hand-defined behavioral elements organized in a four-layer architecture, then feeds them through a two-phase LLM-driven inference engine to produce purchase-intent predictions with fully reproducible, auditable decision trails. The central claim is that by prioritizing element-level transparency, anti-inflation mechanisms (redundancy group caps, bias penalties, cold-start protection), and deterministic reproducibility (sigma=0 for the core engine), the system can serve compliance-sensitive applications where knowing why a prediction was made matters more than squeezing out marginal accuracy gains. The pilot study on 39 sessions achieves 56.4% accuracy—below the 84% majority baseline—but the author frames this not as a failure but as a structural consequence of conservative abstention: the system withholds low-confidence predictions rather than fabricating certainty, producing 22% uncertain outputs by design.","feed_headline":"Audit-First E-Commerce Inference Trades Accuracy for Transparency","feed_subtitle":"A 24-element behavioral library with LLM-driven inference achieves reproducible, auditable predictions by abstaining rather than guessing—56","key_machinery":"The load-bearing machinery is the element library (behavior_elements.json), a single source-of-truth file defining 24 behavioral elements across four layers (Functional, Interaction, Systemic, Contextual), each with computation formulas, constraints, failure modes, and redundancy group assignments. A two-phase LLM engine reads this library: Phase 1 routes 3-9 relevant elements via lightweight summaries (~2-3k tokens), Phase 2 loads full definitions for deep formula execution (~4-8k tokens), and Phase 3 applies Python-based aggregation with redundancy caps and bias penalties identical to the deterministic engine. Three anti-inflation mechanisms enforce signal quality: AdaptiveConstraintMode (","core_discovery":"The paper's central contribution is the demonstration that a predefined library of 24 semantically scoped behavioral elements, combined with a two-phase LLM inference architecture (lightweight element selection followed by deep formula execution), can produce behavioral inferences that are structurally auditable and deterministically reproducible while honestly surfacing its own uncertainty. The key mechanism is the decoupling of signal extraction (deterministic element computation) from task-specific aggregation (pluggable inference targets), mediated by three anti-inflation mechanisms that prevent any single signal from dominating. The bimodal confidence distribution discovered in the pil—","pith_inferences":["The paper's own data suggests the element library's structure may be the bottleneck: GRP_USER_IDENTITY has 8 members against a 30% cap, with E07 (Visitor Loyalty) dominating in ~85% of sessions. This implies that the library's behavioral taxonomy may not be well-calibrated to the actual signal distribution in e-commerce data, and that the hand-coded element definitions encode assumptions about beh","The absolute-value scoring formula (Eq. 10) produces a structural artifact—no_purchase predictions always get confidence=1.0 while purchase predictions get fractional confidence—meaning the system's confidence scores are not calibrated probabilities but architectural byproducts. This suggests the confidence values cannot be used for probabilistic decision-making without recalibration.","The 10-second session threshold, the 120-second dwell-time ceiling, and all weighting coefficients are design constants without ablation support. If any of these are substantially wrong, the entire element computation chain could be systematically biased, and the sigma=0 reproducibility guarantee would faithfully reproduce the wrong answer every time."],"forward_implications":["If the auditability-first design is viable, compliance-sensitive domains (finance, healthcare, insurance) could adopt similar element-library architectures to satisfy regulatory requirements for explainable automated decisions, even at the cost of predictive accuracy.","The two-phase LLM loading pattern—lightweight routing followed by on-demand deep analysis—suggests a general token-efficient architecture for any LLM inference task involving large structured knowledge bases, reducing context window pressure from ~16k to ~6-13k tokens per session.","The conservative abstention policy (22% uncertain outputs) implies that deployment in high-stakes decisions would require a human-in-the-loop fallback for the uncertain cases, effectively making the system a triage tool rather than an autonomous predictor.","The bimodal confidence pattern (unreliable purchase predictions vs. 85% precision no-purchase detection) suggests the framework could be repurposed as a high-precision negative filter—confidently identifying non-buyers—while deferring purchase predictions to complementary systems."],"fun_headline_variants":["24 Behavioral Elements Enable Auditable E-Commerce Intent Inference","Reproducible LLM Inference Via Shared Element Library Over E-Commerce Sessions","Predefined Element Library Trades Marginal Accuracy for Structural Auditability","Two-Phase LLM Inference with 24-Element Architecture Surfaces Uncertainty Instead of Guess"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The 24 hand-coded behavioral elements and their fixed weighting coefficients capture semantically distinct, non-redundant signals that are structurally valid for inference—a claim the paper itself acknowledges has no ablation support and is undermined by documented group imbalance in the element library.","fun_headline_variants_meta":{"raw":{"variants":["24 Behavioral Elements Enable Auditable E-Commerce Intent Inference","Reproducible LLM Inference Via Shared Element Library Over E-Commerce Sessions","Predefined Element Library Trades Marginal Accuracy for Structural Auditability","Two-Phase LLM Inference with 24-Element Architecture Surfaces Uncertainty Instead of Guessing"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":656,"prompt_tokens":587,"completion_tokens":69,"prompt_tokens_details":null},"tokens_in":587,"tokens_out":69,"duration_ms":37534,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-05T01:24:01.587863+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"Run an ablation where the design constants (element weights, normalisation ceilings, thresholds, redundancy group caps) are randomly perturbed within reasonable ranges; if predictive performance and confidence distributions remain essentially unchanged, the specific parameter values are not load-bearing and the element library's structure is decorative rather than functional. Conversely, if performance changes substantially, the hand-tuned constants are doing unacknowledged work that should be validated.","supporting_citations":[],"review_version":1}