{"id":"5539825c-9122-4e95-ac1a-7b1e6d502a99","arxiv_id":"2608.01377","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CraftAlign uses 304 writing features and an energy model to choose which aspects of an AI story to revise, and its guidance improves perceived human-likeness in a small human study.","lead":"CraftAlign is a new system that scores AI-written stories on 304 writing and narrative features, searches for feature changes that make the story look more human, and turns those changes into editing instructions for another language model. In a small human test, its revisions were selected as most human-like 45.6% of the time versus 30.6% for the strongest baseline, though the model-based scores that drive it are partly self-referential.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Independent evidence for the headline claim is a 20-group human study with no significance tests; the model-space Table 2 is self-referential because the same energy model selects and scores the guidance.","rationale":"The reader's weakest-assumption pointed at schema/annotation validity. I see that as secondary: the human study, if reliable, would validate the full pipeline even if the schema is imperfect, since it tests perceived human-likeness of revised stories against count-matched random guidance. The truly load-bearing point is that the model-space results in Table 2 cannot be interpreted as independent confirmation, because the evaluator is the selection objective. The paper itself states that it relies on the human study for independence. That study is small and lacks inferential statistics. I therefore focus on whether the 15-point advantage over Humanize Only is real. Existing counts per reviewer suggest a paired t-test on reviewers might be borderline, but reviewer-level clustering understates group-level variance, so a proper group-clustered permutation test is needed. If that test is significant, the central claim survives and the reader's CONDITIONAL verdict stands; if not, the claim is unverified. This does not change the verdict category, but it sharpens the condition: report the significance test and a non-circular model-space evaluation.","tokens_in":23263,"tokens_out":10335,"duration_ms":102429,"concrete_test":"Re-analyze the existing human data with a cluster-robust paired permutation test: for each of the 20 groups, compute the mean CraftAlign-minus-Humanize selection count across the 8 reviewers; permute condition labels within group (or use a group-level bootstrap) to get an exact p-value and 95% CI for the difference. If p > 0.05 or the CI includes 0, the claimed advantage is not established and an additional pre-registered human study with ≥50 groups is needed before accepting the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2 is presented as cross-editor evidence, but every metric there uses the same energy model that greedily selected CraftAlign's transitions: ΔD and win rate are the optimizer's objective, and 'target realized' is measured against the CraftAlign-selected target set for all conditions (§F). Random Guidance is therefore penalized for not realizing instructions it was never given, and any partial instruction-following by CraftAlign registers as positive gain. The paper acknowledges this circularity and leans on the human study. That study contains 20 guidance-eligible groups and reports only mean Selection Rate with reviewer-level min-max ranges; no p-value, CI, or effect-size uncertainty is given. Under the two-of-five selection protocol, chance selection is 40%; CraftAlign's 45.6% is above Humanize Only's 30.6% by 15 points, but reviewer-level counts are heterogeneous (R5 preferred Humanize 13 to CraftAlign 10; R6 preferred Random 9 to CraftAlign 10), so the advantage may not be significant once clustered by group. Thus the strongest, cleanest evidence for the central claim is currently a small descriptive human study.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"CraftAlign proposes a feature-grounded framework for detecting and revising AI-written stories. It fine-tunes Qwen3.5-9B with 304 feature-specific heads to predict the StoryScope feature profile of a story, then trains a prompt-optional, class-conditional energy model that scores how well a predicted feature configuration matches Human versus AI writing. At inference, the framework fixes Human as target, performs a budgeted greedy search over schema-valid single-feature transitions that maximize the Human-directed energy gain, renders the selected transitions as natural-language instructions from a curated guidance dictionary, and passes them to a general-purpose LLM editor for full-story rewriting. The paper evaluates the framework in three parts: RQ1 measures feature-estimation fidelity; RQ2 measures Human/AI classification with and without the prompt; RQ3 measures revision guidance across three editors in model space and through a small human study. The headline claims are that predicted profiles retain downstream information, the energy model reliably distinguishes Human from AI patterns, and energy-selected targeted guidance outperforms generic rewriting and count-matched random guidance across editors and in human assessment.","tokens_in":23610,"tokens_out":5137,"duration_ms":50106,"significance":"If the central revision claim holds, the paper makes a useful contribution: it demonstrates that an explicit, auditable feature space can serve as an interface between automatic evaluation and full-story revision guidance, rather than stopping at a source label or a holistic score. The design has real strengths. RQ1 and RQ2 are carefully set up with prompt-level splits, out-of-sample predicted profiles, and multiple complementary metrics; the finding that predicted profiles retain most of the reference-profile information is credible and well supported. The paper also deserves credit for explicitly acknowledging the circularity of the model-space revision evaluation and for including a human study as an independent check. The guidance dictionary, the renderability constraints, and the count-matched Random Guidance baseline are thoughtful controls that isolate the value of direction selection. However, the central revision claim currently rests on a model-space analysis that is self-referential and a human study that is small and reported without inferential statistics. The feature-schema premise that the StoryScope LLM-annotated features capture human-likeness is also validated onl","major_comments":[{"comment":"The model-space cross-editor analysis is self-referential in a way that directly affects the abstract's 'across editors' claim. The target set used for Target realized, Non-target drift, and ΔD in Table 2 is the set selected by the same energy model whose D score is the objective being maximized. Random Guidance is evaluated against a target set it never received, so its lower Target realized (47% vs. 55%) partly reflects a mismatch of selection rather than a failure to follow instructions. The paper acknowledges this in §RQ3, but the conclusion still credits the cross-editor evidence. Please either (a) define the target set independently of the search model, e.g., using human-annotated transitions; (b) re-score revisions with a separate evaluator not used during selection; or (c) explicitly restrict the cross-editor claim to the human study. As written, Table 2 cannot independently supp","section":"RQ3, Table 2, Appendix F"},{"comment":"The human study is the only independent evidence for the central revision claim, but it is reported without inferential statistics. With 20 prompt groups and 8 reviewers selecting 2 of 5 stories, the chance selection rate is 40%. CraftAlign's 45.6% is only 5.6 points above chance, and the 15.0-point gap to Humanize Only (30.6%) has no confidence interval or p-value. Reviewer-level counts are heterogeneous: R5 chose Humanize 13 times and CraftAlign 10, while R6 chose Random 9 times and CraftAlign 10. The aggregate difference may not be significant once clustered by group. Please report a significance test that accounts for the grouped structure (e.g., bootstrap by prompt group or a mixed-effects logistic regression), and include effect sizes and confidence intervals. Without this, the claim that human readers perceive CraftAlign as more human-like is not established.","section":"Human evaluation, Figure 4, Table 5"},{"comment":"The target of the entire guidance pipeline is the Human class in the 304-feature StoryScope space, but the reference profiles used to train the feature estimator are produced by Gemini-3-Flash, not by human critics. If this schema or the LLM annotations are biased, then moving predicted features toward 'Human' in this space optimizes the wrong objective. The small human study is the only external validation of this premise. I suggest adding a validation step: show that a sample of StoryScope features correlates with human judgments of human-likeness, or that energy gain predicts human selections beyond the current 20 groups. This is a correctness-risk concern rather than an internal inconsistency; the paper should make the dependency explicit and test it.","section":"Method / Story Feature Estimation (RQ1) and Energy Model"}],"minor_comments":[{"comment":"The Shared Rewrite Prompt says the revised story must contain 85–115% of the original word count, but Appendix F defines the length-eligibility interval as 0.6–1.4 (60–140%). Please align the two definitions.","section":"Appendix E / Appendix F"},{"comment":"The planner in Appendix E is referred to as Gemini-3.5-Flash, while the human evaluation section and dataset description refer to Gemini-3-Flash. Clarify whether the planner and the editor are the same model and use consistent naming.","section":"Appendix E / Human evaluation"},{"comment":"The baseline names 'Profile-only' and 'Prompt-cond.' are not defined in the main text. Add a sentence or a table footnote explaining what these energy specialists are and how they differ from Joint-Zero and Joint-Prompt.","section":"Table 1"},{"comment":"The left panel reports numeric values for each feature type but the caption does not explain how the percentages are computed. State explicitly that the panel shows average primary scores (F1, κ, Jaccard, or inverse MAE) by feature type.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's core architecture and RQ1/RQ2 evidence are solid, and the revision idea is interesting. The main risk is that the headline revision claim is currently supported by a self-referential model-space evaluation and a small human study without inferential statistics. I would be willing to accept after the authors add proper statistical analysis of the human study and either de-emphasize or independently validate the cross-editor model-space results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the evaluation half of this paper is sound and the revision half is a real integration contribution, but the headline revision claim rests on a small descriptive human study, not on Table 2. Worth a serious referee, with the human evaluation and release of artifacts as the main revision demands.\n\nWhat's actually new: translating StoryScope's 304-feature space from a diagnostic into an intervention space. The local Qwen estimator reproduces reference profiles well enough that downstream Human/AI classification only drops from 95 to 93 Macro-F1, and the joint prompt-optional energy model is within 0.2 points of the reference-profile benchmark. Those numbers are clean, and the isolation protocol (prompt-level splits, out-of-sample profiles) is done properly.\n\nThe also-novel piece is the inference pipeline: structured single-feature perturbations, greedy positive-gain selection, a curated guidance dictionary, and full-story rewriting by a separate LLM editor. The baselines are the right ones—generic rewrite and count-matched random guidance—and the cross-editor setup is a good idea.\n\nSoft spots. Table 2 is not independent evidence for the revision claim. The same energy function selects the transitions and measures the gain, and 'target realized' is computed against CraftAlign's own selected target set for all conditions, so Random Guidance is judged against targets it was never asked to follow. The paper acknowledges this, and to their credit they supply a human study. But that study is 20 groups with eight reviewers, no significance tests, and the published reviewer counts show real heterogeneity (R5 prefers Humanize 13 to CraftAlign 10; R6 prefers Random 9 to CraftAlign 10). The aggregate 45.6% vs 30.6% is suggestive, not decisive. The underlying feature schema also comes from Gemini-3-Flash annotations, so the whole optimization inherits any bias in that schema; the human study is the only external check and it is small.\n\nMinor but real: no code, data, or full guidance dictionary are released, which matters for a pipeline where the dictionary is a load-bearing component.\n\nWho this is for: people working on AI-text evaluation or creative-writing tools. The paper gives a concrete architecture and is unusually candid about its own circularity. I'd send it to reviewers; with a bigger human eval and released artifacts, the revision claim would be much stronger. As is, I would cite it for the evaluation results and the pipeline design, not as proof that energy-guided feature search beats alternatives.","headline":"Solid evaluation half, plausible revision half, but the headline claim rests on a small descriptive human study rather than Table 2.","tokens_in":24011,"tokens_out":2570,"would_cite":true,"duration_ms":25407,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CraftAlign claims that AI stories become more human-like when revision is driven by energy-selected changes in a 304-feature writing profile, and its targeted guidance beats generic rewriting in both model-space and human tests.","keywords":["AI storytelling","story revision guidance","writing-pattern evaluation","energy-based models","feature-grounded evaluation","narrative features","human-likeness","LLM evaluation"],"falsifier":"Run the same human protocol on a larger held-out sample where guidance is selected to maximize AI-likeness instead of Human-likeness; if readers still prefer those revisions, or if count-matched random guidance matches CraftAlign's selection rate when the sample is large enough, the claim that energy-selected movement toward the Human pattern improves perceived human-likeness collapses.","tokens_in":23209,"feed_emoji":"✍️","tokens_out":7917,"duration_ms":70725,"temperature":0.7,"pith_summary":"This paper tries to establish that the formulaic 'AI flavor' of machine-written fiction can be reduced by treating an explicit 304-feature writing profile as a space for revision, not just diagnosis. CraftAlign first predicts those features for a story, then uses a prompt-optional class-conditional energy model to score how Human-like versus AI-like the feature configuration is, and finally searches for small schema-valid feature changes that lower the Human/AI energy gap. The chosen changes are rendered as natural-language instructions, and a separate LLM editor rewrites the full story with them. The authors report that this targeted guidance beats generic rewriting and count-matched random guidance across three editor models, and that human readers select the CraftAlign revisions far more often than the original or baseline revisions, while still preferring authentic human-authored stories. If the claim holds, a feature classifier plus energy search is enough to turn evaluation into actionable, story-wide revision guidance.","feed_headline":"Targeted feature guidance beats generic rewriting in AI stories","feed_subtitle":"Scoring 304 writing features and steering edits toward human patterns lifts reader selection 21 points over the original draft","key_machinery":"The load-bearing object is the shared class-conditional energy function $E_\\theta(\\hat{\\mathbf{z}}, \\tilde{e}_p, y)$: a small MLP that scores a predicted 304-feature profile $\\hat{\\mathbf{z}}$, an optional prompt embedding $\\tilde{e}_p$, and a candidate label $y\\in\\{\\mathrm{H}, \\mathrm{A}\\}$ with a scalar. The difference $D(\\hat{\\mathbf{z}}, \\tilde{e}_p)=E_\\theta(\\cdot,\\mathrm{H})-E_\\theta(\\cdot,\\mathrm{A})$ is the same quantity used to judge Human/AI pattern and to compare revisions. For each single-feature transition $z\\to z'$, the Human-directed gain $\\Delta D(z\\to z'; \\tilde{e}_p)=D(z,\\tilde{e}_p)-D(z',\\tilde{e}_p)$ drives greedy selection; up to five such transitions become natural-lang","core_discovery":"The central discovery, stated on the paper's own terms, is that explicit writing features can serve as a structured intervention space rather than only a diagnostic layer. CraftAlign's local feature estimator reproduces the 304-feature StoryScope profiles well enough that downstream probes retain nearly all Human/AI information (93% Macro-F1 with predicted profiles versus 95% with reference profiles), and its jointly trained energy model reaches 93.86% Macro-F1 when the writing prompt is available. In revision, guidance selected by the largest Human-directed energy gain produces positive mean movement with all three tested editors (average $\\Delta D=2.00$ versus $0.24$ for count-matched rand","pith_inferences":["Beyond the paper: the 304-feature space could be used as a controllable stylistic dial—the same energy-guided search, run in reverse, would make a draft more AI-like, which authors might use to imitate or satirize machine-text conventions.","Beyond the paper: the evaluation-to-revision recipe is domain-generic; any text domain with an explicit feature taxonomy and a class-conditional energy model could convert a black-box quality judgment into prioritized rewrite instructions, e.g., academic prose or dialogue.","Beyond the paper: a natural next experiment is isolating which feature dimensions drive the human preference—ablating single selected transitions would show whether information release, theme explicitness, or causality carries most of the 21-point selection gain.","Beyond the paper: the 20-group human study is small; a larger study with more prompts and reviewers would tell whether the observed gap generalizes and whether the human reference advantage (73.8%) can be further closed by longer search budgets."],"forward_implications":["Feature-grounded guidance transfers across editor models: the same selected transitions improve revisions produced by three different LLM editors, so the method is not tied to one rewrite engine.","Predicted profiles, not human reference annotations, are sufficient for deployment: the local estimator preserves nearly all downstream Human/AI signal, making the pipeline applicable to new stories where no reference profile exists.","Prompt conditioning strengthens evaluation (0.69 Macro-F1 points) without hurting prompt-free performance, so the same checkpoint can be deployed with or without the original prompt.","Randomly selected valid transitions do not help: count-matched random guidance stays near the original in human selection (25.6% vs 24.4%), so targeted selection is the active ingredient.","The same energy signal motivates training story-generation models with feature-grounded rewards, a direction the paper explicitly identifies as the next step."],"supporting_citations":[{"why":"Supplies the 304-feature StoryScope schema, the prompt-grouped Human/AI dataset, and the reference profiles the feature estimator is trained on; the central claim operates in this feature space.","marker":"Russell et al. 2026"},{"why":"Supplies the energy-based modeling formulation that the class-conditional energy function and Human-directed gain inherit.","marker":"LeCun et al. 2006"},{"why":"Supplies the type-aware structured perturbations for discrete and mixed variables used to construct schema-valid single-feature candidates during search.","marker":"Schröder et al. 2024"},{"why":"Supplies the listwise softmax ranking formulation used as an auxiliary training objective so that the human story ranks first within each prompt group.","marker":"Cao et al. 2007"},{"why":"Represents the artifact-focused revision approach that CraftAlign contrasts with story-wide feature guidance.","marker":"Chakrabarty, Laban, and Wu 2025"},{"why":"Supplies the CORN ordinal regression method used for the ordinal feature prediction heads in the local estimator.","marker":"Shi, Cao, and Raschka 2023"}],"fun_headline_variants":["Feature-guided story revision beats generic rewrite by 21 pts","CraftAlign: scoring 304 features steers AI stories to human craft","Targeted feature shifts outperform generic rewriting in AI tales","Energy-guided feature moves improve AI story quality","Feature-aware editing lifts AI narrative quality over baselines"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The whole pipeline stands or falls on whether the 304-feature StoryScope schema, annotated by Gemini-3-Flash rather than human critics, actually captures what makes a story feel human-authored—if the schema misses the real drivers of 'AI flavor,' the energy-guided revisions optimize the wrong target, and the 20-group human study is the only external check.","fun_headline_variants_meta":{"raw":{"variants":["Feature-guided story revision beats generic rewrite by 21 pts","CraftAlign: scoring 304 features steers AI stories to human craft","Targeted feature shifts outperform generic rewriting in AI tales","Energy-guided feature moves improve AI story quality","Feature-aware editing lifts AI narrative quality over baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000682,"raw_usage":{"total_tokens":2939,"prompt_tokens":757,"completion_tokens":2182,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":2102}},"tokens_in":501,"tokens_out":2182,"duration_ms":19363,"temperature":1.0,"reasoning_tokens":2102,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:15:11.619858+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same human protocol on a larger held-out sample where guidance is selected to maximize AI-likeness instead of Human-likeness; if readers still prefer those revisions, or if count-matched random guidance matches CraftAlign's selection rate when the sample is large enough, the claim that energy-selected movement toward the Human pattern improves perceived human-likeness collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the energy-based modeling formulation that the class-conditional energy function and Human-directed gain inherit."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the listwise softmax ranking formulation used as an auxiliary training objective so that the human story ranks first within each prompt group."}],"review_version":1}