{"id":"81bcf696-358e-4d52-b2ea-f1370b07e0b7","arxiv_id":"2505.23788","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"FUA-LLM uses DPO on expert-labeled fair use preference data to make LLMs generate compliant, helpful responses to copyright-sensitive prompts.","lead":"This paper fine-tunes open-source language models with Direct Preference Optimization on a lawyer-curated dataset so they answer copyright-sensitive prompts with fair-use-aligned, useful responses instead of outright refusals. The proposed FUA-LLM framework reports a better trade-off between infringement risk and response utility than refusal-based baselines, and introduces two new metrics to measure that trade-off.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation may measure memorization rather than generalization: no train/test split is described between the 9,000 FairUseDB prompts and the 'diverse test set' in Section 5, so the reported CAH gains and 20% reduction could be artifacts of evaluating on training prompts.","rationale":"I read the paper's central claim as: FUA-LLM, trained with DPO on lawyer-validated preference data, substantially reduces copyright-infringing outputs while preserving utility, and this is demonstrated by the Utility and CAH metrics and human evaluation. For that claim to hold, two conditions are needed: (1) the evaluation must measure generalization to prompts not seen in training, and (2) the metrics must be valid and reproducible. The reader identified the second condition as the weakest assumption, specifically that the Risk-of-Infringement score R comes from an unspecified classifier. I agree this is a real problem: Tables 1, 3, and 4 all depend on R, and if R is miscalibrated the numeric advantages could change or vanish. However, the first condition, the train/test separation, is even more load-bearing because it undermines the claim regardless of how R is defined. The paper reports no held-out split, no deduplication, and no independent test prompt generation. The phrase 'test set comprising prompts from the same nine infringement categories' is consistent with sampling from the same pool used for training. If that is the case, then even a perfect, fully specified risk classifier would not rescue the result: high scores on training-like prompts would be a memorization artifact, not evidence of fair-use alignment. The human evaluation in Table 2 partly helps, because legal experts rated FUA-LLM's own generations, but it lacks a baseline comparison and is reported only for FUA-LLM, so it cannot establish superiority over SHIELD or MemFree. The appendix examples also contain legal errors, such as treating Shelley's 'Ode to the West Wind' as if it were still under copyright while also suggesting a 'public domain version' might exist; this does not directly affect the quantitative claim but further weakens confidence in the legal grounding. Overall, the reader's REJECT verdict is appropriate, and my analysis supports it. I would not change the verdict, but I would sharpen the primary concern from 'unspecified R' to 'uncontrolled train/test overlap,' with the unspecified R as a compounding factor.","tokens_in":28915,"tokens_out":2569,"duration_ms":26216,"concrete_test":"Obtain the released FairUseDB and the evaluation prompts; compute exact overlap and near-duplicate similarity (e.g., embedding or n-gram overlap) between the DPO training set and the test set. Then retrain FUA-LLM from the SFT checkpoint on a random 80/20 split of FairUseDB and evaluate on the held-out 20% plus a newly generated set of 500 prompts per category from an independent prompt-generation pipeline that has not been used to create any training data. If FUA-LLM's CAH advantage over SHIELD and MemFree persists on the non-overlapping, independently generated prompts, the central claim survives; if the advantage collapses, the reported gains are training-set artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is not the unspecified Risk-of-Infringement classifier R itself, although that is serious, but the apparent absence of any held-out evaluation set. Section 3.1 constructs FairUseDB as 9,000 legally sensitive prompts, each with a preferred and a rejected response, and then trains FUA-LLM via DPO on exactly this dataset. Section 5 then evaluates FUA-LLM on 'a diverse test set comprising prompts from the same nine infringement categories' but never states that this test set is disjoint from the 9,000 training prompts, never reports a train/test split, and never describes any deduplication or near-duplicate removal. If the test prompts overlap with or are drawn from the same generation pipeline as the training prompts, then the high Utility and CAH scores in Tables 1, 3, and 4, as well as the claimed 'up to 20%' reduction in problematic outputs, can be explained by the model memorizing preferred responses rather than by learning fair-use-aligned behavior. This is more fundamental than the unvalidated R classifier: replacing R with a fully specified, calibrated classifier would not fix a result measured on training data. The two issues compound, since R is never identified, but a clean evaluation split is a precondition for any quantitative claim about generalization.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FUA-LLM, a fair-use-aligned generation framework for LLMs. The authors construct FairUseDB, a 9,000-prompt preference dataset across nine copyright-infringement categories with lawyer-validated preferred responses, fine-tune Qwen-2.5-7B (and apply the same pipeline to LLaMA-3.1-8B and Gemma-3-9B) via SFT on fair-use court cases followed by DPO, and deploy the resulting model behind a lightweight agentic router. They introduce two evaluation metrics, Weighted Penalty Utility and Compliance-Aware Harmonic Mean (CAH), and report experiments in which FUA-LLM outperforms MemFree and SHIELD on these metrics while retaining low verbatim overlap. A human expert evaluation is also reported.","tokens_in":29187,"tokens_out":4685,"duration_ms":43356,"significance":"If the central claims held, the paper would make a useful contribution: it attacks an important problem (copyright-safe generation without utility-degrading refusal), provides a concrete alignment recipe built on expert-annotated preference data, and proposes metrics that are more sensitive than surface-level overlap. The collaboration with IP lawyers and the breadth of the nine infringement categories are genuine strengths. However, the evaluation as presented is not yet sufficient to support the headline claims. The two quantitative pillars—the reported Utility/CAH gains and the expert-preference result—rest on an unspecified risk classifier, an unverified train/test split, and a human study without baseline comparisons. The paper is therefore more a promising research proposal than a validated system at this stage.","major_comments":[{"comment":"The paper never establishes that the evaluation set is disjoint from the training set. Section 3.1 describes FairUseDB as 9,000 prompts generated and verified by legal experts, with SFT and DPO performed on exactly these data. Section 5 then evaluates on 'a diverse test set comprising prompts from the same nine infringement categories' without stating the test-set size, the split ratio, or any deduplication procedure. If the test prompts are drawn from the same generation pipeline or overlap with the 9,000 FairUseDB prompts, the high Utility/CAH values and the 'up to 20%' reduction claim could reflect memorization of preferred responses rather than learned fair-use alignment. The authors must report an explicit held-out split, verify no near-duplicate overlap (e.g., via embedding similarity or exact-match checks), and ideally evaluate on categories or prompt families not seen in Dpref.","section":"§3.1, §5, Tables 1, 3, 4"},{"comment":"Both proposed metrics are defined in terms of a Risk-of-Infringement score R described only as 'a probability from a classifier indicating that text infringes.' The classifier is never identified (model name, training data, or accuracy), and the Helpfulness score H is left as 'human ratings or an LLM-as-a-Judge' without specifying which was used for the reported numbers. The parameters α, β are said to be 'chosen by stakeholders,' but the values used in Tables 1, 3, and 4 are not given. Every quantitative compliance result in the paper is therefore unreproducible: without the classifier and the parameter settings, the reported Utility and CAH differences could change arbitrarily. The authors must provide a complete specification of R (including calibration and validation), report the chosen α and β, and ideally show sensitivity of the main conclusions to these choices.","section":"§4.1, Tables 1, 3, 4"},{"comment":"The human evaluation as presented does not support the claim that 'FUA-LLM's outputs are substantially preferred' over state-of-the-art methods. Table 2 lists only FUA-LLM's mean/percentile ratings per infringement category; there are no ratings for the base model, MemFree, or SHIELD, no pairwise preference counts, and no statistical significance tests. The surrounding text says 'each model's outputs were rated on a 1–10 scale,' which conflicts with the table caption. A blinded comparison across all competing methods is required before the preference claim can be made.","section":"§5, Table 2"},{"comment":"The claimed legal grounding is undermined by demonstrable legal errors. In Figure 7, FUA-LLM refuses to provide the first stanza of Percy Bysshe Shelley's 'Ode to the West Wind' because the poem is 'highly creative and ... strongly protected under copyright.' Shelley's poem is in the public domain (the author died in 1822), so the model's refusal is legally wrong. Relatedly, §1 refers to 'Section 107 of the United States Constitution,' but Section 107 is a provision of Title 17 of the U.S. Code, not the Constitution. These errors indicate that the SFT grounding on court cases does not reliably transfer to basic copyright-status determination, and they undercut the paper's central claim that FUA-LLM generates 'legally sound' outputs. The authors should correct the legal citation and investigate why the model misclassifies public-domain material, ideally adding public-domain prompts to the dataset and evaluation.","section":"Appendix Figures 7, 30; §1"},{"comment":"The preference construction contains a potential self-distillation loop: the 'chosen' responses in FairUseDB are generated by the SFT model fine-tuned on the fair-use court-case data, and the DPO phase then optimizes the model to prefer those self-generated outputs over the instruct-model 'rejected' outputs. The authors do not discuss whether this introduces bias (e.g., the DPO target reflects the SFT model's own stylistic and reasoning habits rather than a robust legal preference). While the lawyers' manual labeling provides some external validation, the paper should address this issue and, where possible, compare DPO learned from a different choice-generator (e.g., a strong general model or purely expert-authored responses) to show the effect is not an artifact of self-preference.","section":"§3.1–§3.2"}],"minor_comments":[{"comment":"The Utility value reported for Qwen-2.5-7B + SHIELD (5.26 ± 0.07) is implausible relative to all other Utility values in the same table (which lie between 0.02 and 0.5) and relative to the range implied by the definition Utility = αH − βR. This appears to be a typographical error and should be corrected.","section":"Table 1"},{"comment":"The text in §5 states that 'Table 2 breaks down CAH scores across the nine legally defined prompt categories,' but Table 2 is the expert human evaluation table; the CAH category breakdown is actually Table 4. The in-text reference should be updated.","section":"§5, Table 2 vs. Table 4"},{"comment":"The header 'FUA-LLM (FUA-LLM)' appears in the Llama-3.1-8B and Gemma-3-9B blocks, which is a typo for 'FUA-LLM'.","section":"Table 3"},{"comment":"The DPO loss displayed in Section 2 appears malformed: the expression 'log eβ log π(...)' does not correspond to the standard DPO objective. Please replace it with the correct formula and use a distinct symbol for the DPO temperature (currently β is also used in the CAH metric, which is confusing).","section":"§2, DPO objective"},{"comment":"The paper says 'we choose the harmonic mean structure for CAH' but does not state how the balancing parameter β₂ is set for the reported results; please provide the exact value (or the protocol for choosing it) and clarify whether the same β is used across all tables.","section":"§4.1, proposed metrics"},{"comment":"The conclusion claims 'we cut impermissible outputs by up to 20 percent,' but no table directly presents this reduction as a fraction of impermissible outputs; the supporting result should be made explicit and tied to a specific table and metric.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The evaluation gap is severe enough that the current version would not be publishable as is. However, the problems (held-out split, classifier specification, human-study baselines) are fixable within the scope of the manuscript, so I recommend major revision rather than rejection. I would also ask the editor to stress to the authors that the legal grounding claims must be verified by a legally qualified reviewer; the Shelley public-domain error is a red flag for the reliability of the entire 'legally grounded' narrative. The paper would benefit from releasing FairUseDB and the evaluation code to enable reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a look, but not for the headline numbers. The real contribution is FairUseDB, a 9,000-pair preference dataset built with IP lawyers across nine infringement categories, and the idea of using DPO to teach fair-use-aligned generation instead of relying on refusal filters. That's a legitimate new artifact and a sensible direction.\n\nThe soft spots are in the evaluation, and they are load-bearing. The proposed Utility and CAH metrics depend on a Risk-of-Infringement score R from a classifier that is never identified in the paper. No training data, no accuracy, no calibration. So every number in Tables 1, 3, and 4 inherits whatever that classifier does, and the 'up to 20% reduction' claim is therefore unanchored. More fundamental, the paper never states whether the test set used in Section 5 is disjoint from the 9,000 prompts used for DPO. The test prompts come from the same nine categories and the same pipeline, so the gains could be memorization rather than generalization. That's a basic precondition for any quantitative claim, and it's missing.\n\nThere's also a legal slip in the appendix: FUA-LLM refuses to supply 'Ode to the West Wind' by Shelley, which is public domain. That undercuts the claim that the model has internalized fair-use doctrine. And the human evaluation in Table 2 only rates FUA-LLM outputs; there are no baseline comparisons, so the 'expert confirmation' doesn't tell us much about relative performance.\n\nWhat the paper does well: it engages the prior literature fairly, the self-distillation design (SFT on court cases, then DPO on self-generated preferences) is reasonable, and the dataset, if released, would be useful to the community. The concerns above are fixable: specify the classifier, report a clean held-out split, add baseline comparisons to the human study, and correct the legal errors.\n\nWho is this for? Researchers working on copyright alignment and LLM safety. As written, the central claims are not supported. I'd send it to peer review only with the expectation of major revision, because the dataset and direction merit a serious look. But I would not cite the current version, and I would tell anyone asking that the numbers are not trustworthy until the evaluation is cleaned up.","headline":"Useful dataset, sensible idea, but the evaluation's foundation—unnamed classifier and unspecified test set—means the headline results aren't yet credible.","tokens_in":29746,"tokens_out":4014,"would_cite":false,"duration_ms":27365,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning an open LLM with lawyer-validated preference data on nine copyright-sensitive prompt types teaches it to answer with fair-use-compliant alternatives instead of refusing, cutting problematic outputs by up to 20 percent.","keywords":["fair use","copyright infringement","Direct Preference Optimization","LLM alignment","FairUseDB","copyright compliance evaluation","transformative generation"],"falsifier":"Run FUA-LLM and SHIELD on the same nine-category test set but score infringement with a publicly available, calibrated copyright detector or with independent lawyer ratings instead of the paper's unnamed R classifier; if FUA-LLM's CAH no longer exceeds SHIELD's, the central claim is false.","tokens_in":28726,"feed_emoji":"⚖️","tokens_out":7577,"duration_ms":58195,"temperature":0.7,"pith_summary":"The paper aims to show that copyright-safe LLM behavior does not require blanket refusal: an open model can be fine-tuned to respond to prompts that touch on copyrighted works with transformative, legally defensible content such as paraphrase, summary, or public-domain pointers. To that end, the authors built FairUseDB, a dataset of 9,000 expert-validated preference triples spanning nine infringement scenarios (verbatim excerpts, fan fiction, unauthorized translation, style imitation, derivative endings, and others), grounded in U.S. fair-use doctrine. Using Direct Preference Optimization on a model first trained on 252 fair-use court proceedings, they report that FUA-LLM reduces problematic outputs by up to 20 percent compared with refusal-based baselines while scoring higher on two new metrics, Utility and Compliance-Aware Harmonic Mean, that balance helpfulness against infringement risk. A blinded review by ten copyright lawyers rated FUA-LLM's outputs around 8 on a 10-point scale for being both compliant and helpful.","feed_headline":"LLM trained on fair use cuts infringing outputs by 20%","feed_subtitle":"Lawyer-validated preferences teach open models to offer transformative, useful alternatives instead of blunt refusals.","key_machinery":"The central mechanism is FairUseDB, a dataset of 9,000 expert-validated preference triples (18,000 responses) across nine infringement scenarios, built on an SFT model trained on 252 US fair-use court proceedings, and applied via Direct Preference Optimization (DPO), a fine-tuning method that directly optimizes the model toward preferred responses without a separate reward model. The paper also introduces the metrics Utility = αH − βR and CAH = (1+β²)HS/(β²S+H), where H is helpfulness and S = 1 − R is safety derived from an infringement-risk classifier; these metrics carry the evaluation of the compliance-utility tradeoff.","core_discovery":"The central discovery is that preference alignment with legally curated data changes what a model does when asked for protected content: instead of regurgitating a passage or issuing a generic refusal, FUA-LLM produces an original, transformative response that explains why the verbatim version would not qualify as fair use and then offers a lawful alternative. Across LLaMA-3.1-8B, Gemma-3-9B, and Qwen-2.5-7B, the DPO-fine-tuned models achieve CAH scores of 0.68–0.92 across all nine infringement categories, with the highest gains in derivative work, unauthorized translation, and fan fiction, where n-gram-based filters like MemFree and SHIELD degrade. The authors further claim that these gains persist under prefix and direct probing, and that the proposed Utility and CAH metrics expose the low helpfulness of refusal-based methods that simple LCS or refusal-rate metrics miss.","pith_inferences":["Because the paper evaluates only US fair use and only the nine curated categories, the compliance behavior is untested on other jurisdictions' copyright exceptions (e.g., fair dealing, quotation rights) and on prompt types outside the taxonomy.","The two-stage agentic deployment (topic analysis plus web search) is never ablated; a head-to-head single-model variant would reveal whether the web-search step is necessary for the reported gains or merely an added latency cost.","The central reported percentages depend on an unnamed infringement-risk classifier R; substituting a public, calibrated detector in the Utility and CAH formulas would provide an independent check of whether the 20% reduction is real."],"forward_implications":["FUA-LLM's pattern of steering prompts toward paraphrase, summary, and public-domain guidance can be reproduced across open LLaMA, Gemma, and Qwen models, suggesting the alignment recipe is model-agnostic.","Using the nine-category FairUseDB test set and the proposed Utility and CAH metrics, comparisons of future copyright-safe systems can be reported on a single legally grounded scale instead of refusal rate or string overlap.","If the reported CAH advantage holds, providers can deploy open models in sensitive products without a blanket refusal policy, preserving much of the models' helpfulness for ordinary users.","The court-case SFT followed by preference DPO offers a concrete template for injecting legal doctrine into generation policy, which could be repeated for other jurisdictions or IP types."],"supporting_citations":[{"why":"Supplies Direct Preference Optimization, the training objective that carries FUA-LLM's preference alignment.","marker":"Rafailov et al., 2024"},{"why":"SHIELD is the main refusal-based baseline; the paper's claims of superior compliance and utility are measured against it.","marker":"Liu et al., 2024a"},{"why":"MemFree is the verbatim-memorization-prevention baseline whose n-gram filtering FUA-LLM is compared with.","marker":"Ippolito et al., 2023"},{"why":"Provides the four fair-use factors (Section 107) that the expert-validated preference labels are grounded in.","marker":"United States Copyright Office, 2024"}],"fun_headline_variants":["Fair-use aligned LLM cuts copyright violations by 20%","LLM with legal preferences reduces infringing outputs 20%","DPO on fair use data: 20% fewer copyright breaches","New alignment method cuts LLM copyright infringement 20%","Fair use training: LLMs offer lawful alternatives, cut violations 20%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central numbers depend on a 'Risk-of-Infringement' score produced by a classifier that the paper never names, describes, or validates; if that classifier is miscalibrated, the reported 20% reduction in problematic outputs and the CAH advantage could disappear.","fun_headline_variants_meta":{"raw":{"variants":["Fair-use aligned LLM cuts copyright violations by 20%","LLM with legal preferences reduces infringing outputs 20%","DPO on fair use data: 20% fewer copyright breaches","New alignment method cuts LLM copyright infringement 20%","Fair use training: LLMs offer lawful alternatives, cut violations 20%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1470,"prompt_tokens":955,"completion_tokens":515,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":426}},"tokens_in":571,"tokens_out":515,"duration_ms":10984,"temperature":1.0,"reasoning_tokens":426,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:19:06.700675+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FUA-LLM and SHIELD on the same nine-category test set but score infringement with a publicly available, calibrated copyright detector or with independent lawyer ratings instead of the paper's unnamed R classifier; if FUA-LLM's CAH no longer exceeds SHIELD's, the central claim is false.","supporting_citations":[],"review_version":1}