{"paper":{"title":"The Frequency Confound in Language-Model Surprisal and Metaphor Novelty","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Word frequency beats surprisal at predicting metaphor novelty","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Omar Momen, Sina Zarrie{\\ss}","submitted_at":"2026-05-07T16:20:37Z","abstract_excerpt":"Language-model (LM) surprisal is widely used as a proxy for contextual predictability and has been reported to correlate with metaphor novelty judgments. However, surprisal is tightly intertwined with lexical frequency. We explore this interaction on metaphor novelty ratings using two different word frequency measures. We analyse surprisal estimates from eight Pythia model sizes and 154 training checkpoints. Across settings, word frequency is a stronger predictor of metaphor novelty than surprisal. Across training stages, the surprisal--novelty association peaks at an early stage and then fall"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Across settings, word frequency is a stronger predictor of metaphor novelty than surprisal. Across training stages, the surprisal--novelty association peaks at an early stage and then falls again, mirroring a similarly timed increase in the surprisal--frequency association.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That the collected metaphor novelty ratings reflect genuine human judgments of novelty rather than being influenced by frequency biases in the chosen stimuli or rater pool.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Word frequency is a stronger predictor of metaphor novelty than LM surprisal, with the surprisal-novelty association peaking early in training before declining.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Word frequency beats surprisal at predicting metaphor novelty","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"3084aca524dd82844da8e49073c1cec78a4f7e639d2d90664943c4804091a1af"},"source":{"id":"2605.06506","kind":"arxiv","version":2},"verdict":{"id":"29124f55-2244-449b-9356-642e9adf08e1","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-08T10:01:10.713876Z","strongest_claim":"Across settings, word frequency is a stronger predictor of metaphor novelty than surprisal. Across training stages, the surprisal--novelty association peaks at an early stage and then falls again, mirroring a similarly timed increase in the surprisal--frequency association.","one_line_summary":"Word frequency is a stronger predictor of metaphor novelty than LM surprisal, with the surprisal-novelty association peaking early in training before declining.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That the collected metaphor novelty ratings reflect genuine human judgments of novelty rather than being influenced by frequency biases in the chosen stimuli or rater pool.","pith_extraction_headline":"Word frequency beats surprisal at predicting metaphor novelty"},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2605.06506/integrity.json","findings":[],"available":true,"detectors_run":[{"name":"doi_title_agreement","ran_at":"2026-05-19T18:01:19.713467Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"doi_compliance","ran_at":"2026-05-19T12:36:33.096723Z","status":"completed","version":"1.0.0","findings_count":0}],"snapshot_sha256":"ab632927009a077881146d5bada87c207f3929efa7df94f6a1405dffce394194"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}