{"work":{"id":"4e7ac831-1344-4b1d-910b-7698f7d1c019","openalex_id":"https://openalex.org/W4403788929","doi":"10.48550/arxiv.2409.12822","arxiv_id":"2409.12822","raw_key":null,"title":"Language Models Learn to Mislead Humans via RLHF","authors":null,"authors_text":"Wen, J","year":2024,"venue":"cs.CL","abstract":"Language models (LMs) can produce errors that are hard to detect for humans, especially when the task is complex. RLHF, the most popular post-training method, may exacerbate this problem: to achieve higher rewards, LMs might get better at convincing humans that they are right even when they are wrong. We study this phenomenon under a standard RLHF pipeline, calling it \"U-SOPHISTRY\" since it is Unintended by model developers. Specifically, we ask time-constrained (e.g., 3-10 minutes) human subjects to evaluate the correctness of model outputs and calculate humans' accuracy against gold labels. On a question-answering task (QuALITY) and programming task (APPS), RLHF makes LMs better at convincing our subjects but not at completing the task correctly. RLHF also makes the model harder to evaluate: our subjects' false positive rate increases by 24.1% on QuALITY and 18.3% on APPS. Finally, we show that probing, a state-of-the-art approach for detecting Intended Sophistry (e.g. backdoored LMs), does not generalize to U-SOPHISTRY. Our results highlight an important failure mode of RLHF and call for more research in assisting humans to align them.","external_url":"https://arxiv.org/abs/2409.12822","cited_by_count":4,"metadata_source":"pith","metadata_fetched_at":"2026-08-05T02:28:24.338817+00:00","pith_arxiv_id":"2409.12822","created_at":"2026-05-10T13:10:26.232127+00:00","updated_at":"2026-08-05T02:28:24.338817+00:00","title_quality_ok":true,"display_title":"Language models learn to mislead humans via rlhf","render_title":"Language models learn to mislead humans via rlhf"},"hub":{"state":{"work_id":"4e7ac831-1344-4b1d-910b-7698f7d1c019","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":14,"external_cited_by_count":null,"distinct_field_count":4,"first_pith_cited_at":"2025-03-14T23:50:34+00:00","last_pith_cited_at":"2026-07-07T06:59:30+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-07-10T16:54:28.581281+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":2}],"polarity_counts":[{"context_polarity":"background","n":1},{"context_polarity":"support","n":1}],"runs":{},"summary":{},"graph":{},"authors":[]}}