{"id":"d935e03e-8928-45e0-b82e-8d66f014c2bf","arxiv_id":"2606.04867","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"AICompanionBench is a new public dataset of annotated AI companion conversations used to benchmark LLMs as judges, revealing strong performance on explicit harms but limitations on nuanced risks like manipulation.","lead":"This paper creates AICompanionBench, a dataset of 2,123 real Replika conversations labeled for nine safety risks, and tests 20 LLMs as judges for spotting unsafe AI companion chats. A smart generalist might read it to see how well current AI can monitor growing companion platforms for harm.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Ground-truth labels from human-AI collaboration on Reddit-sourced Replika chats lack reported validation or agreement metrics.","rationale":"The reader's weakest_assumption directly identifies the same load-bearing point. Because the full text is now stated to be available, the absence of any validation statistics in the abstract-level description remains the decisive gap; confirming or refuting it requires only the concrete re-annotation check above. No other internal inconsistency is visible from the provided material.","tokens_in":1740,"tokens_out":298,"duration_ms":21830,"concrete_test":"Select a stratified random sample of 200 conversations; have two independent human annotators (no LLM assistance) re-label them using the same nine-category scheme; compute Cohen's kappa per category. If any category kappa < 0.65, the ground-truth reliability is insufficient to support the headline performance claims.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The benchmark's utility for evaluating LLMs-as-judges rests entirely on the 2,123 conversations being correctly labeled across the nine categories. The abstract states these labels were produced via human-AI collaboration, yet no inter-annotator agreement, expert validation subset, disagreement-resolution protocol, or bias audit (Reddit selection + AI assistance) is referenced. Without these, it is impossible to know whether the reported LLM struggles on \"manipulation\" or false positives on benign cases reflect model limitations or label noise.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces AICompanionBench, the first public benchmark of 2,123 real-world Replika conversations sourced from Reddit and annotated via human-AI collaboration for nine fine-grained safety risk categories (sexual behavior, antisocial behavior, physical aggression, verbal aggression, substance abuse, self-harm and suicide, control, manipulation, and no-harm). It evaluates 20 open- and closed-source LLMs as judges on this dataset, reporting substantial variation in performance with stronger models achieving high overall accuracy yet struggling on nuanced categories such as manipulation and producing false positives on benign conversations. The work positions the benchmark as a resource for AI companion safety research and makes the dataset publicly available.","tokens_in":1823,"tokens_out":493,"duration_ms":21667,"significance":"If the annotations are shown to be reliable, the release of a publicly available, fine-grained safety benchmark for human-AI companion interactions would be a useful contribution to the growing literature on LLM-based safety monitoring. The evaluation across 20 models provides concrete comparative data on current capabilities for explicit versus implicit harm detection. The public dataset release is a clear strength that enables reproducibility and follow-on work.","major_comments":[{"comment":"Annotation methodology section: the paper states that the 2,123 conversations were labeled through human-AI collaboration across the nine categories but provides no inter-annotator agreement metrics, expert validation subset, disagreement-resolution protocol, or bias audit for Reddit sourcing and AI assistance. This is load-bearing because the benchmark's utility for evaluating LLMs-as-judges rests entirely on the quality of these ground-truth labels.","section":"Annotation methodology section"},{"comment":"Results section (model performance on manipulation and benign cases): without reported validation of the labels, the claims that LLMs 'struggle with nuanced categories such as manipulation' and incorrectly flag benign conversations cannot be distinguished from possible label noise.","section":"Results section"}],"minor_comments":[{"comment":"The GitHub repository path contains 'anonymousresearcher2026'; update to a stable, non-anonymous link for publication.","section":null},{"comment":"Add the exact LLM-as-judge prompt template and temperature settings used in the evaluations to support reproducibility.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"Thank you for the constructive feedback and the recommendation for major revision. We agree that the annotation methodology requires substantially more detail to establish the reliability of the ground-truth labels, which is foundational to the benchmark. We address each major comment below and will revise the manuscript to incorporate additional documentation and qualifications.","responses":[{"response":"We acknowledge that the current manuscript provides insufficient detail on the annotation process. The description is limited to noting human-AI collaboration without metrics or protocols. In the revised version, we will expand this section to fully describe the annotation workflow, include any inter-annotator agreement metrics that can be computed from the process used, document the disagreement-resolution approach, report on an expert validation subset, and add a bias audit discussion for both the Reddit data collection and the AI assistance component. These additions will directly address the load-bearing concern about label quality.","revision_made":"yes","referee_comment":"[Annotation methodology section] Annotation methodology section: the paper states that the 2,123 conversations were labeled through human-AI collaboration across the nine categories but provides no inter-annotator agreement metrics, expert validation subset, disagreement-resolution protocol, or bias audit for Reddit sourcing and AI assistance. This is load-bearing because the benchmark's utility for evaluating LLMs-as-judges rests entirely on the quality of these ground-truth labels."},{"response":"The referee is correct that the absence of label validation metrics makes it difficult to separate model limitations from potential annotation issues, particularly for nuanced categories. We will revise the results and discussion sections to include an explicit limitations paragraph on label quality, qualify the claims about struggles with manipulation and false positives on benign cases by referencing possible noise, and tie these qualifications to the expanded methodology details. This will prevent overinterpretation of the findings.","revision_made":"yes","referee_comment":"[Results section] Results section (model performance on manipulation and benign cases): without reported validation of the labels, the claims that LLMs 'struggle with nuanced categories such as manipulation' and incorrectly flag benign conversations cannot be distinguished from possible label noise."}],"tokens_in":1423,"tokens_out":452,"duration_ms":27553,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"This paper puts out a new dataset of over 2,000 Replika conversations labeled for nine safety risks and tests how well 20 LLMs can judge them. The dataset is public, which is the useful part.\n\nIt does a decent job collecting real conversations from Reddit and breaking down risks into categories like sexual behavior, manipulation, and self-harm. Evaluating multiple models shows that even strong ones miss nuanced cases and overflag safe chats, which matches what people worry about with AI companions.\n\nThe weak point is the ground truth. The labels came from human-AI collaboration, yet there are no reports on agreement rates, expert checks, or how they resolved disagreements. The stress-test note is right on this: without those checks, we cannot tell if the models' problems with manipulation come from the models or from label errors. Reddit data plus AI assistance adds possible biases that are not addressed.\n\nThe work is for people building or studying safety tools for chatbots aimed at users. It gives them a starting point to test their own judges, but anyone using the benchmark will have to do their own validation first.\n\nI would send this to peer review. The topic matters and the data release is concrete, but referees should require the annotation validation details before it can be taken as a solid benchmark.","headline":"The paper releases a new public dataset of Replika chats labeled for nine safety risks and tests 20 LLMs on it, but the annotation process has no reported validation or agreement metrics.","tokens_in":2265,"tokens_out":349,"would_cite":false,"duration_ms":31345,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"AICompanionBench introduces the first public dataset of annotated human-AI companion conversations for testing LLMs as safety judges.","keywords":["AI companion safety","LLM as judge","benchmark dataset","safety risk categories","Replika conversations","manipulation detection","unsafe interaction detection","human-AI collaboration annotation"],"falsifier":"A new set of independent human annotators re-labeling a random subset of the conversations and finding systematic disagreement on manipulation or control labels would show the ground truth is unreliable.","tokens_in":2642,"feed_emoji":"🤖","tokens_out":750,"duration_ms":21537,"temperature":0.7,"pith_summary":"The paper creates a benchmark of 2,123 real Replika conversations labeled across nine safety categories such as manipulation, self-harm, and verbal aggression. It then runs twenty open and closed LLMs as judges on these labeled examples to measure how well they detect unsafe content. Results indicate that stronger models reach high overall accuracy on clear harms yet consistently underperform on subtle risks and over-flag safe exchanges. This matters because growing AI companion platforms need reliable automated checks, and a shared test set makes it possible to track progress on those checks. The work supplies the dataset publicly so others can run their own evaluations.","feed_headline":"Benchmark reveals LLMs miss subtle manipulation in AI companion chats","feed_subtitle":"Dataset of 2,123 real conversations shows top models catch explicit harms but struggle with nuanced risks and false positives.","key_machinery":"The AICompanionBench dataset of Reddit-sourced Replika conversations annotated with nine fine-grained safety risk categories, used as ground truth to score LLM judges.","core_discovery":"AICompanionBench is the first publicly available benchmark dataset of 2,123 real-world Replika conversations annotated through human-AI collaboration across nine safety risk categories: sexual behavior, antisocial behavior, physical aggression, verbal aggression, substance abuse, self-harm and suicide, control, manipulation, and no-harm. Evaluations of twenty state-of-the-art LLMs under an LLM-as-judge setup show substantial performance variation, with stronger models achieving high overall accuracy yet still struggling with nuanced categories such as manipulation and with benign conversations incorrectly flagged as harmful. The findings indicate that current LLMs detect explicit harmful con","pith_inferences":["The benchmark could be expanded to conversations from other companion platforms to test whether the observed LLM weaknesses are general.","Models fine-tuned on the dataset might close the gap on manipulation detection, which would be a direct test of the benchmark's usefulness.","If false-positive rates remain high, platforms may still need human review for borderline cases even after adopting LLM judges.","Similar fine-grained annotation methods could apply to safety evaluation in other conversational AI domains beyond companions."],"forward_implications":["Stronger LLMs can already identify explicit harmful content in AI companion conversations with high accuracy.","LLMs continue to miss implicit unsafe interactions such as manipulation even when overall accuracy looks good.","Some safe conversations are misclassified as harmful by current models, creating false positives.","A public benchmark makes it possible to compare and improve safety monitoring systems for AI companions in a consistent way.","Insights from the evaluations can direct work toward better detection of nuanced risks rather than only obvious ones."],"fun_headline_variants":["AICompanionBench tests LLMs on Replika safety risks","Dataset benchmarks 20 LLMs for AI companion harms","LLMs vary on detecting manipulation in chat benchmark","Nine categories in Replika conversation safety dataset","Benchmark evaluates LLM judges on 2123 companion chats"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The human-AI collaboration annotations of the 2,123 conversations supply accurate and unbiased ground truth labels for the nine safety risk categories.","fun_headline_variants_meta":{"raw":{"variants":["AICompanionBench tests LLMs on Replika safety risks","Dataset benchmarks 20 LLMs for AI companion harms","LLMs vary on detecting manipulation in chat benchmark","Nine categories in Replika conversation safety dataset","Benchmark evaluates LLM judges on 2123 companion chats"]},"model":"grok-4.3","cost_usd":0.003909,"raw_usage":{"total_tokens":2035,"prompt_tokens":728,"num_sources_used":0,"completion_tokens":74,"cost_in_usd_ticks":39087000,"prompt_tokens_details":{"text_tokens":728,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1233,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":728,"tokens_out":74,"duration_ms":9886,"temperature":1.0,"reasoning_tokens":1233,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T05:42:58.289489+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A new set of independent human annotators re-labeling a random subset of the conversations and finding systematic disagreement on manipulation or control labels would show the ground truth is unreliable.","supporting_citations":[],"review_version":1}