REVIEW 50 cited by
LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Studying how people interact with large language models (LLMs) in real-world scenarios is increasingly important due to their widespread use in various applications. In this paper, we introduce LMSYS-Chat-1M, a large-scale dataset containing one million real-world conversations with 25 state-of-the-art LLMs. This dataset is collected from 210K unique IP addresses in the wild on our Vicuna demo and Chatbot Arena website. We offer an overview of the dataset's content, including its curation process, basic statistics, and topic distribution, highlighting its diversity, originality, and scale. We demonstrate its versatility through four use cases: developing content moderation models that perform similarly to GPT-4, building a safety benchmark, training instruction-following models that perform similarly to Vicuna, and creating challenging benchmark questions. We believe that this dataset will serve as a valuable resource for understanding and advancing LLM capabilities. The dataset is publicly available at https://huggingface.co/datasets/lmsys/lmsys-chat-1m.
Forward citations
Cited by 50 Pith papers
-
Efficiency and Cost Alignment in Batched LLM Serving via Resource-Fair Scheduling
A resource-fair batching policy (ISJL) that keeps co-batched LLM requests within a token-progress window is proved 3/4-competitive in an offline model and empirically outperforms FCFS, SJF, and LJF on throughput and latency.
-
K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs
K12-KGraph is a textbook-derived knowledge graph that powers a new benchmark revealing LLMs' poor curriculum cognition and a small training corpus that outperforms general instruction data on educational tasks.
-
Understanding Refusal in Language Models with Sparse Autoencoders
Refusal in Gemma-2-2B and Llama-3.1-8B is mediated by a small set of SAE features, harm features causally activate refusal features, and adversarial jailbreaks suppress those refusal features.
-
The AI Gap: How Socioeconomic Status Affects Language Technology Interactions
Higher self-reported socioeconomic status is associated with shorter, more abstract, work-oriented chatbot prompts, while lower SES users show more concrete, polite, general-purpose interactions, patterns that may dee...
-
JailbreaksOverTime: Detecting Jailbreak Attacks Under Distribution Shift
Jailbreak detection models drift over time, and a weekly self-trained detector plus an unsupervised behavioral monitor can keep false negatives near 0.3 to 0.4 percent at a 0.1 to 1 percent false positive rate.
-
Marconi: Prefix Caching for the Era of Hybrid LLMs
A prefix cache for hybrid attention-SSM LLMs that selectively checkpoints SSM states and evicts by FLOP efficiency, improving token hit rates and time-to-first-token.
-
From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set
Using per-example in-context demonstrations built from historical same-source human MQM ratings makes an LLM judge dramatically better at fine-grained MT evaluation on WMT'23 and WMT'24.
-
Measuring Non-Adversarial Reproduction of Training Data in Large Language Models
In non-adversarial settings, popular LLMs reproduce 7-15% of characters from online sources on average, versus far less for humans; worst-case generations can match 100% of their content verbatim.
-
Different Feedback, Different Updates: Selective Self-Learning from User Interactions for Large Language Models
SLIFT decomposes user feedback into Fix, Spec, and Null parts, then trains a Generalist adapter for fixes and a Specialist adapter for optional refinements, improving LLMs on MemoryBench and WildFB.
-
Calling the Bluff: Detecting Ever-Shifting Harmful Chat Dialogue via Ordered Reasoning Chain Regularization
BRACE detects harmful chat dialogue by regularizing a classifier with an ordered reasoning chain of topic, indicator, severity, and type, reaching 0.934 macro F1 on the authors' 9,000-dialogue test set.
-
The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Data
Fairness collapse: gender-occupation bias worsens in the first iterations of recursive synthetic-data training, ahead of visible model-collapse indicators.
-
CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement
A chest X-ray VLM co-trained with classification and grounding heads, tuned with DAPO reinforcement learning, and augmented with deterministic measurement tools outperforms prior radiology VLMs on report generation, V...
-
RH-RAG: Trustworthy Long-Form Generation for Privacy-Constrained Settings
A multi-agent RAG framework that adds planning, bounded memory, and NLI-based revision to local 7-8B models, reported to improve faithfulness and coherence in long-form generation.
-
The One-Word Census: Answer-Choice Conformity Across 44 Language Models
Forty-four language models asked to name one thing per category converge on the same modal answers far more than people do, with newest flagships most conformist and persona-tuned models most divergent.
-
Economic Evaluations of Language Models
Using O*NET and 4.5M chatbot conversations plus synthetic prompts, EconEvals measures LM performance on U.S. work activities and predicts substantial time savings in 47% of occupations, with usage lagging.
-
Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders
Coverage of sparse-autoencoder-identified task features predicts post-training performance and can guide synthesis of small, high-impact datasets (2,000 vs. 300,000 samples).
-
After Talking with 1,000 Personas: Learning Preference-Aligned Proactive Assistants From Large-Scale Persona Interactions
A two-stage framework — category-structured fine-tuning on LLM-simulated personas plus on-device activation steering — improves proactive-assistant timing and perceived quality, though the biggest gains are measured w...
-
Auditing LLM Editorial Bias in News Media Exposure
Compared with Google News, GPT-4o-Mini, Claude-3.7-Sonnet, and Gemini-2.0-Flash surface fewer unique news outlets, distribute attention more unevenly, and lean ideologically in system-specific ways.
-
A global log for medical AI
MedLog defines a nine-field, syslog-style event log for clinical AI, intended to support real-world surveillance and auditing; the four-deployment validation claimed in the abstract is absent from the body.
-
Developer-LLM Conversations: An Empirical Study of Interactions and Generated Code Quality
Analyzing 82,845 real ChatGPT coding chats shows generated code frequently has language-specific issues, with some quality problems persisting or worsening over multiple turns.
-
VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routing
VoltanaLLM saves up to 36.3% GPU energy in LLM serving by per-iteration, phase-aware frequency scaling and state-space routing that preserve latency SLOs.
-
TweakLLM: A Routing Architecture for Dynamic Tailoring of Cached Responses
A small language model can rewrite cached large-model responses to fit similar new queries, preserving quality while cutting inference cost.
-
Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreaking
Across 2M+ in-the-wild LLM conversations, jailbreak attempts show no higher complexity than normal chats, and assistant toxicity has declined over time, suggesting bounded attack sophistication.
-
Arch-Router: Aligning LLM Routing with Human Preferences
Arch-Router, a 1.5B fine-tuned generative model, matches chat queries to user-defined domain-action policies and reports higher accuracy than several proprietary models on adapted routing benchmarks.
-
Evaluating the Sensitivity of LLMs to Prior Context
Prior conversational context, especially from a different knowledge domain, can sharply reduce LLM multiple-choice accuracy, and repeating the task near the query mitigates the drop.
-
EdgeWisePersona: A Dataset for On-Device User Profiling from Natural Language Interactions
EdgeWisePersona is a new synthetic dataset and benchmark for reconstructing structured smart-home user routines from multi-session dialogues, on which large LLMs clearly outperform small on-device models.
-
PROMPTEVALS: A Dataset of Assertions and Guardrails for Custom Production Large Language Model Pipelines
A new dataset of production-style prompts with matching assertion criteria, and a benchmark where fine-tuned 7-8B models beat GPT-4o at generating those criteria.
-
DeepThink: Aligning Language Models with Domain-Specific User Intents
DeepThink improves domain-specific QA by synthesizing conversation-based training data and refining answers with retrieval-augmented feedback, beating a GPT-4-turbo+RAG assistant by 7.92% on advertising-domain real us...
-
Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation
Cache-Craft reuses stored attention states for repeated RAG text chunks with selective recomputation, cutting redundant LLM prefill computation by 51% over prefix caching while retaining roughly 90% of answer quality.
-
Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons
A fine-tuned 14B LLM judge, trained with scenario-based prompts and controlled instruction generation, approaches GPT-4's human-agreement performance, and the paper documents why scaling distillation data can fail.
-
Why human-AI relationships need socioaffective alignment
The authors propose that AI alignment must account for the social and emotional relationships people form with personalized, agentic AI, and outline a 'socioaffective alignment' agenda.
-
IC-Cache: Efficient Large Language Model Serving via In-context Caching
IC-Cache reuses historical large-model responses as in-context examples so small models can handle a larger share of serving traffic without losing quality, improving throughput and latency.
-
Liquid: Language Models are Scalable and Unified Multi-modal Generators
Liquid extends existing LLMs with VQGAN image tokens and shows unified visual understanding and generation can scale, with the language-versus-image trade-off shrinking as model size grows.
-
APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Serving
A CPU-based simulator that searches hybrid data, pipeline, and tensor parallelism plans for LLM serving, claiming up to 3.37x faster plans than heuristics in simulation and 10.7% average speedup-prediction error.
-
BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching
BlendServe combines resource-aware batching with prefix sharing using a resource-aware prefix tree and dual scanner, achieving up to 1.44x throughput vs vLLM/SGLang in offline LLM inference.
-
GEM-Bench: A Benchmark for Ad-Injected Response Generation within Generative Engine Marketing
GEM-Bench offers the first benchmark for ad-injected LLM responses, and its experiments indicate generate-then-inject pipelines beat prompt-based ad injection on LLM-judged quality.
-
Reviving Your MNEME: Predicting The Side Effects of LLM Unlearning and Fine-Tuning via Sparse Model Diffing
Sparse model diffing on task-agnostic text can flag which academic, safety, or behavioral categories a fine-tuned or unlearned LLM has unintentionally changed.
-
Language Matters: How Do Multilingual Input and Reasoning Paths Affect Large Reasoning Models?
Large reasoning models default to English or Chinese as internal 'reasoning hubs', and forcing non-hub reasoning lowers math accuracy, especially for low-resource languages, while sometimes improving safety and cultur...
-
PromptShield: Deployable Detection for Prompt Injection Attacks
PromptShield reports a 65.3% true positive rate at 0.1% false positive rate for prompt injection detection, more than six times the best prior model, on its own out-of-distribution evaluation split.
-
Lightweight Safety Classification Using Pruned Language Models
Intermediate-layer features of small LLMs plus a penalized logistic regression classifier achieve high F1 scores on content safety and prompt injection classification with very few labeled examples, per the paper's ex...
-
Beyond Direct Identifiers: Probabilistic Privacy Risk Estimation for Privacy-Conscious LLM Query Delegation
Optimizing a privacy-preserving LLM delegation pipeline with an LLM-based k-anonymity objective improves quality for most models, but the objective itself contributed little variation and privacy gains are not robustl...
-
Improving Aviation Safety Analysis: Automated HFACS Classification Using Reinforcement Learning with Group Relative Policy Optimization
GRPO fine-tuning of Llama 3.1 8B improves multi-label HFACS classification of aviation narratives, reaching 18% exact match and 88% partial match on a 100-sample test set.
-
DialogueForge: LLM Simulation of Human-Chatbot Dialogue
DialogueForge generates synthetic human-chatbot dialogues by pitting an inquirer LLM against a responder LLM, and finds that fine-tuned small models can approach GPT-4o-level realism on LLM-judged metrics.
-
Toward Real-World Chinese Psychological Support Dialogues: CPsDD Dataset and a Co-Evolving Multi-Agent System
CPsDD is a 68K-dialogue Chinese psychological support dataset with strategy annotations, and CADSS is a multi-agent system reporting state-of-the-art results on Chinese and English emotional support tasks.
-
Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Models
A GRPO decoding framework that cuts memory via micro-batched KV-cache reuse and improves decoding-round efficiency with predicted-length scheduling, at the cost of serialization.
-
Is There a Case for Conversation Optimized Tokenizers in Large Language Models?
Retraining LLM tokenizers on chatbot conversation data reduces token counts by 5-10% on conversational text with minimal impact on general text.
-
Analysis of Indic Language Capabilities in LLMs
A desk-research review finds that LLM performance is strongest for Hindi, Bengali, Marathi, Telugu, and Tamil, and recommends prioritizing these five languages for safety benchmarks.
-
Adaptations of AI models for querying the LandMatrix database in natural language
On a roughly 60-question Land Matrix benchmark, Codestral-22B with an agentic LLM pipeline generates the most syntactically valid REST and GraphQL queries, though overall accuracy remains modest.
-
Multi-Bin Batching for Increasing LLM Inference Throughput
Grouping LLM requests into bins by predicted execution time before forming batches provably raises inference throughput, approaching the theoretical maximum as bin count grows.
-
LP Data Pipeline: Lightweight, Purpose-driven Data Pipeline for Large Language Models
This paper presents a CPU-only data curation pipeline for LLMs, but the central claim of high-quality output is not supported by any training or quality evaluation.
Discussion (0). Continue with ORCID to comment.