Introduces BonaFide benchmark of 3,066 ground-truth labeled CoTs showing most faithfulness metrics perform near chance with biases and poor scaling to longer chains.
super hub Mixed citations
A coefficient of agreement for nominal scales.Educational and Psychological Measurement, 20(1):37–46
Mixed citation behavior. Most common role is background (33%).
34
Pith papers citing it
30.9k
external citations · Crossref
Background
33% of classified citations
hub tools
citation-role summary
background 3
method 2
dataset 1
citation-polarity summary
authors
co-cited works
representative citing papers
Causal state binding is introduced as a framework that predicts action control in language agents, validated across large benchmarks and SWE-bench Lite where adding the measure raised issue-to-file hit@3 AUC from 0.873 to 0.935.
ProactBench measures LLM conversational proactivity in three phases using 198 multi-agent dialogues and finds recovery behavior hard to predict from existing benchmarks.
HackerSignal aggregates 7.45M documents from hacker communities, exploit databases, vulnerability reports, and fixes into a public benchmark for temporal OOD CVE linkage and exploit classification.
VulKey introduces hierarchical expert knowledge abstractions to guide LLMs in vulnerability repair, reporting 31.5% accuracy on PrimeVul (7.6% above best baseline) and strong results on Vul4J.
EO-Gym supplies an executable multimodal environment and 9k-trajectory benchmark that turns Earth Observation into a tool-using, multi-step reasoning task, revealing that current VLMs struggle on temporal and cross-sensor workflows while fine-tuning lifts Pass@3 from 0.49 to 0.74.
Computer-use agents show attack success rates above 90% on benign instructions that produce harm via context or execution, with safety-aligned Claude 4.5 Sonnet at 73% ASR rising to 92.7% in multi-agent deployments.
LLMs produce stable cognitive distortion labels that improve downstream model performance, paired with a kappa-based framework for dataset-agnostic evaluation in subjective NLP tasks.
On 2,520 programming tasks, matched Qwen general and coder models reliably raise Bloom cognitive demand but fail to lower it, so execution skill does not imply educational control.
Gemini 3.0 Pro with rubric prompts reached ICC 0.888 agreement with human graders on low-complexity Linux/bash responses but lower agreement at higher taxonomy levels across 1200 student answers from three expert raters.
A multi-reference audit framework for LLM translations of the Pali Canon uses embedding drift from a human reference centroid to triage candidates for LLM-judge adjudication, showing drift correlates with major error rates and model-specific differences in the high-drift tail.
Analysis of SATD in Dockerfiles shows 27% of admissions and 40% of repayments are coupled to non-Dockerfile artifacts, with coupled events repaid faster overall and external dependencies as a key trigger.
Each tested LLM shows its own characteristic unreliability when engaging in repair during extended math-question dialogues.
Ground-truth evaluation of eight debloaters shows dynamic tools remove up to 94% of code that should be kept and static tools show high false retention from over-approximation.
Synthetically formalizing information needs into topics with descriptions and narratives improves LLM relevance assessor agreement with humans and reduces over-labeling of relevant documents on TREC Deep Learning and Robust04.
Large-scale review mining of 1M+ comments from 171 Gen-AI apps using an LLM framework reveals top topics plus three opportunities and three challenges for developers.
Introduces a new English dataset from r/AskParents and r/needadvice annotated for advice sentences plus preliminary models showing pre-trained LMs outperform rule-based systems but the task remains challenging.
Content analysis of #GymTok videos finds positive associations between muscularity, perceived harm, and engagement metrics, implying TikTok algorithms may amplify muscular ideals and risky behaviors.
On binary verdicts, Pearson, Spearman, Kendall's tau-b, phi, and the Matthews correlation are a single statistic, so most multi-metric agreement reports repeat one number under different names.
Develops a section-aware hallucination detection method for LLM bug report summaries using synthetic injection on the BugsRepo dataset from Mozilla projects, reporting up to 0.89 Macro-F1 at report level.
Systematic study of inter-agent communication in LLM multi-agent systems shows reasoning and verification are critical for performance, with a new augmentation technique recovering 86.2% of failures.
LLMs achieve near-human agreement (k=0.794 vs human-human k=0.872) on annotating Mandarin narrative macrostructure with the MAIN framework, reducing time by 65 percent but showing lower reliability on young adult narratives with greater lexical variation.
LLMs can detect usability content in user reviews with F-scores comparable to humans, though performance depends strongly on prompt design.
Embodied LLM agents exhibit emergent collaborative behaviors indicating mental models of partners in a color-matching game, detected via LLM judges and supported by positive user feedback.
citing papers explorer
-
Faithfulness Metrics Don't Measure Faithfulness: A Meta-Evaluation with Ground Truth
Introduces BonaFide benchmark of 3,066 ground-truth labeled CoTs showing most faithfulness metrics perform near chance with biases and poor scaling to longer chains.
-
Causal state binding predicts action control in language agents
Causal state binding is introduced as a framework that predicts action control in language agents, validated across large benchmarks and SWE-bench Lite where adding the measure raised issue-to-file hit@3 AUC from 0.873 to 0.935.
-
ProactBench: Beyond What The User Asked For
ProactBench measures LLM conversational proactivity in three phases using 198 multi-agent dialogues and finds recovery behavior hard to predict from existing benchmarks.
-
HackerSignal: A Large-Scale Multi-Source Dataset Linking Hacker Community Discourse to the CVE Vulnerability Lifecycle
HackerSignal aggregates 7.45M documents from hacker communities, exploit databases, vulnerability reports, and fixes into a public benchmark for temporal OOD CVE linkage and exploit classification.
-
VulKey: Automated Vulnerability Repair Guided by Domain-Specific Repair Patterns
VulKey introduces hierarchical expert knowledge abstractions to guide LLMs in vulnerability repair, reporting 31.5% accuracy on PrimeVul (7.6% above best baseline) and strong results on Vul4J.
-
EO-Gym: A Multimodal, Interactive Environment for Earth Observation Agents
EO-Gym supplies an executable multimodal environment and 9k-trajectory benchmark that turns Earth Observation into a tool-using, multi-step reasoning task, revealing that current VLMs struggle on temporal and cross-sensor workflows while fine-tuning lifts Pass@3 from 0.49 to 0.74.
-
The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agents
Computer-use agents show attack success rates above 90% on benign instructions that produce harm via context or execution, with safety-aligned Claude 4.5 Sonnet at 73% ASR rising to 92.7% in multi-agent deployments.
-
Towards Consistent Detection of Cognitive Distortions: LLM-Based Annotation and Dataset-Agnostic Evaluation
LLMs produce stable cognitive distortion labels that improve downstream model performance, paired with a kappa-based framework for dataset-agnostic evaluation in subjective NLP tasks.
-
From Execution to Education: A Bloom-Aligned Framework for Measuring Educational Control in LLMs
On 2,520 programming tasks, matched Qwen general and coder models reliably raise Bloom cognitive demand but fail to lower it, so execution skill does not imply educational control.
-
Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach
Gemini 3.0 Pro with rubric prompts reached ICC 0.888 agreement with human graders on low-complexity Linux/bash responses but lower agreement at higher taxonomy levels across 1200 student answers from three expert raters.
-
From Outliers to Errors: Auditing Pali-to-English LLM Translations with Multi-Reference Adjudication
A multi-reference audit framework for LLM translations of the Pali Canon uses embedding drift from a human reference centroid to triage candidates for LLM-judge adjudication, showing drift correlates with major error rates and model-specific differences in the high-drift tail.
-
Beyond the Tip of the Iceberg: Understanding SATD in Dockerfiles through the Lens of Co-evolution
Analysis of SATD in Dockerfiles shows 27% of admissions and 40% of repayments are coupled to non-Dockerfile artifacts, with coupled events repaid faster overall and external dependencies as a key trigger.
-
Talking to a Know-It-All GPT or a Second-Guesser Claude? How Repair reveals unreliable Multi-Turn Behavior in LLMs
Each tested LLM shows its own characteristic unreliability when engaging in repair during extended math-question dialogues.
-
Revisiting Code Debloating with Ground Truth-based Evaluation
Ground-truth evaluation of eight debloaters shows dynamic tools remove up to 94% of code that should be kept and static tools show high false retention from over-approximation.
-
Formalized Information Needs Improve Large-Language-Model Relevance Judgments
Synthetically formalizing information needs into topics with descriptions and narratives improves LLM relevance assessor agreement with humans and reduces over-labeling of relevant documents on TREC Deep Learning and Robust04.
-
Understanding the Challenges and Opportunities of Generative AI Apps: An Empirical Study
Large-scale review mining of 1M+ comments from 171 Gen-AI apps using an LLM framework reveals top topics plus three opportunities and three challenges for developers.
-
Help! Need Advice on Identifying Advice
Introduces a new English dataset from r/AskParents and r/needadvice annotated for advice sentences plus preliminary models showing pre-trained LMs outperform rule-based systems but the task remains challenging.
-
The Body as Status: Muscularity, Engagement, and Body Image Risk on #GymTok
Content analysis of #GymTok videos finds positive associations between muscularity, perceived harm, and engagement metrics, implying TikTok algorithms may amplify muscular ideals and risky behaviors.
-
Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why
On binary verdicts, Pearson, Spearman, Kendall's tau-b, phi, and the Matthews correlation are a single statistic, so most multi-metric agreement reports repeat one number under different names.
-
Empirical Analysis and Detection of Hallucinations in LLM-Generated Bug Report Summaries
Develops a section-aware hallucination detection method for LLM bug report summaries using synthetic injection on the BugsRepo dataset from Mozilla projects, reporting up to 0.89 Macro-F1 at report level.
-
What Do Agents Communicate? Characterizing Information Exchange in Multi-Agent Systems
Systematic study of inter-agent communication in LLM multi-agent systems shows reasoning and verification are critical for performance, with a new augmentation technique recovering 86.2% of failures.
-
LLMs for automatic annotation of Mandarin narrative transcripts
LLMs achieve near-human agreement (k=0.794 vs human-human k=0.872) on annotating Mandarin narrative macrostructure with the MAIN framework, reducing time by 65 percent but showing lower reliability on young adult narratives with greater lexical variation.
-
User Reviews as a Source for Usability Requirements: A Precursor Study on Using Large Language Models
LLMs can detect usability content in user reviews with F-scores comparable to humans, though performance depends strongly on prompt design.
-
Evaluating Generative Models as Interactive Emergent Representations of Human-Like Collaborative Behavior
Embodied LLM agents exhibit emergent collaborative behaviors indicating mental models of partners in a color-matching game, detected via LLM judges and supported by positive user feedback.
-
Annotation Quality in Aspect-Based Sentiment Analysis: A Case Study Comparing Experts, Students, Crowdworkers, and Large Language Model
Expert re-annotations of a German ABSA dataset serve as ground truth to evaluate how students, crowdworkers, and LLMs affect inter-annotator agreement and downstream performance on ACSA and TASD tasks using BERT, T5, and LLaMA models.
-
A Multimodal Text- and Graph-Based Approach for Open-Domain Event Extraction from Documents
MODEE is a multimodal system that integrates graphs with LLM embeddings to outperform prior open-domain event extraction methods on large datasets.
-
Using Learning Progressions to Guide AI Feedback for Science Learning
Learning progression-derived rubrics produce AI feedback on student science writing that matches expert rubric quality in key dimensions.
-
Data Comics for Education: Evaluating Effectiveness, Benefits, and the Ethics of AI-Assisted Creation
Within-subjects study with 60 students found superior information retrieval and insight comprehension with GenAI-assisted data comics over conventional visualizations, independent of prior literacy, alongside engagement benefits and ethical concerns.
-
Comparison of Modern Multilingual Text Embedding Techniques for Hate Speech Detection Task
Supervised models using embeddings like jina and e5 reach up to 92% accuracy on multilingual hate speech detection, substantially outperforming anomaly detection, while PCA to 64 dimensions preserves most performance in the supervised case.
-
Omakase: proactive assistance with actionable suggestions for evolving scientific research projects
Omakase monitors project documents to infer timely queries and distills research reports into actionable suggestions that users rated significantly more useful than raw reports.
-
Training Data Size Sensitivity in Unsupervised Rhyme Recognition
With sufficient training data, the unsupervised RhymeTagger outperforms human inter-annotator agreement in rhyme recognition across seven languages while LLMs without phonetic awareness perform poorly.
-
Do Small Language Models Know When They're Wrong? Confidence-Based Cascade Scoring for Educational Assessment
Verbalized confidence from small LMs enables cost-effective cascade routing for automated educational scoring, matching large-model accuracy at 76% lower cost when discrimination is strong.
- SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models
- Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents