Introduces the first large-scale multimodal benchmark MedLayXPlain-122K showing medical VLMs suffer significant lay-register degradation while general VLMs lack clinical precision.
super hub Mixed citations
BERTScore: Evaluating Text Generation with BERT
Mixed citation behavior. Most common role is background (41%).
201
Pith papers citing it
2,045
external citations · Pith
Background
41% of classified citations
abstract
We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTScore is more robust to challenging examples when compared to existing metrics.
hub tools
citation-role summary
background 8
method 7
baseline 1
other 1
citation-polarity summary
claims ledger
- abstract We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BE
authors
co-cited works
representative citing papers
NeuroQA is a large-scale 3D brain MRI visual question answering benchmark with verified image-grounded QA pairs, multi-domain coverage, and baseline evaluations showing current models lag behind text-only performance.
WearBCI provides the first multimodal dataset of wearable EEG signals under varied motion conditions with benchmarks for artifact removal and behavior analysis.
ReplicatorBench evaluates LLM agents on replicating social and behavioral science claims across retrieval, computation, and interpretation stages, finding strength in experiment execution but weakness in resource retrieval.
Sentence-BERT adapts BERT with siamese and triplet networks to produce sentence embeddings for efficient cosine-similarity comparisons, cutting computation time from hours to seconds on similarity search while matching BERT accuracy.
EHR-derived standardized patients and dual-track evaluation reveal LLMs trail clinicians by 37.28 points on full psychiatric encounters, with mental-status assessment the main bottleneck.
LePaX enables high-resolution chest X-ray report generation by learning to allocate resolution to diagnostically relevant regions and fusing high-res patches back into global features without increasing token count.
A power-calibrated statistical framework gives closed-form links from KGW watermark parameters (γ, δ) to detection power and KL distortion, enabling principled Pareto-optimal selection.
Releases SEFORA corpus of instructor feedback on college writing and UniMatch evaluation showing no LLM configuration exceeds 0.4 F1 in matching instructor priorities.
VSE perturbs images only to probe visual ambiguity in VLMs, clusters outputs into semantic prototypes, and computes mass-weighted dispersion, outperforming prior entropy methods on five VQA benchmarks across five models.
LLMs are applied in a generative pipeline for extracting, normalizing, and interpreting eligibility criteria from securities prospectuses, achieving up to 91% precision in document-level decisions with a conservative bias.
C3-Bench supplies a multi-domain dataset and LLM-based evaluation protocol that exposes systematic failures in existing change captioning models outside their training regimes.
Transition-aware best-of-N sampling embeds report sentences as sets, computes directional transition vectors via set-to-set distances, and scores candidates by proximity to ground-truth training transitions.
HALAS is a human-annotated dataset of ASR hallucinations on unprocessed real audio that shows simple metrics outperform current detection methods at 81% ROC-AUC versus 53.1% F1.
Curiosity-oriented linguistic interventions in LLM tutoring dialogues increased exploratory learner behaviors up to 2.4x across 270 conversations spanning multiple models and domains.
RouteJudge introduces an open platform for preference-based evaluation of LLM routers via pairwise user comparisons, along with the ORBIT toolbox for standardized routing workflows.
PaperGuard benchmark shows multimodal AI reviewers are vulnerable to domain-specific attacks on text and figures and proposes a chunk-based embedding defense.
Alignment defenses adapted from DPO and GRPO mitigate property inference attacks on LLMs while preserving utility.
AuditFraudBench is a new enforcement-grounded benchmark with three tasks for testing whether LLMs can detect fraudulent misstatements by reasoning over financial figures, disclosure framing, and known manipulation patterns.
Introduces IndoRad-VQA dataset and reports 8-25% performance gap in medical VLMs between English and Indonesian radiology VQA prompts.
Introduces the Matter to Mechanism benchmark of 2,645 structured instances and a composite metric suite for evaluating AI co-scientists on problem-to-hypothesis reasoning in battery materials research.
A per-token feature from temperature-induced changes in LLM token distributions predicts within-prompt creativity rank at Spearman rho 0.918 vs LLM judges and 0.870 vs humans, outperforming perplexity, entropy, top-1 margin, and compression baselines.
DEPO formulates detector-evasive paraphrasing as a constrained MDP and solves it via Lagrangian primal-dual RL with GRPO-style updates to achieve evasion while satisfying a semantic-preservation constraint.
Prompting LLMs with test-taking strategies for true/false factuality checks reduces tokens by over 80%, matches strong baselines on two benchmarks with SOTA on one, and enables fine-tuned SLMs to perform similarly at low cost with rationales.
citing papers explorer
-
MEDLAYXPLAIN: Benchmarking the Expert-Lay Gap in Medical Vision-Language Models
Introduces the first large-scale multimodal benchmark MedLayXPlain-122K showing medical VLMs suffer significant lay-register degradation while general VLMs lack clinical precision.
-
NeuroQA: A Large-Scale Image-Grounded Benchmark for 3D Brain MRI Understanding
NeuroQA is a large-scale 3D brain MRI visual question answering benchmark with verified image-grounded QA pairs, multi-domain coverage, and baseline evaluations showing current models lag behind text-only performance.
-
WearBCI Dataset: Understanding and Benchmarking Real-World Wearable Brain-Computer Interfaces Signals
WearBCI provides the first multimodal dataset of wearable EEG signals under varied motion conditions with benchmarks for artifact removal and behavior analysis.
-
ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences
ReplicatorBench evaluates LLM agents on replicating social and behavioral science claims across retrieval, computation, and interpretation stages, finding strength in experiment execution but weakness in resource retrieval.
-
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Sentence-BERT adapts BERT with siamese and triplet networks to produce sentence embeddings for efficient cosine-similarity comparisons, cutting computation time from hours to seconds on similarity search while matching BERT accuracy.
-
MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters
EHR-derived standardized patients and dual-track evaluation reveal LLMs trail clinicians by 37.28 points on full psychiatric encounters, with mental-status assessment the main bottleneck.
-
Seeing What Matters: Lesion-Aware High-Resolution Patch Discovery and Fusion for Chest X-ray Report Generation
LePaX enables high-resolution chest X-ray report generation by learning to allocate resolution to diagnostically relevant regions and fusing high-res patches back into global features without increasing token count.
-
Beyond Heuristic Tuning: Power-Calibrated LLM Watermarking
A power-calibrated statistical framework gives closed-form links from KGW watermark parameters (γ, δ) to detection power and KL distortion, enabling principled Pareto-optimal selection.
-
SEFORA: Student Essays with Feedback Corpus and LLM Feedback Evaluation Framework
Releases SEFORA corpus of instructor feedback on college writing and UniMatch evaluation showing no LLM configuration exceeds 0.4 F1 in matching instructor priorities.
-
Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?
VSE perturbs images only to probe visual ambiguity in VLMs, clusters outputs into semantic prototypes, and computes mass-weighted dispersion, outperforming prior entropy methods on five VQA benchmarks across five models.
-
LLM-Based Examination of Eligibility Criteria from Securities Prospectuses at the German Central Bank
LLMs are applied in a generative pipeline for extracting, normalizing, and interpreting eligibility criteria from securities prospectuses, achieving up to 91% precision in document-level decisions with a conservative bias.
-
C3-Bench: A Context-Aware Change Captioning Benchmark
C3-Bench supplies a multi-domain dataset and LLM-based evaluation protocol that exposes systematic failures in existing change captioning models outside their training regimes.
-
Transition-Aware best-of-N sampling for Longitudinal Chest X-ray Reports
Transition-aware best-of-N sampling embeds report sentences as sets, computes directional transition vectors via set-to-set distances, and scores candidates by proximity to ground-truth training transitions.
-
HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systems
HALAS is a human-annotated dataset of ASR hallucinations on unprocessed real audio that shows simple metrics outperform current detection methods at 81% ROC-AUC versus 53.1% F1.
-
Curiosity as Linguistic Intervention: Using LLM Tutoring Dialogues to Influence Exploratory Learning Behavior
Curiosity-oriented linguistic interventions in LLM tutoring dialogues increased exploratory learner behaviors up to 2.4x across 270 conversations spanning multiple models and domains.
-
RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing
RouteJudge introduces an open platform for preference-based evaluation of LLM routers via pairwise user comparisons, along with the ORBIT toolbox for standardized routing workflows.
-
Does AI Reviewer See the Full Picture? Attacking and Defending Multimodal Peer Review
PaperGuard benchmark shows multimodal AI reviewers are vulnerable to domain-specific attacks on text and figures and proposes a chunk-based embedding defense.
-
Alignment Defends LLMs from Property Inference Attacks
Alignment defenses adapted from DPO and GRPO mitigate property inference attacks on LLMs while preserving utility.
-
AuditFraudBench: Benchmarking Audit Judgment in Detecting Fraudulent Misstatements
AuditFraudBench is a new enforcement-grounded benchmark with three tasks for testing whether LLMs can detect fraudulent misstatements by reasoning over financial figures, disclosure framing, and known manipulation patterns.
-
Does Language Shift Break Medical Vision-Language Models? Indonesian Radiology Visual Question Answering Case Study
Introduces IndoRad-VQA dataset and reports 8-25% performance gap in medical VLMs between English and Indonesian radiology VQA prompts.
-
Matter to Mechanism: A Benchmark for AI Co-Scientists in Materials and Battery Research
Introduces the Matter to Mechanism benchmark of 2,645 structured instances and a composite metric suite for evaluating AI co-scientists on problem-to-hypothesis reasoning in battery materials research.
-
Before and After Temperature: A Distributional View of Creative LLM Generation
A per-token feature from temperature-induced changes in LLM token distributions predicts within-prompt creativity rank at Spearman rho 0.918 vs LLM judges and 0.870 vs humans, outperforming perplexity, entropy, top-1 margin, and compression baselines.
-
Detector-Evasive LLM Paraphrasing via Constrained Policy Optimization
DEPO formulates detector-evasive paraphrasing as a constrained MDP and solves it via Lagrangian primal-dual RL with GRPO-style updates to achieve evasion while satisfying a semantic-preservation constraint.
-
Teaching Language Models to Check Grounded Claim Factuality with Human Test-Taking Strategies
Prompting LLMs with test-taking strategies for true/false factuality checks reduces tokens by over 80%, matches strong baselines on two benchmarks with SOTA on one, and enables fine-tuned SLMs to perform similarly at low cost with rationales.
-
SliceWorld: A Predictive and Controllable World-State Model for CT Report Generation
SliceWorld introduces a world-state model for CT report generation that uses predictive and factor-aware objectives on axial slice sequences.
-
Graphs of Research: Citation Evolution Graphs as Supervision for Research Idea Generation
GoR extracts citation DAGs using position, frequency, predecessor links and time, then fine-tunes Qwen2.5-7B on 498 seed papers to generate ideas, claiming SOTA over gpt-4o baselines via LLM judges.
-
WirelessSenseLLM: Zero-Shot Human Activity Understanding by Bridging Wireless Signals and Human Language
WirelessSenseLLM bridges unsegmented Wi-Fi CSI signals to LLMs via a CSI-to-Language Adapter for zero-shot human activity understanding and reasoning.
-
EpiGraph: Building Generalists for Evidence-Intensive Epilepsy Reasoning in the Wild
EpiGraph creates a heterogeneous epilepsy knowledge graph that boosts LLM performance on clinical reasoning tasks by 30-41% in pharmacogenomics when used with Graph-RAG.
-
Dataset Watermarking for Closed LLMs with Provable Detection
A new watermarking method for closed LLMs boosts random word-pair co-occurrences via rephrasing and detects the signal statistically in outputs, working reliably even when the watermarked data is only 1% of fine-tuning tokens while preserving utility.
-
Led to Mislead: Adversarial Content Injection for Attacks on Neural Ranking Models
CRAFT is a supervised LLM framework using retrieval-augmented generation, self-refinement, fine-tuning, and preference optimization to create fluent adversarial content that boosts target ranks in neural ranking models, outperforming baselines on MS MARCO and TREC benchmarks with cross-architecture
-
Identifying and Characterizing Semantic Clones of Solidity Functions
A code-and-comment analysis method detects semantic clones in Solidity functions with 59% overall precision (84% for same-name functions) and 97% recall on 300k contracts, plus LLM summaries for uncommented code.
-
Analysis and Explainability of LLMs Via Evolutionary Methods
Evolutionary trees from LLM weights recover ground-truth training topologies and identify key datasets and layers through phenotypic analysis.
-
EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMs
EmoTrans is a new video benchmark with four progressive tasks that measures how well current multimodal LLMs handle dynamic emotion transitions rather than static recognition.
-
ArgRE: Formal Argumentation for Conflict Resolution in Multi-Agent Requirements Negotiation
ArgRE embeds abstract argumentation into multi-agent requirements negotiation to deliver argument-level traceability, higher evaluator-rated justifications, and improved compliance coverage over heuristic baselines.
-
EVENT5Ws: A Large Dataset for Open-Domain Event Extraction from Documents
EVENT5Ws is a new large-scale, manually verified open-domain event extraction dataset that benchmarks LLMs and demonstrates cross-context generalization.
-
Evaluating Remote Sensing Image Captions Beyond Metric Biases
Unfine-tuned MLLMs outperform fine-tuned models on remote sensing image captioning when captions are scored by their ability to reconstruct the source image, and a training-free self-correction method achieves SOTA performance.
-
Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI
A new multi-frame VQA benchmark on volumetric MRI demonstrates that bounding-box supervised fine-tuning improves spatial grounding in VLMs over zero-shot baselines.
-
TaxPraBen: A Scalable Benchmark for Structured Evaluation of LLMs in Chinese Real-World Tax Practice
TaxPraBen is a new benchmark with 14 datasets and a structured evaluation method for measuring LLM performance on Chinese real-world tax tasks and scenarios.
-
Right at My Level: A Unified Multilingual Framework for Proficiency-Aware Text Simplification
Re-RIGHT trains a 4B policy model with vocabulary coverage, semantic preservation, and coherence rewards to perform proficiency-aware lexical simplification in four languages without parallel corpora.
-
TableNet A Large-Scale Table Dataset with LLM-Powered Autonomous
TableNet is a new large-scale table dataset created via LLM multi-agent generation, combined with diversity-based active learning that achieves competitive performance on its test set and superior results on real-world tables using fewer samples than baselines.
-
Integrating Domain-Specialized Language Models with AI Measurement Tools for Deterministic Atomic-Resolution Experimentation
Domain-specialized small language models enable deterministic atomic-resolution scanning probe microscopy control with 99.3% command accuracy, lower computational cost, and better domain performance than larger general models.
-
On the Factual Consistency of Text-based Explainable Recommendation Models
A prompting pipeline and statement-level metrics show that six state-of-the-art text-based explainable recommendation models achieve high semantic similarity but very low factual consistency on Amazon review data.
-
IPQA: A Benchmark for Core Intent Identification in Personalized Question Answering
IPQA is a new benchmark that measures how well models identify core user intents from history in personalized question answering, finding that performance is poor and declines with greater question complexity.
-
Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks
Neural-MedBench reveals sharp performance drops in state-of-the-art VLMs on reasoning-intensive neurology tasks compared to conventional classification benchmarks, with reasoning failures dominating errors.
-
OpenLifelogQA: An Open-Ended Multi-Modal Lifelog Question-Answering Dataset
Presents OpenLifelogQA, a new multimodal lifelog QA dataset with 14,187 pairs from 18 months of real-world data, plus baseline results on LLaVA-NeXT-Interleave.
-
Smoothie: Smoothing Diffusion on Token Embeddings for Text Generation
Smoothie performs diffusion by smoothing token embeddings based on semantic similarity, outperforming prior diffusion models on sequence-to-sequence and unconditional text generation tasks.
-
Topic-Based Watermarks for Large Language Models
A topic-guided watermarking scheme partitions the LLM vocabulary into topic-aligned token subsets and green-lists relevant tokens based on the input prompt to embed detectable marks while preserving text quality and improving robustness to attacks.
-
On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces
Aligning adversarial perturbations with the near-null singular directions of intermediate linear layers in transformer VLMs yields stronger attacks than existing feature- and output-space methods.
-
Search-based Testing of Vision Language Models for In-Car Scene Understanding
Search-based optimization over rendered in-cabin scenes finds up to 10× more VLM failures and up to 3.6× higher failure-cluster coverage than random generation for question answering and captioning.
-
Identifying and Resolving Pitfalls of Knowledge-Based VQA Benchmarks: Auditing, Repairing, and Augmenting
Audit of KB-VQA benchmarks reveals systematic violations of answer derivability, question clarity, and visual disambiguation assumptions, with new repair and multi-entity augmentation protocols producing different model performance trends.