ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models

Abhranil Chandra; Manish Gupta; Pawan Goyal; Sreevatsa Muppirala; Yaswanth Narsupalli

RECORD

ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models

Yaswanth Narsupalli, Abhranil Chandra, Sreevatsa Muppirala, Manish Gupta, Pawan Goyal

Computation and Language · Submitted 16 Jul 2024

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim The framework not only improves the accuracy of these evaluations, surpassing previous benchmarks but also generates constructive feedback.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2407.12877 (version 2)

Pith IDHPDBYNJW, Pith's permanent record of this paper and its review

CitationAbhranil Chandra, Manish Gupta, Pawan Goyal, Sreevatsa Muppirala, Yaswanth Narsupalli. "ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models." arXiv:2407.12877 (2024). https://pith.science/paper/HPDBYNJW

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 8,761 input tokens.

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.17 · human participants 0.08 · controlled comparison 0.54 · randomized assignment 0.01 · states quantitative result 0.98 · compares with prior results 0.94 · mentions code or data availability 0.99.

The abstract, sentence by sentence

arXiv listing abstract v2. Results and the main claim are underlined; show every role.

Assessing the quality of outputs generated by generative models, such as large language models and vision language models, presents notable challenges. Traditional methods for evaluation typically rely on either human assessments, which are resource-intensive, or automatic metrics that often show a low correlation with human judgment. Another common approach is to use deep learning systems, which not only consume a substantial amount of compute and time but also require extensive training data. In this study, we introduce a tuning-free framework called ReFeR, designed to evaluate generative outputs, including both text and images, by leveraging a 2-level hierarchy of LLMs and VLMs themselves. We rigorously evaluate our framework, ReFeR, across four diverse evaluation tasks. The framework not only improves the accuracy of these evaluations, surpassing previous benchmarks but also generates constructive feedback. Interestingly, the framework is also applicable to reasoning tasks. Experiments on four reasoning tasks demonstrate superior collective reasoning abilities of the framework. We present two variants of the framework: ReFeR-Turbo, optimized for accelerated performance, and ReFeR-Lite, offering a more cost-effective solution. ReFeR-Lite is $\sim7.7\times$ more efficient while being comparably accurate to ReFeR-Turbo. We make code, data and PIP package publicly available. See this PIP URL https://pypi.org/project/refer-agents/ and this Git URL https://github.com/yaswanth-iitkgp/ReFeR_Code .

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Assessing the quality of outputs generated by generative models, such as large language models and vision language models, presents notable challenges.	background 1.00	0.05	0.02	asserted 1.00
s1	Traditional methods for evaluation typically rely on either human assessments, which are resource-intensive, or automatic metrics that often show a low correlation with human judgment.	background 0.96	0.05	0.02	asserted 0.99
s2	Another common approach is to use deep learning systems, which not only consume a substantial amount of compute and time but also require extensive training data.	background 0.99	0.05	0.02	asserted 1.00
s3	In this study, we introduce a tuning-free framework called ReFeR, designed to evaluate generative outputs, including both text and images, by leveraging a 2-level hierarchy of LLMs and VLMs themselves.	contribution 0.88	0.88	0.11	asserted 1.00
s4	We rigorously evaluate our framework, ReFeR, across four diverse evaluation tasks.	contribution 0.55	0.95	0.04	asserted 0.98
s5	The framework not only improves the accuracy of these evaluations, surpassing previous benchmarks but also generates constructive feedback.	result 0.99	0.93	0.53	asserted 1.00
s6	Interestingly, the framework is also applicable to reasoning tasks.	contribution 0.52	0.68	0.06	asserted 0.97
s7	Experiments on four reasoning tasks demonstrate superior collective reasoning abilities of the framework.	result 1.00	0.95	0.17	asserted 1.00
s8	We present two variants of the framework: ReFeR-Turbo, optimized for accelerated performance, and ReFeR-Lite, offering a more cost-effective solution.	contribution 0.84	0.91	0.06	asserted 0.99
s9	ReFeR-Lite is $\sim7.7\times$ more efficient while being comparably accurate to ReFeR-Turbo.	result 0.99	0.94	0.12	asserted 1.00
s10	We make code, data and PIP package publicly available.	contribution 0.90	0.86	0.02	asserted 0.99
s11	See this PIP URL https://pypi.org/project/refer-agents/ and this Git URL https://github.com/yaswanth-iitkgp/ReFeR_Code .	contribution 0.92	0.57	0.02	asserted 0.61

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99.

What was examined, and what was not

sources read

arXiv listing abstract v2

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

Assessing the quality of outputs generated by generative models, such as large language models and vision language models, presents notable challenges. Traditional methods for evaluation typically rely on either human assessments, which are resource-intensive, or automatic metrics that often show a low correlation with human judgment. Another common approach is to use deep learning systems, which not only consume a substantial amount of compute and time but also require extensive training data. In this study, we introduce a tuning-free framework called ReFeR, designed to evaluate generative outputs, including both text and images, by leveraging a 2-level hierarchy of LLMs and VLMs themselves. We rigorously evaluate our framework, ReFeR, across four diverse evaluation tasks. The framework not only improves the accuracy of these evaluations, surpassing previous benchmarks but also generates constructive feedback. Interestingly, the framework is also applicable to reasoning tasks. Experiments on four reasoning tasks demonstrate superior collective reasoning abilities of the framework. We present two variants of the framework: ReFeR-Turbo, optimized for accelerated performance, and ReFeR-Lite, offering a more cost-effective solution. ReFeR-Lite is $\sim7.7\times$ more efficient while being comparably accurate to ReFeR-Turbo. We make code, data and PIP package publicly available. See this PIP URL https://pypi.org/project/refer-agents/ and this Git URL https://github.com/yaswanth-iitkgp/ReFeR_Code .

Human review

How Pith checks papers → · Full audit of this paper →

How the literature uses it

1 arXiv paper cite this in 2 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 9 sentences without a confident label

Uses the method or tool of 4 papers

All the baselines were implemented and evaluated using the scripts provided by [ref] in their official repository.

Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaboration · Implementation Details · 0.98
The baselines like CLIPScore [ref] , ImageReward [ref] , PickScore [ref] are implemented following the codes provided in their official repositories.

CLIPScore: A Reference-free Evaluation Metric for Image Captioning · Implementation Details · 0.99
The baselines like CLIPScore [ref] , ImageReward [ref] , PickScore [ref] are implemented following the codes provided in their official repositories.

ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation · Implementation Details · 0.99

Uses data or values from 5 papers

For NLG evaluation, we test our framework on SummEval [ref] for summarization evaluation, and TopicalChat [ref] for dialogue generation evaluation.

USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation · Datasets · 0.95
For multimodal evaluation, we compare our framework on evaluating two types of task, image-to-text using ICQD (Image Caption Quality Dataset) [ref] and text-to-image generation using AGIQA-1k by [ref] .

Quality Estimation for Image Captions Based on Large-scale Human Evaluations · Datasets · 0.93
For multimodal evaluation, we compare our framework on evaluating two types of task, image-to-text using ICQD (Image Caption Quality Dataset) [ref] and text-to-image generation using AGIQA-1k by [ref] .

A Perceptual Quality Assessment Exploration for AIGC Images · Datasets · 0.93

Compares with 8 papers

Lastly, ReFeR also beats single agent methods (zero-shot CoT [ref] , self correction [ref] ), and multi-agent methods like multi-agent debate [ref] and multi-agent peer review [ref] on 3 out of 4 reasoning datasets, clearly outperforming on average while keeping lower costs than baselines.

Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaboration · Introduction · 0.99
Lastly, ReFeR also beats single agent methods (zero-shot CoT [ref] , self correction [ref] ), and multi-agent methods like multi-agent debate [ref] and multi-agent peer review [ref] on 3 out of 4 reasoning datasets, clearly outperforming on average while keeping lower costs than baselines.

Large Language Models Cannot Self-Correct Reasoning Yet · Introduction · 0.99
Lastly, ReFeR also beats single agent methods (zero-shot CoT [ref] , self correction [ref] ), and multi-agent methods like multi-agent debate [ref] and multi-agent peer review [ref] on 3 out of 4 reasoning datasets, clearly outperforming on average while keeping lower costs than baselines.

Improving Factuality and Reasoning in Language Models through Multiagent Debate · Introduction · 0.99

Cites as background 9 papers

[ref] developed a framework inspired by academic peer review, emphasizing iterative improvement through peer feedback.

Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaboration · Related Work · 0.87
Self-Correct [ref] is a single-agent approach that enables an LLM to iteratively evaluate its own outputs, identify errors, and refine its responses through self-reflection.

Large Language Models Cannot Self-Correct Reasoning Yet · Reasoning · 0.83
Multi-Agent Debate [ref] involves a group of agents, where each agent observes the solutions provided by others, updates its own solution accordingly, and repeats this process through multiple iterations.

Improving Factuality and Reasoning in Language Models through Multiagent Debate · Reasoning · 0.95

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.