Pith. sign in

REVIEW 1 cited by

Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on Consistency with Human Preferences

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2303.07610 v1 pith:RPM3OMBT submitted 2023-03-14 cs.CL

Yunjie Ji , Yan Gong , Yiping Peng , Chao Ni , Peiyan Sun , Dongyu Pan , Baochang Ma , Xiangang Li This is my paper · ORCID
classification cs.CL
keywords chatgptcontenthumanpreferencesrankrankingabilitycapability
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

As a natural language assistant, ChatGPT is capable of performing various tasks, including but not limited to article generation, code completion, and data analysis. Furthermore, ChatGPT has consistently demonstrated a remarkable level of accuracy and reliability in terms of content evaluation, exhibiting the capability of mimicking human preferences. To further explore ChatGPT's potential in this regard, a study is conducted to assess its ability to rank content. In order to do so, a test set consisting of prompts is created, covering a wide range of use cases, and five models are utilized to generate corresponding responses. ChatGPT is then instructed to rank the responses generated by these models. The results on the test set show that ChatGPT's ranking preferences are consistent with human to a certain extent. This preliminary experimental finding implies that ChatGPT's zero-shot ranking capability could be used to reduce annotation pressure in a number of ranking tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Explingo: Explaining AI Predictions using Large Language Models

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Explingo uses GPT-4o to convert SHAP explanations into natural-language narratives and to grade them on four quality metrics, with a few exemplars improving style but nudging correctness down.

Pith tools