{"id":"586eaedd-ba8d-45fa-b7e9-f44cbbf4b146","arxiv_id":"2504.18572","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"BELL applies known prompting techniques and simple text-similarity metrics to rank seven LLMs on 'explainability' using math questions from OpenOrca.","lead":"This paper proposes a benchmark called BELL that scores how explainable large language models are by running them with several reasoning prompts and measuring coherence, uncertainty, and similarity to reference answers. It reports scores for seven models, but the metrics are loosely defined and the results have no error bars, so the benchmark is not yet a validated measurement tool.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sections 4.7 and 5 never define Coherence(E_i) or Uncertainty(E_i); without those definitions the OverallScore formula is not computable, so BELL is not yet a standardized benchmarking technique.","rationale":"Reader's REJECT remains appropriate. The strongest claim is that BELL standardizes explainability evaluation; a standardized benchmark must have computable metrics. The paper's own equations leave two of three OverallScore terms undefined, so no one can determine what is being measured. The reader's chosen weakest assumption—OpenOrca reference as ground truth—is a real validity concern, but it presupposes the score is computable. My concern is prior and definitional. The linked code might fix this, but without examining it, the assertion of open-source implementation is not demonstrated evidence in the manuscript. This is a specification gap, not an attack on the authors. Thus no verdict change.","tokens_in":9941,"tokens_out":3393,"duration_ms":33474,"concrete_test":"Clone the linked Infosys-Responsible-AI-Toolkit repository, locate the responsible-ai-llm-explain implementation, and extract or run the exact functions for coherence and uncertainty on one dataset sample. Independently recompute the OverallScore and Hallucination Score for GPT-4's CoT row from the code. If these metrics are not implemented or the Table 1 value is not reproduced, the central claim fails; if they are fully specified and reproducible, this objection is withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that BELL is a standardized benchmark for LLM explainability. For that to hold, the OverallScore in Section 5 must be a well-defined function of observable outputs. The equation is OverallScore = (1/n) * sum((Coherence(E_i) + Uncertainty(E_i) + CosSim(E_i, B_i))/3). Section 4.7 gives an explicit formula only for Cosine Similarity (eq. 1). Coherence and Uncertainty are described in qualitative prose ('logically consistent and contextually aligned', 'how confident the model is'), but no scoring function, scale, measurement procedure, or implementation details are specified. The Hallucination Score is also defined in terms of an 'Average of Evaluation Metrics' and 'Average Similarity Score' whose arguments are never pinned down. Consequently the headline numbers in Table/Graph 1-7 cannot be reproduced or independently checked from the manuscript. If the GitHub repository resolves this, the code could supply the missing specification; but the submitted text alone does not define a benchmark. This is more load-bearing than the reference-ground-truth issue: even if OpenOrca B_i were a perfect gold standard, an undefined coherence or uncertainty term would still make OverallScore non-computable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes BELL (Benchmarking the Explainability of Large Language Models), a framework for evaluating LLM explainability using thought-eliciting techniques (CoT, ThoT, ReRead, CoVe, and Hallucination) on the OpenOrca dataset. The proposed score averages coherence, uncertainty, and cosine similarity to an OpenOrca reference response; a hallucination score is then subtracted from this average to produce a final model score. Results are reported for seven models in tabular and bar-chart form, and the authors claim that BELL is a standardized benchmark and that larger models such as GPT-4 outperform smaller ones. The paper also points to an open-source repository as the implementation of the techniques and metrics.","tokens_in":10170,"tokens_out":7744,"duration_ms":68103,"significance":"If fully specified and validated, a reproducible benchmark for LLM explanation quality would be a valuable contribution to the interpretability and safety literature. The paper has useful starting points: it addresses a real need, covers several thought-eliciting techniques, provides a comparative table of seven models, and releases an implementation. However, as written, BELL is not yet a standardized benchmarking technique because core components of the scoring formula are undefined and the reported results cannot be reproduced from the manuscript. The missing metric definitions and lack of validation are load-bearing issues that substantially reduce the current significance of the work.","major_comments":[{"comment":"The central scoring formula is not computable from the manuscript. Section 4.7 gives a formal definition only for Cosine Similarity (Eq. 1); the entries for Coherence and Uncertainty are qualitative prose (\"logically consistent and contextually aligned\" and \"how confident the model is\") with no scoring function, scale, or measurement procedure. The Hallucination Score formula, Hallucination Score = 1 - (0.8 * Average of Evaluation Metrics) - (0.2 * Average Similarity Score), is also undefined because \"Average of Evaluation Metrics\" and \"Average Similarity Score\" are never specified. Since Section 5's OverallScore includes Coherence(E_i) and Uncertainty(E_i), the benchmark cannot be implemented, reproduced, or independently checked from the submitted text. This directly contradicts the paper's claim of introducing a \"standardized benchmarking technique.\"","section":"§4.7 and §5 (OverallScore formula)"},{"comment":"The Model_Score formula is under-specified and appears numerically inconsistent. Model_Score = Avg(OverallScore) - Hallucination(E_i) mixes a model-level average with a per-response hallucination term, and the table's Model Score values cannot be derived from the preceding columns: for GPT-4 the average of the five technique scores (85.28, 92.39, 91.91, 91.37, 85.14) is approximately 89.2, and subtracting the reported hallucination value 19.42 does not yield the reported model score 87.78. Moreover, since the Hallucination Score is itself defined as a linear combination of evaluation metrics and cosine similarity, and the OverallScore is an average of coherence, uncertainty, and cosine similarity, the final ranking is a reweighting of the same input measurements rather than an independent assessment of hallucination. The paper needs to specify exactly what enters each average and provide a derivation that reproduces Table 1.","section":"§5 (Model_Score formula, Table 1)"},{"comment":"The choice of OpenOrca GPT-4/GPT-3.5 completions as the baseline B_i is not justified. Cosine similarity between the generated explanation and this reference is used as a component of the explainability score, but no evidence is provided that these references are gold-standard explanations or that embedding similarity is a valid proxy for transparency. Without such evidence, the benchmark measures closeness to a particular reference corpus, not explainability as claimed. The authors should validate the metric against human judgments or established interpretability benchmarks.","section":"§4.6, §4.7 (cosine similarity to OpenOrca baseline)"},{"comment":"The reported experiments lack essential details for the comparative claims. No sample size (number of OpenOrca math questions), model versions (e.g., exact GPT-4 API version), inference parameters (temperature, max tokens, seed), embedding model, or G-Eval configuration are given, and the bar charts show no error bars, confidence intervals, or significance tests. Consequently the conclusion that \"GPT-4 consistently outperformed others\" is unsupported by the reported evidence.","section":"§5, Table/Graph 1-7"}],"minor_comments":[{"comment":"The section numbering jumps from \"4 Experimental Setup\" to \"4.6 Dataset\", with no Subsections 4.1-4.5; the numbering should be corrected so that the metrics subsection is consistently referenced.","section":"Section 4"},{"comment":"There are inconsistencies in capitalization and terminology: \"ReRead Thot\" appears alongside \"ReRead ThoT\", and \"Thread-of-Thought\" appears alongside \"Thread of Thought\"; the figures labelled \"Graph 1-7\" are bar charts and should be labelled as figures with axis labels and units.","section":"Throughout"},{"comment":"Sections 3.5 and 3.6 describe GoT and LoT, and the introduction lists them among the evaluated techniques, but Section 5 reports results only for CoT, ThoT, ReRead variants, CoVe, and Hallucination; the authors should either include results for GoT and LoT or revise the claimed contributions accordingly.","section":"Sections 3.5, 3.6, and 5"},{"comment":"Reference [37] is cited for the OpenOrca dataset, but the entry is for the Orca paper; the dataset itself should be cited with its version and URL, and several other reference entries (e.g., [4], [19]) are incomplete.","section":"References"},{"comment":"The term \"Hallucination\" is used both as a thought-eliciting technique in Fig. 1 and as a metric in Section 4.7; hallucination is a failure mode, not a reasoning technique, and this conflation should be clarified.","section":"Fig. 1 and Section 4.7"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as a technical report or white paper rather than a peer-reviewed benchmark paper. The submitted text alone does not define a benchmark: the core scoring formula is not computable, the hallucination component is circular, and the reported model rankings cannot be reproduced. I would be open to a resubmission if the authors supply precise metric definitions, a reproducible experimental protocol, and validation of the scores against human judgments or existing interpretability benchmarks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a white paper from Infosys proposing BELL, a benchmark that scores LLM explanations by combining cosine similarity to OpenOrca references with coherence and uncertainty. The packaging is the only new thing; the pieces are all borrowed. That said, the paper has a real problem: the two most important metrics, coherence and uncertainty, are never defined. The OverallScore formula in Section 5 is therefore not computable from the text. The stress-test note gets this right. I checked the GitHub link—it's a generic toolkit URL, and while the paper claims the code is there, the paper itself doesn't point to a specific directory that resolves the missing definitions. So as submitted, the headline results in Graphs 1-7 cannot be reproduced.\n\nWhat the paper does well: it's clearly written, compares a sensible set of prompt-based reasoning techniques across seven models, and the idea of a single explainability score for model selection is practically useful for regulated industries. Releasing open-source code is the right instinct. The cosine similarity metric is standard and reproducible. The related work is broad, if shallow.\n\nSoft spots beyond the undefined metrics: the hallucination score is an unvalidated linear combination of 'evaluation metrics' and 'similarity score,' which are never pinned down, and it's then subtracted from the overall score—so the final Model_Score is just a reweighting of the same three inputs. The table doesn't obviously compute from the stated formulas: for GPT-4, Hallucination is 19.42 and Model Score is 87.78; if Model Score is average overall minus hallucination, the implied average is over 100, which is impossible. So there's a mismatch between the formula and the numbers. Also, the abstract name-checks Graph-of-Thought and Logic-of-Thought, but the experiments only cover CoT, ThoT, ReRead CoT/ThoT, and CoVe. That overclaims. And the ground truth concern—OpenOrca's GPT-written answers as gold explanations—is real, though secondary: cosine similarity to a reference is not explainability.\n\nNone of these flaws are fatal if the code fills in the gaps. But the burden is on the authors. As a benchmark proposal, the paper lacks validation: no error bars, no human agreement, no sensitivity analysis. It reads like an internal report, not a rigorous evaluation.\n\nWho is this for? Practitioners in industry who want a quick way to compare models for transparency-sensitive applications. It doesn't add new science. I'd give it a skeptical peer review: send it to a referee who can check whether the code really defines the metrics, and let the authors revise or retract. It deserves a serious referee, but the current version would need major work before acceptance.","headline":"A practical white paper on benchmarking LLM explainability, but the core scoring formula is underspecified and the numbers don't compute as written; the open-source code is the only path to salvage.","tokens_in":10701,"tokens_out":2427,"would_cite":false,"duration_ms":22862,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BELL proposes a single score—the average of coherence, uncertainty, and cosine similarity against OpenOrca reference answers—as a standardized measure of LLM explainability.","keywords":["Explainability","Interpretability","Benchmarking","Chain-of-Thought","Thread-of-Thought","Thought elicitation","OpenOrca","Model transparency"],"falsifier":"Take a fixed set of explanations whose quality is independently judged by human experts, compute BELL scores on the same set, and check whether the rankings agree. If a model judged clearer scores lower on BELL, the benchmark is not measuring explainability as claimed.","tokens_in":9782,"feed_emoji":"🧠","tokens_out":3861,"duration_ms":32440,"temperature":0.7,"pith_summary":"This paper introduces BELL, a benchmark that assigns a single explainability score to a large language model by averaging three quality metrics over a set of responses. The score combines coherence, measured self-uncertainty, and cosine similarity between the model's explanation and a reference answer from the OpenOrca dataset. The authors argue that this composite score lets researchers and practitioners compare models, as well as thought-eliciting techniques such as Chain-of-Thought, Thread-of-Thought, ReRead, and Chain-of-Verification, on how transparent their reasoning is. The contribution is the packaging of several known prompting strategies and similarity metrics into one reproducible scoring recipe.","feed_headline":"One formula ranks LLMs by explainability","feed_subtitle":"BELL averages coherence, uncertainty, and similarity to reference answers to score model transparency.","key_machinery":"The central object is the overall score formula: $$\\text{OverallScore} = \\frac{1}{n}\\sum_{i=1}^{n} \\frac{\\text{Coherence}(E_i) + \\text{Uncertainty}(E_i) + \\text{CosSim}(E_i, B_i)}{3}$$ with the OpenOrca response $B_i$ serving as the reference explanation. Coherence and uncertainty are computed for each generated explanation; cosine similarity is computed in the model's embedding space; a separate hallucination score is subtracted to form the model score. The benchmark's load-bearing move is using this single average as the operational definition of explainability.","core_discovery":"The paper's central claim is that explainability can be quantified as an average of three measurable components, computed per response and then across the dataset: coherence of the generated explanation, uncertainty of the model about that explanation, and cosine similarity between the explanation's embedding and the embedding of a reference answer. The overall score is presented as a standardized benchmarking technique, with hallucination subtracted separately to produce a final model score. On the math subset of OpenOrca, the authors report that larger models such as GPT-4 outscore smaller models, and that Thread-of-Thought and ReRead variants generally yield higher scores than plain Chain-of-Thought.","pith_inferences":["A testable extension is to compare BELL scores against human expert ratings on the same explanations; if the ranking diverges, the cosine-similarity component would need reweighting or replacement.","The formula implicitly assumes the OpenOrca reference answer is the canonical explanation; in domains where many equally good explanations exist, embedding similarity could penalize valid alternatives.","Because the same embedding space is used for similarity and for measuring uncertainty, BELL may partly reward stylistic conformity to GPT-generated phrasing rather than reasoning quality.","The framework could be extended to non-mathematical domains by swapping the reference dataset, but the reference dependency would remain."],"forward_implications":["If BELL is accepted, model selection for high-stakes tasks can be partly driven by a single explainability number rather than qualitative inspection.","The benchmark provides a common yardstick for comparing thought-eliciting techniques, so a technique's benefit can be measured by its effect on the composite score.","Releasing the implementation lets other teams reproduce the scores and extend the metric set.","The math-focused evaluation on OpenOrca gives an initial ranking that suggests larger models produce more transparent explanations on this subset."],"supporting_citations":[{"why":"Supplies the reference responses (OpenOrca, GPT-4/GPT-3.5 completions) used as the baseline $B_i$ for cosine similarity.","marker":"[37]"},{"why":"Defines Chain-of-Thought prompting, one of the thought-eliciting techniques BELL evaluates.","marker":"[31]"},{"why":"Defines Thread-of-Thought, a technique the experiments show producing some of the highest scores.","marker":"[32]"},{"why":"Defines ReRead, which the paper combines with CoT and ThoT to form two of the evaluated techniques.","marker":"[35]"},{"why":"Defines Chain-of-Verification, the verification technique whose hallucination-reduction behavior BELL measures.","marker":"[36]"},{"why":"Defines Graph-of-Thought, presented as part of the benchmark's technique set.","marker":"[33]"},{"why":"Defines Logic-of-Thought, presented as part of the benchmark's technique set.","marker":"[34]"}],"fun_headline_variants":["BELL benchmark: score LLM explainability via coherence, uncertainty, similarity","New metric ranks LLMs on clarity of reasoning, not just answers","Explainability score for LLMs: coherence + certainty + reference alignment","BELL: a formula to measure how explainable an LLM really is","Rank LLMs by explainability with BELL's three-factor score"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes that OpenOrca's GPT-written reference answers are the correct gold standard for explanation quality, and that cosine similarity to those answers measures explainability; if either fails, the score measures something other than explainability.","fun_headline_variants_meta":{"raw":{"variants":["BELL benchmark: score LLM explainability via coherence, uncertainty, similarity","New metric ranks LLMs on clarity of reasoning, not just answers","Explainability score for LLMs: coherence + certainty + reference alignment","BELL: a formula to measure how explainable an LLM really is","Rank LLMs by explainability with BELL's three-factor score"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001073,"raw_usage":{"total_tokens":4376,"prompt_tokens":713,"completion_tokens":3663,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":329,"completion_tokens_details":{"reasoning_tokens":3568}},"tokens_in":329,"tokens_out":3663,"duration_ms":22967,"temperature":1.0,"reasoning_tokens":3568,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:16:43.083446+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed set of explanations whose quality is independently judged by human experts, compute BELL scores on the same set, and check whether the rankings agree. If a model judged clearer scores lower on BELL, the benchmark is not measuring explainability as claimed.","supporting_citations":[{"cited_title":"Orca: Progressive Learning from Complex Explanation Traces of GPT-4","cited_arxiv_id":null,"evidence_quote":"Supplies the reference responses (OpenOrca, GPT-4/GPT-3.5 completions) used as the baseline $B_i$ for cosine similarity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines ReRead, which the paper combines with CoT and ThoT to form two of the evaluated techniques."}],"review_version":1}