INFERENCEDYNAMICS: Efficient Routing Across LLMs through Structured Capability and Knowledge Profiling

Baixuan Xu; Chunkit Chan; Chunyang Li; Haochen Shi; Qiang Yang; Tao Fan; Tianshi Zheng; Weiqi Wang; Yangqiu Song

REVIEW 5 major objections 6 minor 2 cited by

INFERENCEDYNAMICS: Efficient Routing Across LLMs through Structured Capability and Knowledge Profiling

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A router that profiles each model's knowledge and capability beats every single candidate by 1.28 points on average across four held-out benchmarks while spending about 80% of the best model's budget.

desk verdict Useful routing extension with a new dataset, but the evaluation lacks baselines and error bars, so the 1.28-point claim is not yet established. read the letter →

arxiv 2505.16303 v1 pith:WCS4PX7I submitted 2025-05-22 cs.CL

Haochen Shi , Tianshi Zheng , Weiqi Wang , Baixuan Xu , Chunyang Li , Chunkit Chan , Tao Fan , Yangqiu Song

show 1 more author

Qiang Yang

This is my paper

classification cs.CL

keywords LLMroutingmodelselectioncapabilityprofilingknowledgeRouteMixparameter-freecost-performancetrade-offout-of-distributiongeneralization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

InferenceDynamics is an LLM routing framework that claims to choose the best model for a query by profiling each model's knowledge and capability on a fixed 'Index Set', then matching unseen queries to those profiles. The paper's central result is that this profile-based router beats the strongest single model by 1.28 average points on MMLU-Pro, GPQA, BigGenBench, and LiveBench, without any router training, and at roughly 80% of the best model's cost under default settings. The authors' motivating concern is that the LLM ecosystem now contains many specialized models; if routing can be rebuilt cheaply as new models and new knowledge domains appear, users can exploit that specialization without retraining a router.

What carries the argument

The central object is the ranked knowledge-capability score pair. For a model $M_t$ and an index query $x_i$, each element $e$ (a knowledge item or capability) receives a per-query score $S^\alpha_\beta(M_t, x_i, e) = \sum_{j=1}^{|K_{x_i}|} (s^i_t - \beta c^i_t)\, \mathbb{1}(e = k_j)\, \alpha^{j-1} / \sum_{m=1}^{|K_{x_i}|} \alpha^{m-1}$, where $s^i_t$ is the model's average evaluation score over $K$ trials and $c^i_t$ its average cost. These per-query scores are averaged over all Index Set queries containing $e$ to give $S^\alpha_\beta(M_t, D, e)$. For a new query $x$, the router forms a knowledge score $KS^\alpha(M_t, x)$ and an analogous capability score $CS^\alpha(M_t, x)$ by weighting the per-element aggregates with the query's ranked labels, then returns $\arg\max_{M_t} (\gamma\, KS^\alpha(M_t, x) + \delta\, CS^\alpha(M_t, x))$. The machinery is deliberately parameter-free: no router weights are learned, and the only tuned numbers are the hyperparameters $\alpha$, $\beta$, $\gamma$, and $\delta$, together with the label-generation, embedding-consolidation, and frequency-threshold steps.

What would settle it

Take a new out-of-distribution benchmark whose knowledge domains are absent from RouteMix, so that most queries fall into the 'Other' bucket; if InferenceDynamics's average score then drops to the level of random routing or below the best single model, the transfer-from-index-set assumption would be refuted.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that routing can be decided by a simple aggregation of historical scores. For each query, an auxiliary LLM labels the capabilities and knowledge domains required; each candidate model's scores on index queries carrying those labels are converted into per-label proficiency scores; and the router simply picks the model that maximizes a weighted sum of knowledge and capability scores. The claim is that this transfer works on out-of-distribution queries: across MMLU-Pro, GPQA, BigGenBench, and LiveBench, the mixed router averaged 74.55, which is 1.28 points above the best single candidate, and under a cost penalty it stayed near the best model's performance while spending about half as much.

Load-bearing premise

The router assumes that a model's past scores on index queries carrying the same predicted knowledge or capability labels predict its score on a brand-new query, even when those labels come from a smaller auxiliary model and rare knowledge is lumped into an 'Other' bucket.

Editorial extensions

If this is right

A new LLM can be added to the candidate pool by scoring it on the Index Set; the router itself needs no retraining.
New knowledge domains can be incorporated by computing a model's aggregate score for that domain and folding it into future routing decisions.
Routing by knowledge alone performs best on knowledge-intensive benchmarks (top on GPQA), while routing by capability alone performs best on capability-driven tasks (top on BigGenBench); the mixed strategy combines both advantages.
Applying a cost penalty barely degrades average performance while cutting spending to roughly half the best single model's budget, with text-generation tasks more sensitive to the penalty than question-answering tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An implication the paper leaves implicit is that the same Index Set profiling could be published once per model, letting many downstream routers reuse a shared capability map without re-running inference.
Because rare knowledge is folded into an 'Other' bucket, the claim is really about label reuse; a stress test with adversarial queries whose knowledge is genuinely novel would reveal how much of the 1.28-point margin survives without any matching index entries.
The router's sensitivity to the auxiliary labeler could be probed cheaply by corrupting a fraction of generated labels; the margin should shrink smoothly with label noise if the transfer assumption holds.
The ranked-weighting step is a soft attention over capability and knowledge labels; a hybrid that combines this profile-based score with a small learned router could preserve fast adaptation while closing any remaining gap to trained baselines.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a legitimate extension of the capability-profiling line of work (Model-SAT, Re-Task) with a useful new dataset, but the evaluation as written cannot support the headline 1.28-point margin. The core idea is simple: use an auxiliary LLM to tag queries with ranked knowledge and capability elements, aggregate per-model scores over an Index Set, and route by weighted sum. That is sensible, and the dynamic-pool result is a nice property.

The RouteMix dataset—20 benchmarks for indexing plus four for evaluation—is a real asset for the community. The rank-weighted aggregation formula is a reasonable way to combine multiple knowledge elements, and the paper is refreshingly clear about how new models can be added by evaluating on the Index Set.

The problems are in the experimental validation. Most importantly, there is no comparison to any existing router—not RouteLLM, not Model-SAT, not RouterDC. Without those baselines, we cannot tell whether this is better than prior work or just a new heuristic. Second, there are no error bars or significance tests. The average gain of 1.28 points comes from winning on GPQA and LiveBench and losing on MMLU-Pro and BigGenBench; with no variance estimates, that margin could easily be noise. Third, the paper calls the method parameter-free, but alpha, beta, gamma, delta, the cosine threshold, and the frequency cutoff are all hand-chosen. They should be tuned on a validation split, or at least acknowledged as hyperparameters.

The stress-test concern about the 'Other' bucket is valid. The paper says low-frequency and unseen knowledge elements become 'Other', but never defines the score assigned to 'Other'. If many evaluation queries hit this bucket, the knowledge score becomes a generic average and the routing advantage has little to do with knowledge profiling. The authors need to either report the fraction of queries routed through 'Other' or show that the knowledge scores are still discriminative. The OOD claim is also weakened by the fact that MMLU-Pro and LiveBench overlap with Index Set benchmarks.

Overall, the idea is sound and the dataset is valuable, but the evidence needs a substantial upgrade. I would send it out to a serious referee, with the expectation of major revisions. If the authors add baselines, uncertainty quantification, and clarify the 'Other' handling, this could be a useful contribution to the routing literature.

Referee Report

5 major / 6 minor

Summary. The paper proposes InferenceDynamics, an LLM routing framework that profiles candidate models along capability and knowledge dimensions extracted by an auxiliary LLM (GPT-4o-mini). The method computes per-query, per-element performance scores on an Index Set of 20 benchmarks, aggregates them into knowledge and capability scores for each model, and routes each unseen query by the weighted combination in Eq. (2). The authors introduce the RouteMix dataset (Index Set plus four OOD evaluation benchmarks) and report that their Mixed Routing strategy beats the best single LLM by 1.28 points on average while using about 80% of the cost (or about half under a cost penalty). They also present a dynamic-pool scalability experiment and an analysis of router-selected models.

Significance. If fully substantiated, InferenceDynamics would be a useful addition to LLM routing: it is training-free, naturally accommodates new models, and its knowledge/capability decomposition is well motivated. The paper contributes RouteMix, a broad 24-dataset benchmark, and provides model-selection and dynamic-pool analyses that are informative. However, the current evaluation does not compare against any existing routing algorithm, lacks error bars or significance tests, and leaves a central mechanism (the 'Other' knowledge bucket) undefined. These gaps mean that the headline 1.28-point improvement over the best single model is not yet established as robust or as evidence of knowledge-based OOD generalization.

major comments (5)

[§4.2 / Eq. (1)] The treatment of 'Other' knowledge is never specified. §4.2 states that knowledge entries with frequency below 10 and previously unseen knowledge are classified as 'Other', but the paper does not define how Eq. (1) accounts for 'Other' nor report how often evaluation queries fall into this bucket. If 'Other' is simply dropped, then queries whose extracted knowledge is all unseen yield zero knowledge scores for every model, and routing is decided by the capability term alone. In that case the reported 1.28-point gain could reflect generic capability routing rather than knowledge-aware routing. Please define the score for 'Other' and report the coverage of matched knowledge elements on each evaluation benchmark.
[§4.4 / Table 1] The experiments compare InferenceDynamics only against single LLMs and random routing; no existing routing algorithm (RouteLLM, RouterDC, C2MAB-V, P2L, Model-SAT, or EmbedLLM) is used as a baseline. The related work motivates the novelty by contrast with these methods, but without a numerical comparison the claimed advantage over the state of the art in routing is unquantified. Additionally, every result appears to come from a single run, with no error bars, confidence intervals, or significance tests. The 1.28-point margin over Gemini-1.5-Pro could be within sampling variability, particularly on LiveBench where the reported score is an average over six categories.
[§4.1.1 / §4.2] The label 'parameter-free' is inaccurate. The routing decision depends on the hyperparameters α, β, γ, δ, on the cosine similarity threshold 0.6 for knowledge consolidation, on the frequency cutoff 10 for the 'Other' bucket, and on the performance thresholds for translation/summarization listed in Appendix A. The paper does not describe how these values were chosen or whether a held-out validation set was used. If they were tuned with knowledge of the evaluation benchmarks, the out-of-distribution claim is weakened; if they are arbitrary defaults, their sensitivity should be shown through ablations.
[§3.4 / §4.1.2] The claim of generalization to out-of-distribution queries is not validated for genuinely novel knowledge elements. The Index Set contains MMLU, MATH-500, GSM8K, and BigCodeBench, while the evaluation set includes MMLU-Pro and LiveBench with largely overlapping categories; GPQA and BigGenBench may also share topical structure. The paper provides no analysis of the fraction of evaluation queries whose extracted knowledge elements match the Index Set only through the 'Other' bucket or fall below the cosine threshold. Without such an analysis, the reported improvement could be driven by near-duplicate topics instead of transfer to new knowledge domains.
[§4.5 / Fig. 3] The cost-constrained results are not reproducible as reported. The Cost Ratio is defined as total cost relative to the best-performing single LLM, but the paper does not specify the cost model, token counts, or API pricing used to compute it. The reader cannot determine whether the 'approximately half the budget' figure includes the router's knowledge/capability generation overhead or only the inference cost of the selected model. Please provide the exact cost computation and the numeric performance/cost values behind Fig. 3.

minor comments (6)

[§3.2 / Appx. D] The list of capabilities in §3.2 includes seven items, but the generation prompt in Appendix D lists eight, adding 'In-context Learning' between Coding and Multilingual; these should be made consistent.
[Fig. 3] The x-axis of Figure 3 appears to show the β penalty coefficient, but the axis label and caption mention 'Increasing Tasks'; please correct the label to match the actual experimental variable.
[Table 1] The 'Random' routing baseline is given as a single point estimate; reporting the mean and standard deviation over multiple random seeds would provide a reference scale for interpreting the 1.28-point improvement.
[Appx. A] The performance threshold for translation and summarization tasks is mentioned but never quantified; please state the actual threshold values or the rule used to determine correctness.
[Fig. 4] The figure caption contains 'inRouteMix' (missing space) and the label 'IDOOD' in the figure is unclear; rephrase the caption and clarify the ID/OOD split.
[General] The code availability statement says the source code 'will be made publicly available'; for a journal submission, an anonymous repository link or a review-stage artifact would allow the evaluation to be verified.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: routing scores are derived from the independent Index Set, and the evaluation benchmarks are external to the scoring construction.

full rationale

The central derivation chain is not circular. Routing scores are built exclusively from the Index Set: per-query scores s_i^t are measured by evaluating each candidate model on D_index (Sec. 3.3), knowledge/capability labels are generated by GPT-4o-mini, and the aggregate scores S^alpha_beta(M_t, D, k) average observed Index Set performances over queries sharing element k. An unseen evaluation query x enters Eq. (1) only through the predicted label sets K_x; the model's performance on x is never used to construct the score. The four evaluation benchmarks (MMLU-Pro, GPQA, BigGenBench, LiveBench) are held out and reported as OOD, so the 1.28-point average improvement over the best single LLM is an independent empirical result, not a quantity that Eq. (1) reproduces by definition. The one self-citation in the paper (Wang et al. 2024a, in the related-work discussion of multi-agent systems) is contextual and not load-bearing. The hand-chosen hyperparameters alpha, beta, gamma, delta, the knowledge deduplication and frequency thresholds, and the unspecified treatment of 'Other' knowledge are validity and robustness concerns (e.g., possible mild post hoc selection or weak transfer to truly novel knowledge), but they do not amount to circularity: no fitted parameter is renamed as a prediction, and no equation is equivalent to its input by construction.

Assumptions & free parameters 7 free parameters · 4 assumptions · 1 invented entities

The central routing scheme depends on four hand-set hyperparameters, two thresholds, an unreported binarization threshold, and an unreported number of trials, alongside assumptions that the predefined capability taxonomy and the 'Other' knowledge bucket are sufficient. These choices are not validated on held-out data, which inflates the apparent generality of the method.

free parameters (7)

alpha (α) = 0.5 (default)
Rank attenuation factor controlling how quickly lower-ranked knowledge and capability elements lose influence. Chosen by hand in Section 4.2 and not tuned on a validation set.
beta (β) = 0 (default); varied 5-25 in Section 4.5
Cost penalty coefficient. Introduced in Section 3.3; its value is the main knob in the cost-constrained experiments.
gamma (γ) and delta (δ) = 1.0 each
Weights for knowledge and capability scores in the final routing decision (Equation 2). Set by default and not optimized.
Cosine similarity threshold = 0.6
Used in Section 4.2 to merge semantically similar knowledge entries. Value chosen by hand without sensitivity analysis.
Knowledge frequency threshold = 10
Knowledge entries with frequency lower than 10 are mapped to 'Other'. Chosen in Section 4.2 without justification.
Performance threshold for translation and summarization = not reported
Mentioned in Appendix A to binarize outputs for translation and summarization metrics; the threshold value is not specified.
Number of trials K = not reported
The score s_t_i is an average over K independent trials in Section 3.3; K is not reported, so the variance of the scores is unknown.

assumptions (4)

domain assumption The capability taxonomy from Minaee et al. (2024) is sufficient to capture the dimensions along which models differ for routing.
Section 3.2 and the prompt in Appendix D constrain the capability labels to eight predefined categories, excluding any capability not in that list.
ad hoc to paper The knowledge space is unbounded, but the labels produced by GPT-4o-mini, after deduplication and the 'Other' bucket, preserve the information needed for routing.
Section 4.2: low-frequency and unseen knowledge is folded into 'Other', assuming that this aggregated representation still discriminates between models.
ad hoc to paper The weighted rank aggregation with geometric decay (α=0.5) is an appropriate functional form for combining per-element scores.
Equation (1) and Section 3.3 define the aggregation; this form is assumed rather than derived or validated.
domain assumption The aggregate scores computed on the Index Set transfer to unseen OOD queries that share knowledge or capability labels.
This is the core transfer assumption behind Equation (2); the paper does not test what happens when a query's knowledge is entirely novel.

invented entities (1)

'Other' knowledge category
purpose: A catch-all bucket for knowledge elements that appear fewer than 10 times in the Index Set or are unseen at test time.
Section 4.2 introduces this category; it posits that all rare or novel knowledge can be represented by the models' aggregate performance on the pooled rare-knowledge subset, without providing evidence that this preserves routing signal.

how reviews work

0 comments

Cite this review

Pith. "Pith review of INFERENCEDYNAMICS: Efficient Routing Across LLMs through Structured Capability and Knowledge Profiling." pith.science (2026). https://pith.science/paper/WCS4PX7I

@misc{pith2026250516303,
  author       = {Pith},
  title        = {Pith review of: INFERENCEDYNAMICS: Efficient Routing Across LLMs through Structured Capability and Knowledge Profiling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WCS4PX7I}},
  note         = {Machine review of arXiv:2505.16303}
}

read the original abstract

Large Language Model (LLM) routing is a pivotal technique for navigating a diverse landscape of LLMs, aiming to select the best-performing LLMs tailored to the domains of user queries, while managing computational resources. However, current routing approaches often face limitations in scalability when dealing with a large pool of specialized LLMs, or in their adaptability to extending model scope and evolving capability domains. To overcome those challenges, we propose InferenceDynamics, a flexible and scalable multi-dimensional routing framework by modeling the capability and knowledge of models. We operate it on our comprehensive dataset RouteMix, and demonstrate its effectiveness and generalizability in group-level routing using modern benchmarks including MMLU-Pro, GPQA, BigGenBench, and LiveBench, showcasing its ability to identify and leverage top-performing models for given tasks, leading to superior outcomes with efficient resource utilization. The broader adoption of Inference Dynamics can empower users to harness the full specialized potential of the LLM ecosystem, and our code will be made publicly available to encourage further research.

Figures

Figures reproduced from arXiv: 2505.16303 by the authors.

**Figure 2.** LLM performances across 20 datasets in RouteMix. Dataset labels including "PlanBench" indicate subsets of the PlanBench benchmark. For detailed metric information, refer to Appx. §A. In this formulation, the hyperparameter α serves to attenuate the influence of less critical knowledge elements, based on their rank j. The hyperparameter β acts as a coefficient penalizing higher computational costs. The denominator,… view at source ↗

**Figure 3.** Performance Ratio (%) and Cost Ratio (%) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Distribution of knowledge domains across 24 datasets in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Comparative distribution of router-selected models. Lighter colors signify a higher selection ratio for a [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Routing Performance (%) in Dynamic LLM Pools. 5.2 Knowledge Distribution As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

VDAR-Router: Adaptive LLMs Routing via Verbalized Query Difficulty Analysis Retrieval
cs.CL 2026-07 conditional novelty 6.0 of 10

VDAR-Router improves LLM routing by retrieving past queries with similar verbalized difficulty descriptions and ranking models by performance minus cost.
Context Reasoner: Incentivizing Reasoning Capability for Contextualized Privacy and Safety Compliance via Reinforcement Learning
cs.CL 2025-05 conditional novelty 5.0 of 10

A Context Reasoner pipeline that cold-starts LLMs on distilled legal reasoning and applies PPO with a rule-based compliance reward improves performance on CI-based legal compliance benchmarks and transfers to general ...

Reference graph

Works this paper leans on

13 extracted references · 5 canonical work pages · cited by 2 Pith papers

[1]

Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general
[2]

You should refer to the score rubric

After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric
[3]

Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5)

The output format should look as follows: "Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5)"
[4]

(...) [RESULT] <score>

Please do not generate any other opening, closing, and explanations. The instruction to evaluation: example question Response to evaluate: example solution Reference Answer (Score 5): reference score Score Rubrics: Criteria: criteria Description of a Score 1 response: score1 description Description of a Score 2 response: score2 description Description of ...

work page
[6]

CoRR, abs/2406.12793

Chatglm: A family of large language mod- 12 els from GLM-130B to GLM-4 all tools. CoRR, abs/2406.12793. Kai Zhang, Liqian Peng, Congchao Wang, Alec Go, and Xiaozhong Liu. 2024. LLM cascade with multi-objective optimal consideration. CoRR, abs/2410.08014. Yi-Kai Zhang, Ting-Ji Huang, Yao-Xiang Ding, De- Chuan Zhan, and Han-Jia Ye. 2023. Model spider: Learn...

arXiv 2024
[7]

arXiv preprint arXiv:2406.15877

Bigcodebench: Benchmarking code genera- tion with diverse function calls and complex instruc- tions. arXiv preprint arXiv:2406.15877. 13 A Benchmark Overview Table Table 2: Overview of Benchmarks, Data Processing, Prompts, and Metrics Benchmark Name Data Processing Manner Prompt Type Metric Used ARC (Clark et al., 2018) Sample 500 instances according to t...

arXiv 2025
[12]

Identify and list the *LLM Capabilities* from the definitions above that are directly and significantly re- quired to effectively address the query

work page
[13]

List the selected Capabilities first, ranked from most important to least important

Identify and list the general *Knowledge Domains* (e.g., categories, subject areas) most pertinent to solving the problem presented in the query. List the selected Capabilities first, ranked from most important to least important. Then, list the identified Knowledge Domains, also ranked from most important to least important. *Do not provide any justifica...

work page 2016

Show all 13 references

[2018]

In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, Brussels, Belgium, October 31 - November 4, 2018 , pages 1797–1807

Don’t give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, Brussels, Belgium, October 31 - November 4, 2018 , pages 1797–1807. ...

2018
[2020]

CoRR, abs/2009.03300

Measuring massive multitask language under- standing. CoRR, abs/2009.03300. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Ja- cob Steinhardt. 2021. Measuring mathematical prob- lem solving with the MATH dataset. In Proceedin...

2009 arXiv
[2021]

CoRR, abs/2110.14168

Training verifiers to solve math word prob- lems. CoRR, abs/2110.14168. Xiangxiang Dai, Jin Li, Xutong Liu, Anqi Yu, and John C. S. Lui. 2024. Cost-effective online multi- llm selection with versatile reward models. CoRR, abs/2405.16587. Dujian Ding, Ankur Mallick, Chi Wang, R...

2024 arXiv
[2023]

Legalbench: A collaboratively built bench- mark for measuring legal reasoning in large language models. In Advances in Neural Information Pro- cessing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -...

2023
[2024]

CoRR, abs/2403.05530

Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. CoRR, abs/2403.05530. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R. Bowman. 2023. GPQA: A graduate-level...

2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general

[2] [2]

You should refer to the score rubric

After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric

[3] [3]

Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5)

The output format should look as follows: "Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5)"

[4] [4]

(...) [RESULT] <score>

Please do not generate any other opening, closing, and explanations. The instruction to evaluation: example question Response to evaluate: example solution Reference Answer (Score 5): reference score Score Rubrics: Criteria: criteria Description of a Score 1 response: score1 description Description of a Score 2 response: score2 description Description of ...

work page

[5] [6]

CoRR, abs/2406.12793

Chatglm: A family of large language mod- 12 els from GLM-130B to GLM-4 all tools. CoRR, abs/2406.12793. Kai Zhang, Liqian Peng, Congchao Wang, Alec Go, and Xiaozhong Liu. 2024. LLM cascade with multi-objective optimal consideration. CoRR, abs/2410.08014. Yi-Kai Zhang, Ting-Ji Huang, Yao-Xiang Ding, De- Chuan Zhan, and Han-Jia Ye. 2023. Model spider: Learn...

arXiv 2024

[6] [7]

arXiv preprint arXiv:2406.15877

Bigcodebench: Benchmarking code genera- tion with diverse function calls and complex instruc- tions. arXiv preprint arXiv:2406.15877. 13 A Benchmark Overview Table Table 2: Overview of Benchmarks, Data Processing, Prompts, and Metrics Benchmark Name Data Processing Manner Prompt Type Metric Used ARC (Clark et al., 2018) Sample 500 instances according to t...

arXiv 2025

[7] [12]

Identify and list the *LLM Capabilities* from the definitions above that are directly and significantly re- quired to effectively address the query

work page

[8] [13]

List the selected Capabilities first, ranked from most important to least important

Identify and list the general *Knowledge Domains* (e.g., categories, subject areas) most pertinent to solving the problem presented in the query. List the selected Capabilities first, ranked from most important to least important. Then, list the identified Knowledge Domains, also ranked from most important to least important. *Do not provide any justifica...

work page 2016