Pith. sign in

REVIEW 3 major objections 8 minor 91 references

StaICC: Standardized Evaluation for Classification Task in In-context Learning

T0 review · 3 major / 8 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes StaICC, a standardized evaluation toolkit for in-context classification, and claims that on 29 modern language models it reveals clean log-linear scaling laws of classification performance with model parameter count.

desk verdict Useful ICL evaluation toolkit with overclaimed scaling-law validation. read the letter →

arxiv 2501.15708 v3 pith:MN5D2QMU submitted 2025-01-27 cs.CL cs.AI

Hakaze Cho , Naoya Inoue This is my paper · ORCID
classification cs.CLcs.AI
keywords in-contextlearningclassificationbenchmarkevaluationtoolkitprompttemplatesensitivitylabel-noiserobustnessscalinglawsprobabilitycalibrationdiagnostic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

In-context learning papers report widely varying numbers for the same model and dataset because datasets, prompt templates, demonstration order, and data splits all differ. StaICC is a proposed fix: ten common single-sentence classification datasets, a fixed meta-template with dataset-specific connectors and verbalizers, a frozen demonstration sampler, and stable-over-trial inputs. The paper's central claim is that, under this fixed setting, the true-label probability of 29 modern language models follows a clear log-linear scaling law against parameter count, confirming that the benchmark measures a real capability rather than template artifacts. A companion diagnostic benchmark measures prediction bias, template and sampling robustness, and label-noise sensitivity. If the claim holds, papers can compare models and ICL-improving methods on common ground and meta-analyze results across studies.

What carries the argument

The load-bearing object is the meta-template: a fixed prompt skeleton with exchangeable attributes (instruction, x-prefix, y-prefix, y-affix, query prefix, label space), instantiated with dataset-specific connectors and verbalizers. Around it sit a frozen pre-processer that splits each raw dataset into calibration, demonstration, and test sets, and a frozen demonstration sampler guaranteeing zero variance across trials. StaICC-Normal applies this to 10 datasets with four metrics (accuracy, true-label probability, macro-F1, ECE-1); StaICC-Diag reuses the meta-template to construct pseudo-query bias probes, nine template variants, eight demonstration samples, and label-noise corrupted demonstrations, yielding entropy, consistency, and GLER slope diagnostics. The scaling-law check is the argument that the fixed template is faithful: if larger models perform better under it, the benchmark is presumed discriminative.

What would settle it

Re-run the StaICC-Normal evaluation with the same 29 models but replace only the default template with several alternative natural-language templates; if the log-linear scaling law disappears or the model ranking changes substantially across templates, the scaling-law validation would fail. A simpler direct check is to find a smaller model that consistently outperforms a larger model across all 10 datasets under the default template, which would contradict the claimed clean scaling relation.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes StaICC-Normal as a standardized benchmark and StaICC-Diag as a diagnostic companion. Using the fixed default template, the paper measures accuracy, true-label probability, macro-F1, and ECE-1 over 29 LMs and reports $R^2 = 0.692$ for the log-linear fit of true-label probability against model parameters, treating this scaling law as evidence that the simple default template is discriminative. It further reports that ICL-improving methods such as SA-ICL and Hidden Calibration outperform more often across models, while no scaling law appears for the accuracy improvements these methods provide. On the diagnostic side, contextual and domainal bias decrease and template and sampling robustness increase with scale, but label-noise sensitivity (GLER) increases, meaning larger models are more affected by falsified demonstration labels.

Load-bearing premise

The central assumption is that the single fixed default prompt template, a simple meta-template with dataset-specific connectors, is representative enough to compare models and methods fairly; if that template obscures real differences, the reported rankings and the scaling law used to validate the benchmark would be artifacts of the template choice.

Editorial extensions

If this is right

  • Any ICL classification paper can report results on StaICC and be directly comparable with the 29-model baselines, removing the need to rerun baselines in inconsistent settings.
  • The observed log-linear scaling law means classification ICL performance is predictable from parameter count under this fixed setting, at least within the tested model families.
  • The absence of a scaling law for ICL-improving methods implies their gains are model-size-dependent and must be benchmarked per model rather than extrapolated to larger scales.
  • Larger models' higher GLER indicates label-noise sensitivity grows with scale, so demonstrations must be curated more carefully for large LMs.
  • Diagnostic sub-metrics separate bias from robustness, so a method can be optimized for faithful probabilities (lower ECE-1) without necessarily improving accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: adopting StaICC as a community standard would also standardize one family of simple prompt templates, so the benchmark would benefit from periodic revalidation against alternative templates to prevent entrenching template-specific rankings.
  • The reported weak correlation between accuracy and contextual or domainal bias implies that calibration methods should be judged primarily by calibration metrics rather than accuracy; this is a direct extension of the paper's correlation analysis and could be tested by comparing methods on ECE-1.
  • The GLER result suggests a testable extension: if demonstrations for larger models are filtered to high-confidence labels, the scale-dependent accuracy drop under label noise should be reduced.
  • The toolkit opens interfaces to modify all conditions, so a natural next step would be reporting model rankings averaged over the nine template variants from the diagnostic, which the paper does not currently present as a primary ranking.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper proposes StaICC, a standardized evaluation toolkit for in-context classification, consisting of StaICC-Normal (10 widely used single-sentence classification datasets with a fixed prompt template, data split, and demonstration order) and StaICC-Diag (diagnostic tasks for prediction bias, prompt sensitivity, sampling robustness, and label-noise robustness). The authors evaluate 29 language models and 9 ICL-improving methods, reporting accuracy, TLP, Macro-F1, ECE-1, and several diagnostic metrics. Their central claim is that the observed log-linear scaling law of TLP against model parameters confirms the discriminability and robustness of StaICC. The paper also reports method-wise comparisons and a multivariate correlation analysis of metrics.

Significance. If the evaluation protocol is reliable, StaICC addresses a genuine need: it provides a concrete, reproducible, and stable-over-trial setting for in-context classification, mitigating the cross-paper inconsistencies documented in Table 1 and Fig. 1. The release of the toolkit, the fixed splits, and the explicit measurement of 29 models and 9 methods are practical contributions that many ICL researchers would use. The diagnostic sub-benchmark is a useful addition, and the paper is honest about its limitations and about the non-transferability of optimal templates. However, the central validation of the benchmark via the scaling-law check is currently the weakest link, because it is based on a single fixed template and a moderate R² value over a heterogeneous model set, and because the paper itself provides evidence (Fig. 2, StaICC-Diag) that template variation has large effects.

major comments (3)
  1. [Section 3.1, Fig. 4] The validation of StaICC is circular: the scaling law is produced by running StaICC and is then used to 'confirm its discriminability.' As stated in Section 3.1, 'we will calibrate it through scaling laws,' which does not provide an external criterion. Moreover, the evidence for a 'clear' scaling law is a single log-linear fit to TLP over 29 heterogeneous points (R²=0.692), which is moderate and computed across different model families, base and instruction-tuned variants, and AWQ-quantized large models. This does not establish that the fixed template is discriminative. A concrete test is available within the paper: StaICC-Diag already evaluates 9 prompt templates per dataset. The authors should report the scaling-law fit (slope and R²) for TLP or Macro-F1 for each of the 9 templates, or a per-model-family breakdown, and show that the scaling relationship is stable. Without such evidence, the central claim that the scaling law 'confirms the robustness of StaICC' is not supported.
  2. [Abstract and Section 3.2] The wording 'clear scaling laws of ICL classification performance against the model parameter numbers' overstates the reported results. The paper only shows a scaling-law plot for TLP (Fig. 4), not for Accuracy or Macro-F1, which are the headline classification metrics in Table 2. The diagnostic results in Fig. 7 show R² values of 0.003, 0.038, 0.026, and 0.087 for Contextual Bias, Domain Bias, Template Robustness, and Sample Robustness, respectively, which are effectively zero. Even the TLP fit at R²=0.692 leaves substantial unexplained variance. The authors should either report scaling fits for all StaICC-Normal metrics (Accuracy, TLP, Macro-F1, ECE-1) or qualify the claim to refer specifically to TLP rather than 'ICL classification performance' in general.
  3. [Section 2.2, Fig. 2; Section 3.3] The paper's own evidence undermines the assumption that a single fixed template is a neutral or representative choice. Fig. 2 shows that the connector variation can push Macro-F1 of Falcon-7B-Instruct down to the level of Pythia-160M, and the StaICC-Diag template-robustness consistency is only 80–90% across 9 templates. The fixed default template in StaICC-Normal is one arbitrary point in this space, so the scaling law observed on it may be an artifact of that choice. The paper cites Voronov et al. (2024) for exactly this phenomenon but does not quantify how much discriminability is retained by the chosen template. A minimum requirement is to report the ranking stability of the 29 models across the 9 template variants, or at least to show that the scaling slope does not change sign or vanish when the template is varied.
minor comments (8)
  1. [Section 3.1] The heading contains a typo: 'Methdology' should be 'Methodology'.
  2. [Section 1.1] 'theroically' should be 'theoretically' in the sentence 'some works try to find the principle of ICL theroically'.
  3. [Fig. 4 caption] The legend categories 'Normal', 'AWQ', and 'Instruct' are not defined in the caption; please clarify what each category denotes and how the R² value is computed.
  4. [Table 3] The 'Cons.(%)' column label for Template Robustness and Sample Robustness is ambiguous; the metric is the average majority-prediction rate across templates or demonstration samples, not a standard consistency coefficient. Please define the calculation in the caption.
  5. [Fig. 8] The correlation matrix appears as plain text (e.g., 'Corr: 0.954***') rather than a formatted figure; as rendered, the figure is hard to read and the layout is confusing. A standard heatmap with numeric annotations would be clearer.
  6. [Section 3.3, GLER] The GLER metric is described as the slope of a linear regression of accuracy against label-noise rate p, but Table 3 labels it as 'Slope/0.1'. Please clarify the units and how the linear regression is performed (e.g., are the five p values treated as equally spaced?).
  7. [Table 8] The augmented results table has column headers concatenated without spaces (e.g., 'ModelGPT2Llama2GPT2'); this is a formatting artifact that should be fixed in the camera-ready version.
  8. [Section 3.1] The phrase 'we will calibrate it through scaling laws' is informal and could be misinterpreted; use 'validate' instead of 'calibrate' to avoid confusion with probability calibration methods.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the scaling-law validation of StaICC is an internal but falsifiable sanity check, and the paper's self-citations are not load-bearing for the benchmark's validity.

full rationale

The paper's central validation move — using a log-linear scaling law of True Label Probability against model size to 'preliminarily confirm' StaICC's discriminability — is an internal consistency check with real falsifiable content, not a reduction by construction. The authors state that if larger models did not perform better on StaICC, the benchmark's discriminability would be in doubt (Section 3.1). That conditional criterion is not an equation into which the conclusion is substituted; it is an empirical pattern that could have failed. No parameter is fitted to a subset of data and then reported as a prediction, and no definition ties StaICC's robustness to the scaling-law outcome. The benchmark's datasets, templates, and metrics are specified independently of the later measurements. The only self-citations — Cho et al. (2024) for Hidden Calibration as an evaluated method and Cho et al. (2025) for agreement with the GLER result — are experimental comparisons or literature alignment, and neither carries the burden of justifying the benchmark's validity. They are also externally checkable results, not unverified premises. The paper openly concedes the limits of its own contribution ('dirty works', small novelty), which further undercuts any claim that a self-referential authority is being used to force a conclusion. Under the required standard of exhibiting a specific reduction or a fitted-input-called-prediction step, no such step is present here. The main weakness — that a monotone model-size trend does not by itself prove template discriminability — is a question of evidential strength, not circularity, and is already acknowledged by the paper's own 'preliminarily' qualifier.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The benchmark rests on manually chosen settings (k, splits, templates, method budgets) rather than on fitted parameters. No parameter is tuned to produce the scaling law; the circularity comes from using that scaling law to validate the benchmark itself. No new entities are introduced.

free parameters (7)
  • k (number of demonstrations) = 4 (implied by GLER example, p=0.5, k=4)
    Manual design choice; all benchmark results depend on how many demonstrations are included.
  • Calibration/demonstration/test split sizes = 1024/4096 (FP:512)/512 per dataset (Table 5)
    Chosen by hand to define the data regime; affects every metric.
  • Additional data budget for calibration methods = 128 samples
    Set for fair comparison among four methods that require extra inference; changing this budget could change rankings.
  • Batch size for Batch Calibration = 128
    Hyperparameter for one evaluated baseline; it influences that method's results.
  • Sample length for Domain Calibration = 64 tokens
    Hyperparameter for the Domain Calibration baseline.
  • Number of bins for ECE-1 = 10
    Standard but arbitrary binning for the calibration metric.
  • Label noise rates for GLER = 5 uniformly sampled p in [0,1]
    The GLER slope depends on these sampled noise rates.
assumptions (5)
  • domain assumption The probability of the next token over the verbalizer label set is a valid classification output distribution.
    Used to define o(xq) in Eq. 1; ignores tokenization or vocabulary biases except through the methods evaluated.
  • ad hoc to paper The simplest valid prompt template is a faithful, representative baseline for evaluating ICL.
    Section 3.1: 'we follow Occam's Razor, using the simplest but valid template'; this is a design choice, not a proven optimality.
  • ad hoc to paper Observing scaling laws on a benchmark is sufficient to confirm its discriminability.
    Section 3.1: 'if we find larger models perform better on StaICC, we can preliminarily confirm its discriminability'; assumes scaling is a benchmark-quality test rather than a property of the tasks.
  • ad hoc to paper The 10 selected datasets are representative of in-context classification tasks.
    Dataset selection based on usage frequency in Table 1; representativeness is asserted and limited to single-sentence English classification.
  • domain assumption MAUVE score is an appropriate measure of sampling unbiasedness.
    Appendix B.2 uses MAUVE=0.9802/0.9808 to claim close distributions between splits.

how reviews work

0 comments
Cite this review

Pith. "Pith review of StaICC: Standardized Evaluation for Classification Task in In-context Learning." pith.science (2026). https://pith.science/paper/MN5D2QMU

@misc{pith2026250115708,
  author       = {Pith},
  title        = {Pith review of: StaICC: Standardized Evaluation for Classification Task in In-context Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MN5D2QMU}},
  note         = {Machine review of arXiv:2501.15708}
}
read the original abstract

Classification tasks are widely investigated in the In-Context Learning (ICL) paradigm. However, current efforts are evaluated on disjoint benchmarks and settings, while their performances are significantly influenced by some trivial variables, such as prompt templates, data sampling, instructions, etc., which leads to significant inconsistencies in the results reported across various literature, preventing fair comparison or meta-analysis across different papers. Therefore, this paper proposes a standardized and easy-to-use evaluation toolkit (StaICC) for in-context classification. Including, for the normal classification task, we provide StaICC-Normal, selecting 10 widely used datasets, and generating prompts with a fixed form, to mitigate the variance among the experiment implementations. To enrich the usage of our benchmark, we also provide a sub-benchmark StaICC-Diag for diagnosing ICL from several aspects, aiming for a more robust inference processing.

Figures

Figures reproduced from arXiv: 2501.15708 by the authors.

Figure 1
Figure 1. Summaries of experimental results in the literature: ICL results are disjoint in the literature. Left: Accuracies of various models with different sizes on specified datasets, where the results do not comply with the scaling laws w.r.t. the model scaling. Right: Accuracies of the same model (GPT2-XL) on vanilla ICL inference from different papers, where results have a considerable range even on the same model. tasks… view at source ↗
Figure 2
Figure 2. The influence of various prompt variables on ICL. Details are shown in Appendix B.2. This section claims our proposition: the inconsistency in experimen￾tal settings leading to inconsistent results hinders comparison and meta-analysis across the literature. We analyze how the inconsis￾tency happens so that we can propose principles for our benchmark to reduce these inconsistencies. The disjoint in dataset selection.… view at source ↗
Figure 3
Figure 3. Major schematic diagram of data pre-processing and input forming in StaICC. Raw datasets are first divided into 3 sub-sets with a frozen pre-processer for calibration, demonstration, and query. Based on these sub-sets, ICL inputs are built with a frozen demonstration sampler, and a frozen prompt template under a meta-template, shown in gray for an example. The disjoint in numerical results. We also investigate the d… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: TLP results against model parameter numbers. As the main objective of this paper, to test the basic prediction capacity of an ICC inference o(xq), we build StaICC-Normal with standard input form and typical metrics for classification tasks. Metric. We apply 4 metrics f…
Figure 5
Figure 5. Figure 5: Absolute accuracy (%) improvement of various ICL-improving methods against model parameter numbers [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Evaluation process for (left) contextual bias, where empty pseudo queries (x˙) are included in the input; and (right) domainal bias, where pseudo queries (x˙) consist of token sequences i.i.d. sampled from the token distribution of all real instances x. Entropy is calc…
Figure 7
Figure 7. Figure 7: Diagnostic results against model parameter numbers [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Multivariate covariate analysis on the metrics in this paper. Spearman correlation is calculated. Rethinking the influence of bias and robust￾ness of ICL. We visualize the statistical covariate among the metrics evaluated in this paper, shown in [PITH_FULL_IMAGE:figur…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

91 extracted references · 22 canonical work pages

  1. [1]

    Qwen2 technical report. 2024. URL http://arxiv.org/abs/2407.10671v4

    arXiv 2024

  2. [2]

    Enhancing In-context Learning via Linear Probe Calibration

    Momin Abbas, Yi Zhou, Parikshit Ram, Nathalie Baracaldo, Horst Samulowitz, Theodoros Salonidis, and Tianyi Chen. Enhancing in-context learning via linear probe calibration. In International Conference on Artificial Intelligence and Statistics, pp.\ 307--315. PMLR, 2024. URL http://arxiv.org/abs/2401.12406v1

    work page Pith review arXiv 2024

  3. [3]

    Falcon-40B : an open large language model with state-of-the-art performance

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B : an open large language model with state-of-the-art performance. 2023. URL https://huggingface.co/tiiuae/falcon-40b

    2023

  4. [4]

    Dbpedia: A nucleus for a web of open data

    S \"o ren Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: A nucleus for a web of open data. In international semantic web conference, pp.\ 722--735. Springer, 2007. URL https://link.springer.com/content/pdf/10.1007/978-3-540-76298-0_52.pdf

    work page doi:10.1007/978-3-540-76298-0_52.pdf 2007

  5. [5]

    Explaining neural scaling laws

    Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121 0 (27): 0 e2311878121, 2024. URL http://dx.doi.org/10.1073/pnas.2311878121

    doi:10.1073/pnas.2311878121 2024

  6. [6]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  7. [7]

    S em E val-2019 task 5: Multilingual detection of hate speech against immigrants and women in T witter

    Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. S em E val-2019 task 5: Multilingual detection of hate speech against immigrants and women in T witter. In Jonathan May, Ekaterina Shutova, Aurelie Herbelot, Xiaodan Zhu, Marianna Apidianaki, and Saif M. Moh...

    2019

  8. [8]

    The fifth pascal recognizing textual entailment challenge

    Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7: 0 8, 2009. URL https://tac.nist.gov/publications/2009/additional.papers/RTE5_overview.proceedings.pdf

    2009

Show all 91 references
  1. [9]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In Intern...

    2023 arXiv

  2. [10]

    GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow , March 2021

    Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow , March 2021. URL https://doi.org/10.5281/zenodo.5297715. If you use this software, please cite it using these metadata

    2021 doi

  3. [11]

    Icl-markup: Structuring in-context learning using soft-token tags

    Marc-Etienne Brunet, Ashton Anderson, and Richard Zemel. Icl-markup: Structuring in-context learning using soft-token tags. In R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023. URL https://www.semanticscholar.org/paper/1eddc4e366323423098...

    2023

  4. [12]

    On the relation between sensitivity and accuracy in in-context learning

    Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. On the relation between sensitivity and accuracy in in-context learning. In 2023 Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 155--167. Association for Computational Linguistics (ACL), ...

    2023 arXiv

  5. [13]

    Token-based decision criteria are suboptimal in in-context learning

    Hakaze Cho, Yoshihiro Sakai, Mariko Kato, Kenshiro Tanaka, Akira Ishii, and Naoya Inoue. Token-based decision criteria are suboptimal in in-context learning. arXiv preprint arXiv:2406.16535, 2024. URL http://arxiv.org/abs/2406.16535v2

    2024 arXiv

  6. [14]

    Revisiting in-context learning inference circuit in large language models

    Hakaze Cho, Mariko Kato, Yoshihiro Sakai, and Naoya Inoue. Revisiting in-context learning inference circuit in large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2410.04468

    2025 arXiv

  7. [15]

    In-context learning with transformers: Softmax attention adapts to function lipschitzness

    Liam Collins, Advait Parulekar, Aryan Mokhtari, Sujay Sanghavi, and Sanjay Shakkottai. In-context learning with transformers: Softmax attention adapts to function lipschitzness. arXiv preprint arXiv:2402.11639, 2024. URL https://arxiv.org/abs/2402.11639

    2024 arXiv

  8. [16]

    The pascal recognising textual entailment challenge

    Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pp.\ 177--190. Springer, 2005. URL https://www.semanticscholar.org/paper/e808f28d411a958c5db81ceb111beb2638698f47

    2005

  9. [17]

    Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers

    Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023. U...

    2023 arXiv

  10. [18]

    Hate Speech Dataset from a White Supremacy Forum

    Ona de Gibert, Naiara Perez, Aitor Garc \' a-Pablos, and Montse Cuadros. Hate Speech Dataset from a White Supremacy Forum . In Proceedings of the 2nd Workshop on Abusive Language Online ( ALW 2) , pp.\ 11--20, Brussels, Belgium, October 2018. Association for Computational Ling...

    2018 doi

  11. [19]

    The commitmentbank: Investigating projection in naturally occurring discourse

    Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pp.\ 107--124, 2019. URL https://www.semanticscholar.org/paper/39e801ca0dbc69c3697f1...

    2019

  12. [20]

    Automatically constructing a corpus of sentential paraphrases

    Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (IWP2005), 2005. URL https://www.semanticscholar.org/paper/475354f10798f110d34792b6d88f31d6d5cb099e

    2005

  13. [21]

    A survey on in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022. URL https://arxiv.org/abs/2301.00234

    2022 arXiv

  14. [22]

    Mitigating label biases for in-context learning

    Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. Mitigating label biases for in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 14014--14031, 2023. URL http://arxiv.org/abs/2305.19148v3

    2023 arXiv

  15. [23]

    The third pascal recognizing textual entailment challenge

    Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pp.\ 1--9, 2007. URL http://dl.acm.org/ft_gateway.cfm?id=1654538&type=pdf

    2007

  16. [24]

    Demystifying prompts in language models via perplexity estimation

    Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 10136--10148, 2023. URL http://arxiv.org/abs/2212.04037v2

    2023 arXiv

  17. [25]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv e-prints, pp.\ arXiv--2407, 2024. URL http://arxiv.org/abs/2407.21783v3

    2024 arXiv

  18. [26]

    Pre-training to learn in context

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Pre-training to learn in context. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 4849--4870, 2023. URL http://arxiv.org/pdf/2305.09137

    2023 arXiv

  19. [27]

    On calibration of modern neural networks

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp.\ 1321--1330. PMLR, 2017. URL http://arxiv.org/abs/1706.04599v2

    2017 arXiv

  20. [28]

    The second pascal recognising textual entailment challenge

    R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, volume 7, pp.\ 785--79...

    2006

  21. [29]

    Understanding in-context learning via supportive pretraining data

    Xiaochuang Han, Daniel Simig, Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz, and Tianlu Wang. Understanding in-context learning via supportive pretraining data. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...

    2023 arXiv

  22. [30]

    Prototypical calibration for few-shot learning of language models

    Zhixiong Han, Yaru Hao, Li Dong, Yutao Sun, and Furu Wei. Prototypical calibration for few-shot learning of language models. In The Eleventh International Conference on Learning Representations, 2023 b . URL http://arxiv.org/abs/2205.10183v2

    2023 arXiv

  23. [31]

    Structured prompting: Scaling in-context learning to 1,000 examples

    Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. Structured prompting: Scaling in-context learning to 1,000 examples. arXiv preprint arXiv:2212.06713, 2022. URL http://arxiv.org/abs/2212.06713v1

    2022 arXiv

  24. [32]

    Toward semantics-based answer pinpointing

    Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin-Yew Lin, and Deepak Ravichandran. Toward semantics-based answer pinpointing. In Proceedings of the First International Conference on Human Language Technology Research, 2001. URL https://www.aclweb.org/anthology/H01-1069

    2001

  25. [33]

    In-context convergence of transformers

    Yu Huang, Yuan Cheng, and Yingbin Liang. In-context convergence of transformers. arXiv preprint arXiv:2310.05249, 2023. URL https://arxiv.org/abs/2310.05249

    2023 arXiv

  26. [34]

    Opt-iml: Scaling language model instruction meta learning through the lens of generalization

    Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.1...

    2022 arXiv

  27. [35]

    An information-theoretic analysis of in-context learning

    Hong Jun Jeon, Jason D Lee, Qi Lei, and Benjamin Van Roy. An information-theoretic analysis of in-context learning. arXiv preprint arXiv:2401.15530, 2024. URL https://arxiv.org/abs/2401.15530

    2024 arXiv

  28. [36]

    Generative calibration for in-context learning

    Zhongtao Jiang, Yuanzhe Zhang, Cao Liu, Jun Zhao, and Kang Liu. Generative calibration for in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 2312--2333, 2023. URL https://www.semanticscholar.org/paper/bd2bbaa226be8fe6564e878e26...

    2023

  29. [37]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. URL http://arxiv.org/abs/2001.08361v1

    2001 arXiv

  30. [38]

    In-context learning learns label relationships but is not conventional learning

    Jannik Kossen, Yarin Gal, and Tom Rainforth. In-context learning learns label relationships but is not conventional learning. In The Twelfth International Conference on Learning Representations, 2024. URL http://arxiv.org/abs/2307.12375v4

    2024 arXiv

  31. [39]

    The winograd schema challenge

    Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012. URL https://www.semanticscholar.org/paper/128cb6b891aee1b5df099acb48e2efecfcff689f

    2012

  32. [40]

    What do language models learn in context? the structured task hypothesis

    Jiaoda Li, Yifan Hou, Mrinmaya Sachan, and Ryan Cotterell. What do language models learn in context? the structured task hypothesis. arXiv preprint arXiv:2406.04216, 2024. URL http://arxiv.org/abs/2406.04216v3

    2024 arXiv

  33. [41]

    Finding support examples for in-context learning

    Xiaonan Li and Xipeng Qiu. Finding support examples for in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 6219--6235, 2023. URL http://arxiv.org/abs/2302.13539v3

    2023 arXiv

  34. [42]

    Unified demonstration retriever for in-context learning

    Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, GUOTONG XIE, Xiaoling Wang, and Xipeng Qiu. Unified demonstration retriever for in-context learning. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023. URL http://arxiv.org/abs/2305.04320v2

    2023 arXiv

  35. [43]

    Learning question classifiers

    Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics , 2002. URL https://www.aclweb.org/anthology/C02-1150

    2002

  36. [44]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 0 87--...

    2024

  37. [45]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Archit...

    2022

  38. [46]

    Let's learn step by step: Enhancing in-context learning ability with curriculum learning

    Yinpeng Liu, Jiawei Liu, Xiang Shi, Qikai Cheng, and Wei Lu. Let's learn step by step: Enhancing in-context learning ability with curriculum learning. arXiv preprint arXiv:2402.10738, 2024. URL http://arxiv.org/abs/2402.10738v2

    2024 arXiv

  39. [47]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1...

    2022 arXiv

  40. [48]

    P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65, 2014. URL http://arxiv.org/abs/1307.5336v2

    2014 arXiv

  41. [49]

    Which examples to annotate for in-context learning? towards effective and efficient selection

    Costas Mavromatis, Balasubramaniam Srinivasan, Zhengyuan Shen, Jiani Zhang, Huzefa Rangwala, Christos Faloutsos, and George Karypis. Which examples to annotate for in-context learning? towards effective and efficient selection. arXiv preprint arXiv:2310.20046, 2023. URL http:/...

    arXiv

  42. [50]

    Noisy channel language model prompting for few-shot text classification

    Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompting for few-shot text classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 5316--5330, 2022 ...

    2022 arXiv

  43. [51]

    Metaicl: Learning to learn in context

    Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.\ 2791--2809, 2022 b ....

    2022

  44. [52]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.\...

    2022

  45. [53]

    S em E val-2018 task 1: Affect in tweets

    Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. S em E val-2018 task 1: Affect in tweets. In Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, and Marine Carpuat (eds.), Proceedings of the 12th Internationa...

    2018 doi

  46. [54]

    Obtaining well calibrated probabilities using bayesian binning

    Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015. URL https://ojs.aaai.org/index.php/AAAI/article/download/9602/9461

    2015

  47. [55]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022. URL https://arxiv.org/abs/2209.11895

    2022 arXiv

  48. [56]

    What in-context learning" learns" in-context: Disentangling task recognition and task learning

    Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning" learns" in-context: Disentangling task recognition and task learning. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023. URL http://arxiv.org/pdf/2305.09731

    2023 arXiv

  49. [57]

    Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales

    Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pp.\ 115--124, 2005. URL http://arxiv.org/abs/...

    2005 arXiv

  50. [58]

    Mauve: Measuring the gap between neural text and human text using divergence frontiers

    Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34: 0 4816--4828, 2021....

    2021

  51. [59]

    In-context learning with iterative demonstration selection

    Chengwei Qin, Aston Zhang, Anirudh Dagar, and Wenming Ye. In-context learning with iterative demonstration selection. arXiv preprint arXiv:2310.09881, 2023. URL http://arxiv.org/abs/2310.09881v4

    2023 arXiv

  52. [60]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019. URL https://www.semanticscholar.org/paper/9405cc0d6169988371b2755e573cc28650d14dfe

    2019

  53. [61]

    Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36, 2024

    Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36, 2024. URL http://arxiv.org/abs/2304.15004v2

    2024 arXiv

  54. [62]

    Smith, Luke Zettlemoyer, Wen tau Yih, and Mike Lewis

    Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Wen tau Yih, and Mike Lewis. In-context pretraining: Language modeling beyond document boundaries. In The Twelfth International Conference on Learning Representat...

    2024

  55. [63]

    Why larger language models do in-context learning differently? arXiv preprint arXiv:2405.19592, 2024 b

    Zhenmei Shi, Junyi Wei, Zhuoyan Xu, and Yingyu Liang. Why larger language models do in-context learning differently? arXiv preprint arXiv:2405.19592, 2024 b . URL http://arxiv.org/abs/2405.19592v1

    2024 arXiv

  56. [64]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Proces...

    2013

  57. [65]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on mach...

    2023 arXiv

  58. [66]

    Smith, and Tao Yu

    Hongjin SU, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Selective annotation makes language models better few-shot learners. In The Eleventh International Conference on Learning Represe...

    2023

  59. [67]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational L...

    2023 arXiv

  60. [68]

    Taguchi methods: orthogonal arrays and linear graphs; tools for quality engineering

    Genichi Taguchi and Seiso Konishi. Taguchi methods: orthogonal arrays and linear graphs; tools for quality engineering. ASI press, 1987

    1987

  61. [69]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. URL http://arxiv.org/a...

    2023 arXiv

  62. [70]

    In-context learning demonstration selection via influence analysis

    Minh-Hao Van, Xintao Wu, et al. In-context learning demonstration selection via influence analysis. arXiv preprint arXiv:2402.11750, 2024. URL http://arxiv.org/abs/2402.11750v2

    2024 arXiv

  63. [71]

    Mind your format: Towards consistent evaluation of in-context learning improvements

    Anton Voronov, Lena Wolf, and Max Ryabinin. Mind your format: Towards consistent evaluation of in-context learning improvements. arXiv preprint arXiv:2401.06766, 2024. URL http://arxiv.org/abs/2401.06766v3

    2024 arXiv

  64. [72]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, 2019. URL https://www.aclweb.o...

    2019

  65. [73]

    Gpt-j-6b: A 6 billion parameter autoregressive language model, 2021

    Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A 6 billion parameter autoregressive language model, 2021. URL https://huggingface.co/EleutherAI/gpt-j-6b

    2021

  66. [74]

    Label words are anchors: An information flow perspective for understanding in-context learning

    Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp....

    2023 arXiv

  67. [75]

    Sida Wang and Christopher D. Manning. Baselines and bigrams: simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers - Volume 2, ACL '12, pp.\ 90–94, USA, 2012. Association for Co...

    2012

  68. [76]

    Dai, and Quoc V Le

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=gEZrGCozdqR

    2022

  69. [77]

    Symbol tuning improves in-context learning in language models

    Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, et al. Symbol tuning improves in-context learning in language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,...

    2023 arXiv

  70. [78]

    Larger language models do in-context learning differently

    Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846, 2023 b . URL http://arxiv.org/abs/2303.03846v2

    2023 arXiv

  71. [79]

    The learnability of in-context learning

    Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. Advances in Neural Information Processing Systems, 36, 2024. URL http://arxiv.org/abs/2303.07895v1

    2024 arXiv

  72. [80]

    Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering

    Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volum...

    2023 arXiv

  73. [81]

    \ k\ NN prompting: Beyond-context learning with calibration-free nearest neighbor inference

    Benfeng Xu, Quan Wang, Zhendong Mao, Yajuan Lyu, Qiaoqiao She, and Yongdong Zhang. \ k\ NN prompting: Beyond-context learning with calibration-free nearest neighbor inference. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.ne...

    2023

  74. [82]

    Ground-truth labels matter: A deeper look into input-label demonstrations

    Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, and Taeuk Kim. Ground-truth labels matter: A deeper look into input-label demonstrations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,...

    2022 arXiv

  75. [83]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. URL https://arxiv.org/abs/2205.01068

    2022 arXiv

  76. [84]

    Character-level convolutional networks for text classification

    Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015. URL https://www.semanticscholar.org/paper/51a55df1f023571a7e07e338ee45a3e3d66ef73e

    2015

  77. [85]

    Noisyicl: A little noise in model parameters calibrates in-context learning

    Yufeng Zhao, Yoshihiro Sakai, and Naoya Inoue. Noisyicl: A little noise in model parameters calibrates in-context learning. arXiv preprint arXiv:2402.05515, 2024. URL https://arxiv.org/abs/2402.05515

    2024 arXiv

  78. [86]

    Calibrate before use: Improving few-shot performance of language models

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pp.\ 12697--12706. PMLR, 2021. URL http://arxiv.org/abs/2102.09690v2

    2021 arXiv

  79. [87]

    Batch calibration: Rethinking calibration for in-context learning and prompt engineering

    Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine A Heller, and Subhrajit Roy. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. In The Twelfth International Conference on Learning Representations, 2024. URL https:/...

    2024

  80. [88]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  81. [89]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  82. [90]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  83. [91]

    , " * write output.state after.block = add.period write newline

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2010

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.