MaXIFE: Multilingual and Cross-lingual Instruction Following Evaluation

Jing Chang; Jinglu Hu; Liang Li; Xiu Jiang; Yile Liu; Ziwei Ma

REVIEW 3 major objections 7 minor 1 cited by

MaXIFE: Multilingual and Cross-lingual Instruction Following Evaluation

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 23-language benchmark tests whether LLMs follow instructions outside English.

desk verdict A genuinely useful 23-language instruction-following benchmark whose cross-lingual rankings are confounded by instruction feasibility in CJK and Indic languages; worth refereeing, but only if the data ships and the templates are fixed. read the letter →

arxiv 2506.01776 v2 pith:KP76CAWG submitted 2025-06-02 cs.CL cs.AI

Yile Liu , Ziwei Ma , Xiu Jiang , Jinglu Hu , Jing Chang , Liang Li This is my paper

classification cs.CLcs.AI

keywords MaXIFEmultilingualinstructionfollowingcross-lingualevaluationLLMbenchmarkrule-basedmodel-basedlow-resourcelanguagesparallelprompts

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MaXIFE is a benchmark designed to measure how well large language models follow instructions in 23 languages at once, using 795 parallel prompts, 47 instruction types, and 1,667 verifiable instruction tasks. The paper's central claim is that this is the first multilingual and cross-lingual instruction-following benchmark, filling a gap left by single-language evaluation sets. If the claim is right, researchers can now compare instruction-following ability on identical prompts across languages and see where models genuinely succeed or fail outside English. The reported baseline results suggest that instruction-following quality tracks language resource levels, with large drops in low-resource languages and notable gains when weaker models receive instructions in English.

What carries the argument

The load-bearing mechanism is the parallel instruction dataset: 47 instruction templates organized into 11 categories, templated with fill-in-the-blank parameters, then combined with 795 basic questions and translated with native-speaker quality control. Scoring splits into deterministic rule-based checks for 32 instruction subcategories, with language-specific adaptations such as Bengali numerals and CJK full-width punctuation, and a Claude-3.5 Sonnet model-based judge for semantic categories like style, tone, and language switching. Loose and Strict scores aggregate instruction-level compliance, and the parallel structure is what makes cross-language comparisons possible.

What would settle it

For a language at the bottom of the score table, such as Quechua, have two independent teams of native speakers produce separate translations of the same prompts, run the same models on both versions, and compare scores; if the two versions produce larger score differences than the reported gap between high-resource and low-resource languages, then translation variance, not model capability, is driving the result.

Watch

Extended reading notes

Core claim

The paper argues that existing instruction-following benchmarks test only one language at a time, usually English, and therefore miss real differences in how models handle other languages. MaXIFE is presented as the first benchmark that evaluates instruction following in both multilingual and cross-lingual settings: the same 795 questions and 47 instruction templates are translated into 23 languages, each question is paired with one to three instructions, and every response is scored automatically. The results on five commercial models show a consistent resource-level gradient, with average loose scores of about 78.7 percent for high-resource languages versus 62.0 percent for low-resource languages, and a steep decline for GPT-3.5 Turbo in particular. The paper also reports cross-lingual experiments where instructions stay in English while responses must be produced in another language, finding that this setup substantially improves GPT-3.5 Turbo's performance in low-resource languages, from about 31.5 percent to 52.7 percent on average.

Load-bearing premise

The paper's cross-lingual comparisons stand on the assumption that the translated instructions are semantically and functionally equivalent across all 23 languages, an assumption the authors themselves qualify by noting that subtle semantic deviations may persist in extremely low-resource languages like Quechua.

Editorial extensions

If this is right

Researchers can directly compare instruction-following quality across 23 languages on identical prompts, exposing strengths and weaknesses that English-only tests hide.
The benchmark provides reusable baseline numbers for commercial models, so future models can be measured against a fixed multilingual standard.
Low-resource language gaps are quantified per instruction type, pointing to specific weaknesses such as punctuation handling in non-Latin scripts.
Cross-lingual results suggest that for models with weaker multilingual training, an English-instruction plus target-language-response mode can improve instruction following in low-resource languages.
The templated design and extension scripts let the benchmark grow to additional languages and instruction types.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Extension: The paper's 'first of its kind' framing depends on the completeness of its literature review; if any comparable benchmark already exists, the novelty claim narrows to the specific combination of parallel prompts, rule-based scoring, and cross-lingual mode.
Extension: Because the paper's own limitations section concedes that subtle semantic deviations may exist in very low-resource languages like Quechua, the cleanest reading is that cross-language score differences mix genuine capability gaps with possible translation artifacts, and the magnitude of the artifact remains unmeasured.
Extension: A natural follow-up test would reverse the cross-lingual direction, giving instructions in the target language and requiring responses in English, or testing pairs of non-English languages, to separate language comprehension from generation ability.
Extension: The rule-based scoring recipes for script-specific punctuation and word counting could be adopted by other multilingual benchmarks, making the evaluation infrastructure reusable beyond this dataset.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The headline result is real but overstated. MaXIFE gives NLP a genuinely new resource — 795 parallel prompts with 47 verifiable instruction types across 23 languages, most of them never covered by an instruction-following benchmark — and the appendix documents the scoring machinery in unusual detail: rubrics, judge prompts, language-specific word-counting rules, and human consistency numbers at 94.6–97.3%. That is the paper's real value; it is a usable blueprint, if the data ever ships.

The problem is the parallelism assumption, and the stress-test note is right — I checked the per-category tables. "The first word of your response must be X" scores 0.00% for all five models in Chinese (Table 37) and Japanese (Table 45), against 87–90% in English. Repeat categories collapse the same way in Korean and Tamil. That is not a model-capability deficit; the template was not localized for languages without space-delimited words, even though Section B.2 shows the authors knew word counting had to be language-adaptive. When a chunk of instructions is structurally infeasible in a language, the aggregate loose score stops being comparable across languages, and Table 1's resource-level gradient becomes partly an artifact of instruction feasibility. The Limitations section's "subtle semantic deviations in Quechua" understates what is systematic in the tables.

Three smaller soft spots. Section 4.1 lists "marks" in both the rule-based and model-based sets — a genuine internal contradiction, quick fix. Claude-3.5 Sonnet is both an evaluated model and the judge; the human consistency check mitigates but does not eliminate self-preference. And the cross-lingual headline runs only on Gemini-1.5 Flash and GPT-3.5 Turbo, chosen after a pilot, so the claim that English instructions help in low-resource languages rests on two models picked post hoc. Add to that no data or code release, which makes all of this unverifiable, and the Section 2 "first benchmark" claim is asserted rather than demonstrated against prior work.

I largely agree with the reader's conditional verdict. One difference in emphasis: the benchmark core is sound enough to build on — the evaluation method is a known pattern, but the coverage is new — and the flaw, while load-bearing for cross-lingual comparisons, is concentrated in specific subcategories rather than the whole design. With data release, template fixes for CJK and Indic languages, and per-category reporting, this becomes a solid resource.

Recommendation: send to peer review. A referee can extract real value, but acceptance should require releasing the data and addressing the feasibility confound.

Referee Report

3 major / 7 minor

Summary. The paper introduces MaXIFE, a multilingual and cross-lingual instruction-following benchmark built from 795 basic questions and 47 instruction templates translated into 23 languages, producing 1,667 scored instruction instances. Evaluation combines rule-based scoring for 32 instruction subcategories and model-based scoring for 15 subcategories, using Claude-3.5 Sonnet as the model-based judge. Five commercial LLMs are evaluated, and the paper reports language-level, category-level, and cross-lingual results, including two models run in an English-instruction/target-language-response setting. The central claim is that MaXIFE is the first benchmark allowing parallel assessment of instruction-following across languages and that its per-language scores measure genuine cross-lingual capability differences.

Significance. If the parallel-corpus assumption and scoring validity hold, MaXIFE would be a substantial resource: it has an unusually detailed translation pipeline with native-speaker review, explicit rule-based rubrics in the appendix, human-model consistency checks, and a reusable template structure that supports adding languages and instructions. The reported baseline results also document large gaps between high- and low-resource languages. However, the benchmark's core value depends on the functional equivalence of the 47 instructions across languages, and the appendix data contain systematic patterns that contradict this equivalence. The structural infeasibility of word-based instructions in non-space-delimited languages, together with an unsupported 'first benchmark' claim and an unaddressed self-preference risk in the model-based judge, mean the current version does not yet support the paper's strongest conclusions.

major comments (3)

[Section 3.1 / A.5; Tables 37, 38, 45, 46, 54] The parallelism claim is contradicted by the reported per-instruction scores. keywords:first_word is 0.00% for all five models in Chinese (Table 37) and Japanese (Table 45) but 90.32% for GPT-4o in English (Table 38); repeat:first_last_same and repeat:last_sentence are 0.00% for all models in Korean (Table 46), with similarly collapsed values in Tamil and Telugu (Tables 54 and 55). Because these instructions cannot be satisfied under the structural properties of the language (e.g., absence of space-delimited words, or sentence-boundary definitions that do not transfer), all models necessarily score zero. Language-level Loose Scores in Table 1 are therefore confounded by instruction feasibility rather than measuring only model capability. The paper should localize such instructions (e.g., first-character variants for CJK, adapted repeat definitions for scripts without sentence-final punctuation), or explicitly exclude infeasible items from cross-language comparisons and report per-instruction feasibility.
[Section 2] The claim that MaXIFE is 'the first multilingual/cross-lingual instruction-following task benchmark' is not adequately substantiated. Existing multilingual instruction-following benchmarks, including multilingual IFEval extensions and the Multi-IF benchmark, should be cited and explicitly differentiated. If the claim is retained, the related-work section should establish what MaXIFE adds over these prior resources; otherwise, the paper should be reframed as a new multilingual benchmark rather than the first one.
[Section 4.3 / B.3] The model-based evaluator is Claude-3.5 Sonnet, which is also one of the evaluated models. The human validation in Section B.3 uses only 100 instructions per language and reports aggregate consistency rather than per-judge bias. In the English validation, Claude's model-based score (99.1%) is notably higher than its human score (94.7%), consistent with mild self-preference. The paper should report agreement between human and model-based scores separately for each evaluated model and for the 15 model-based instruction types, and discuss how any systematic over-scoring affects cross-model comparisons.

minor comments (7)

[Section 4.1] 'Marks' appears in both the rule-based list and the model-based list of categories, while 'content' is omitted from the model-based list; this appears to be a typo but should be corrected.
[Section 5.2] The statement that 795 prompts and 47 instruction types 'ensure the statistical significance of the evaluation' is not justified; with 1,667 total instruction instances, per-subcategory counts are small. The paper should report the number of instances per instruction subcategory and provide confidence intervals or a power analysis.
[Sections B.4 and B.5] The rule-based scoring formulas use arbitrarily chosen quadratic penalty coefficients (0.1, 0.2, 0.03) and component weights; because Loose Scores are computed from these values, the paper needs a sensitivity analysis or an explicit justification for the chosen constants.
[Section 3.2] The resource-level classification combines four dimensions without stating their relative weights or the exact decision rule; since Table 1 groups languages by these levels, the rubric should be made fully explicit.
[Section 6.3 / Tables 58-59] Full cross-lingual experiments are only run on GPT-3.5 Turbo and Gemini-1.5 Flash; conclusions about the benefit of English instructions for low-resource languages should be explicitly qualified as model-specific rather than general.
[General] The dataset and code are not linked in the manuscript; a data-availability statement and release URL are needed for reproducibility and for independent verification of the parallelism concern.
[General] Minor typographical issues include 'Gemini-1.5 Proper' in Section 6.1.2, 'colokial' in the Malay instruction list, and an undefined use of the up/down arrow notation in Table 1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark scores are computed from deterministic rules plus model-based judgments externally validated against human ratings; no derived quantity reduces to a fitted input or self-citation chain.

full rationale

The paper's central outputs are Loose and Strict scores for 1,667 verifiable instructions across 23 languages. These scores are produced by explicit rule-based checkers (Section B.4) and by a Claude-3.5 Sonnet judge for semantic categories (Section 4.3), with no parameter fitted to the target scores and no prediction that is a renamed input. The only overlap between evaluated system and evaluator is that Claude-3.5 Sonnet is both a baseline model and the model-based judge. This is a methodological risk, not a circular derivation, because the paper reports human/model consistency of 97.3%, 95.2%, and 94.6% for high-, medium-, and low-resource languages (Tables 8-10), which provides external grounding outside the judged model's own preferences. The claim of being the first multilingual/cross-lingual instruction-following benchmark rests on a survey of prior work (Section 2), not on a self-referential uniqueness theorem. Reported anomalies such as near-zero 'first_word' scores for Chinese and Japanese indicate a possible construct-validity or translation-equivalence problem, and the paper's own Limitations section admits that 'subtle semantic deviations may exist in extremely low-resource languages like Quechua.' That is a correctness concern about parallelism, not a circularity of the kind where an output equals an input by construction. No quoted equation or parameter fit can be exhibited that reduces a headline result to its own definition, so the appropriate finding is no significant circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

MaXIFE is an evaluation artifact rather than a postulated physical entity; no new particles, forces, dimensions, or conserved quantities are introduced. The main hand-chosen inputs are scoring coefficients and component weights in Section B.4. The load-bearing domain assumptions are translation equivalence, judge reliability, script-group correctness, and survey representativeness.

free parameters (6)

Scoring penalty coefficient for frequency and format items = 0.1
Hand-chosen in Section B.4 for keywords frequency, markdown_highlight, ordered_list, markdown_title, and emoji frequency; changes would alter partial-credit scores and model rankings.
Scoring penalty coefficient for repetition and paragraph items = 0.2
Hand-chosen in Section B.4 for paragraph_end, before_answer, last_sentence, sentence_n_times, and all_sentences_twice; changes would alter partial-credit scores.
Scoring penalty coefficient for punctuation items = 0.03
Hand-chosen in Section B.4 for no_commas, replace_with_exclamations, end_with_semicolons, and replace_with_asterisks; changes would alter punctuation-related scores.
Scoring penalty coefficient for citation bracket counts = 0.3
Hand-chosen in Section B.4 for citation square_brackets; changes would alter citation-related scores.
Scoring penalty coefficient for length limits = 20
Hand-chosen in Section B.4 for length max_words and range_words; changes would alter length-related scores.
Component weights in composite rule-based instructions = various (0.3/0.15/0.4, 0.5/0.5, 0.1/0.9, 0.7/0.3)
Hand-chosen in Section B.4 for keywords together, addition_at_end, title_brackets, markdown_title, start_from_zero, and emoji banned; changes would alter composite instruction scores.

assumptions (4)

domain assumption Native-speaker translation and quality control preserve semantic equivalence of all 47 instruction templates across 23 languages.
Invoked in Section 3.1 and A.5 to justify cross-language comparability; the paper's own Limitations section concedes possible 'subtle semantic deviations' in low-resource languages.
domain assumption Claude-3.5 Sonnet provides instruction-following scores that agree with human judgment across all languages.
Used in Section 4.3 to convert subjective instructions into numeric scores; validated only on 100 samples for three languages, not for all 23.
domain assumption The hand-written rule-based scorers correctly implement linguistic checks for each script and language family.
Section B.2 groups languages into word-count and keyword-matching families; a wrong grouping would bias language-level scores.
domain assumption A questionnaire of 31 respondents is representative of real-world instruction distributions.
Used in Section A.3 to source Basic Questions and Instructions; a biased sample would propagate into the benchmark.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MaXIFE: Multilingual and Cross-lingual Instruction Following Evaluation." pith.science (2026). https://pith.science/paper/KP76CAWG

@misc{pith2026250601776,
  author       = {Pith},
  title        = {Pith review of: MaXIFE: Multilingual and Cross-lingual Instruction Following Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KP76CAWG}},
  note         = {Machine review of arXiv:2506.01776}
}

read the original abstract

With the rapid adoption of large language models (LLMs) in natural language processing, the ability to follow instructions has emerged as a key metric for evaluating their practical utility. However, existing evaluation methods often focus on single-language scenarios, overlooking the challenges and differences present in multilingual and cross-lingual contexts. To address this gap, we introduce MaXIFE: a comprehensive evaluation benchmark designed to assess instruction-following capabilities across 23 different languages with 1667 verifiable instruction tasks. MaXIFE integrates both Rule-Based Evaluation and Model-Based Evaluation, ensuring a balance of efficiency and accuracy. We applied MaXIFE to evaluate several leading commercial LLMs, establishing baseline results for future comparisons. By providing a standardized tool for multilingual instruction-following evaluation, MaXIFE aims to advance research and development in natural language processing.

Figures

Figures reproduced from arXiv: 2506.01776 by the authors.

**Figure 2.** MaXIFE Structure, its evaluation dataset composition, and evaluation strategy. We provide 795 Basic [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** 11 Instruction Categories and 47 Instruction [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Data Example of Cross-lingual experiment. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models
cs.CL 2025-10 conditional novelty 7.0 of 10

ChiKhaPo is an 8-subtask benchmark that measures word-level comprehension and generation in 2,700+ languages and shows state-of-the-art models perform poorly on low-resource languages.

Reference graph

Works this paper leans on

27 extracted references · 23 canonical work pages · cited by 1 Pith paper

[1]

Both words must appear together (0.3 points)

work page
[2]

exactly N

Languages that use simple repetition forms, such as Malay, Filipino, Indonesian. 3) Languages that use suffix variations, such as Bengali, Hindi. 4) Other languages. For these four different types of languages, we use different keyword matching methods to ensure the correct matching of key- words. B.3 Model-based Validity Verification We sampled 100 instr...

work page
[3]

Have you ever encountered situations where the model did not fully understand or execute your instructions? Please provide examples

work page
[4]

{word}” should appear {natural_relation} {word_num} times. together Your response must contain both “{word1}

Adapting language models for zero-shot learn- ing by meta-tuning on dataset and prompt collections. InFindings of the Association for Computational Linguistics: EMNLP 2021, pages 2856–2878, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Wei...

arXiv 2021
[5]

(Optional)In evaluating the model’s ability to follow instruc- tions, what other aspects do you think deserve attention?

work page
[6]

challenging

(Optional)If you were to design some “challenging” in- structions to test the model, what would you propose?

work page
[8]

I feel the model often cannot follow my word count requirements and keeps going on

(Optional)What is your native language? Compared to English, do you think your native language has any particular characteristics? Thank you for completing this survey! 21 The total number of questionnaire participants was 31 people, of which 19 were male, 12 were female, most people’s age distribution was 21-39 years old, and all respondents had high-fre...

work page 2024
[11]

Each word must meet minimum frequency N (0.15 points each)

work page

Show all 27 references

[12]

words ’cat’ and ’dog’ must appear at least 2 times each, with ’cat’ more frequent

Word1 must appear more frequently than Word2 (0.4 points if both meet minimum N) Total score is the sum of points earned for each requirement (max 1.0). Example: For instruction “words ’cat’ and ’dog’ must appear at least 2 times each, with ’cat’ more frequent”: Response with ...
[13]

Together requirement: +0.3
[14]

Min frequency for cat: +0.15
[15]

Min frequency for dog: +0.15
[16]

do not use the words ’cat’ or ’dog’ in response

Cat more frequent: +0.4 Total score = 1.0 Note: The scoring considers various word forms (plurals, repetitions, suffixes) based on language. 24 keywords: banned The score penalizes the use of forbidden words with a tiered deduction system: • No forbidden words: 1.0 points • On...
[17]

Minimum paragraph count (N)
[18]

response must have at least 3 paragraphs with ’conclusion’ in last sentence of each

Required word appearing in last sentence of each paragraph Scoring formula: Score =max(0,1−0.2E×E) where E is the number of paragraphs that fail the last-sentence requirement. Example: For instruction “response must have at least 3 paragraphs with ’conclusion’ in last sentence...
[19]

Special Unicode ranges for specific scripts format: addition_at_end The score evaluates two requirements for the postscript addition:

Character-based for East Asian languages 3. Special Unicode ranges for specific scripts format: addition_at_end The score evaluates two requirements for the postscript addition:
[20]

Presence of the required text (0.5 points)
[21]

add postscript starting with ‘Note:

Correct placement at the end (0.5 points) Score calculation: When text present AND at end: Score = 1.0 When text present but NOT at end: Score = 0.5 When text NOT present: Score = 0.0 Example: For instruction “add postscript starting with ‘Note:”’: • Response ending with “...e...
[22]

Proper enclosure in brackets (0.1 points)
[23]

title in brackets, max 5 words

Length requirement (0.9 points) Score calculation: When properly enclosed AND length≤M: Score = 1.0 When enclosed but length > M: Score = 0.1 + max(0, 0.9 - 0.1R × R) When not enclosed: Score = 0.0 where: • M = maximum allowed length • R = (L - M) / M (length deviation ratio) ...

2020
[24]

humorous tone

Score 0.0: The response shows no indication of any “humorous tone”. Please note that as a fair and objective evaluator, you should not adhere to an “ambiguous” principle and give a score of 0.7 regardless of circumstances, nor should you be too lenient or too strict. Your scor...
[25]

anger or fury

Score 0.0: The response shows no indication of “anger or fury”. Please note that as a fair and objective evaluator, you should not adhere to an “ambiguous” principle and give a score of 0.7 regardless of circumstances, nor should you be too lenient or too strict. Your scoring ...
[26]

ambiguous

Score 0.7: The response contains only two famous quotes, or the famous quotes that appear in the response are not obvious. 3. Score 0.0: The response contains only one quote, or no famous quotes at all. Please note that as a fair and objective evaluator, you should not adhere ...
[27]

ambiguous

Score 0.7: The response contains two languages, but the expressions in the two languages differ greatly. 3. Score 0.0: The response does not repeat at all, or only uses one language. Please note that as a fair and objective evaluator, you should not adhere to an “ambiguous” pr...
[2020]

InInternational Conference on Machine Learn- ing, pages 4411–4421

Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisa- tion. InInternational Conference on Machine Learn- ing, pages 4411–4421. PMLR. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng ...

2024 arXiv
[2021]

InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265–1285, Online and Punta Cana, Dominican Republic

The perils of using Mechanical Turk to evalu- ate open-ended text generation. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265–1285, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Shiyan...

2021 arXiv
[2024]

Journal of Machine Learning Research, 25(70):1–53

Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. U...

2020 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Both words must appear together (0.3 points)

work page

[2] [2]

exactly N

Languages that use simple repetition forms, such as Malay, Filipino, Indonesian. 3) Languages that use suffix variations, such as Bengali, Hindi. 4) Other languages. For these four different types of languages, we use different keyword matching methods to ensure the correct matching of key- words. B.3 Model-based Validity Verification We sampled 100 instr...

work page

[3] [3]

Have you ever encountered situations where the model did not fully understand or execute your instructions? Please provide examples

work page

[4] [4]

{word}” should appear {natural_relation} {word_num} times. together Your response must contain both “{word1}

Adapting language models for zero-shot learn- ing by meta-tuning on dataset and prompt collections. InFindings of the Association for Computational Linguistics: EMNLP 2021, pages 2856–2878, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Wei...

arXiv 2021

[5] [5]

(Optional)In evaluating the model’s ability to follow instruc- tions, what other aspects do you think deserve attention?

work page

[6] [6]

challenging

(Optional)If you were to design some “challenging” in- structions to test the model, what would you propose?

work page

[7] [8]

I feel the model often cannot follow my word count requirements and keeps going on

(Optional)What is your native language? Compared to English, do you think your native language has any particular characteristics? Thank you for completing this survey! 21 The total number of questionnaire participants was 31 people, of which 19 were male, 12 were female, most people’s age distribution was 21-39 years old, and all respondents had high-fre...

work page 2024

[8] [11]

Each word must meet minimum frequency N (0.15 points each)

work page