Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

QuArch, a 2,671-question expert-validated benchmark for computer architecture, shows frontier LLMs have largely mastered recall but fall short on analyze, design, and implement skills — and fine-tuning on it improves a realistic memory-hier

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 08:09 UTC pith:2Y3VB7CG

load-bearing objection QuArch is a genuinely useful first benchmark for computer-architecture reasoning, but the headline recall-vs-reasoning gap rests on unvalidated LLM skill labels and a format confound, and the abstract promises a fine-tuning experiment that never appears in the body. the 3 major comments →

arxiv 2510.22087 v2 pith:2Y3VB7CG submitted 2025-10-24 cs.AR cs.AIcs.LGcs.SE

QuArch: A Benchmark for Evaluating LLM Reasoning in Computer Architecture

Shvetank Prakash , Andrew Cheng , Mark Mazumder , Arya Tschand , Varun Gohil , Jeffrey Ma , Jason Yik , Zishen Wan
show 19 more authors
Jessica Quaye Elisavet Lydia Alvanaki Avinash Kumar Chandrashis Mazumdar Tuhin Khare Alexander Ingare Ikechukwu Uchendu Radhika Ghosal Abhishek Tyagi Chenyu Wang Andrea Mattia Garavagno Sarah Gu Alice Guo Grace Hur Luca P. Carloni Tushar Krishna Ankita Nayak Amir Yazdanbakhsh Vijay Janapa Reddi
This is my paper
classification cs.AR cs.AIcs.LGcs.SE
keywords computer architectureLLM evaluationreasoning benchmarkexpert-validated QAskills taxonomymemory hierarchy designLLM-as-a-judge
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper builds the first benchmark aimed at computer-architecture knowledge and reasoning in LLMs: 2,671 expert-validated QAs spanning processor design, memory, interconnects, and related topics, organized by four skills. Evaluating frontier models, it finds recall accuracy is 83-89% across models, while analyze, design, and implement scores average 32-39 points lower, with design showing the widest gap. It also reports that fine-tuning on QuArch transfers to a realistic memory hierarchy design task, giving up to 1.99x more area-efficient and 40% more viable solutions. A sympathetic reader would take away that the field now has a reusable instrument to measure whether LLMs can actually reason about architecture trade-offs, not just recall facts — and the answer so far is mostly no.

Core claim

On the paper's own terms, the central discovery is a measurable split: frontier LLMs have "largely mastered recall" (83-89% accuracy) but "fall short on advanced skills" (analyze 34-72%, design 18-87%, implement 28-71% depending on model). The drop is consistent, and design questions — despite being few — are the most discriminative. Reasoning-oriented variants of models gain 23-34 points on higher-order skills relative to non-reasoning variants, versus only about 3 points on recall, suggesting test-time deliberation specifically helps architectural reasoning. The paper further argues the benchmark is trustworthy: LLM-as-a-judge grading of free responses agrees with domain-expert grading at

What carries the argument

The four-skill taxonomy — Recall, Analyze, Design, Implement — is the organizing mechanism. Each QA carries context, constraints, and often a figure, and is labeled into one skill. The benchmark combines synthetic generation, exam parsing, and expert crowdsourcing, all expert-validated; free-response grading uses an LLM judge validated against human experts (85.35% agreement vs 90.7% human-human). The skill split is what turns a QA collection into a reasoning diagnostic.

Load-bearing premise

The load-bearing premise is that the skill labels are accurate: the paper's central recall-vs-reasoning comparison is computed from labels assigned by an LLM (Section 2.3, Appendix D.4), not by human experts; if those labels are noisy, part of the apparent reasoning gap could be label error.

What would settle it

Have a panel of computer-architecture experts independently classify a random sample (say 300-500) of QuArch questions into Recall/Analyze/Design/Implement without seeing the LLM labels, then recompute model accuracies with expert labels. If expert-LLM label agreement falls well below the ~91% human-human agreement reported, or if the recall-vs-reasoning accuracy gap shrinks materially under the expert labels, the paper's central claim is weakened.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • QuArch provides a common yardstick for tracking future LLM progress in computer architecture, with skill-level and topic-level scores rather than a single number.
  • Because design is the most discriminative skill, small sets of hard design questions can serve as an efficient early signal of architectural competence.
  • The strong recall but weak reasoning pattern implies that training data coverage is not the bottleneck; targeted training on analysis/design/implementation or more test-time reasoning is needed.
  • Fine-tuning on QuArch transfers to realistic memory-hierarchy design (up to 1.99x area efficiency, 40% more viable solutions), suggesting benchmark improvement can translate to downstream design quality.
  • The validation of LLM-as-a-judge against human experts at near-human agreement supports scalable grading for free-form architecture answers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial: the benchmark's headline gap is only as clean as the LLM-assigned skill labels; a human relabeling study would be a cheap, decisive check.
  • Editorial: the abstract's fine-tuning transfer claim (1.99x, 40%) is not described anywhere in the main text; if it is meant to be a headline result, it needs its own methods and results section.
  • Editorial: the same Recall/Analyze/Design/Implement taxonomy could extend to adjacent systems fields (OS, networks, compilers), and the observed model-specific topic strengths hint that multi-model routing would beat any single model.
  • Editorial: models' tendency to assume unconventional defaults (e.g., word-level rather than byte-level addressing) suggests a testable, practical fix: training or prompting with explicit default conventions, and QuArch could measure that improvement directly.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces QuArch, a 2,671 question-answer benchmark for computer architecture, organized along four skill categories (Recall, Analyze, Design, Implement) and covering text-only and multimodal items. The benchmark is assembled from synthetic generation, academic exams, and expert crowdsourcing, with all items reported as expert-validated. Ten frontier models are evaluated zero-shot; the authors report uniformly high recall (83–89%) but substantially lower higher-order skill accuracies, and they validate an LLM-as-a-judge protocol against human expert grading on 908 free-response judgments (85.35% agreement). The abstract additionally claims that fine-tuning on QuArch improves a memory-hierarchy design task by up to 1.99x area efficiency and 40% more viable solutions, but this experiment does not appear in the manuscript body.

Significance. If the central claims hold, QuArch would be a genuinely useful asset for the computer-architecture and LLM-evaluation communities: the benchmark is large, expert-validated, public, and spans a neglected domain. The LLM-as-a-judge validation (85.35% agreement vs. 90.7% human-human agreement) is a solid methodological contribution, and the detailed prompt templates in Appendix D support reproducibility. The failure-mode analyses in Section 4.3 and Appendix C are instructive and valuable. However, the headline 'recall is mastered, reasoning is not' rests on skill labels that have not been human-audited, and the abstract's fine-tuning result has no supporting experiment in the manuscript. Both issues are load-bearing for the paper's claimed contributions and must be addressed before the results can be fully credited.

major comments (3)
  1. [§2.3 / Appendix D.4] The central recall-vs-reasoning dissociation depends on assignment of every question to a skill category. Section 2.3 says these labels are 'derived from LLM labeling (Appendix D.4),' but Appendix D.4 reports no human audit, no inter-annotator agreement, and no error analysis. The sentence in D.4 that the prompt 'ensures that classification is reliable across reviewers' is an assertion, not a measured property. Since Table 3, Section 4.2, and the QUARCH-RECALL/QUARCH-REASONING split are all computed over these labels, and since Design+Implement comprise only about 100 questions combined, label noise could shift or even create the reported skill gap. The LLM-as-judge validation in §4.4 validates grading, not skill labeling. Please add a human audit of a stratified random sample (with at least two annotators, reporting agreement), or explicitly weaken the skill-level claims to 'LLM-provisi
  2. [Abstract / body] The abstract states that fine-tuning on QuArch leads to 'up to 1.99x more area-efficient solutions and up to 40% more viable solutions overall' on a realistic memory hierarchy design task. No such experiment is described anywhere in the manuscript: there is no fine-tuning section, no task definition, no baseline, no model, no data split, and no result table. This is a central claimed contribution of the abstract and appears to be unsupported by the current submission. Either add the full experiment with methodology and results, or remove the claim from the abstract and any other summary text.
  3. [§4.2 / Table 3] Section 4.2, item (2) states that 'GPT-5 stands as an extreme outlier on design QAs at 89%,' but Table 3 reports GPT-5 Design accuracy as 86.7%. The 89.0 value in Table 3 belongs to Recall. This is a factual inconsistency in a headline number. Correct the text or the table, and please do a final pass for consistency between the main text, tables, and abstract (which also gives slightly different overall ranges: 34–72% vs. 34–73%).
minor comments (4)
  1. [Title/abstract] The title in the manuscript body reads 'FOREVALUATING' and 'INCOMPUTERARCHITECTURE' with inconsistent spacing and capitalization; clean this up.
  2. [§2.2] There is a stray comma and missing reference: 'Prompt details are in Appendix D.6 and D.7, .' Also consider giving Appendix D.5 a forward reference in the topic-classification discussion in §2.3.
  3. [Table 3 / Appendix B.3] Reporting per-skill sample sizes (N for Recall, Analyze, Design, Implement) would substantially help readers calibrate the small Design/Implement subsets; the text mentions ~100 combined but the table does not show N.
  4. [§4.4] Claude 3.7 Sonnet serves both as the LLM judge and as one of the evaluated models. The human-validation result mitigates circularity, but a one-sentence discussion of this choice would be helpful, e.g., whether judge-model performance was checked for systematic bias on its own outputs.

Circularity Check

0 steps flagged

No significant circularity; QuArch is an externally validated benchmark. Minor self-citations and a partially overlapping LLM judge are not load-bearing.

full rationale

QuArch is an empirical measurement instrument rather than a derivation, so the central claims do not reduce to their inputs by construction. The headline recall-vs-reasoning result is computed over fixed, pre-assigned skill labels (Section 2.3, Appendix D.4). Even though those labels are produced by GPT-4o rather than audited by humans, the labels are not fitted to model accuracies; the reported gap is a measured outcome, not an identity. The LLM-as-a-judge loop is the only structurally self-referential element: Claude 3.7 Sonnet serves as judge while a Claude model is also evaluated. The paper breaks this loop with an external human-expert audit on 100 QAs / 908 responses (85.35% LLM-human agreement vs. 90.7% human-human, Section 4.4), so the judge's verdicts are not merely self-confirmations. Self-citations (Tschand et al. 2025 for 'architecture-semantics of code execution'; Reddi & Yazdanbakhsh 2025 for source scarcity) supply vocabulary and framing and are not load-bearing for the benchmark's validity. The abstract's fine-tuning transfer numbers ('up to 1.99x more area-efficient solutions and up to 40% more viable solutions') are unaccompanied by methodology in the provided text; this is a missing-support/completeness concern rather than circularity. No equation-level reduction, fitted-parameter renaming, or load-bearing self-citation chain is present.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

This is an empirical benchmark paper, so the load-bearing assumptions are about data quality, label quality, judge generalization, and contamination, not mathematical axioms. No free parameters are fitted in the benchmark itself; the main hand-chosen choices are the n=3 sampling, judge model, and partial-credit rounding rule.

axioms (4)
  • domain assumption Expert-validated ground-truth answers are correct.
    Every score in the benchmark treats the provided solution as the unique correct answer; an error in any validated QA propagates directly to model accuracy (Section 2.2).
  • domain assumption GPT-4o skill labels accurately classify questions as Recall/Analyze/Design/Implement.
    The headline recall-vs-reasoning gap and all per-skill tables rely on LLM labeling without human audit (Section 2.3, Appendix D.4).
  • domain assumption LLM-as-a-judge generalizes from a 100-question human-validation sample to the full 1,124 FRQ set.
    Agreement with humans was measured on 8.9% of free-response questions (908 graded responses); the rest are graded by the LLM judge without human checks (Section 4.4).
  • domain assumption Evaluated models had no prior exposure to QuArch questions.
    Questions are sourced from public exams, manuals, and crowdsourcing, but no contamination analysis is provided (Section 2.2).

pith-pipeline@v1.3.0-alltime-deepseek · 44554 in / 10077 out tokens · 111977 ms · 2026-08-04T08:09:26.078614+00:00 · methodology

0 comments
read the original abstract

The field of computer architecture, which bridges high-level software abstractions and low-level hardware implementations, remains absent from current large language model (LLM) evaluations. To this end, we present QuArch (pronounced 'quark'), the first benchmark designed to facilitate the development and evaluation of LLM knowledge and reasoning capabilities specifically in computer architecture. QuArch v1.0 provides a comprehensive collection of 2,671 expert-validated question-answer (QA) pairs covering various aspects of computer architecture, including processor design, memory systems, and interconnection networks. Our evaluation reveals that while frontier models possess domain-specific knowledge, they struggle with skills that require higher-order thinking in computer architecture. Frontier model accuracies vary widely (from 34% to 73%) on these advanced questions, highlighting persistent gaps in architectural reasoning across analysis, design, and implementation QAs. Furthermore, via fine-tuning we find that QuArch can translate to improved performance on a realistic memory hierarchy design task, resulting in up to 1.99x more area-efficient solutions and up to 40% more viable solutions overall. By holistically assessing fundamental skills, QuArch provides a foundation for building and measuring LLM capabilities that can accelerate innovation in computing systems. The QuArch benchmark and leaderboard are publicly available at: https://quarch.ai/.

Figures

Figures reproduced from arXiv: 2510.22087 by Abhishek Tyagi, Alexander Ingare, Alice Guo, Amir Yazdanbakhsh, Andrea Mattia Garavagno, Andrew Cheng, Ankita Nayak, Arya Tschand, Avinash Kumar, Chandrashis Mazumdar, Chenyu Wang, Elisavet Lydia Alvanaki, Grace Hur, Ikechukwu Uchendu, Jason Yik, Jeffrey Ma, Jessica Quaye, Luca P. Carloni, Mark Mazumder, Radhika Ghosal, Sarah Gu, Shvetank Prakash, Tuhin Khare, Tushar Krishna, Varun Gohil, Vijay Janapa Reddi, Zishen Wan.

Figure 1
Figure 1. Figure 1: Reported results (Vals AI, Inc., 2025) for different models across Q [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: QUARCH QA Skills Framework. The benchmark evaluates four complementary com￾petencies: Recall, Analyze, Design, and Implement. QAs in QUARCH contain relevant context describing the scenario, constraints, and figures when appropriate. The illustrative example shows how distinct question styles derived from the same context can probe different skills. Together, these competencies form the building blocks requ… view at source ↗
Figure 3
Figure 3. Figure 3: We construct QUARCH with a three-pronged approach including a blend of synthetic data generation, community crowdsourcing, and academic exams. All QAs are validated by a human expert to curate QUARCH’s final benchmark set of 2,671 question-answer pairs. memory subsystem will exhibit more performance bottlenecks than if the two were optimized to￾gether. Thus, a benchmark should capture topic breadth to prop… view at source ↗
Figure 5
Figure 5. Figure 5: Breakdown of skill, format, and modality distributions in Q [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of topics in QUARCH. Architecture Topic Diversity. QUARCH captures diverse topics in 13 core areas de￾rived from key themes in modern computer architecture research ( [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Confusion matrix com￾paring LLM-as-a-Judge with domain￾expert human grading on FRQ. Motivation. Semantically equivalent and correct solu￾tions to the same FRQ can differ in phrasing, as shown in Appendix C.6. Since full manual grading by domain experts is intractable, we employ LLM-as-a-judge for QUARCH. Human Validation. While LLM-as-a-judge has gained popularity for evaluating FRQ-style questions (Lee et… view at source ↗
Figure 7
Figure 7. Figure 7: QuArch Website Home Page A LLM USAGE Language models were employed to refine the prose (e.g., grammar, clarity, and style) and to check formatting compliance with venue guidelines. Apart from their explicit roles described in the paper, namely for synthetic QA generation, exam parsing assistance, and evaluation (LLM-as-a-Judge), LLMs were not used to originate substantive scholarly content. All benchmark c… view at source ↗
Figure 8
Figure 8. Figure 8: QuArch Website Submit Question Instructions [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: QuArch Website Submit Question Portal 18 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: QuArch Website LLM Testing on Question Submission [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: QuArch Website Question Submission Scoring [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: QuArch Website Admin Approvals [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: QuArch Website Exam Collection Portal 20 [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Topic-wise characterization of frontier models, categorized by skill. In order, the group [PITH_FULL_IMAGE:figures/full_fig_p022_14.png] view at source ↗
Figure 16
Figure 16. Figure 16: Caption for image 2 You may also use the following macros if required: • STOREADR n: Replace the address field of location n with the contents of the accumulator • LOADADR n: Load the address field of location n into the accumulator Write a macro for LISTPUSH, which pushes the node pointed to by the accumulator to the head of the list. LISTPUSH takes one argument, the memory address of the new node, which… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AgentDSE: Reasoning-Augmented Architectural Design Space Exploration

    cs.AR 2026-06 unverdicted novelty 7.0

    AgentDSE uses an LLM agent in a simulator-in-the-loop setup to achieve competitive or superior architectural designs with up to 100x fewer evaluations than traditional black-box optimization methods.

  2. Economy of Minds: Emerging Multi-Agent Intelligence with Economic Interactions

    cs.CL 2026-06 unverdicted novelty 6.0

    An economy of agents using auctions and wealth accumulation produces emergent multi-step reasoning that outperforms monolithic baselines on five agentic tasks.

Reference graph

Works this paper leans on

44 extracted references · cited by 2 Pith papers

  1. [1]

    • The elementA[m, k]is accessed within the innermost loop (for k in [0, K))

    Matrix A (MxK). • The elementA[m, k]is accessed within the innermost loop (for k in [0, K)). • For a fixedm, the rowA[m, k]is accessed for eachkfrom0toK-1. Thus, for each iteration of the outer loop (for m in [0, M)), the same row is not re-referenced untilMiterations have passed. • Within the inner loop, each element of a row ofAis accessed once. So for ...

  2. [2]

    •B[k, n]is accessed within the innermost loop (for k in [0, K))

    Matrix B (KxN). •B[k, n]is accessed within the innermost loop (for k in [0, K)). 30 Preprint under review • For a fixedn,B[k, n]is accessed for eachk. Each element of a column is accessed once askvaries, and re-referenced afterKiterations for a differentk. • Considering the loop structure: for a givenkandn,B[k, n]is accessed for every min[0, M). Thus, it ...

  3. [3]

    A[m, k]is accessed for everynin[0, N)... it is reusedNtimes,

    Matrix Z (MxN). •Z[m, n]is accessed within the innermost loop (for k in [0, K)). • For a fixedm, n,Z[m, n]is updatedKtimes. It is not re-referenced until the computation for that pair is completed, which happens afterKiterations of the innermost loop. Hence, the RI forZisK. Therefore, the reuse intervals are: • RI of A =N • RI of B =M • RI of Z =K These v...

  4. [4]

    if (B[i]<8888) (condition check) 2.A[i] =A[i]∗C[i](executed ifB[i]<8888) 3.A[i] =A[i] +B[i](executed ifB[i]<8888) 4.C[i] =B[i] + 1(executed ifB[i]<8888)

  5. [5]

    Student D

    if (B[i]>8888) (condition check) 6.A[i] =A[i]∗B[i](executed ifB[i]>8888) SIMD Execution with Branches With 4096 iterations and 64 threads per warp, we have 64 warps total. In SIMD execution: • All threads in a warp execute the same instruction simultaneously • When branches occur, threads not taking a path become inactive • The warp must execute all paths...

  6. [6]

    It writes to a memory range that includes the target addresses

  7. [7]

    It writes zeros to memory (even if to too many locations)

  8. [8]

    closer to being correct,

    Code Snippet (B) completely fails by writing to only one wrong address Answer: (A)— Though flawed, it’s the only option that writes to multiple memory locations in the vicinity of the target range. Question Summary and Relevance:The task required writing zeroes to memory locations from 0x00004000through0x00005000– a range of exactly 1024 word-aligned addr...

  9. [9]

    Two’s complement of 11000110: invert→00111001, add 1→00111010

    SumA+Bexpressed in 8-bit two’s complement notation: A+B= 11000110 To convert this to decimal: since the MSB is 1, it is negative. Two’s complement of 11000110: invert→00111001, add 1→00111010. Decimal value =32 + 16 + 8 + 2 = 58. Therefore, 11000110 represents−58

  10. [10]

    From the table,c out 7 = 1

    Carry Flag (CF) after A + B on x86: The Carry Flag is set if there is a carry-out from the MSB. From the table,c out 7 = 1. Therefore, CF = 1 (set)

  11. [11]

    Memory→C

    Overflow Flag (OF) after A + B on x86: Overflow is set ifc in 7⊕c out 7 = 1. From the table,c in 7 = 1,c out 7 = 1, so OF= 1⊕1 = 0. Therefore, OF = 0 (not set). Final Answer: • SumA+B:11000110(−58in decimal) • Carry Flag: Set (CF = 1) • Overflow Flag: Not set (OF = 0) Question Summary and Relevance:The question under consideration asks students to compute...

  12. [12]

    - Cache State: If the tail pointer is not in the cache, it will transition from I (Invalid) to S (Shared) or E (Exclusive) if it is not shared

    P3 LD tail ptr: - State Transition: The producer loads the tail pointer into its cache. - Cache State: If the tail pointer is not in the cache, it will transition from I (Invalid) to S (Shared) or E (Exclusive) if it is not shared

  13. [13]

    - Cache State: This write will transition the cache line to M (Modified)

    P4 ST message: - State Transition: The producer writes the message to the tail location. - Cache State: This write will transition the cache line to M (Modified)

  14. [14]

    - Cache State: The tail pointer cache line will transition to M (Modified) in the producer’s cache

    P6 ST new tail: - State Transition: The producer updates the tail pointer. - Cache State: The tail pointer cache line will transition to M (Modified) in the producer’s cache

  15. [15]

    - Cache State: The lock cache line will transition back to I (Invalid) or S (Shared) depending on other cores’ states

    P7 ST release lock: - State Transition: The producer releases the lock. - Cache State: The lock cache line will transition back to I (Invalid) or S (Shared) depending on other cores’ states. Consumer Side:1. C2 LD tail ptr: - State Transition: The consumer loads the tail pointer into its cache. - Cache State: If the tail pointer is not in the cache or is ...

  16. [16]

    Producer Acquires Lock (P1 TST try lock): - Transition from I or S to M (1 transition)

  17. [17]

    Producer Loads Tail Pointer (P3 LD tail ptr): - Transition from I to S or E (1 transition)

  18. [18]

    Producer Writes Message (P4 ST message): - Transition from S or E to M (1 transition)

  19. [19]

    Producer Updates Tail Pointer (P6 ST new tail): - Transition from S or E to M (1 transi- tion)

  20. [20]

    Producer Releases Lock (P7 ST release lock): - Transition from M to I (1 transition)

  21. [21]

    — 45 Preprint under review Data/Permission Transfers:- Memory to Cache (Memory→C): When the tail pointer is initially loaded into the cache

    Consumer Loads Tail Pointer (C2 LD tail ptr): - Transition from I to S (1 transition), potentially causing an invalidation in the producer’s cache (1 additional transition). — 45 Preprint under review Data/Permission Transfers:- Memory to Cache (Memory→C): When the tail pointer is initially loaded into the cache. - Cache Invalidation (C invalidates P): Wh...

  22. [22]

    Producer: - Acquire lock: I/S→M (1 transition) - Load tail pointer: I→S/E (1 transition) - Write message: S/E→M (1 transition) - Update tail pointer: S/E→M (1 transition) - Release lock: M→I (1 transition)

  23. [23]

    - 2 transitions on the consumer side

    Consumer: - Load tail pointer: I→S (1 transition) - Invalidate producer’s tail pointer: M →I (1 transition) Total State Transitions per Message: - 5 transitions on the producer side. - 2 transitions on the consumer side. Total: 7 state transitions per message in the steady state. Operation Producer Cache State TransitionConsumer Cache State TransitionTran...

  24. [24]

    Will produce value for R1

    RS A (earliest): Both sources are valid with values 20 and 30. Will produce value for R1. Operation must be ADD R1, ?, ?; since both values are ready: ADD R1, #20, #30 (or using registers with those values)

  25. [25]

    Will produce value for R2

    RS B: SRC1 valid with value 20; SRC2 waiting for RS F. Will produce value for R2. Operation must be ADD R2, #20, ?

  26. [26]

    Will produce value for R5

    RS D: SRC1 waiting for RS B (which produces R2); SRC2 waiting for RS E. Will produce value for R5. Operation must be ADD R5, R2, ?. Multiplication Reservation Stations:

  27. [27]

    Will produce value for R4

    RS C (earliest): Both sources waiting for RS A (which produces R1). Will produce value for R4. Must be using the same value twice: MUL R4, R1, R1

  28. [28]

    Destination register unknown

    RS F: SRC1 waiting for RS A (which produces R1); SRC2 valid with value 20. Destination register unknown. Operation must be MUL ?, R1, #20

  29. [29]

    Reduced Bus Contention and Cache Invalidation

    RS E: SRC1 waiting for RS C (which produces R4); SRC2 waiting for RS A (which produces R1). Destination register unknown. Operation must be MUL ?, R4, R1. Completed Table Reservation Station A: Opcode = ADD, Source1 = #20, Source2 = #30, Destination = R1 Reservation Station B: Opcode = ADD, Source1 = #20, Source2 = ?, Destination = R2 Reservation Station ...

  30. [30]

    Bus doesn’t need to be aware of them

  31. [31]

    No extra hardware (tied to 1) Correct Frontier Model Response: The LOAD RESERVE/STORE CONDITIONAL (LdR/StC) pair is preferable over atomic read-test-modify instructions like TEST&SET for the following reasons:

  32. [32]

    This results in frequent bus traffic and cache invalidations for contending processors, degrading performance

    Reduced Bus Contention and Cache Invalidation TEST&SET performs an atomic read-modify-write operation, which always writes to memory even if the lock acquisition fails. This results in frequent bus traffic and cache invalidations for contending processors, degrading performance. 51 Preprint under review LdR/StC avoids unnecessary writes: – The StC instruc...

  33. [33]

    This creates a scalability bottleneck in multi-processor systems

    No Bus Locking During Critical Section TEST&SET requires locking the memory bus for the entire atomic operation, blocking all other memory accesses during execution. This creates a scalability bottleneck in multi-processor systems. LdR/StC operates without bus locking: – LdR is a standard load instruction, and StC is a conditional store. – The hardware tr...

  34. [34]

    Read the question carefully and understand what is being asked

  35. [35]

    Use the provided context to inform your answer

  36. [36]

    Show your reasoning and work where appropriate

  37. [37]

    Be precise and accurate in your response

  38. [38]

    student LLM,

    If you’re unsure about something, acknowledge the uncertainty Question Context: {context} {context images placeholder} Question: {question} Please provide your answer: This prompt is used to simulate the “student LLM,” where models act as exam-takers solving ar- chitecture questions. The emphasis is on structured, step-by-step reasoning, accuracy, and cla...

  39. [39]

    The original question and context

  40. [40]

    The correct solution

  41. [41]

    LLM-as-a-Judge

    A student’s answer to the question Your task is to carefully evaluate whether the student’s answer is correct, partially correct, or incorrect by comparing it to the provided solution. Evaluation criteria: • CORRECT: The answer is accurate, complete, and demonstrates proper under- standing • PARTIALLY-CORRECT: The answer shows some understanding but has s...

  42. [42]

    If part of another problem is included, ignore this and only focus on the current pdf file

    A PDF file containing one problem of an exam. If part of another problem is included, ignore this and only focus on the current pdf file

  43. [43]

    , ‘‘subproblem context figures

    A .json-styled txt file containing the problem’s extracted text. It may contain some or all of the following empty-list fields:‘‘problem context figures", ‘‘subproblem context figures",‘‘subproblem solution figures", and‘‘problem solution figures"

  44. [44]

    ) – The sub-problem question (‘‘subproblem context figures

    PNG images containing tables, diagrams, circuit schematics, or block diagrams that may or may not pertain to this problem. The names of the images provided are as follows, in order:{images}. The given problem may be a standalone problem or consist of multiple sub-problems. Each problem or sub-problem may contain: – Main context figures which are necessary...