{"id":"52a45d8b-d4f3-493a-9f89-36ed57a4a704","arxiv_id":"2506.22679","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Decoder-only Llama-3.1 with in-context examples outperformed encoder-only sentiment models at detecting micro-behaviors in NASA team transcripts, though the few-shot setup sampled examples from the same teams as the test turns.","lead":"This study tested whether large language models can detect uplifting and discouraging micro-behaviors in team conversations from simulated space missions using only text transcripts. The best Llama-3.1 model reached 44% macro F1 on 3-way and 68% on binary classification, but the few-shot examples came from the same teams being tested.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Llama's few-shot evaluation samples in-context examples from the same team as each test turn, while encoder-only models are evaluated on held-out teams, so the reported comparison is not team-independent and may vanish under a held-out-team protocol.","rationale":"The paper's central claim is that instruction-tuned Llama-3.1 outperforms fine-tuned encoder-only models for micro-behavior detection. The most load-bearing condition for that claim is that the two model families are compared under an equivalent generalization protocol. That condition fails. In Section 4.1, encoder-only models are fine-tuned and evaluated with leave-one-team-out cross-validation; in Section 4.2, Llama's few-shot examples are explicitly sampled from the same team as each test turn, and results are aggregated over all turns except the in-context examples. This makes the Llama evaluation a within-team, in-distribution few-shot task, whereas the encoder evaluation is a cross-team, out-of-distribution task. Since in-context examples from the same team can encode team-specific vocabulary, conversational style, and label patterns, the reported performance gap is not diagnostic of model capability. The absence of confidence intervals and significance tests across the many configurations explored (k, m, coding definitions) compounds this, because the headline numbers are best-selection maxima. I agree with the reader's identification of this weak assumption. The appropriate resolution is a held-out-team few-shot evaluation for Llama; until then the central comparison is not valid, supporting the reader's REJECT verdict. This is not an accusation of bad faith; it is a straightforward protocol mismatch that the paper itself states in Section 4.2.","tokens_in":7955,"tokens_out":3598,"duration_ms":64521,"concrete_test":"Re-run the decoder-only evaluation with a strictly held-out-team protocol: for each test turn, sample the m few-shot examples only from conversations of the other four teams (m=3,5,7; k=3,4; with and without coding definitions), keeping all other settings identical, and compute macro F1 per task. Also report per-team macro F1 and bootstrap confidence intervals for both encoders and Llama. If Llama's held-out-team macro F1 is not better than, or not significantly different from, RoBERTa's 41% and DistilBERT's 57%, the claimed superiority is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 states that \"The few-shot examples for generating the micro-behavior corresponding to a conversational turn are randomly sampled from the conversations of the same team that the turn belongs to,\" and results are reported over \"all conversational turns except the in-context learning examples across all teams.\" Thus Llama-3.1 is evaluated in a within-team few-shot regime: for each test turn, the prompt contains labeled examples drawn from the same team's conversations, potentially from the same task session. In contrast, the encoder-only baselines (Section 4.1) are evaluated with leave-one-team-out cross-validation: the model is fine-tuned on four teams and tested on the held-out fifth team. The two pipelines answer different questions: Llama is asked to classify a turn after seeing examples from that same team's style and label distribution, while RoBERTa/DistilBERT must generalize to an entirely unseen team. This asymmetry alone can explain the reported Llama advantage (44% vs 41% for 3-way; 68% vs 57% for binary), because same-team in-context examples provide team-specific lexical and conversational cues that are unavailable in the held-out-team setting. The paper also selects the best configuration across k, m, and coding-definition variants (Figure 4) without reporting variance or significance tests, so the numeric comparisons are not established. The central claim that decoder-only LLMs are more feasible for micro-behavior detection is not supported by a valid comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a feasibility study of large language models (LLMs) for detecting micro-behaviors (uplifting, discouraging, neither) in conversational turns from transcripts of five teams in NASA's Human Exploration Research Analog (HERA) missions. It compares zero-shot, fine-tuned, and paraphrase-augmented fine-tuned encoder-only models (RoBERTa, DistilBERT) with few-shot prompting of Llama-3.1-8B-Instruct. The authors report best macro F1-scores of 44% (Llama) versus 41% (RoBERTa) for 3-way classification and 68% (Llama) versus 57% (DistilBERT) for binary classification, concluding that decoder-only LLMs with few-shot examples are more feasible for this task. The paper also reports that encoder-only models fail to detect discouraging turns and that paraphrase augmentation is not consistently beneficial.","tokens_in":8248,"tokens_out":5207,"duration_ms":59512,"significance":"If the reported comparison were valid, the paper would provide a useful empirical data point for the team-process and spoken-language-understanding communities: an 8B instruction-tuned model with a few in-context examples outperforming fine-tuned encoders on a real, high-stakes collaborative task. The dataset from the HERA analog mission is a valuable resource, and the task of detecting both positive and negative micro-behaviors from text alone is relatively unexplored. The paper's principal strength is its exploration of several modeling paradigms on a practically relevant corpus. However, the central claim is undermined by an evaluation asymmetry between the decoder and encoder pipelines, and by the absence of any variance or significance analysis. As presented, the paper does not establish that decoder-only LLMs are more feasible for micro-behavior detection; the observed advantage may be an artifact of the experimental protocol. With a corrected protocol and proper statistical reporting, the study could still be a meaningful contribution.","major_comments":[{"comment":"The few-shot evaluation for Llama draws in-context examples from the same team as the test turn (Section 4.2: 'The few-shot examples ... are randomly sampled from the conversations of the same team that the turn belongs to'), whereas the encoder-only models are evaluated with leave-one-team-out cross-validation (Section 4.1: 'holding out 1 team and randomly splitting the input contexts from the remaining 4 teams'). This is an asymmetric comparison: Llama sees team-specific lexical and conversational cues in its prompt that are not available to the encoder baselines, which must generalize to an entirely unseen team. The reported gaps (44% vs 41%; 68% vs 57%) could be explained by this protocol mismatch rather than by model capability. To support the abstract's central claim, the authors must re-evaluate Llama under a matched held-out-team protocol, e.g., sampling few-shot examples only from the training teams and testing on the held-out team, or otherwise give the encoder models access to the same within-team information.","section":"Section 4.2 (few-shot sampling) and Section 4.1 (cross-validation)"},{"comment":"No confidence intervals, per-fold variance, or significance tests are reported anywhere in the results. The dataset is small (five teams), and the headline differences are modest (3 points in 3-way macro F1). Moreover, the best configurations are selected post hoc from a grid over context length k, number of examples m, and whether coding definitions are included; Figure 4 shows many configurations, and Tables 1-3 report only selected maxima. This selection without correction or variance reporting makes the reported superiority of Llama unsubstantiated. The authors should provide per-fold results, a bootstrap or permutation test over teams (or at minimum over turns), and either pre-specify the configuration selection rule or report all configurations with appropriate multiplicity corrections.","section":"Section 5 and Figure 4"},{"comment":"It is not stated whether the binary classification for Llama excludes 'neither' turns, as is done for DistilBERT (Section 4.1: 'binary classification on conversational turns that are valenced, i.e., labeled as either discouraging or uplifting'). If the two binary tasks use different data subsets, the comparison in Figure 4(b) versus Table 2 is invalid. The authors should state explicitly which turns are included in the binary evaluation for each model and ensure the evaluation sets are identical.","section":"Section 4.2 (binary task)"},{"comment":"Table 3 compares precision and recall between 'the best RoBERTa configuration' and 'the best Llama-3.1 configuration', but the specific configurations are not identified in the table or the text. Without knowing which k, m, weighting, and cross-validation fold produced these numbers, and without any uncertainty estimate, the claim of 'significantly better recall' (Section 5) is unsupported. The authors should identify the exact configurations and provide error bars or significance tests for these per-class metrics.","section":"Table 3"}],"minor_comments":[{"comment":"The paper states that k=2 previous turns and m>7 few-shot examples were excluded because of 'poor results' or 'diminishing returns', but the corresponding results are not shown. Reporting these in an appendix would make the claim verifiable.","section":"Section 4.1, 4.2"},{"comment":"The use of off-the-shelf sentiment models as zero-shot sequence classifiers maps sentiment labels to micro-behavior labels; this is a strong assumption, because sentiment is not equivalent to the context-dependent micro-behavior construct. A sentence acknowledging this limitation would help calibrate reader expectations.","section":"Section 4.1"},{"comment":"The word 'significantly' is used without any statistical test ('significantly better recall'); please replace with a quantitative statement or add a supporting test.","section":"Section 5"},{"comment":"The prompt example in Figure 3 would be easier to follow if the location where the coding definition (when used) is inserted were marked clearly in the figure, since the paper reports that including definitions changes performance.","section":"Figure 3"},{"comment":"Reference [8] is missing publication venue information; please complete it. Also, reference [3] appears in a non-standard format; please use the venue's citation style.","section":"References"},{"comment":"The conclusion states that 'LLMs are feasible for this task' without noting that the results are based on a single analog mission corpus with five teams; adding a caveat about generalizability would be more accurate.","section":"Abstract / Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The central comparison of the paper is invalid as designed, but the flaw is correctable by re-running the Llama few-shot evaluation with held-out-team sampling and by adding statistical significance measures. If the re-evaluated results continue to show an advantage for Llama, the paper would be a worthwhile contribution; if the advantage disappears, the paper's conclusions would need to be substantially revised. Given the scope and the effort required, a major revision is appropriate, but the authors should be prepared for the possibility that the reported superiority is not reproducible under a matched protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what you should know: the paper's headline result—Llama-3.1 beating RoBERTa/DistilBERT on micro-behavior detection—is not supported by a valid comparison. The few-shot examples for Llama are sampled from the same team as the test turn (Section 4.2), while the encoder baselines are evaluated with leave-one-team-out cross-validation. That is an apples-to-oranges setup. The 44% vs 41% and 68% vs 57% gaps could easily come from team-specific lexical/style cues in the prompt rather than from decoder-only superiority.\n\nThat said, the paper does something useful. It applies decoder-only in-context learning to a genuinely new task—detecting uplifting and discouraging micro-behaviors in oral multiparty team transcripts from NASA's HERA analog missions. The dataset is real, the class imbalance is stark (3.3% discouraging), and the paper is transparent about failures, including encoder models never predicting the discouraging class and paraphrase augmentation sometimes hurting. It reports per-class precision/recall, which is more than many feasibility studies do.\n\nThe soft spots beyond the protocol mismatch: no confidence intervals or significance tests, the best numbers are selected across a grid of k, m, and coding-definition variants without variance reporting, and no code or data is released. The fine-tuning results for Llama are mentioned as a drastic failure but not shown, which is fine but worth noting. None of these are fatal on their own, but the evaluation design is a load-bearing flaw.\n\nWho is this for: anyone working on team communication analytics or NLP for high-stakes training environments. The paper gives a clear picture of what does not work in this low-resource, imbalanced setting. If the authors redo the decoder-only evaluation with a true held-out-team few-shot protocol (no same-team examples), report variance, and release code, the result would be worth a serious look. As is, I would not cite the comparative claim.\n\nRecommendation: send to peer review, but be prepared for a major revision. This is not a desk reject—the task and data deserve scrutiny—but the central claim needs a corrected evaluation before publication.","headline":"Same-team few-shot examples undermine the headline Llama comparison; the task and data are real and the paper is fixable.","tokens_in":8789,"tokens_out":2286,"would_cite":false,"duration_ms":24592,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Decoder-only LLMs with few-shot examples detect team micro-behaviors from transcripts better than fine-tuned encoder-only models, reaching 44% and 68% macro F1.","keywords":["large language models","micro-behavior detection","team communication","in-context learning","few-shot classification","space missions","transcript analysis","Llama-3.1"],"falsifier":"Hold out an entire crew from the few-shot examples as well as from the test set, and check whether Llama-3.1's macro F1 on that crew's turns stays near 44% (three-way) and 68% (binary).","tokens_in":7746,"feed_emoji":"🚀","tokens_out":11073,"duration_ms":104532,"temperature":0.7,"pith_summary":"The paper asks whether large language models can read conversation transcripts from simulated space missions and pick out the small, momentary signs that crew members are uplifting or discouraging one another. It compares fine-tuned encoder-only models (RoBERTa for three-way classification, DistilBERT for binary classification) against a much larger decoder-only model (Llama-3.1) prompted with a handful of labeled examples. The central finding is that the decoder-only route is more feasible: its best configurations reach 44% on the three-way task and 68% on the binary task (macro F1, the average of per-class F1 scores), while the encoder-only models fail to detect the rare discouraging category almost entirely, with recall near 1% in the best configuration. If this result holds, subtle team dynamics can be monitored from text alone in settings where audio is unavailable and labeled data are too scarce to fine-tune large models.","feed_headline":"Few-shot Llama-3.1 beats fine-tuned encoders on micro-behaviors","feed_subtitle":"It catches rare discouraging turns from space-mission transcripts at 68% binary macro F1.","key_machinery":"The central mechanism is few-shot in-context prompting: the target turn is embedded in a prompt that also contains its event type, three or four previous turns, the following response, and several labeled examples of each micro-behavior class, and an instruction-tuned causal language model is asked to generate a single label. The paper attributes the decoder-only advantage to autoregressive token-by-token prediction, which can assign high probability to the distinctive words that mark a rare class such as discouraging speech, whereas encoder-only models pool information across the whole sequence and tend to wash out those signals. Supporting devices include weighted cross-entropy loss and paraphrase-augmented fine-tuning for the encoders, and 8-bit quantization and truncated generation for the decoder-only model.","core_discovery":"Using transcripts from five crews in a 45-day simulated space mission, the paper compares two routes to micro-behavior classification: fine-tuning encoder-only language models and few-shot prompting of an instruction-tuned decoder-only language model. Its central result is that the decoder-only model outperforms the fine-tuned encoders, improving macro F1 from 41% to 44% in the three-way task and from 57% to 68% in the binary task, with the gain concentrated in recall for the rare discouraging class (28% versus 1% in the best configurations). The paper interprets this as evidence that decoder-only models can detect subtle, underrepresented micro-behaviors from text transcripts alone, and that in-context learning with a handful of examples is a viable alternative to fine-tuning on small, imbalanced datasets.","pith_inferences":["The reported decoder-only advantage may be partly inflated because the few-shot examples are drawn from the same team as the turn being classified; a test with examples taken only from other crews would give a fairer estimate of cross-team generalization.","The token-level explanation suggests that deliberately selecting in-context examples rich in rare-class vocabulary, or balancing examples across classes, could push recall for discouraging speech higher still.","The binary result of 68% is close to a usable screening signal, but operational use would require setting a cost for confusing uplifting and discouraging turns, since the two errors have very different consequences for team support.","With only five crews in the data, team-level variance is a serious unknown; leave-multiple-teams-out evaluation or additional analog-mission transcripts would show how stable the 44% and 68% figures are."],"forward_implications":["Transcript-only analysis of team communication can identify valenced micro-behaviors with moderate macro F1 without needing audio, which matters for space missions and other high-stakes settings where text is the only accessible data.","The rare discouraging class, which fine-tuned encoders almost never detect, is recoverable by a decoder-only model with as few as seven in-context examples.","For small, imbalanced datasets, few-shot prompting of a large instruction-tuned model can beat fine-tuning of smaller encoder-only models, avoiding the catastrophic forgetting observed when the 8-billion-parameter decoder was fine-tuned.","Three previous conversational turns appear sufficient context; larger context windows and more in-context examples give diminishing returns or degrade performance.","Paraphrase-based data augmentation is not reliably beneficial and can hurt performance on very small datasets, as the DistilBERT binary results show."],"supporting_citations":[{"why":"Supplies the Llama-3.1-8B-Instruct model that produces the paper's best few-shot results.","marker":"[24]"},{"why":"Supplies the RoBERTa architecture used for the three-way encoder-only baseline.","marker":"[18]"},{"why":"Supplies the DistilBERT architecture used for the binary encoder-only baseline.","marker":"[19]"},{"why":"Supplies the sentiment-tuned RoBERTa checkpoint that is zero-shot tested and fine-tuned for three-way classification.","marker":"[20]"},{"why":"Supplies the SST-2-tuned DistilBERT checkpoint used for binary classification.","marker":"[21]"},{"why":"Establishes the in-context learning paradigm that the decoder-only few-shot prompts rely on.","marker":"[23]"},{"why":"Provides the adapted Smith and Griffiths framework from which the uplifting/discouraging violation labels are taken.","marker":"[10]"},{"why":"Supports the definition and coding of micro-behaviors in team settings used to annotate the transcripts.","marker":"[11]"}],"fun_headline_variants":["Llama-3.1 edges out fine-tuned encoders on micro-behavior detection","Few-shot Llama-3.1 tops fine-tuned models for team talk micro-behaviors","Decoder-only LLM wins on rare micro-behaviors in space transcripts","Llama-3.1 outperforms fine-tuned encoders on micro-behavior spotting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that drawing the few-shot examples from the same team as the turn being classified gives a fair estimate of how the model would perform on a team it has never seen.","fun_headline_variants_meta":{"raw":{"variants":["Llama-3.1 edges out fine-tuned encoders on micro-behavior detection","Few-shot Llama-3.1 tops fine-tuned models for team talk micro-behaviors","Decoder-only LLM wins on rare micro-behaviors in space transcripts","Llama-3.1 outperforms fine-tuned encoders on micro-behavior spotting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000359,"raw_usage":{"total_tokens":1929,"prompt_tokens":915,"completion_tokens":1014,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":921}},"tokens_in":531,"tokens_out":1014,"duration_ms":6810,"temperature":1.0,"reasoning_tokens":921,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:00:41.121990+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out an entire crew from the few-shot examples as well as from the test set, and check whether Llama-3.1's macro F1 on that crew's turns stays near 44% (three-way) and 68% (binary).","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RoBERTa architecture used for the three-way encoder-only baseline."},{"cited_title":"Automated detection of racial microaggressions using machine learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the DistilBERT architecture used for the binary encoder-only baseline."},{"cited_title":"Finding mi- croaggressions in the wild: A case for locating elusive phenom- ena in social media posts,","cited_arxiv_id":null,"evidence_quote":"Supplies the sentiment-tuned RoBERTa checkpoint that is zero-shot tested and fine-tuned for three-way classification."},{"cited_title":"Leveraging bias in pre- trained word embeddings for unsupervised microaggression de- tection,","cited_arxiv_id":null,"evidence_quote":"Supplies the SST-2-tuned DistilBERT checkpoint used for binary classification."},{"cited_title":"Towards identification of microaggressions in real-life and scripted conversations, using context-aware machine learning techniques,","cited_arxiv_id":null,"evidence_quote":"Establishes the in-context learning paradigm that the decoder-only few-shot prompts rely on."},{"cited_title":"Chain-of-thought prompting elic- its reasoning in large language models,","cited_arxiv_id":null,"evidence_quote":"Supports the definition and coding of micro-behaviors in team settings used to annotate the transcripts."}],"review_version":1}