{"id":"369793b6-120f-49fa-812f-6c587d2e8144","arxiv_id":"2501.03295","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A no-training LLM pipeline with retrieval-augmented feature selection and few-shot in-context learning predicts penicillin concentration and melt flow rate while generating explanations and confidence scores.","lead":"This paper proposes an LLM-based soft sensor that predicts industrial quality variables from text prompts instead of trained models, using a few example cases and retrieved domain knowledge. It reports competitive accuracy plus self-explanations and uncertainty scores on penicillin fermentation and polypropylene production datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RAC retrieval may leak test targets into context: IPDVS is built from the same 200 samples, with no stated test exclusion, potentially inflating Table 6 gains.","rationale":"The reader's weakest_assumption correctly identifies the most load-bearing risk: the RAC retrieval pool may contain the test samples themselves or near-duplicates, making the headline Table 6 improvement an artifact of label leakage. This is a direct threat to the strongest_claim's retrieval-augmented component, and it is concrete because the paper explicitly describes building the IPDVS from the same 200 samples used in FSC without stating test exclusion. I agree with the reader's conditional verdict rather than escalating to rejection: the FSC results (Table 5) provide independent, albeit weaker, evidence that a prompt-based LLM can be competitive, and the leakage concern is falsifiable with a clean split. Secondary issues (no error bars, temperature 0 vs. probabilistic confidence intervals, missing code/prompts) are real but less central; the proposed concrete test would settle whether the RAC advantage survives a clean protocol. If it does not survive, the paper should either reposition RAC as a retrieval-augmented interpolator rather than evidence for ICL soft sensing, or remove the SOTA claim for that variant.","tokens_in":19554,"tokens_out":2637,"duration_ms":27302,"concrete_test":"Re-run the LLM-UFSS-RAC experiments with a strict temporal split: for each dataset, hold out the last 20% of each batch (or a contiguous time block) as test samples, build the IPDVS only from the remaining earlier samples, and retrieve neighbors from that split IPDVS. Also run a leave-one-out control where the exact test sample is deleted from the IPDVS. Recompute Table 6 MAE/RMSE for LLM-UFSS-RAC and RFR under identical retrieved contexts. If the LLM-UFSS-RAC advantage over RFR drops below 10% relative improvement, the reported RAC gains are attributable to retrieval leakage, not to ICL reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of state-of-the-art predictive performance rests in part on LLM-UFSS-RAC results (Table 6). The Experimental settings state: 'the 200 samples from LLM-UFSS-FSC were used to construct the IPDVS, matching each test sample with similar ones from the IPDVS to form the context.' No statement excludes test samples from the IPDVS, and both datasets are time-series-like (batch fermentation trajectories and a continuous polypropylene process) with highly autocorrelated auxiliary variables. In IndPensim, penicillin concentration evolves smoothly and monotonically after 24h; nearest-neighbor retrieval in feature space will tend to return temporally adjacent samples whose target values are very close to the query target. The LLM can then generate predictions by copying or interpolating those retrieved labels, so the reported MAE reductions (45.93% and 16.23% over RFR at 10-shot) may reflect target leakage through retrieval rather than ICL-based soft-sensing ability. This does not invalidate the FSC results in Table 5, which use random context samples, but RAC is presented as the primary retrieval-augmented contribution and the main evidence for 'finer-grained' reasoning. Without a demonstrable test/retrieval separation, the RAC component of the strongest_claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLM-FUESS, a two-stage soft-sensor framework built on large language models with in-context learning instead of supervised training. Stage one, LLM-ZAVS, uses retrieval-augmented generation from an industrial knowledge vector store to select auxiliary variables zero-shot. Stage two, LLM-UFSS, formats numerical samples as text prompts, uses either random few-shot contexts (FSC) or retrieval-augmented contexts from an industrial process data vector store (RAC), and produces point predictions, confidence intervals, confidence scores, and textual self-explanations. Experiments on the IndPensim penicillin fermentation simulator and a polypropylene production dataset compare the method against PCR, k-NN, MLP, and random forest regression, and report lower MAE/RMSE in several configurations, plus robustness to missing values and ablations of prompt components.","tokens_in":19820,"tokens_out":3397,"duration_ms":119025,"significance":"If the reported results hold, the paper would be a noteworthy demonstration that a no-training, prompt-based LLM pipeline can compete with trained regression models for industrial quality prediction, with added interpretability and uncertainty outputs. The paper has genuine strengths: it evaluates against external numerical baselines rather than only self-comparisons, tests multiple LLMs in an ablation, uses raw unnormalized inputs, and provides detailed prompt templates and JSON output formats that support reproducibility. However, the significance is currently tempered by a plausible data-leakage path in the retrieval-augmented variant, by the absence of any variance or significance analysis for the headline comparisons, and by the self-referential nature of the confidence-score validation.","major_comments":[{"comment":"The RAC results in Table 6 and Figures 8-10 rest on a retrieval protocol that may leak target information into the context. The paper states that 'the 200 samples from LLM-UFSS-FSC were used to construct the IPDVS' but never states that the test samples are excluded from this pool. Both datasets are time-series-like, and in IndPensim the penicillin concentration evolves smoothly and monotonically after 24 h, so nearest-neighbor retrieval in feature space will tend to return temporally adjacent samples whose target values are very close to the query target. The LLM could then produce predictions by copying or interpolating the retrieved labels, inflating the reported 10-shot MAE reductions of 45.93% and 16.23% over RFR. This is a load-bearing issue for the 'state-of-the-art predictive performance' claim; the authors should either demonstrate that test samples were excluded from the IPDVS or rerun the RAC experiments with a retrieval pool that provably excludes all test samples and their near-duplicates.","section":"Experimental settings, paragraph 3 and Analysis of LLM-UFSS-RAC"},{"comment":"All headline comparisons are reported as single point estimates, with no standard deviations, confidence intervals, or significance tests for either the proposed method or the baselines. Because LLM outputs are stochastic even at temperature 0 and the baselines involve hyperparameter selection, statements such as 'achieved the lowest MAE and RMSE across both datasets' are not statistically supported. The authors should report repeated-run means with variability and, where appropriate, paired significance tests across the same test samples.","section":"Tables 5-7 and Figure 16"},{"comment":"The confidence score is generated by the same LLM that produces the prediction it is meant to assess, so the observed alignment between high confidence and low error in Figure 11b is partly self-referential. Similarly, the higher confidence scores for LLM-UFSS-RAC in Table 8 may reflect the similarity of retrieved contexts rather than calibrated uncertainty. The trustworthiness claim needs an external calibration check, for example comparing confidence scores against empirical error rates or using a held-out calibration set, rather than only showing that the model's self-reported confidence correlates with its own errors.","section":"Confidence score output analysis, Figure 11 and Table 8"},{"comment":"The RFE row for linear regression on IndPensim reports MAE 0.046, the best value in that column, but R² = 0.096, the worst value in that column and inconsistent with such a low MAE. This is likely a reporting error, but it directly affects the claim that LLM-ZAVS is competitive with numerical feature selection methods. The authors should verify the entries in Table 3 and correct the inconsistent value.","section":"Table 3, IndPensim LR row for RFE"}],"minor_comments":[{"comment":"The paper sets temperature to 0 for consistency but then constructs 95% confidence intervals by repeated experiments; please clarify how output diversity is obtained under temperature 0 (for example API nondeterminism or sampling parameters other than temperature) and whether the same randomness affects the single-evaluation RAC results.","section":"Experimental settings, implementation details"},{"comment":"The caption of Figure 13b describes a local explanation of 'oxygen uptake rate for melt flow rate', but oxygen uptake rate is an IndPensim variable and the surrounding text describes the variable's effect on penicillin concentration; the caption or the dataset association should be corrected.","section":"Figure 13b caption and Analysis of explanation results"},{"comment":"The text says each set of 20 test samples in Figure 6 uses the same 20 training samples for contextualization, while the experimental settings describe 200 randomly chosen samples forming 10 contexts of 20 samples; the relationship between the figure subset and the full protocol should be stated explicitly.","section":"Figure 6 and Experimental settings"},{"comment":"There are numerous typos that should be corrected, including 'feasubke' in the LLM-UFSS methods section, 'Performace' in Table 6, 'comparision' in Table 5, 'raitos' in Table 7, 'pediction' and 'dfferent' in Figure 16, and 'reults' in the Introduction.","section":"Throughout"},{"comment":"The statement that each RAC test sample is evaluated only once means that per-sample confidence intervals are not available for RAC, limiting direct uncertainty comparisons between FSC and RAC; this limitation should be acknowledged where the uncertainty visualizations are discussed.","section":"Experimental settings, LLM-UFSS-RAC"}],"recommendation":"major_revision","confidential_remarks":"The central idea is within scope for a machine-learning application paper, and the FSC results may survive the leakage concern, but the RAC protocol as written has a checkable leakage path that must be resolved before the SOTA claim can be accepted. The lack of repeated-run statistics is also a standard but necessary fix. I do not see grounds for rejection if the authors can verify the retrieval/test separation and correct the Table 3 inconsistency."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper is a serious application-level contribution: it's the first soft sensor framework built entirely on LLM in-context learning, with retrieval-augmented feature selection, few-shot prediction, self-explanation, and uncertainty outputs. If it works, it gives process-control practitioners a no-training alternative that tolerates missing values and explains itself. The authors build two datasets (penicillin fermentation, polypropylene), run ablations on prompt components and LLM choices, and propose a consistency metric for feature selection. That is real engineering work, and the idea is worth taking seriously.\n\nThe soft spots are concrete, not manufactured. The biggest one is the RAC retrieval protocol. The IPDVS is constructed from the same 200 samples used for testing, and the paper never states that test samples are excluded from the retrieval pool. On a smooth fermentation trajectory, nearest-neighbor retrieval will often return temporally adjacent samples whose target values are nearly identical to the query. The LLM can interpolate those labels, so the Table 6 gains (45.93% and 16.23% MAE reduction over RFR) may reflect target leakage rather than ICL reasoning. This needs an explicit train/test split in the vector store.\n\nSecond, the uncertainty quantification is internally inconsistent. The implementation sets temperature to 0 'for consistency', but then repeats each test sample 10 times to build confidence intervals. At temperature 0, repeated outputs are identical, so any confidence interval is degenerate. The same contradiction shows up in the feature selector: ASCS 0.75 with five runs implies variability, which temperature 0 would not produce. Either the temperature setting is wrong or the repetition protocol is incomplete; as written, the uncertainty claims are not supported.\n\nThird, the headline 'state-of-the-art' is not established. Tables 5-7 report point estimates only, with no variance or significance tests. The FSC results in Table 5 are not affected by the retrieval leak, but they still need error bars before I'd trust the ranking.\n\nThe paper does not ship code or exact prompts, which hurts reproducibility given how sensitive LLM outputs are to prompt wording.\n\nWho is this for? Practitioners who wonder whether LLM-based soft sensors can replace trained regressors, and researchers working on LLMs for tabular/structured data. The idea deserves a serious referee, but the paper needs major revisions: fix the retrieval protocol, correct the temperature/repetition story, add error bars, and release artifacts. I'd send it to review with a clear request to verify those points.","headline":"A genuinely novel LLM-based soft sensor framework with a promising no-training pipeline, but the RAC retrieval protocol and the temperature-0 uncertainty setup both need fixing before the SOTA claim is credible.","tokens_in":20355,"tokens_out":2700,"would_cite":false,"duration_ms":24809,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a two-stage LLM pipeline using in-context learning can replace supervised training for industrial soft sensing, achieving state-of-the-art prediction errors on two process datasets while generating explanations and…","keywords":["soft sensor","in-context learning","large language models","retrieval-augmented generation","uncertainty quantification","self-explanation","industrial process monitoring","few-shot prediction"],"falsifier":"Repeat the LLM-UFSS-RAC protocol with an explicitly held-out test set: build the IPDVS only from training samples, forbid any retrieved neighbor from being within a short time window of the test sample, and compare the resulting MAE with random forest trained on the same retrieved samples. If the gap closes or reverses, the reported advantage came from target leakage rather than in-context reasoning.","tokens_in":19371,"feed_emoji":"🏭","tokens_out":6362,"duration_ms":54771,"temperature":0.7,"pith_summary":"This paper claims that soft sensors, models that estimate hard-to-measure quality variables from easy-to-measure process variables, can be built without any supervised training by reformulating the task as text-based in-context learning. The proposed LLM-FUESS pipeline has two stages: LLM-ZAVS selects auxiliary variables zero-shot using domain knowledge retrieved from a vector store, and LLM-UFSS turns a few labeled samples into text demonstrations in a prompt to predict the target variable. On a penicillin fermentation simulation and a polypropylene production dataset, the paper reports that this prompt-only method achieves lower mean absolute error and root mean square error than random forest, MLP, k-NN, and PCR baselines, handles up to 50% missing values, and supplies confidence intervals, confidence scores, and human-readable explanations. If correct, the result means an industrial soft sensor could be deployed by filling in a prompt template, with no model training, no normalization, and no imputation.","feed_headline":"Prompt-only LLM predicts plant quality without training a model","feed_subtitle":"It retrieves domain knowledge, reads text examples, and returns predictions with confidence intervals.","key_machinery":"The load-bearing mechanism is in-context learning applied to text-formatted structured data: each training sample becomes a line such as 'Aeration rate: 6.5, Sugar feed rate: 0.1, ..., Penicillin concentration: 15.2', and the LLM predicts the primary variable for a test line from a handful of such demonstrations in the prompt. Supporting machinery includes the Industrial Knowledge Vector Store and the Industrial Process Data Vector Store, both enabled by retrieval-augmented generation, the AVS-PT and SS-PT fill-in-the-blank prompt templates with role, chain-of-thought, and emotional-stimulus prompts, and two uncertainty channels: repeated sampling to build confidence intervals and a prompted confidence score. The paper's Average Selection Consistency Score (ASCS) measures how consistently LLM-ZAVS picks the same variables across repeated runs.","core_discovery":"On the paper's own terms, the central discovery is that large language models can act as soft sensors through in-context learning alone, matching or exceeding conventional trained regressors without parameter updates. The authors introduce and evaluate LLM-FUESS, a two-stage framework in which LLM-ZAVS performs zero-shot auxiliary variable selection with retrieval-augmented generation from an industrial knowledge vector store, and LLM-UFSS performs few-shot prediction by formatting structured process data as text and inserting the selected variables and retrieved similar samples into a soft-sensor prompt template. In the fixed-context variant, LLM-UFSS-FSC achieves the lowest MAE and RMSE on both datasets among all compared methods; in the retrieval-augmented variant, LLM-UFSS-RAC lowers MAE further by retrieving similar samples from an industrial process data vector store, while also producing confidence scores and confidence intervals that track prediction error. The paper also reports that prompting the model to explain its reasoning and rate its confidence improves accuracy, and that replacing missing values with 'N/A' preserves useful predictions at missing rates up to 50%.","pith_inferences":["The strongest caution a reader should carry is about the retrieval-augmented experiments: the paper builds the IPDVS from the same 200 samples used in the fixed-context variant and does not state that test samples are excluded, so nearest-neighbor retrieval could return the test sample itself or a temporally adjacent point whose target value is nearly identical, which would inflate the reported ga","A fair comparison against trained baselines would require a strict split: build the data vector store only from training samples, ensure retrieved neighbors are not within a few time steps of the test point, and re-run both variants.","If the leakage concern is resolved and the gains persist, the same template could be tested for online streaming soft sensing, where the retrieved context naturally contains the most recent samples rather than randomly drawn historical ones.","The zero-shot variable selector's competitive results suggest an interesting direction: using LLMs as a prior feature-selection step before any numerical model, which could reduce dimensionality and preprocessing costs even in conventional pipelines."],"forward_implications":["If the central claim holds, soft sensor models can be developed for a new process by editing a few placeholders in a prompt template, removing the need for dataset-specific training pipelines and machine-learning expertise.","The absence of parameter updates means the usual failure modes of trained soft sensors, such as overfitting, underfitting, gradient instability, and sensitivity to initialization, are bypassed by construction.","Missing values can be passed through as 'N/A' rather than imputed, so data preprocessing is simplified while retaining predictive accuracy at missing rates up to 50%.","Operators would receive not only a prediction but a confidence interval, a confidence score, and a natural-language explanation, giving them information about when to trust the output.","Choice of underlying large language model matters: the paper's ablation shows a large performance gap between weaker and stronger base models, so the benefit is tied to the capability of the base model."],"supporting_citations":[{"why":"Supplies the default API-based large language model used throughout the LLM-FUESS experiments.","marker":"[26]"},{"why":"Defines in-context learning, the core paradigm that lets the soft sensor predict without parameter updates.","marker":"[35]"},{"why":"Provides the implicit Bayesian inference framing used to justify why in-context demonstrations transfer to new tasks.","marker":"[36]"},{"why":"Establishes retrieval-augmented generation, the technique behind both the knowledge vector store and the process data vector store.","marker":"[38]"},{"why":"Introduces chain-of-thought prompting, which the templates use to generate step-by-step explanations and improved predictions.","marker":"[40]"},{"why":"Supplies the IndPensim penicillin fermentation simulation used as one of the two case-study datasets.","marker":"[45]"},{"why":"Provides the stronger alternative base model tested in the ablation study that shows LLM choice affects performance.","marker":"[27]"}],"fun_headline_variants":["LLM-based soft sensor skips training, adds uncertainty estimates","In-context learning turns LLMs into soft sensors with confidence","Few-shot LLM soft sensor with retrieval and self-explanation","LLM soft sensor: no parameter updates, just prompts and retrieval","Uncertainty-aware LLM soft sensor predicts from text examples"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The retrieval-augmented results assume the test samples are not part of, or near-duplicates of, the 200 samples stored in the Industrial Process Data Vector Store, because the prompt returns the nearest stored samples' target values.","fun_headline_variants_meta":{"raw":{"variants":["LLM-based soft sensor skips training, adds uncertainty estimates","In-context learning turns LLMs into soft sensors with confidence","Few-shot LLM soft sensor with retrieval and self-explanation","LLM soft sensor: no parameter updates, just prompts and retrieval","Uncertainty-aware LLM soft sensor predicts from text examples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1431,"prompt_tokens":1075,"completion_tokens":356,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":691,"completion_tokens_details":{"reasoning_tokens":269}},"tokens_in":691,"tokens_out":356,"duration_ms":3988,"temperature":1.0,"reasoning_tokens":269,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:59:18.292874+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the LLM-UFSS-RAC protocol with an explicitly held-out test set: build the IPDVS only from training samples, forbid any retrieved neighbor from being within a short time window of the test sample, and compare the resulting MAE with random forest trained on the same retrieved samples. If the gap closes or reverses, the reported advantage came from target leakage rather than in-context reasoning.","supporting_citations":[{"cited_title":"A survey on in-context learning","cited_arxiv_id":null,"evidence_quote":"Defines in-context learning, the core paradigm that lets the soft sensor predict without parameter updates."},{"cited_title":"An explanation of in-context learning as implicit bayesian inference","cited_arxiv_id":null,"evidence_quote":"Provides the implicit Bayesian inference framing used to justify why in-context demonstrations transfer to new tasks."},{"cited_title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","cited_arxiv_id":null,"evidence_quote":"Establishes retrieval-augmented generation, the technique behind both the knowledge vector store and the process data vector store."},{"cited_title":"The development of an industrial -scale fed-batch fermentation simulation","cited_arxiv_id":null,"evidence_quote":"Supplies the IndPensim penicillin fermentation simulation used as one of the two case-study datasets."},{"cited_title":"Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context","cited_arxiv_id":null,"evidence_quote":"Provides the stronger alternative base model tested in the ablation study that shows LLM choice affects performance."}],"review_version":1}