{"id":"34ec1674-5b43-484b-aa32-ed62853fe296","arxiv_id":"2411.18324","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RITA, an offline framework built on a fine-tuned RoBERTa NER model, identifies IoT critical objects from requirement texts and reports higher F1 than ChatGPT on four of seven categories.","lead":"RITA is an offline, open-source framework that automatically finds important devices, services, and resources in IoT requirement documents, then connects them to threats and countermeasures. The paper claims it beats ChatGPT on four of seven identification categories while keeping sensitive data on-site.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The four-of-seven advantage may not generalize: 200 of 300 validation phrases are GPT-3-generated via the same pipeline used for RITA's training data, and the remaining 100 are from the same dataset's test split; the comparison is in-distribution, not a real-world test.","rationale":"The reader's weakest_assumption identified exactly this issue. I agree that the provenance of the validation set is the single most load-bearing vulnerability because it threatens the external validity of the headline comparison. I considered other candidates: the absence of significance testing, partial-overlap scoring, and the unvalidated threat and countermeasure components. Lack of confidence intervals is real but secondary, because even with intervals, an in-distribution comparison would not support the abstract's practical conclusion. Partial-overlap scoring could affect both models, but it is not what makes the comparison ungeneralizable. The unvalidated Components 2 and 3 are a scope limitation rather than a defect in the stated comparison. The proposed check, an independent human-annotated corpus with confidence intervals, directly tests whether the four-of-seven result survives a distribution shift. If it does not, the central empirical claim should be weakened to 'competitive on a synthetic in-distribution benchmark' and the framework's real-world utility remains unverified. Since the reader already conditioned acceptance on addressing such issues, I recommend no change to the verdict.","tokens_in":6207,"tokens_out":5440,"duration_ms":49197,"concrete_test":"Independent re-evaluation: collect 300 to 500 real IoT requirement sentences from sources not used in training, for example CrowdRe held-out documents, public smart-home or industrial IoT specifications, or fresh practitioner-written user stories. Have two annotators label ICO categories per Table I and measure inter-annotator agreement. Run the released RITA model and ChatGPT under the paper's few-shot prompt on this held-out corpus, and report per-category F1 with bootstrap 95% confidence intervals plus McNemar's test for paired differences. If RITA still beats ChatGPT in the same four categories with non-overlapping intervals, the generalization concern is resolved; if the advantage shrinks, reverses, or falls within noise, the original claim is an artifact of the in-distribution validation set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparative claim rests entirely on a 300-phrase validation set whose provenance makes it an in-distribution test for RITA, not a sample of real IoT requirement documents. Section III states that '100 of these phrases represented Storylines phrases... The storyline phrases were taken from the test group of the dataset. The other 200 phrases were generated using GPT-3.' Section IV then says 'From the test portion of the dataset, 300 phrases were chosen randomly,' which is ambiguous, but either way two-thirds of the set is LLM-generated. The training data for RITA was itself created with an analogous ChatGPT query pipeline ('we queried chatGPT to find examples... asked for commercially available options... asked for the phrases'), so the 200 generated validation phrases come from nearly the same distribution the model was fine-tuned on. The remaining 100 phrases share the dataset's annotation conventions. As a result, Table II measures how well a domain-fine-tuned model performs on its own training distribution relative to a general few-shot model; it does not establish superiority on real storyline, user-story, or requirement documents. The abstract's phrase 'using both human-annotated and ChatGPT-generated test data' overstates independence: the generated phrases' ground truth is derived from the generation target, not from independent human annotation. If real IoT documents differ in vocabulary, sentence structure, or entity density, the reported four-of-seven advantage may not carry over.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents RITA, an offline, open-source framework for supporting resilient IoT application design. The framework has three components: (1) a fine-tuned RoBERTa-based NER model that identifies IoT Critical Objects (ICOs) in seven categories from storyline, user-story, and requirement texts; (2) a relational database that maps ICOs to threats; and (3) a mitigation-strategy database that maps threats to countermeasures. The authors evaluate the NER component against ChatGPT (GPT-3) on a 300-phrase validation set and report that RITA achieves higher F-score in four of seven categories (actuator, sensor, network resource, service). They also contribute a large annotated dataset and release the code and data publicly.","tokens_in":6521,"tokens_out":6820,"duration_ms":59590,"significance":"If the stated results hold, RITA would be a useful practical tool for early-stage IoT resilience design, with concrete advantages in privacy, reproducibility, and offline deployment. The open-source release of the dataset, the fine-tuned model, and the framework is a real strength, as is the evaluation of a seven-way fine-grained ICO taxonomy. However, the significance is currently limited by two factors: the validation set is largely in-distribution for RITA, and the framework's threat-correlation and mitigation-recommendation components are not empirically evaluated at all. The paper's central comparative claim is therefore narrower than the framework-level claim in the abstract, though still informative as an in-domain baseline comparison.","major_comments":[{"comment":"The validation set is not a representative sample of real IoT requirement documents. Section III states that 200 of the 300 validation phrases were generated using GPT-3, and that the remaining 100 storylines were taken from the test group of the dataset used to train RITA. Because RITA's training data was itself created using an analogous ChatGPT query pipeline, the generated validation phrases come from nearly the same distribution the model was fine-tuned on, and the 100 dataset phrases share the dataset's annotation conventions. Table II therefore measures in-distribution performance for RITA relative to a general few-shot model, not superiority on real-world requirement documents. To support the abstract's general claim, the authors should evaluate on an out-of-distribution, independently collected human-authored corpus and report per-category F1 there. They should also state clearly that the generated phrases' ground truth is the generation target rather than independent human annotation.","section":"Section III-IV, Table II"},{"comment":"The four-of-seven comparison is reported as point estimates without any uncertainty quantification. The per-category F1 differences are small in several cases (e.g., Sensor 1.00 vs 0.96, Service 0.60 vs 0.47), and the number of validation phrases per category is not reported. The authors should provide per-category counts of true positives, false positives, and false negatives, and report confidence intervals or bootstrap estimates so that the reader can assess whether the observed differences are statistically reliable.","section":"Table II"},{"comment":"The true-positive criterion is too lenient for NER evaluation. The paper states that 'if the response predicted the right category and had overlap it's deemed a true positive.' Partial span overlap (e.g., predicting 'sensor' for the gold span 'temperature sensor') is counted as a correct identification. This can inflate F1 for both systems and may differentially favor the fine-tuned model if it systematically produces shorter or longer spans. The authors should report exact-match F1 as an additional metric, or at least report span-boundary overlap statistics, so that the comparison is not driven by a scoring convention.","section":"Section IV (scoring)"},{"comment":"The framework's central claim is that RITA automatically identifies ICOs, correlates threats, and recommends countermeasures, but only the NER component (Component 1) is empirically evaluated. Components 2 and 3 (the threat database and mitigation database) are not tested for coverage or correctness. The authors should evaluate these components, for example by comparing the recommended countermeasures for a sample of ICO-threat pairs against a manually constructed gold standard, or by reporting precision and recall of the threat and mitigation retrieval steps.","section":"Section I and Section IV"},{"comment":"The composition of the validation set is described inconsistently. Section III says that 100 phrases were storylines taken from the test group and 200 were generated using GPT-3, while Section IV says 'From the test portion of the dataset, 300 phrases were chosen randomly.' These descriptions cannot both be correct. The authors must provide a single precise description of the validation set, including counts per document type and per ICO category, and make the exact generation prompts and filtering steps available in the repository.","section":"Section III vs Section IV"},{"comment":"The abstract claims RITA 'outperformed ChatGPT in four of seven ICO categories,' while Section I (contribution iv) states that 'RITA framework which has a performance similiar to GPT-3 Model (baseline) on the task of IoT Critical Objects identification in 4 categories out of the 7.' 'Outperformed' and 'similar' are different claims. The authors should reconcile these statements with the actual pattern of results in Table II (better in four categories, worse in three).","section":"Abstract vs Section I"}],"minor_comments":[{"comment":"There are several spelling and typographical errors: 'performace' (Section I), 'chatPGT' (Section IV), 'trainig' (Section III), and 'dependance' (Section I). These should be corrected.","section":"Throughout"},{"comment":"The framework name 'Spacy framework' should be written as 'spaCy framework.'","section":"Section III"},{"comment":"The table uses inconsistent numerical precision: the RITA test column has nine decimal places while the validation columns have two. Use consistent rounding throughout and add the number of validation phrases per category.","section":"Table II"},{"comment":"Reference [15] is cited to support the claim of a 'large-sized dataset,' but the cited paper on glossary term extraction does not appear to support this claim; consider citing the dataset repository or the prior work [1] instead.","section":"References"},{"comment":"The paper uses 'ChatGPT' and 'GPT-3' interchangeably. Clarify that the comparison model is GPT-3 accessed through the ChatGPT interface, so that the reader understands the exact system evaluated.","section":"Section III-IV"},{"comment":"The example output ('a3144e hall effect sensor switch', 'ACTUATOR') is inconsistent with Table I, where a Hall effect sensor would fall under 'Sensor' (a device that gathers information). Please correct the example or clarify the category definition.","section":"Section III"}],"recommendation":"major_revision","confidential_remarks":"The paper's validation strategy relies heavily on the authors' own prior dataset [1] and taxonomy [8], and the 300-phrase validation set is partly generated by the same model family (GPT-3/ChatGPT) used to create the training data. This is a legitimate correctness-risk concern but not disqualifying; the comparison is still an empirical F1 evaluation. The main value of the paper is the open-source framework and dataset release. The recommendation of major_revision reflects the need for an out-of-distribution evaluation, uncertainty estimates, and evaluation of the threat/mitigation components; with those additions the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RITA is a workmanlike contribution: a fine-grained seven-category taxonomy for IoT critical objects, a dataset of 66,108 annotated phrases, a fine-tuned RoBERTa NER model, and an offline open-source framework that wires NER output to threat and mitigation databases. The authors also release code and data, which makes the work reproducible. That part is real and useful.\n\nThe empirical headline is narrower and softer than the abstract implies. The four-of-seven advantage over ChatGPT is computed on a 300-phrase validation set where 200 phrases were generated by GPT-3 (same model family as the baseline) and 100 come from the test split of the dataset RITA was trained on. That is an in-distribution comparison, not a test on real storyline or requirement documents. The paper itself acknowledges that GPT-3 should have the upper hand 'since many of the phrases were generated using chatGPT itself,' which is points in favor of the authors' honesty, but the abstract's phrasing 'using both human-annotated and ChatGPT-generated test data' overstates independence. There are no error bars or significance testing, and partial-overlap is scored as a true positive, which flatters both systems but especially the one that must localize entities.\n\nThe soft spots are moderate, not fatal. For the narrow claim — RITA outperforms ChatGPT on this validation set in four categories — the table supports it. For any broader claim about real-world generalization, the evidence is not there. The threat and countermeasure databases are not evaluated at all, so the framework part beyond NER is an argument from construction, not from test.\n\nWho is this for: people building practical IoT resilience tooling and anyone needing a labeled NER corpus for IoT requirements. It deserves a serious referee, but the revision should include an independently sampled test set, exact-match or clearly defined scoring, and uncertainty reporting. The dataset release alone justifies not desk-rejecting it.","headline":"A useful dataset and tooling contribution with a narrow, honestly-reported result that the abstract oversells; peer review should happen but the evaluation needs an independent test set.","tokens_in":7016,"tokens_out":2044,"would_cite":true,"duration_ms":17203,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a fully offline, open-source framework named RITA can automate the first stages of resilient IoT design: identifying IoT Critical Objects (ICOs) in requirement documents, mapping them to relevant threats, and…","keywords":["IoT Critical Objects","Named Entity Recognition","RoBERTa","resilience design","threat analysis","countermeasure selection","ChatGPT comparison","offline machine learning"],"falsifier":"Collect an independent set of human-written IoT requirements from a different source, run both RITA and ChatGPT on it under the same output-format constraints, and check whether RITA still wins four of seven categories and whether any false negatives appear.","tokens_in":6023,"feed_emoji":"🤖","tokens_out":7004,"duration_ms":52764,"temperature":0.7,"pith_summary":"This paper claims that a fully offline, open-source framework named RITA can automate the first stages of resilient IoT design: identifying IoT Critical Objects (ICOs) in requirement documents, mapping them to relevant threats, and recommending countermeasures. The system uses a fine-tuned RoBERTa named-entity recognition model that classifies ICOs into seven categories, then queries a relational database built from an existing resilience taxonomy. In an empirical comparison, RITA achieved higher F-scores than ChatGPT on four of the seven categories — actuators, sensors, network resources, and services — on a validation set of 300 phrases that included both human-annotated and ChatGPT-generated text. The practical interest is that this workflow runs locally, so sensitive requirement documents never leave the site, and it produces reproducible outputs instead of ChatGPT's variable answers.","feed_headline":"Offline IoT framework beats ChatGPT on 4 of 7 categories","feed_subtitle":"RITA runs on-site, keeps data private, and gives consistent outputs for resilient IoT design.","key_machinery":"The central mechanism is a fine-tuned RoBERTa NER model that classifies phrases into seven ICO categories, paired with a relational database of 138 threats and 97 countermeasure examples organized by the ADD4RIOT meta-model. The NER model does the entity extraction; the database does the threat correlation and mitigation recommendation. The evaluation uses F-score to account for class imbalance and a few-shot prompt template to align ChatGPT's output format with the model's.","core_discovery":"The paper's central claim is that a comparatively small, fine-tuned RoBERTa model can match or beat a general-purpose GPT-3-based assistant on the niche task of identifying IoT Critical Objects. On the 300-phrase validation set, RITA achieved higher F-scores in actuator, sensor, network resource, and service identification, despite the fact that two-thirds of those validation phrases were generated by GPT-3 itself. The paper further claims that the extracted entities can be turned into threat and mitigation recommendations by querying a database built from the ADD4RIOT taxonomy, all without sending data to any cloud service.","pith_inferences":["A natural extension is to test RITA on requirements from industrial or medical IoT domains, where entity distributions differ from web-derived phrases, to see whether the four-category advantage persists.","The threat and countermeasure database could be reused independently of the NER model as a design-time risk checklist.","Because the validation set includes GPT-generated phrases, a purely human-authored evaluation set might narrow the reported four-of-seven margin.","The claim of zero false negatives on the validation set deserves a stress test with ambiguous phrases that break entity boundaries."],"forward_implications":["Because RITA wins on actuators, sensors, network resources, and services, it can serve as a reliable offline filter for these entity types in requirement documents.","Organizations handling sensitive requirements can run the entire pipeline on-site, eliminating data-sharing concerns.","Repeated runs on the same input produce identical labels, which supports standardised resilience assessments across teams.","The released source code and dataset allow others to retrain the model on their own requirements corpora."],"supporting_citations":[{"why":"Provides the original four-category annotated dataset and the NLP extraction approach that RITA extends to seven categories.","marker":"[1]"},{"why":"Defines the ADD4RIOT meta-model whose taxonomy is encoded in the threat and countermeasure database used by RITA's later components.","marker":"[8]"},{"why":"Specifies the RoBERTa architecture and pretraining procedure that the fine-tuned NER model is based on.","marker":"[14]"}],"fun_headline_variants":["Offline RITA framework beats ChatGPT on 4 of 7 IoT tasks","RITA: private on-site AI outperforms ChatGPT in key IoT categories","Small RoBERTa model tops ChatGPT for IoT critical object spotting","RITA: automated IoT resilience design without internet dependence","On-site NER framework beats ChatGPT for IoT threat mitigation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the 300-phrase validation set reflects real-world IoT requirement documents, even though 200 of those phrases were generated by GPT-3 (the same model family as ChatGPT) and the remaining 100 come from the same dataset used to train and test RITA.","fun_headline_variants_meta":{"raw":{"variants":["Offline RITA framework beats ChatGPT on 4 of 7 IoT tasks","RITA: private on-site AI outperforms ChatGPT in key IoT categories","Small RoBERTa model tops ChatGPT for IoT critical object spotting","RITA: automated IoT resilience design without internet dependence","On-site NER framework beats ChatGPT for IoT threat mitigation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1280,"prompt_tokens":873,"completion_tokens":407,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":319}},"tokens_in":489,"tokens_out":407,"duration_ms":4052,"temperature":1.0,"reasoning_tokens":319,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:18:02.611056+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect an independent set of human-written IoT requirements from a different source, run both RITA and ChatGPT on it under the same output-format constraints, and check whether RITA still wins four of seven categories and whether any false negatives appear.","supporting_citations":[{"cited_title":"Automated extraction of iot critical objects from iot storylines, requirements and user stories via nlp,","cited_arxiv_id":null,"evidence_quote":"Provides the original four-category annotated dataset and the NLP extraction approach that RITA extends to seven categories."},{"cited_title":"An Architectural Design Decision Model for Resilient IoT Application","cited_arxiv_id":"2306.10429","evidence_quote":"Defines the ADD4RIOT meta-model whose taxonomy is encoded in the threat and countermeasure database used by RITA's later components."}],"review_version":1}