{"id":"127a93d7-f9c2-4957-a5c5-10fef4d13f93","arxiv_id":"2505.01627","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning GPT-3.5 Turbo on 681 OSDR parts yields about 89% accuracy on held-out OSDR data and produces function labels for ABC parts, although the ABC labels are never checked against ground truth.","lead":"This paper fine-tunes GPT-3.5 Turbo on labeled mechanical parts to automatically assign functions to unlabeled CAD parts. The application is sensible, but the headline accuracy is undermined by test-set-based model selection and by a lack of any ground truth for the newly labeled ABC parts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ABC function labels are never validated against ground truth; the paper's own Discussion (Sec. 6) concedes the 99% in-class rate is a limitation, contradicting the abstract's 'high-quality' claim.","rationale":"The reader's weakest_assumption captures the core gap: no validation of ABC labels. I agree. The paper itself, in Section 6, labels the 99% in-class result as a limitation, creating a direct contradiction with the abstract's 'high-quality' claim. That self-referential note should be weighed heavily. The inverse scaling in Table 4 is an additional red flag: if the 89% test accuracy with 10% training is real, it is remarkable and needs explanation; if it is an artifact of test-set leakage or selection, the source-domain evidence is undermined. Either way, the CONDITIONAL verdict from the reader is appropriate: the paper should not be accepted as-is, but the method could be validated with a simple human study. Therefore, I do not change the verdict.","tokens_in":11612,"tokens_out":8665,"duration_ms":81690,"concrete_test":"Select a random sample of 100 ABC parts from the matched set (Sec. 5.1.2). Have two independent mechanical design experts annotate each part with one of the eight function classes (Table 2) using only the part name, assembly name, and a thumbnail or CAD view, blinded to the model's predictions. Measure inter-annotator agreement and model-expert agreement (e.g., Cohen's kappa or exact accuracy). If model-expert accuracy is below a pre-specified threshold (e.g., 70%) or kappa below 0.6, the abstract's claim of high-quality functional data fails. This directly tests the transfer assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the domain-adapted LLM 'can generate high-quality functional data' on ABC rests entirely on the untested assumption that labels transferred from OSDR are correct. The only quantitative evidence is that 99% of ABC outputs fall inside the eight predefined function classes (Fig. 8). As the paper states in Section 6, this 'depends heavily on the predefined function labels, limiting the domain-adapted classifier's ability to distinguish other possible function classes' — a constraint-satisfaction check, not a correctness measure. There is no ground truth for ABC functions, no human evaluation, and no comparison with a baseline (e.g., the GNN in Ferrero et al., 2022). The name-based matching in Section 5.1.2 selects ABC parts whose names appear in OSDR, but the same part name can have different functions in different assemblies, so matched names do not guarantee semantic equivalence. Additionally, the OSDR evaluation itself is questionable: Table 4 shows test accuracy decreasing from 0.89 (10% training) to 0.63 (90% training), an unexplained inverse scaling that may stem from test-set-based hyperparameter selection (Sec. 4) or from changing test sets. If the OSDR numbers are artifacts, the transfer has no support at all. Thus, the 'high-quality functional data' claim is unsupported by the presented evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a supervised domain-adaptation framework that fine-tunes GPT-3.5 Turbo on the Oregon State Design Repository (OSDR) to classify mechanical assembly component functions into eight predefined classes, and then applies the fine-tuned model to the ABC CAD dataset to generate function labels. The central claimed results are an OSDR test accuracy of about 89% when using only 10% of the training data (Table 4) and that 99% of the ABC outputs fall within the predefined function classes (Figure 8). The authors interpret these results as evidence that the domain-adapted LLM can generate high-quality functional data that supports early-phase conceptual design.","tokens_in":11878,"tokens_out":4036,"duration_ms":45651,"significance":"If the accuracy and transfer claims were validated, the proposed pipeline would offer a practical, scalable alternative to manual function annotation in mechanical design, and the paper provides a useful case study of fine-tuning a commercial LLM on a design repository. The experimental setup is clearly described, and the authors provide detailed metric tables and comparisons with pre-trained GPT models, which is a strength. However, two load-bearing issues prevent the central claim from being accepted as evidence: the OSDR results are obtained by selecting hyperparameters on the test set, and the ABC 'evaluation' never checks the semantic correctness of the transferred labels. The paper's own Discussion section concedes the key limitation. If the authors add a proper validation protocol and external or human-validated assessment of the ABC labels, the contribution would be significantly strengthened.","major_comments":[{"comment":"The optimization procedure selects hyperparameters by maximizing evaluation metrics computed on the held-out test set (Eq. (7) defines M on D_test, and Eq. (8) chooses H' by the 'highest values of evaluation metrics'). This makes the reported 89% test accuracy an optimistic selection over the hyperparameter grid rather than an unbiased estimate of generalization. The inverse scaling in Table 4, where test accuracy drops from 0.89 at 10% training data to 0.63 at 90%, is unexplained and is exactly the pattern one would expect if each training-size configuration was tuned on the test set. A proper evaluation should use a separate validation split for hyperparameter selection, a fixed test set, and ideally repeated runs with variance reporting.","section":"Section 4, Eq. (8), and Table 4"},{"comment":"The evaluation on the ABC dataset is not an accuracy evaluation: the only quantitative check is that 99% of the predicted labels fall inside the eight predefined function classes. Because the model's output layer is a softmax over exactly those eight classes (Eq. (3)), a high in-class rate is a syntactic constraint-satisfaction check, not a measure of semantic correctness. The name-based matching between OSDR and ABC does not guarantee functional equivalence, since the same part name can have different functions in different assemblies. There is no ground-truth function label for ABC parts, no human evaluation, and no comparison with an alternative classifier such as the GNN of Ferrero et al. [7]. The claim that the model generates 'high-quality functional data' therefore rests entirely on an untested transfer assumption.","section":"Section 5.1.2 and Figure 8"},{"comment":"The paper's own Discussion states that 'the DA prediction results heavily depend on the predefined function labels, limiting the domain-adapted classifier's ability to distinguish other possible function classes.' This limitation directly contradicts the abstract's assertion that the domain-adapted LLM 'can generate high-quality functional data.' The 99% in-class rate is presented in Section 5.3 as a success, but Section 6 acknowledges it as a constraint of the label set rather than evidence of quality. The central conclusion should be revised to state that the model produces labels within a predefined taxonomy, not that the labels are correct or high-quality.","section":"Section 6 and Abstract"},{"comment":"The macro-averaged F1 scores in Table 4 are substantially lower than the weighted scores (e.g., 0.64 vs. 0.89 for the 10% training run), reflecting poor performance on rare classes. Figure 8 shows that the ABC predictions concentrate heavily on 'channel' and 'support', matching the OSDR class imbalance. This means that the reported high weighted accuracy is driven by majority classes, and the claim that the model provides comprehensive functional annotations for mechanical parts is not supported for minority function classes.","section":"Section 5.3, Table 4, and Section 6"}],"minor_comments":[{"comment":"There are typographical errors in the text: 'numebr' should be 'number' and 'evalution' should be 'evaluation'.","section":"Section 4, around Eq. (5)-(7)"},{"comment":"The notation in the loss and update equations is inconsistent: the double sum in Eq. (5) iterates over epochs and batches, but Eq. (6) updates W_y with a single gradient expression without epoch or batch indices. The equations should be aligned to make the optimization loop clear.","section":"Section 4, Eq. (5)-(6)"},{"comment":"The tables are difficult to parse because multiple metrics are packed into one row with no separating blank columns. Splitting the tables or using a clearer multi-row layout would improve readability.","section":"Table 3 and Table 4"},{"comment":"The bar chart in Figure 8 lacks axis labels and a caption that explains what quantity is being plotted (counts vs. percentages). Adding axis labels and a caption would make the figure self-contained.","section":"Figure 8"},{"comment":"The comparison with pre-trained GPT models is reported as single-point accuracies without confidence intervals or repeated runs, so it is unclear whether the improvements are statistically meaningful.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is currently not supported by the evidence in the manuscript. The OSDR results are likely optimistic due to test-set-based model selection, and the ABC 'evaluation' is only a label-set constraint check. The authors themselves acknowledge the key limitation in Section 6, which is encouraging, but the abstract and conclusion do not reflect this. A major revision could address the issue: add a validation split for hyperparameter selection, report results with variance, and validate ABC labels through human annotation or an external functional-basis benchmark. If these fixes are not possible, the paper's claims should be substantially narrowed. Given the clarity of the presentation and the potential practical relevance, I do not recommend rejection, but the current version overclaims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a routine fine-tuning paper with an overreaching abstract. The one thing to know is that the OSDR numbers are probably real but the ABC claim is not supported.\n\nWhat's new: fine-tuning GPT-3.5 Turbo on OSDR and applying it to ABC is a new application. The OSDR held-out test shows a large jump from zero-shot GPT-3.5 (4% accuracy) to fine-tuned (89% at 10% training data). That is a useful data point for anyone working on function annotation in mechanical design. The authors also deserve credit for being candid in the Discussion: they admit the ABC evaluation only checks that outputs fall within the predefined label set and that this limits the classifier.\n\nSoft spots, in order of severity:\n\n1. The ABC labels are never validated against ground truth or human judgment. The 99% in-class rate is a constraint-satisfaction check, not correctness. The abstract's 'high-quality functional data' is contradicted by the paper's own limitation statement.\n\n2. The 89% OSDR accuracy is suspect because it was selected across training sizes. Table 4 shows test accuracy dropping from 89% (10% training) to 63% (90% training). The authors call this overfitting, but a more plausible explanation is that they chose the training size that maximized test accuracy, which is test-set-based selection. No error bars or repeated runs.\n\n3. No comparison with the GNN baseline (Ferrero et al., 2022) on the ABC transfer. The paper cites that work but never benchmarks against it.\n\n4. Minor: the zero-shot GPT-3.5 accuracy of 4% is far below random (12.5%), which suggests prompt/output format mismatch rather than pure inability to classify. The comparison is still valid directionally but the magnitude is inflated.\n\nThe math is generic cross-entropy fine-tuning, so there's no methodological novelty. That's fine — the contribution is the case study.\n\nBottom line: this paper deserves a serious referee because the problem matters and the OSDR result is a real data point. But a referee should push for validation of the ABC labels, confidence intervals, and a benchmark against the GNN. Without those, the paper's main selling point — high-quality ABC annotations — does not hold.\n\nI'd send it to review with a request for major revision, not desk reject.","headline":"A routine fine-tuning paper with a real OSDR result and an overreaching ABC claim: the ABC labels are never validated, so the 'high-quality functional data' headline is not supported.","tokens_in":12416,"tokens_out":2967,"would_cite":false,"duration_ms":28834,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned GPT-3.5 classifies mechanical part functions at 89% accuracy","keywords":["Large language models","Domain adaptation","Function classification","Mechanical assembly components","Functional modeling","Conceptual design","Fine-tuning","CAD dataset"],"falsifier":"Select a random sample of the 6,786 ABC parts labeled by the fine-tuned model, have human designers assign one of the eight function classes to each from the part's name and assembly context, and measure agreement; if agreement is near the OSDR-chance level rather than the 89 percent test accuracy, the name-based transfer assumption fails.","tokens_in":11410,"feed_emoji":"⚙️","tokens_out":6216,"duration_ms":61072,"temperature":0.7,"pith_summary":"This paper tries to show that a general-purpose large language model can be turned into an accurate function classifier for mechanical assembly parts by fine-tuning it on a small set of function-labeled design descriptions. The authors adapt GPT-3.5 Turbo using data from the OSDR design repository, then apply the adapted model to name-matched parts from the ABC CAD dataset. On the OSDR test split, the fine-tuned model reaches about 89 percent accuracy using only 10 percent of the training data, compared with 4 percent for the same pre-trained model. The authors take this as evidence that domain adaptation can produce high-quality synthesized function labels, easing the scarcity of functional data in conceptual design.","feed_headline":"Fine-tuned GPT-3.5 classifies mechanical part functions at 89%","feed_subtitle":"Fine-tuning on 681 design-repository samples beats off-the-shelf GPT models at labeling CAD part functions.","key_machinery":"The carrying mechanism is supervised domain adaptation by fine-tuning, implemented as conditional text prediction over a prompt composed of the function classes, their definitions, a part name, and an assembly name. The fine-tuned model's distribution is shifted toward assembly-design labels by minimizing cross-entropy over the OSDR function-labeled data, and the hyperparameters (epochs, batch size, learning-rate multiplier) are selected by evaluation on a held-out test set. This is what lets the model generate function labels for unlabeled CAD parts: it is the same language-modeling machinery, repurposed with a domain-specific prompt and label vocabulary.","core_discovery":"The central discovery is that supervised domain adaptation via fine-tuning is sufficient to move a foundation language model from general language competence to mechanical-function classification. With a prompt that supplies the eight function classes and their textual definitions and asks for the function of a named part within a named assembly, fine-tuned GPT-3.5 Turbo reaches 89 percent test accuracy on a 10 percent OSDR training sample, and the same model assigns labels to 6,786 ABC parts with 99 percent of outputs falling inside the predefined classes. The paper attributes the improvement to the model's ability to incorporate contextual function definitions, something earlier machine-learning classifiers such as graph neural networks could not do directly. It therefore positions the fine-tuned model as a generator of synthesized, function-labeled design data rather than only a classifier.","pith_inferences":["The 99 percent figure only measures whether predicted labels sit inside the eight-class vocabulary, not whether they are correct; a human validation on a random ABC sample would be the direct test of the 'high-quality' claim.","Name-based matching between OSDR and ABC is the paper's transfer bridge; if the naming conventions differ, accuracy on ABC could be far lower even though OSDR test accuracy is high.","Because the OSDR label distribution is dominated by 'channel' and 'support,' the model will likely over-annotate those classes on any new dataset; rebalancing or prior correction is a natural next step.","A stronger test would compare fine-tuned model labels on ABC parts with labels derived from geometric or visual features, testing whether function can be inferred from form."],"forward_implications":["Function annotation of large unlabeled CAD repositories can be automated with a small labeled source set, replacing manual and subjective annotation.","The injected class definitions make the classifier's reasoning visible: a user can inspect which of the eight function definitions the model is choosing between.","Using less source training data improved test accuracy, so overfitting on the source repository is a real risk and small, balanced samples may be preferable.","The generated ABC labels form a new function-labeled dataset that can support conceptual-design reasoning, retrieval, and downstream behavioral models."],"supporting_citations":[{"why":"Supplies the OSDR data schema and function hierarchy used as the labeled training source.","marker":"[31]"},{"why":"Describes the product design repository system from which the OSDR samples are drawn.","marker":"[32]"},{"why":"Defines the eight function classes and their textual definitions used as the classification vocabulary and prompt context.","marker":"[28]"},{"why":"Provides the ABC CAD model dataset, the target domain whose part and assembly names are matched to OSDR for label generation.","marker":"[33]"},{"why":"Presents the graph-neural-network baseline for component function classification that motivates the LLM-based approach.","marker":"[7]"},{"why":"Review of fine-tuning practices the proposed supervised domain-adaptation pipeline builds on.","marker":"[26]"},{"why":"Supplies the self-attention mechanism the paper invokes to explain how the model uses contextual descriptions in classification.","marker":"[29]"},{"why":"Defines supervised domain adaptation by transferring parameter sets and gradients, the technical framing for the method.","marker":"[27]"}],"fun_headline_variants":["Fine-tuned LLM labels CAD part functions with 89% accuracy","Domain-adapted GPT-3.5 reaches 89% on part function classification","Fine-tuned GPT-3.5 improves part function labeling in assemblies","89% accuracy from fine-tuning language model on design data","Fine-tuned GPT-3.5 synthesizes labeled functional design data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that matching part names between the OSDR training data and the ABC target data transfers the model's classification accuracy to ABC, but the paper never validates any ABC function label against ground truth.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned LLM labels CAD part functions with 89% accuracy","Domain-adapted GPT-3.5 reaches 89% on part function classification","Fine-tuned GPT-3.5 improves part function labeling in assemblies","89% accuracy from fine-tuning language model on design data","Fine-tuned GPT-3.5 synthesizes labeled functional design data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001105,"raw_usage":{"total_tokens":4625,"prompt_tokens":978,"completion_tokens":3647,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":3552}},"tokens_in":594,"tokens_out":3647,"duration_ms":25367,"temperature":1.0,"reasoning_tokens":3552,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:14:10.750555+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Select a random sample of the 6,786 ABC parts labeled by the fine-tuned model, have human designers assign one of the eight function classes to each from the part's name and assembly context, and measure agreement; if agreement is near the OSDR-chance level rather than the 89 percent test accuracy, the name-based transfer assumption fails.","supporting_citations":[{"cited_title":"Introduction of a data schema to support a design reposi- tory,","cited_arxiv_id":null,"evidence_quote":"Supplies the OSDR data schema and function hierarchy used as the labeled training source."},{"cited_title":"Product design support: Exploringadesignrepositorysystem,","cited_arxiv_id":null,"evidence_quote":"Describes the product design repository system from which the OSDR samples are drawn."},{"cited_title":"Development of a functional basisfordesign,","cited_arxiv_id":null,"evidence_quote":"Defines the eight function classes and their textual definitions used as the classification vocabulary and prompt context."},{"cited_title":"ABC: A big CAD model dataset for geometric deep learning,","cited_arxiv_id":null,"evidence_quote":"Provides the ABC CAD model dataset, the target domain whose part and assembly names are matched to OSDR for label generation."},{"cited_title":"Classi- fyingcomponentfunctioninproductassemblieswithgraph neural networks,","cited_arxiv_id":null,"evidence_quote":"Presents the graph-neural-network baseline for component function classification that motivates the LLM-based approach."},{"cited_title":"Supervised domain adaptation by transferring both the parameter set and its gradient,","cited_arxiv_id":null,"evidence_quote":"Defines supervised domain adaptation by transferring parameter sets and gradients, the technical framing for the method."}],"review_version":1}