{"id":"ae774033-7ad0-4667-b5dc-2f695bd8249e","arxiv_id":"2411.16587","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A large language model, prompted with COLREGs rules and fed real-time collision-risk numbers, made the give-way or stand-on decisions for a simulated autonomous surface vehicle in four encounter scenarios.","lead":"This paper hooks OpenAI's GPT-4 into a simulated autonomous ship, letting the language model read collision-risk numbers and decide whether to give way or stand on under maritime COLREGs rules. It is a proof of concept that LLMs can produce explainable, rule-compliant sailing decisions, though the evidence so far is qualitative.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (2) defines an internally inconsistent bearing classifier: head-on lies inside overtaking, and the overtaking sector is ahead rather than abaft the beam; since this mapping is encoded in the prompt, this is the load-bearing weakness.","rationale":"I agree with the reader's weakest_assumption and find it the most load-bearing issue. The central claim is not merely that an LLM can discuss COLREGs; it is that the decision layer classifies encounters correctly online and that the resulting commands yield COLREGs-compliant manoeuvres. Eq. (2) is the only formal classification specification in the paper and it is internally inconsistent: head-on is a subset of overtaking, and the overtaking wedge is placed forward rather than abaft the beam as Rule 13 requires. Because the paper states this mapping is encoded in the prompt, every LLM call is exposed to the contradiction. The four scenarios in Fig. 4 may still have been handled correctly if the LLM's pretrained knowledge overrode the prompt's bad geometry, but the paper provides no test separating 'correct despite the prompt' from 'correct because of the prompt.' The proposed bearing sweep with the exact prompt template is a minimal, concrete way to settle this. The paper has real strengths—public code, a nonlinear ship model, and explainable outputs—and the concern is correctable, so the reader's CONDITIONAL verdict should stand unchanged.","tokens_in":8315,"tokens_out":6411,"duration_ms":64142,"concrete_test":"Extract the exact prompt template from the GitHub repository and run a systematic bearing sweep: for each psi_rel in integer degrees from -180 to 180, with fixed risk, DCPA, TCPA, and range values, prompt GPT-4 at temperature 0.2 and record the classification and action. Check (a) repeated calls with identical inputs are self-consistent, (b) no input maps to two classes, and (c) the labels match COLREGs sectors, especially that an overtaking-astern scenario with psi_rel near 0 deg is not classified as head-on. If any input yields conflicting labels, or the judged sectors deviate from COLREGs, the decision layer's classification foundation is broken.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that the LLM decision layer maintains online COLREGs compliance—rests on the situation classification S(psi_rel) in Section III-A, Eq. (2). That function is not a partition: the head-on interval (-6 deg to 6 deg) is a proper subset of the overtaking interval (-112 deg to 112 deg), so a near-zero bearing is simultaneously 'head-on' and 'overtaking'. The overtaking interval is also centred ahead of the vessel, whereas COLREGs Rule 13 defines overtaking by approach from more than 22.5 deg abaft the beam of the vessel being overtaken. The authors state this mapping is 'encoded in the LLM's prompt architecture', so the contradiction is present in the exact text the LLM receives. No experiment isolates whether the LLM produced its correct give-way/stand-on decisions because it followed Eq. (2) or because its pretrained maritime knowledge overrode the flawed mapping. Since the abstract claims a 'rigorous classification framework' and the results claim COLREGs compliance, this undefined classification is load-bearing: if the LLM is relying on a contradictory sector map, the compliance evidence is anecdotal and not a demonstration of the proposed method. The public GitHub code makes a direct test possible, so the issue is correctable rather than a fundamental rejection.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an LLM-based (GPT-4) high-level decision-making layer for autonomous surface vehicles under COLREGs. The system classifies encounters (head-on, overtaking, crossing) from relative bearing, risk metrics, and state, and outputs explainable give-way/stand-on commands; these commands are executed by a low-level LOS/CTE/COLAV planner and PD controller on a nonlinear vessel model. The authors claim it is the first application of LLM-based explainable decision-making to COLREGs-constrained ASV control, and they demonstrate the architecture in four simulated encounter scenarios (crossing give-way, head-on, overtaking, crossing stand-on) with qualitative trajectory plots and LLM reasoning traces. The code is open-sourced on GitHub.","tokens_in":8628,"tokens_out":2792,"duration_ms":27713,"significance":"The paper addresses a timely problem: translating ambiguous, human-oriented COLREGs rules into machine-executable decisions in an explainable way. The architecture combines LLM reasoning with conventional planning and control in a modular fashion, and the decision traces are human-interpretable, which is a genuine strength. The open-source implementation and the explicit consideration of online risk assessment are also positive. However, the central claim of demonstrated COLREGs compliance rests on qualitative anecdotes rather than systematic evaluation; if the internal inconsistencies in the bearing classification and the absence of quantitative metrics are addressed, the approach could be a useful contribution to the emerging area of LLM-based maritime autonomy. As presented, the evidence is not yet sufficient to support the paper's claims.","major_comments":[{"comment":"The bearing classification mapping B(Ψ_rel) is internally inconsistent: the head-on interval (−6° to 6°) is a proper subset of the overtaking interval (−112° to 112°), so a near-zero relative bearing is simultaneously classified as both head-on and overtaking. Furthermore, the overtaking sector is centered ahead of the vessel, whereas COLREGs Rule 13 defines overtaking as approach from more than 22.5° abaft the beam. Because this mapping is explicitly encoded in the LLM prompt, the decision-maker receives contradictory instructions. The paper neither states how this ambiguity is resolved nor tests cases with bearings near the interval boundaries; it also does not verify which source (Eq. (2) or the LLM's pretrained maritime knowledge) actually drives the decisions. This issue is load-bearing for the 'rigorous classification framework' and 'COLREGs compliance' claims.","section":"Section III-A, Eq. (2)"},{"comment":"The evaluation is entirely qualitative. There are no quantitative metrics (e.g., observed CPA/DCPA/TCPA, cross-track error, heading error, control effort, or number of rule violations), no baseline comparison against a conventional rule-based or optimization-based COLREGs-compliant planner, no repeated trials with random seeds or perturbed initial conditions, and no failure analysis. The abstract claims 'accurate waypoint tracking and feasible control' but no data substantiates these terms. A single illustrative run per scenario cannot support the conclusion that the system 'maintains online COLREGs compliance' in general.","section":"Section IV, Figs. 4-6"},{"comment":"The risk thresholds T_Risk, T_R, T_DCPA, T_TCPA are declared as fixed constants without justification or sensitivity analysis, and the low-level tuning gains K_p, K_d, K_COLAV, μ, w_R, w_β are not reported in the paper (they are only referred to reference [18]). Since the paper claims reproducible results and provides code, these parameter values and the rationale for the threshold choices should be stated explicitly, along with a sensitivity study showing how the decisions change with threshold variations.","section":"Section III-B, Eqs. (3) and (15)"},{"comment":"The example in Fig. 5 is used to 'verify the system's ability to correctly interpret relative bearings,' but the reported relative heading of −150.09° falls outside the overtaking range in Eq. (2) and is thus classified as 'crossing' by default. The example does not demonstrate the resolution of the Eq. (2) ambiguity. Moreover, the paper does not specify how the LLM is instructed to handle headings that are simultaneously in two intervals, so the robustness of the classification remains unverified.","section":"Section IV, crossing give-way example (Fig. 5)"}],"minor_comments":[{"comment":"The abstract states 'first application' and 'first attempt' twice; given that reference [8] already explored LLMs for marine navigation, the novelty claim should be qualified (e.g., 'first to integrate with low-level control and online risk assessment').","section":"Abstract and Introduction"},{"comment":"There are typographical and grammatical issues: 'Also, The absence' has an incorrect capitalization, and 'mirrors trained mariners expertise' should be 'mirrors trained mariners' expertise'.","section":"Section II"},{"comment":"In the sentence 'to generate the control actions (uc in Eg. 6)', 'Eg.' should be 'Eq.'.","section":"Section III-B.3"},{"comment":"The relative bearing values are given without units; please add 'deg' or the degree symbol to avoid ambiguity.","section":"Figs. 5 and 6"},{"comment":"The sentence 'While our fuzzy logic risk assessment model effectively combines navigation parameters into a unified risk metric, but has limitations' contains a double subject; please revise to '...unified risk metric, it has limitations'.","section":"Conclusion"},{"comment":"The paper refers to 'a rigorous classification framework' but Eq. (2) is the only formal specification; consider adding a diagram that clearly partitions the bearing space according to COLREGs, including the abaft-the-beam overtaking sector.","section":"Section III-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is a promising early demonstration of LLM-based COLREGs decision-making, and the open-source code is a plus. However, the internal inconsistency in the bearing classification (Eq. (2)) and the purely qualitative evaluation are substantive problems that must be fixed before the paper can be considered for publication. I would urge the editor to send the revised manuscript back to the same reviewers, as the issues are correctable within the manuscript's scope and do not warrant outright rejection. The novelty claim should also be softened given prior work in [8]."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The quick take: this is a real first demonstration—GPT-4 as a high-level COLREGs decision maker in a closed loop with a nonlinear ship model, risk assessment, and low-level planning/control. The code and prompt templates on GitHub are a genuine asset. But the bearing classification function in Eq. (2) is internally inconsistent, and the evaluation is a handful of single-run scenarios, so the \"COLREGs compliance\" claim is overreach.\n\nWhat is new and good: prior work (COLREGs3) tested LLMs on static scenes without dynamics; this paper actually closes the loop with a first-order Nomoto model, input saturation, disturbances, and a PD controller, and the LLM outputs readable reasoning at 1 Hz. The integration of fuzzy risk indices (DCPA/TCPA/Range) into the prompt is a sensible architecture, and the paper is honest about missing weather and traffic density.\n\nThe soft spot that matters: Eq. (2). The head-on interval (-6° to 6°) is a proper subset of the overtaking interval (-112° to 112°), so any near-zero bearing is both head-on and overtaking. Overtaking is also centered ahead of the vessel, whereas COLREGs Rule 13 defines it as approach from more than 22.5° abaft the beam. The authors say this mapping is \"encoded in the LLM's prompt architecture,\" so the LLM is being asked to reason with a contradictory sector map. There's no experiment showing whether the LLM follows the flawed mapping or falls back on pretrained maritime knowledge. That makes the internal consistency of the method unverified, not just an aesthetic issue.\n\nSecondary: the evaluation is thin. Four scenarios, one run each, no quantitative metrics (minimum DCPA, cross-track error, time-to-collision), no baseline comparison to a conventional rule-based layer, no repeated trials. The LLM's own explanations are used as evidence of compliance, which is self-referential. This is addressable with a corrected sector definition and a systematic benchmark, so it's not a fundamental rejection.\n\nFor a reader: this is a proof of concept for the maritime autonomy community, useful for anyone exploring LLMs for rule-based decision making in safety-critical settings. I'd bring it to a reading group—the flawed Eq. (2) is a good teaching moment. I would not trust the compliance claims without more data, but the idea is sound enough to deserve serious peer review, with major revisions requested: fix the bearing mapping, add quantitative metrics, and moderate the conclusions.","headline":"A genuine first proof-of-concept of LLM-based COLREGs decision-making for ASVs, but Eq. (2)'s contradictory bearing mapping and anecdotal evaluation undercut the compliance claims.","tokens_in":9146,"tokens_out":3210,"would_cite":true,"duration_ms":28416,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a large language model, prompted with COLREGs rules and live collision-risk measurements, can serve as the real-time decision-maker for an autonomous surface vehicle, issuing correct give-way and stand-on commands.","keywords":["large language models","COLREGs","autonomous surface vehicles","collision avoidance","explainable AI","risk assessment","fuzzy logic","waypoint tracking"],"falsifier":"Run repeated head-on encounters with the target dead ahead ($\\psi_{\\mathrm{rel}}=0^\\circ$), which satisfies both the head-on and overtaking intervals of Eq. (2); if the LLM does not consistently return 'head-on, turn starboard', the bearing classification is not self-sufficient.","tokens_in":8112,"feed_emoji":"🚢","tokens_out":11721,"duration_ms":99590,"temperature":0.7,"pith_summary":"Maritime collision regulations are written in natural language for human crews, which makes them hard to encode as fixed rules. The paper argues that this ambiguity is precisely where a large language model belongs: a prompted LLM can read structured encounter data, decide whether the own ship must give way or stand on, and explain its reasoning in plain text. The decision layer runs at 1 Hz while a conventional line-of-sight planner and PD controller execute the chosen manoeuvre on a nonlinear ship model. Simulated crossing, head-on, overtaking, and stand-on scenarios are offered as evidence that the loop remains COLREGs-compliant, tracks waypoints, and keeps the human in the loop through generated explanations. If this holds, COLREGs compliance stops being a hardcoded lookup and becomes a language understanding task.","feed_headline":"LLM calls give-way and stand-on manoeuvres for ships","feed_subtitle":"Prompted with live risk numbers and COLREGs, it explains each turn while a ship controller follows through.","key_machinery":"The central object is the LLM-based decision function $D(\\psi_{\\mathrm{rel}}, S) = \\mathrm{LLM}(\\mathcal{M}, B(\\psi_{\\mathrm{rel}}), \\mathrm{Risk}, S)$, where $\\mathcal{M}$ is a maritime prompt template, $B(\\psi_{\\mathrm{rel}})$ maps relative bearing to one of three encounter classes through a piecewise interval rule, and $S$ is a state vector of situation, action, turning state, and manoeuvre index. This function converts raw encounter geometry and risk into a human-readable rule citation and a steering direction. On the execution side, the mechanism is the composite heading command $\\psi_d = \\psi_{\\mathrm{LOS}} + \\psi_{\\mathrm{CTE}} + \\psi_{\\mathrm{COLAV}}$, where the $\\psi_{\\mathrm{COLAV}}$ term contains the factor $K_{\\mathrm{Dir}}$ that carries the LLM's chosen give-way or stand-on decision down to the controller. The risk index that feeds the LLM is computed by a Z-shaped fuzzy membership function over DCPA, TCPA, and range, with thresholds $T_{\\mathrm{Risk}}$, $T_R$, $T_{\\mathrm{DCPA}}$, and $T_{\\mathrm{TCPA}}$.","core_discovery":"The central claim is that the encounter classification and manoeuvre-selection parts of COLREGs, which resist crisp algorithmic formulation, can be delegated to a large language model without any maritime fine-tuning. At each decision step the model receives the relative bearing $\\psi_{\\mathrm{rel}}$, range $R$, DCPA, TCPA, and a normalised fuzzy risk index $\\mathrm{Risk}(t)\\in[0,1]$, together with a prompt template that encodes Rules 13–17 and a state vector carrying the current situation, action, and turning status. It returns a situation label, a command ('give-way, turn starboard' or 'stand on'), and a reasoning chain. That command sets the direction of a collision-avoidance term in the low-level heading planner, which is tracked by a PD controller on a nonlinear ship model with input saturation and disturbances. The paper reports four encounter scenarios — crossing give-way, head-on, overtaking, and crossing stand-on — in which the decisions match the expected COLREGs behaviour and the vessel completes its waypoint mission.","pith_inferences":["Editorial inference: Because the decision layer is prompt-driven rather than retrained, the same interface could be pointed at inland-waterway rules, port traffic schemes, or company standing orders by swapping the prompt template and threshold vector; this extension is implied by the design but not tested in the paper.","Editorial inference: The bearing partition in Eq. (2) makes a dead-ahead target fall in both the head-on and overtaking intervals and places the overtaking sector ahead of the vessel, so the formulas alone do not fix the encounter class; the LLM's interpretation of the prompt is carrying that unresolved ambiguity, and that is worth testing directly.","Editorial inference: The authors' stated limitations (weather, vessel manoeuvrability, traffic density) suggest a direct stress test: vary those factors inside the fuzzy risk thresholds or as prompt context and check whether the LLM's explanations and actions degrade gracefully or flip discontinuously.","Editorial inference: The 1 Hz decision rate and written reasoning output make the system an attractive platform for human-in-the-loop review, but they also mean that any deployment must bound LLM latency and output variability before the loop can close safely."],"forward_implications":["COLREGs compliance becomes an online property of the loop: the LLM reclassifies the encounter at 1 Hz and can shift between give-way and stand-on as the risk numbers evolve, rather than following a precomputed manoeuvre table.","Every steering decision carries a text explanation that names the governing rule, so the same output that drives the ship could be logged for audit or displayed to a human supervisor.","The low-level planner and controller do not need to understand COLREGs; they execute the direction set by the decision layer, so the same ship model can be reused with different rule sets.","The reported scenarios (crossing give-way, head-on, overtaking, and crossing stand-on) show the architecture covering the four main encounter types without maritime-specific retraining of the LLM."],"supporting_citations":[{"why":"Supplies the nonlinear ship model, local line-of-sight planner, PD controller, and the DCPA/TCPA fuzzy risk calculation that feed the LLM decision layer.","marker":"[18]"},{"why":"Defines the COLREGs convention, the regulatory text the decision-maker must interpret and comply with.","marker":"[15]"},{"why":"Provides a prior simulation-based COLREGs-compliance approach that this work contrasts with online LLM decision-making.","marker":"[16]"},{"why":"Earlier exploration of LLMs for marine navigation that lacked dynamical simulation; the paper positions its online control loop against that gap.","marker":"[8]"},{"why":"Establishes that an LLM-driven framework can process real-time traffic data for vehicle dispatching, motivating the maritime transfer.","marker":"[2]"},{"why":"Supplies the explainable autonomous-driving architecture that the paper adapts to maritime encounters.","marker":"[3]"},{"why":"Shows LLMs interpreting complex driving rules as decision makers, the capability the paper relies on for COLREGs interpretation.","marker":"[13]"}],"fun_headline_variants":["LLM makes ship collision calls with explainable reasoning","First use of LLMs for COLREGs-compliant ship control","No fine-tuning: LLM interprets maritime rules for ASVs","LLM helms autonomous ships using COLREGs and risk data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that encounter classification can be recovered from the relative bearing alone, with the LLM resolving the interval overlap in Eq. (2) correctly.","fun_headline_variants_meta":{"raw":{"variants":["LLM makes ship collision calls with explainable reasoning","First use of LLMs for COLREGs-compliant ship control","No fine-tuning: LLM interprets maritime rules for ASVs","LLM helms autonomous ships using COLREGs and risk data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000291,"raw_usage":{"total_tokens":1751,"prompt_tokens":1048,"completion_tokens":703,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":631}},"tokens_in":664,"tokens_out":703,"duration_ms":8496,"temperature":1.0,"reasoning_tokens":631,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:57:33.996356+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run repeated head-on encounters with the target dead ahead ($\\psi_{\\mathrm{rel}}=0^\\circ$), which satisfies both the head-on and overtaking intervals of Eq. (2); if the LLM does not consistently return 'head-on, turn starboard', the bearing classification is not self-sufficient.","supporting_citations":[{"cited_title":"An integrated risk assessment and collision avoidance methodology for an autonomous catamaran with fuzzy weighting functions,","cited_arxiv_id":null,"evidence_quote":"Supplies the nonlinear ship model, local line-of-sight planner, PD controller, and the DCPA/TCPA fuzzy risk calculation that feed the LLM decision layer."},{"cited_title":"London, U.K.: International Maritime Organization, 1972","cited_arxiv_id":null,"evidence_quote":"Defines the COLREGs convention, the regulatory text the decision-maker must interpret and comply with."},{"cited_title":"Ship collision avoidance and COLREGs compliance using simulation-based control behavior selection with predictive hazard assessment,","cited_arxiv_id":null,"evidence_quote":"Provides a prior simulation-based COLREGs-compliance approach that this work contrasts with online LLM decision-making."},{"cited_title":"COLREG 3 - exploring the potential of large language models in ma- rine navigation systems,","cited_arxiv_id":null,"evidence_quote":"Earlier exploration of LLMs for marine navigation that lacked dynamical simulation; the paper positions its online control loop against that gap."},{"cited_title":"An LLM-driven framework for multiple-vehicle dispatching and navigation in smart city landscapes,","cited_arxiv_id":null,"evidence_quote":"Establishes that an LLM-driven framework can process real-time traffic data for vehicle dispatching, motivating the maritime transfer."},{"cited_title":"Driving with LLMs: Fusing object-level vector modality for explainable autonomous driving,","cited_arxiv_id":null,"evidence_quote":"Supplies the explainable autonomous-driving architecture that the paper adapts to maritime encounters."}],"review_version":1}