Pith.

Cryptography and Security · arXiv:2610.03073 v2 · submitted 2 Oct 2026

SecJev: Bringing Security Expertise to System One Decision Models

Zheng Chen, Fei Yu, Haohao Huang, Yang Li, Anlong Chen, Lei Chen

The paper says

“Security workflows need models that turn complex observations and explicit policies into decisions. System One models introduced by Jev return typed predictions and probabilities; security specialization supplies the domain expertise behind those predictions.”

more of the abstract

We introduce SecJev, to our knowledge the first family of Jev-style decision models specialized for security, spanning 0.8B to 9B parameters. Built on Kev's single-pass candidate scorer, SecJev learns Boolean, choice, and ordered decisions from text, telemetry, and observation histories. We develop SecJev-Corpus to unify source-label prediction and explicit-policy evaluation across 14 tasks and eight sources. It covers tool outputs, traffic, federated updates, consensus, authentication, and vehicle messages. Scene-weighted training adapts the models across these domains while preserving a shared typed decision interface. Security specialization improves every model in the family; SecJev-0.8B outperforms general Kev-9B by 20.51 percentage points in task-macro accuracy. At 0.8B and 4B, SecJev achieves accuracy and latency close to answer-only generative fine-tuning with lower peak allocated GPU memory. SecJev-4B improves on Kev-4B and security-trained generation in prompt-injection and traffic transfer tests. We release adapters, decision heads, SecJev-Corpus, and training and inference code.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 5 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. The corpus and the tool/policy results are real; the traffic source-label numbers are unproven and carry 44% of the headline margin — demand the identity ablation. This line is wrong

2 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at §4.2, §5.2, Table 3, Table 11, Appendix D.1 (b232)

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.60):

“Recorded group boundaries address spe- cific dependencies; residual semantic overlap and transfer beyond the tested sources remain open.”

Pith says: Objection 1 of 2 · major

The two largest test tasks, CICIoT2023 (6,000 questions) and ToN-IoT (3,500 questions), constitute about 44% of the 21,634 test questions, and Table 3 shows the largest individual gains exactly there (25.00 to 91.80 and 25.00 to 98.03 at 0.8B). The paper's own audit (Table 11) reports that both reviewers rate the visible evidence sufficient in only 28 of 100 source-label …

more of objection 1

… questions, with per-reviewer sufficiency near zero on these two tasks (CICIoT 6/20 and 3/20; ToN-IoT 3/20 and 0/20), yet the models score 92-99%. The group-based splits of §4.2 (captures for CICIoT, source hosts for ToN-IoT) prevent row-level leakage but do not rule out a capture- or host-identity shortcut encoded in retained fields (ports, feature signatures, device-type patterns); §4.1 states endpoint identities are omitted for ToN-IoT but does not verify that searchable identity proxies are absent. The disjoint-source results are consistent with this channel: on new IoT-23 captures (Appendix D.1, Table 5, Table 20), SecJev-4B falls from 88.40% (Table 13) to 65.10% with a 66.40% FPR, and per-capture accuracy in Table 22 varies from 29.52% to 98.81%. Because this concerns 44% of the test set and the largest reported gains, the paper needs a decisive in-scope test before the headline claim is accepted: either an identity-feature ablation (permuting or zeroing ports, IP-derived fields, and unique feature signatures on the held-out CICIoT/ToN-IoT questions and measuring the accuracy change) or per-group accuracy on the main test's 8 CICIoT and 6 ToN-IoT groups (Appendix H lists these counts), which would directly confront the capture-memorization hypothesis. I also note that reviewer C's 20/20 label match on ToN-IoT in Table 11 partially cuts against a pure 'unobservable labels' reading and deserves one sentence of discussion in §6.4.

The paper says, at Appendix F, Table 23, §5.2 (b467)

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.88):

“This choice follows inspection of the external scores, so those scores describe exploratory transfer rather than an untouched evaluation of checkpoint selection.”

Pith says: Objection 2 of 2 · major

The reported 9B row in Table 3 uses the epoch-2 checkpoint, which was selected by inspecting exploratory BIPIA and IoT-23 scores rather than by the stated development criterion; Table 23 shows the development-criterion checkpoint is epoch 3, which also has the higher security macro accuracy (94.51 vs 94.08).

more of objection 2

Appendix D.1 correctly freezes the choice before the disjoint-source evaluation and that evaluation is clean, but a reader of §5.2 cannot tell that the 9B main-table numbers do not come from the development-selected model and that the selection was informed by external scores. This is disclosed in the appendix, so the fix is local: add a sentence in §5.2 (or a Table 3 footnote) stating that the 9B checkpoint was chosen after external exploratory inspection and that the Appendix D.1 confirmation set was fixed before new predictions.

5 minor objections

The paper says, at §1, first paragraph

Pith says: minor objection 1 of 5

Typo: 'We useJev-like' should read 'We use Jev-like'.

The paper says, at Table 11 caption

Pith says: minor objection 2 of 5

The combination of high A-C answer agreement with lower A and C label-match counts (e.g., CICIoT 20/20 agreement but 14/14 label matches) is unusual and should be explained in the caption or in §B, since it affects how the audit should be read.

show 3 more

The paper says, at §5.2, Table 3

Pith says: minor objection 3 of 5

The three-setting breakdown (tool-text, source-label, explicit-policy) appears only in prose; consider adding a source-label-only macro gain (4B: 50.53 to 85.19) as an explicit row or footnote so the headline 20.51-point claim can be read net of the near-perfect rule-checking policy tasks.

The paper says, at §5.2, b125 and Appendix H

Pith says: minor objection 4 of 5

Macro-gain bootstrap intervals exclude zero, but the headline comparison 'SecJev-0.8B versus Kev-9B' (b25, b127) is reported without an accompanying interval; since sources have few groups (CICIoT 8, ToN-IoT 6, LANL 7), one interval for this specific comparison would strengthen the claim.

The paper says, at Eq. (1)

Pith says: minor objection 5 of 5

The dimension-normalization term renders as 'p dp' and should be typeset clearly (e.g., divided by sqrt(d_p)) so the formula is unambiguous.

What would settle it

Freeze SecJev-0.8B and SecJev-4B. Re-evaluate the CICIoT and ToN-IoT test splits under two ablations: (1) mask source/destination IP addresses, MACs, and port numbers to constant values, keeping measurement fields (packet and byte counts, sizes, protocol, flags, timing); (2) remove every token that could fingerprint a capture or host while preserving the 13 ToN-IoT telemetry features used for IoT-23. If per-task accuracy falls by more than roughly 10 points from the reported 91.80/91.72% (CICIoT) and 98.03/98.80% (ToN-IoT), the source-label gains are largely per-capture or per-host fingerprints and the 20.51-point gap over Kev-9B must be re-baselined on fingerprint-free inputs.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 5 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 51 s · 27,725 tokens read, 7,031 written
  3. Editor wrote the verdict above 59 s · 28,835 tokens read, 8,894 written
  4. Referee major revision: 2 major objections, 5 minor 1 min 7 s · 28,230 tokens read, 9,277 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The real substance here is the corpus plus the tool-output and policy results; the traffic source-label numbers are the part not to be trusted yet — they carry 44% of the headline margin and sit exactly where the authors' own audit says the labels are not visible.

What's actually new is genuine. First security-specialized Jev-like family, and SecJev-Corpus (170,185 questions, 14 tasks, 8 sources) with unusually careful QA: schema and label checks on everything, independent recomputation of 37,490 policy answers, a 280-question human audit with per-task evidence-sufficiency ratings, and a frozen disjoint-source confirmation set. The reporting is honest — exploratory results kept separate from the confirmation set, per-capture confusion tables published, checkpoint selection on transfer scores disclosed (b467), single runs admitted (b235). That is a usable asset and a good template for transfer reporting.

Where it holds: tool-output injection and explicit-policy tasks. Human reviewers match labels 180/180 there, joint evidence sufficiency 179/180, and BIPIA transfers to new backgrounds at 72-73% with zero false positives. The interface comparison is modest but real — SecJev within ~0.3 points of answer-only generative SFT at lower peak memory. Retention is measured and reported with the confidence degradation included.

The soft spot is the traffic component, and the stress-test concern lands. CICIoT plus ToN-IoT are 44% of the test set (9,500 of 21,634 questions) and contribute ~44% of the 20.51-point margin over Kev-9B: 127 of 287 total margin points. They are also the two tasks where the paper's own audit rates visible evidence jointly sufficient in essentially none of the sampled questions (17+/20 for CICIoT, 20/20 for ToN-IoT) and reviewer-label agreement is low. The disjoint-source IoT-23 check behaves the way a capture-level shortcut predicts: SecJev-4B at 65.10% on new captures with 66.4% FPR, down from ~88% in the exploratory suite, and the near-all-malicious capture 8-1 gets 98.81% with 100% FPR. The paper omits endpoint identities for ToN-IoT but never shows identity proxies (ports, feature signatures) are absent, and the CICIoT preprocessing is documented less. So the 91-99% traffic numbers are underdetermined: leak versus cue. One refinement to the stress-test: the margin survives without those two tasks — SecJev-0.8B still leads Kev-9B by roughly 11 macro points on the other twelve — so the specialization claim is not dead, just the traffic part of it.

Two smaller items. The whole architecture rests on Jev and Kev cited to a blog and a repo; a referee should confirm those bases exist as claimed. And the 9B checkpoint was picked by inspecting transfer scores; the frozen confirmation set mitigates this, so I'd call it minor.

Send it out. The corpus, the QA methodology, and the tool/policy results deserve referee time. Ask for an identity-field ablation on CICIoT/ToN-IoT and a seed-repeat of the 0.8B headline numbers, and the paper is in decent shape.

The strongest objection

The headline gain (b7, b25, b124) is concentrated in the two largest source-label tasks. CICIoT2023 and ToN-IoT contribute 88,000 of 170,185 training questions and 9,500 of 21,634 test questions (about 44%); SecJev reaches 91.80%/98.03% at 0.8B and 91.72%/98.80% at 4B (Table 3). These are exactly the tasks where the paper's own two-reviewer audit (Table 11; b232) finds the visible evidence insufficient in 17/20 (CICIoT) and 20/20 (ToN-IoT) sampled questions, with reviewer-label agreement likewise low (CICIoT 14/20, ToN-IoT 11/20). Human judgment cannot recover these labels from the displayed telemetry, yet the model does so at roughly 92-99%. Either the model exploits cues well beyond human perception, or it keys on per-capture fingerprints — source/destination IPs, ports, MACs, device-type signatures — that identify which capture or host a row came from. The group-based splits (captures for CICIoT, source hosts for ToN-IoT, Section 4.2) prevent row-level test-in-training leakage but do not remove a capture-identity shortcut unless the trimmed fields cannot encode it; the paper asserts endpoint identities are omitted for ToN-IoT but does not demonstrate that searchable identity proxies (ports, unique feature signatures) are absent. The disjoint-source check cuts toward the fingerprint reading: on new IoT-23 captures SecJev-4B reaches 65.10% with 96.6% recall but 66.4% FPR, and new-capture accuracy collapses from the ~88% exploratory level (b236; Appendix D.1), consistent with memorized per-capture patterns failing to transfer rather than with genuine attack semantics. Since Kev scores only 24.83-37.93% on these two tasks (Init., Table 3), the entire 20.51-point gap over Kev-9B (b127) is concentrated here; if the source-label component is fingerprint memorization, the specialization claim and the headline overstate genuine transfer.

The paper is methodologically candid: it discloses checkpoint selection after exploratory scores (b467), unmeasured seed variability (b235), the insufficiency of visible evidence (b232), and the collapse of IoT-23 transfer with high FPR. It also provides genuine independent support in group-stratified bootstrap intervals, a disjoint-source confirmation set frozen before prediction, and a two-reviewer audit agreeing on 261/280 answers. None of this resolves the leakage question because the audit itself is the evidence that the source-label signal is invisible to humans. My convergence on the reader's weakest assumption is deliberate: the Table 3 source-label rows and the 20.51-point headline both hinge on whether the 9,500-question CICIoT/ToN-IoT block reflects genuine inference or capture-identity shortcuts. The reader's conditionality is the right verdict; my concern sharpens it from a general observability doubt into a specific, testable fingerprint channel with a defined ablation. I do not move the verdict because the leakage failure is not demonstrated — the honest outcome is to keep the paper conditional pending the ablation. Should the ablation collapse accuracy, the headline gap would need re-baselining on fingerprint-free inputs; should accuracy hold, the concern closes and the specialization claim stands on the reported numbers.

The referee's report

If correct, the headline result is practically significant: a 0.8B specialized model with a typed decision interface outperforming a 9B general model on security decision tasks would justify small, cheap specialized decision models for security triage, and the released corpus, adapters, and code are a reusable contribution. The paper is unusually careful in several respects that deserve credit: the exploratory-versus-confirmation boundary (Appendix D.1), independent two-reviewer audit with explicit evidence-sufficiency judgments (Table 11), exhaustive policy recomputation of 37,490 questions, matched-precision efficiency measurements, and candid acknowledgment of limitations (single runs, seed variability, checkpoint selection). The credibility hinge is the source-label shortcut question: the largest per-task gains occur on exactly the two tasks (CICIoT2023, ToN-IoT) for which the paper's own audit deems the visible evidence largely insufficient, and 44% of the test set lies there.

Pith's reading of the paper (a machine's summary, not the authors')

The paper sets out to show that a small language model specialized for security decisions can outperform a much larger general-purpose model when both use a typed decision interface rather than free-form text generation. It claims that security specialization improves every task in every model size, with the smallest SecJev model at 0.8B parameters gaining 39.40 percentage points over its general Kev initialization and beating the 9B general Kev model by 20.51 points in task-macro accuracy. The point of the work is that domain expertise and a constrained output format can substitute for raw scale: a security operator could run a cheap, fast model locally to classify tool outputs, traffic, federated updates, consensus evidence, and authentication events with high accuracy. If the claim holds, defensive workflows can get decision-grade predictions at a fraction of the compute footprint of a frontier generalist.

What carries the argument. The engine is Kev's single-pass candidate scorer, used with the Jev-style decision interface. For each question, the model encodes the state, the question, and the list of candidate descriptions in one forward pass; the hidden state at a final decision marker and the hidden states at each candidate delimiter are projected to a common 256-dimensional space, and their dot products become logits that are softmaxed into probabilities over the typed alternatives. A freeze-backbone setup with rank-16 LoRA adapters and a trainable pointer head, trained with scene-weighted cross-entropy over 118,818 questions per epoch, supplies the security expertise; a single calibrated temperature adjusts confidence without changing the winning candidate. SecJev-Corpus provides the supervision, pairing source annotations and recomputed policy answers with questions that preserve each task's decision semantics.

Core claim. SecJev is the first family of Jev-like decision models specialized for security, and its central claim is that specialization over a shared typed interface converts a general single-pass scorer into a security expert across 14 tasks. Built on Kev's candidate scorer, SecJev freezes the backbone and trains rank-16 LoRA adapters plus a pointer head on 170,185 questions drawn from eight sources, covering Boolean, choice, and ordered outputs. The result is that every one of the four model sizes improves on every task; at 4B the task-macro accuracy reaches 94.51%, and the 0.8B model reaches 90.34% task-macro accuracy, exceeding the 9B general model's 69.82% by 20.51 points. The paper also reports that answer-only generative fine-tuning achieves nearly the same accuracy at slightly higher memory, that gains transfer to new prompt-injection and traffic sources, and that the specialized models retain general decision accuracy within about half a point while losing some confidence quality on general tasks.

The premise it rests on. The load-bearing premise is that the group-based data splits (captures, source hosts, simulation runs, and chronological days) together with the removal of endpoint identities fully prevent label leakage, so the high source-label accuracies (CICIoT 91.80%, ToN-IoT 98.03%) reflect genuine inference from visible evidence and not memorized label patterns.

If the paper is right (Pith's inferences, not the authors')

A 0.8B security decision model can replace a 9B general model on these 14 tasks, cutting peak batch-one memory from about 17.1 GiB to 2.56 GiB.

All 14 tasks improve at all four sizes, including near-perfect application of explicit policy rules at 99.87% task-macro accuracy on the six policy tasks at 4B.

Because the scorer is single-pass and never decodes answer tokens, accuracy and latency track answer-only generative fine-tuning while peak memory is lower at both 0.8B and 4B.

General decision accuracy is retained within about half a percentage point after security training, so specialization does not destroy the underlying decision capability.

Gains reproduce on new source groups, with SecJev-4B beating Kev-4B by 17.67 points on BIPIA injection decisions and by 15.10 points on IoT-23 traffic labels, alongside capture-dependent false alarms.

If source-label accuracy rests partly on memorized capture patterns rather than visible evidence, real deployments with new sensors or event formats could fall well short of the reported 91-98% source-label numbers; a test that randomizes group membership while keeping fields identical would expose this.

The same adaptation recipe could plausibly specialize Kev-like models for other high-stakes typed domains such as medical triage, log review, or compliance, since the scorer and corpus-construction method are domain-agnostic.

Because confidence calibration worsens on general tasks after specialization, operators who set confidence thresholds will need per-task and per-deployment calibration rather than one global temperature.

The near-perfect policy-task scores suggest the model has learned to apply a displayed rule to visible evidence; if that is robust, rule-based policy engines could be expressed as typed decisions and audited question by question.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Security workflows need models that turn complex observations and explicit policies into decisions. System One models introduced by Jev return typed predictions and probabilities; security specialization supplies the domain expertise behind those predictions. We introduce SecJev, to our knowledge the first family of Jev-style decision models specialized for security, spanning 0.8B to 9B parameters. Built on Kev's single-pass candidate scorer, SecJev learns Boolean, choice, and ordered decisions from text, telemetry, and observation histories. We develop SecJev-Corpus to unify source-label prediction and explicit-policy evaluation across 14 tasks and eight sources. It covers tool outputs, traffic, federated updates, consensus, authentication, and vehicle messages. Scene-weighted training adapts the models across these domains while preserving a shared typed decision interface. Security specialization improves every model in the family; SecJev-0.8B outperforms general Kev-9B by 20.51 percentage points in task-macro accuracy. At 0.8B and 4B, SecJev achieves accuracy and latency close to answer-only generative fine-tuning with lower peak allocated GPU memory. SecJev-4B improves on Kev-4B and security-trained generation in prompt-injection and traffic transfer tests. We release adapters, decision heads, SecJev-Corpus, and training and inference code.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-05

The authors' main claim, as the classifier picked it out 1.00

“Security specialization improves every model in the family; Secclassifier-0.8B outperforms general Kev-9B by 20.51 percentage points in task-macro accuracy.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.95 · states limitation 0.84 · human participants 0.05 · controlled comparison 0.93 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.98

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Security workflows need models that turn complex observations and explicit policies into decisions. background 1.00 0.06 0.05 asserted 0.99
System One models introduced by Jev return typed predictions and probabilities; security specialization supplies the domain expertise behind those predictions. background 0.91 0.12 0.13 asserted 1.00
We introduce SecJev, to our knowledge the first family of Jev-like decision models specialized for security, spanning 0.8B to 9B parameters. contribution 1.00 0.89 0.15 asserted 0.97
Built on Kev's single-pass candidate scorer, SecJev learns Boolean, choice, and ordered decisions from text, telemetry, and observation histories. method 0.95 0.72 0.06 asserted 1.00
We develop SecJev-Corpus to unify source-label prediction and explicit-policy evaluation across 14 tasks and eight sources. contribution 0.55 0.95 0.07 asserted 1.00
It covers tool outputs, traffic, federated updates, consensus, authentication, and vehicle messages. method 0.64 0.48 0.03 asserted 1.00
Scene-weighted training adapts the models across these domains while preserving a shared typed decision interface. method 0.99 0.86 0.06 asserted 1.00
Security specialization improves every model in the family; SecJev-0.8B outperforms general Kev-9B by 20.51 percentage points in task-macro accuracy. result 1.00 0.94 0.60 asserted 1.00
Comparisons with answer-only generative fine-tuning show close accuracy and latency with lower peak inference memory. result 1.00 0.91 0.09 asserted 1.00
Tests on new source groups reproduce gains over Kev in prompt-injection and traffic decisions, with capture-dependent false alarms. result 0.93 0.93 0.11 asserted 1.00
We release adapters, decision heads, SecJev-Corpus, and training and inference code. contribution 0.99 0.92 0.05 asserted 0.98

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.93

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-05: 1 request, 8,068 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (1)
Figure 1 of the paper
Figure 1. SecJev’s shared decision path and security adaptation. Source annotations and explicit policies provide

Figures are the authors', reproduced from the paper.

Cited by 3 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment