{"id":"ab9d78c8-7f77-44f1-8d1e-e98ed3af68ab","arxiv_id":"1908.09219","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Heterogeneous Relational Kernel Learning builds an interpretable embedding of time series from the standardized BIC scores of all kernels evaluated during relational kernel search.","lead":"This paper introduces a simple way to turn Bayesian kernel learning on multiple time series into an interpretable embedding, so that groups of series with similar structure can be found automatically. The method reuses scores already computed during relational kernel learning, making it nearly free, and it enables clustering, pattern discovery, and anomaly detection.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Row-standardized BIC profiles may encode series length and noise as much as structure, so the embedding's claimed structural semantics are not yet established.","rationale":"The reader's weakest_assumption correctly identifies the comparability of standardized BIC scores as the load-bearing premise. I agree with this assessment. The concern is not that the method is internally inconsistent, but that its central validity condition is unverified. The paper's synthetic experiment varies amplitudes, frequencies, phases, and noise within each class while holding series length fixed and standardizing the series, so it cannot expose length-dependent effects on the BIC penalty. The computational-cost claim is secondary and separately questionable because Algorithm 1 enumerates all grammar expansions rather than reusing a greedy RKL search, but even if that issue were resolved, the structural-comparability issue remains. Since the reader's CONDITIONAL verdict already flags this assumption, no verdict change is needed. The proposed controlled experiment would directly test whether the embedding captures structure across length and noise variation; if it passes, the concern is resolved and the central claim is substantially supported. If it fails, the embedding's usefulness for heterogeneous time series would be seriously undermined, and the verdict should move toward rejection. The method is clearly explained and the experiments are suggestive, so I do not see grounds for outright rejection without this test.","tokens_in":9257,"tokens_out":8400,"duration_ms":89308,"concrete_test":"Generate two structural classes (e.g., sine waves and lines) with lengths n in {50, 200, 1000} and noise levels sigma in {0.1, 0.5, 1.0}, with 10 series per cell. Compute the HRKL embedding exactly as in Algorithm 1 and Section 3, then cluster with HDBSCAN and cosine distance as in Section 4.1. If same-structure series with different lengths and noise levels are not assigned to the same cluster (V-measure well below the equal-length baseline), then the standardized BIC profiles are not structurally comparable across heterogeneous series, and the central claim would need to be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the row-standardized BIC history matrix from an RKL search is an interpretable embedding that captures structural similarity across heterogeneous time series (Section 3). For this to hold, the relative BIC ordering of kernels for a series must reflect the series' generative structure, not nuisance attributes. The BIC for series j and kernel k is -2 log p(d_j|k, theta_hat) + |M| log n_j (Section 2.2). The penalty term scales with log n_j, so for a longer series the penalty differences between complex and simple kernels are amplified. After per-row standardization, this length-dependent scaling is not removed: standardization subtracts the row mean and divides by row standard deviation, but the relative penalty structure across kernels still depends on n_j. Two series with identical structure but different lengths can therefore have different standardized BIC profiles, even before considering noise-level effects on the marginal likelihood scale. The paper's synthetic and Google Trends experiments use series of equal length, and the gait experiment does not systematically vary length while holding structure fixed, so this assumption is never directly tested. The embedding's interpretability claim also depends on column-wise meaning after standardization, which is not established. This is load-bearing because if BIC profiles are dominated by length or noise, the clustering results in Sections 4.1-4.3 would not generalize to the heterogeneous time series the method is designed for.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Heterogeneous Relational Kernel Learning (HRKL), an extension of relational kernel learning (RKL) for time series. During an RKL search, the method records the BIC score of every kernel evaluated on every time series, forming a J-by-K BIC history matrix. After row-standardizing this matrix, each row is used as an embedding of the corresponding time series. The authors argue that because each column corresponds to an interpretable kernel, the embedding is interpretable, and they demonstrate its use for clustering, pattern discovery, and anomaly detection. They further claim that HRKL adds practically no computational cost over a single RKL fit. The central claim is that standardized BIC profiles provide a meaningful measure of structural similarity that allows RKL-style models to handle heterogeneous sets of time series without extra model fitting.","tokens_in":9498,"tokens_out":4501,"duration_ms":46946,"significance":"If the central claim holds, the contribution is genuinely useful: it repurposes intermediate results that are normally discarded, yielding an interpretable embedding at negligible additional cost. The method is simple, clearly specified in Algorithm 1, and the reuse of BIC history is an elegant idea. The experimental results are suggestive, especially the perfect separation in the gait anomaly detection task. The paper also states its computational overhead honestly and quantitatively. However, the evidence is not yet sufficient. The embedding's interpretability rests on an untested comparability assumption about BIC scores across series of different lengths and noise levels, the experiments are single-run with no error bars or significance tests, and one of the three application claims (pattern discovery) is contradicted by the paper's own table, where DTW achieves V-measure 1.000 versus HRKL's 0.821. These issues are fixable, but they are load-bearing for the paper's central claims rather than mere presentation.","major_comments":[{"comment":"The central interpretability claim rests on the assumption that row-standardized BIC profiles encode shared structure rather than series-specific nuisance attributes. The BIC, defined in Section 2.2 as -2 log p(D|M) + |M| log n, has a penalty term that scales with log n_j, so longer series receive systematically different penalty increments between complex and simple kernels. Standardizing each row by subtracting its mean and dividing by its standard deviation does not remove this dependence, and the marginal likelihood scale also depends on noise level and other series-specific factors. The synthetic and Google Trends experiments use series of equal length, and the gait experiment does not systematically vary length while holding structure fixed. Please add a controlled experiment that varies series length and noise level within the same structural class and checks whether the standardized BIC profile remains stable. Without such a test, the embedding's structural semantics, which are the foundation of the method, are not established.","section":"Section 3 and Section 2.2 (BIC definition)"},{"comment":"All quantitative results come from single runs with no error bars, no repeated trials, and no statistical significance tests. This is particularly problematic for the anomaly detection experiment, where Section 4.3.1 states that one time series from each class is 'randomly selected' for corruption; a single random draw cannot support the claim that HRKL 'significantly outperforms' all baselines. Please report means and standard deviations over multiple random corruptions and repeated clustering runs, or at least provide a sensitivity analysis, so the reader can assess whether the reported margins are stable.","section":"Tables 1-3 and Sections 4.1-4.3"},{"comment":"The text contains a direct contradiction about the pattern discovery result. Section 4.2 says 'We find HRKL and DTW both perform best and discover the same overall structures,' while Section 4.2.3 says 'DTW performs the best, followed by HRKL and k-Shape with a tie.' Table 2 shows DTW with V-measure 1.000 and HRKL with V-measure 0.821, tied with k-Shape. The correct statement is that DTW outperforms HRKL on this task. This overstatement should be corrected, since pattern discovery is one of the three headline applications.","section":"Section 4.2 and Table 2"},{"comment":"The claim in Section 4.3.3 that 'SAX BoP also clearly separates the corrupted data from the uncorrupted data' is inconsistent with Table 3, where SAX BoP attains V-measure 0.316. If the intended claim is that the dendrogram in Figure 6b visually shows a separation, that is not a quantitative guarantee and should be stated as such, or the text should be tempered to match the table.","section":"Section 4.3 and Table 3"}],"minor_comments":[{"comment":"Algorithm 1 line 9 says 'cluster(B)', but Section 3 states that the standardized matrix, not the raw BIC matrix, is the embedding. Please make the standardization step explicit in the algorithm.","section":"Algorithm 1"},{"comment":"There is a typo: 'has to potential' should be 'has the potential'.","section":"Section 1"},{"comment":"The caption reads 'The data can is divided into four structural subsets'; this should be 'The data can be divided'.","section":"Figure 3 caption"},{"comment":"The text and figure use 'Heavyside' where the standard spelling is 'Heaviside'.","section":"Section 4.1.1 and Figure 2"},{"comment":"Reference [14] is cited in text as 'Yunseong et al.' but the reference entry lists the first author as 'Hwang Yunseong'; please make the citation style consistent with the reference entry.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short workshop-style manuscript with a simple and appealing idea. The main technical risk is the untested comparability of BIC profiles across series of different lengths and noise levels; this is a correctable but load-bearing weakness. The experimental section also needs repeated runs and statistical support. I recommend major revision rather than rejection, because the core idea is sound enough to deserve a more rigorous evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper does one simple thing that hasn't been done – save the BIC score for every kernel–series pair from a relational kernel search and treat the standardized row as an embedding – and that idea is worth knowing. The paper explains the method cleanly and shows on synthetic data that it clusters by structure rather than parameter values, where DTW and SAX BoP fail. The computational cost argument is honest: the extra work is clustering plus saving a matrix, which is negligible next to GP fits.\n\nWhere it gets soft: the pattern discovery section says HRKL and DTW 'both perform best,' but Table 2 gives DTW a V-measure of 1.000 and HRKL 0.821. That is not a tie. The anomaly detection result is perfect, but it is one run on 15 series with three corrupted; no error bars, no repeated runs, and no statistical testing anywhere. The kernel grammar is described as 'all non-redundant structures,' but the redundancy rule is not specified, and no code is provided, so exact reproduction is not possible.\n\nThe bigger open question, which the paper does not address, is what the standardized BIC profile actually encodes. The BIC penalty term includes |M| log n, so the penalty differences among kernels scale with series length. After row standardization, the profile may still depend on n as well as on structure. All three experiments use series of equal length, so this confound is never tested. This is not a refutation – the embedding could still behave well on mixed-length data – but it is a load-bearing assumption that should be stated and tested.\n\nThe 'circularity' concern does not land: the kernels are fit per series, clustering uses only the scores, and the final cluster kernel is chosen afterwards. That is just ordinary bootstrap-like thinking, not circularity.\n\nSo: the central idea is plausible and the paper gives it a fair first evaluation. It deserves a careful referee, though not a spotless one. The authors should fix the DTW overclaim, add error bars or repeated runs, release code, and either test or clearly limit the claim with respect to series length. With those changes it would be a solid workshop or short-conference paper. As it stands, it is a useful but preliminary contribution.","headline":"A simple, genuinely novel embedding from BIC histories that works on the chosen experiments, with a thin evaluation and an overclaim about DTW.","tokens_in":10031,"tokens_out":4259,"would_cite":false,"duration_ms":38785,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Saving every BIC score during kernel search yields an interpretable embedding for heterogeneous time series.","keywords":["relational kernel learning","heterogeneous time series","Gaussian processes","kernel embedding","time series clustering","anomaly detection","Bayesian information criterion","interpretability"],"falsifier":"Generate time series with the same six structural classes used in the synthetic experiment but vary the length and noise level within each class, then run HRKL; if the clusters separate by length or noise rather than by structural class, the embedding is capturing nuisance factors instead of structure.","tokens_in":9048,"feed_emoji":"📈","tokens_out":6762,"duration_ms":63887,"temperature":0.7,"pith_summary":"The paper introduces Heterogeneous Relational Kernel Learning (HRKL), a method for turning an existing relational kernel learning (RKL) search into an embedding for time series that do not share a single global structure. The key move is to keep the Bayesian Information Criterion score of every kernel evaluated on every time series, arrange these scores in a matrix, standardize each row, and treat each row as a vector representation of that time series. Because each dimension of the vector is tied to one interpretable kernel, the embedding has a built-in meaning: two series are similar when the same kernels fit both well and the same kernels fit both poorly. The authors demonstrate that this representation supports clustering, pattern discovery, and anomaly detection on data sets where a shared-kernel assumption fails, at practically the same computational cost as running RKL once.","feed_headline":"Saved BIC scores become a free embedding for mixed time series","feed_subtitle":"Reusing the existing BIC history matrix yields interpretable clusters at near-zero added cost.","key_machinery":"The load-bearing object is the BIC history matrix $B$, a $J \\times K$ array whose entry $B_{j,k}$ is the Bayesian Information Criterion of a Gaussian process model specified by kernel $k$ and optimized for time series $d_j$. Row-wise standardization, subtracting the row mean and dividing by the row standard deviation, converts each row into the time series embedding. The embedding carries meaning because each column is associated with one kernel from the compositional kernel language, so a high standardized value means that kernel explains that series especially well. The argument runs through this matrix: it converts the bookkeeping of a single model search into a geometric representation, and clustering the rows gives sub-populations without any additional model fits.","core_discovery":"The paper's central claim is that the BIC history matrix, the intermediate output of an RKL search that previous work discarded, already contains enough information to represent each time series by its response to the whole library of candidate kernels. After each row is standardized to zero mean and unit variance, the row encodes a profile: which kernels explain that series unusually well and which explain it unusually poorly. Clustering these profiles separates time series by shared structure rather than by shared parameter values, and the best-scoring kernel for each cluster gives a human-readable description of the sub-population. The paper further claims this is essentially free, because the matrix is produced by a single RKL run and the subsequent clustering cost is small relative to the cubic cost of fitting the Gaussian process models.","pith_inferences":["Beyond the paper: because the embedding is built from model-selection scores, the same construction should work with other selection criteria, such as AIC or cross-validated likelihood; the paper only explores BIC.","Beyond the paper: the per-row standardization makes the embedding a within-series shape profile, so series length and noise level could in principle dominate structure; testing HRKL on series of varying lengths within one structural class would settle this.","Beyond the paper: the embedding could be used for more than clustering, such as low-dimensional visualization of large time-series collections or as a feature representation for supervised tasks; the paper does not explore those uses."],"forward_implications":["HRKL can identify sub-populations in heterogeneous time series and produce a separate interpretable kernel description for each, a task that relational kernel learning and semi-relational kernel learning cannot do without a combinatorial search.","The same RKL run that would find one shared kernel also yields the embedding, so clustering, pattern discovery, and anomaly detection come at a measured extra cost of about 0.07 percent relative to all kernel evaluations.","Each dimension of the embedding is interpretable, so the clusters it produces can be explained in natural language, for example as a linear function or a periodic function whose shape changes smoothly.","On the synthetic clustering benchmark, HRKL separates structural classes such as sine waves, lines, step functions, and sinc functions with a V-measure of 0.836, well above the DTW, SAX BoP, and k-Shape baselines.","On the corrupted gait data, HRKL achieves a perfect separation of corrupted from uncorrupted time series, while the baseline methods do not separate them cleanly."],"supporting_citations":[{"why":"The relational kernel learning method whose search produces the BIC history matrix that HRKL reuses.","marker":"[14]"},{"why":"Defines the compositional kernel language whose interpretable base kernels give each embedding dimension its meaning.","marker":"[3]"},{"why":"Extends that language to natural-language descriptions of kernels, which the paper uses to interpret clusters.","marker":"[5]"},{"why":"HDBSCAN, the density-based clustering algorithm applied to the embedding rows in the synthetic experiment.","marker":"[2]"},{"why":"Provides the Bayesian Information Criterion that fills the BIC history matrix.","marker":"[11]"},{"why":"Supplies the Gaussian process framework that makes each kernel fit a well-defined model comparison.","marker":"[8]"}],"fun_headline_variants":["Reuse BIC scores to cluster mixed time series for free","Free embedding from saved BIC scores for mixed series","BIC history matrix: free interpretable time series features","Clustering mixed time series with no extra cost","Discarded BIC scores become a free cluster embedding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The embedding is only as meaningful as the assumption that a time series's standardized BIC scores against a fixed set of kernels reflect its structure rather than nuisance properties like length, noise level, or amplitude; the paper does not directly test this.","fun_headline_variants_meta":{"raw":{"variants":["Reuse BIC scores to cluster mixed time series for free","Free embedding from saved BIC scores for mixed series","BIC history matrix: free interpretable time series features","Clustering mixed time series with no extra cost","Discarded BIC scores become a free cluster embedding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000355,"raw_usage":{"total_tokens":1831,"prompt_tokens":753,"completion_tokens":1078,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":369,"completion_tokens_details":{"reasoning_tokens":1000}},"tokens_in":369,"tokens_out":1078,"duration_ms":8430,"temperature":1.0,"reasoning_tokens":1000,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:17:28.050197+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate time series with the same six structural classes used in the synthetic experiment but vary the length and noise level within each class, then run HRKL; if the clusters separate by length or noise rather than by structural class, the embedding is capturing nuisance factors instead of structure.","supporting_citations":[{"cited_title":"Automatic Construction of Non Parametric Relational Regression Models for Multiple Time Series","cited_arxiv_id":null,"evidence_quote":"The relational kernel learning method whose search produces the BIC history matrix that HRKL reuses."},{"cited_title":"Tenenbaum, and Zoubin Ghahramani","cited_arxiv_id":null,"evidence_quote":"Defines the compositional kernel language whose interpretable base kernels give each embedding dimension its meaning."},{"cited_title":"Tenenbaum, and Zoubin Ghahramani","cited_arxiv_id":null,"evidence_quote":"Extends that language to natural-language descriptions of kernels, which the paper uses to interpret clusters."},{"cited_title":"Density-Based Clustering Based on Hierarchical Density Estimates","cited_arxiv_id":null,"evidence_quote":"HDBSCAN, the density-based clustering algorithm applied to the embedding rows in the synthetic experiment."},{"cited_title":"Estimating the Dimension of a Model","cited_arxiv_id":null,"evidence_quote":"Provides the Bayesian Information Criterion that fills the BIC history matrix."},{"cited_title":"Rasmussen and Williams - Gaussian Processes for Machine Learning","cited_arxiv_id":null,"evidence_quote":"Supplies the Gaussian process framework that makes each kernel fit a well-defined model comparison."}],"review_version":1}