arxiv: 2604.13056 · v1 · submitted 2026-03-17 · 💻 cs.CL · cs.AI

Recognition: no theorem link

Text-as-Signal: Quantitative Semantic Scoring with Embeddings, Logprobs, and Noise Reduction

Hugo Moreira

Authors on Pith no claims yet

Pith reviewed 2026-05-15 09:26 UTC · model grok-4.3

classification 💻 cs.CL cs.AI

keywords text embeddingssemantic scoringlogprob evaluationdimensionality reductionanomaly detectioncorpus analysisnews monitoring

0 comments

The pith

A pipeline turns text corpora into quantitative semantic signals using embeddings, logprob scoring, and manifold projection.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper demonstrates a workflow that represents each text document as a full embedding scored against a configurable dictionary of semantic dimensions via logprobs, then projects the results onto a noise-reduced manifold. In the case study this produces positioning for 11,922 Portuguese AI news items across six dimensions, supporting both per-document placement and aggregated corpus profiles. The method combines model embeddings with output-space indicators and anomaly detection to create an adaptable identity space for inspection and monitoring tasks. Because the dictionary is user-defined rather than fixed, the same structure can serve different analytical goals without requiring a universal schema. The approach treats text as measurable signals that can be tracked and compared quantitatively.

Core claim

Combining Qwen embeddings, logprob-based scoring over a positional dictionary, UMAP projection after noise reduction, and a three-stage anomaly-detection procedure yields an operational text-as-signal pipeline that supports document-level semantic positioning and corpus-level characterization for AI engineering applications such as corpus inspection and monitoring.

What carries the argument

The text-as-signal workflow that scores full-document embeddings through logprobs on a configurable positional dictionary and projects the scores onto a low-dimensional manifold for structural interpretation.

Load-bearing premise

Logprob scores derived from the positional dictionary remain semantically meaningful and stable after projection onto the noise-reduced manifold without systematic bias from the language model.

What would settle it

Human raters independently scoring a sample of the same documents on the six semantic dimensions and finding low correlation with the pipeline outputs would show the signals are not semantically meaningful.

Figures

Figures reproduced from arXiv: 2604.13056 by Hugo Moreira.

**Figure 1.** Figure 1: Base 2D semantic topography obtained from the UMAP projection of the fulldocument embedding space. semantic coordinates that complement the geometric structure of the embedding manifold. Once computed, they are painted onto the 2D projection so that the same semantic identity can be read spatially across the map. To make the pipeline explicit, let the corpus be D = {d1, . . . , dn}, where each di is a ful… view at source ↗

**Figure 2.** Figure 2: Anomaly-detection sequence used for noise reduction. The first panel repeats the baseline semantic topography, followed by the global distance pass, the local outlier pass, and the structural outlier pass. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Figure 3: K-Means regions before and after anomaly detection and noise reduction. The comparison shows why K-Means is central to the anomaly-detection analysis rather than only a downstream visualization step. The trimmed map that remains after anomaly detection and noise reduction is not only easier to inspect visually. It also provides a more stable substrate for downstream tasks such as region-level characteriza… view at source ↗

**Figure 4.** Figure 4: Relative frequency distribution of semantic centrality scores. The plot compares all articles (grey) with the dataset after removing outliers (orange) and structural noise (blue), showing a prominent peak in the [0.6, 0.75] range. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

**Figure 5.** Figure 5: Representative article-level semantic identity dimensions projected onto the manifold. Across dimensions, opposite poles occupy different parts of the space rather than mixing uniformly across the corpus. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗

read the original abstract

This paper presents a practical pipeline for turning text corpora into quantitative semantic signals. Each news item is represented as a full-document embedding, scored through logprob-based evaluation over a configurable positional dictionary, and projected onto a noise-reduced low-dimensional manifold for structural interpretation. In the present case study, the dictionary is instantiated as six semantic dimensions and applied to a corpus of 11,922 Portuguese news articles about Artificial Intelligence. The resulting identity space supports both document-level semantic positioning and corpus-level characterization through aggregated profiles. We show how Qwen embeddings, UMAP, semantic indicators derived directly from the model output space, and a three-stage anomaly-detection procedure combine into an operational text-as-signal workflow for AI engineering tasks such as corpus inspection, monitoring, and downstream analytical support. Because the identity layer is configurable, the same framework can be adapted to the requirements of different analytical streams rather than fixed to a universal schema.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This paper gives a clear description of a configurable pipeline for semantic text scoring but lacks the validation needed to judge its real performance.

read the letter

This paper outlines a pipeline that converts text into quantitative semantic signals by combining full-document embeddings from Qwen, logprob scoring against a user-defined positional dictionary, UMAP projection, and a three-stage anomaly detection step. The result is an identity space for positioning documents and characterizing corpora, demonstrated on a set of over 11,000 Portuguese AI news articles. What is new is the specific workflow that ties these elements together for tasks like corpus inspection and monitoring. The configurability stands out, allowing the semantic dimensions to be swapped depending on the application rather than relying on a fixed set. The paper does a good job describing the steps in a way that could be implemented, and the focus on operational utility for AI engineering is clear. Using logprobs directly from the model output space is a reasonable choice for deriving indicators without additional training. The main limitation is the absence of quantitative validation. No performance metrics, error rates, or comparisons to baselines are mentioned, which makes it difficult to assess whether the signals are reliable or if the noise reduction actually preserves semantic meaning. The assumption that the logprob scores produce stable coordinates after projection could be a point for further testing. This work is best suited for readers interested in applied methods for text analysis in real-world settings, such as monitoring news flows or document collections. It offers a reusable pattern rather than a breakthrough in theory. I would bring this to a reading group for discussion on practical implementations. I would not cite it yet due to the lack of results, but it deserves peer review to get feedback on strengthening the empirical side.

Referee Report

3 major / 2 minor

Summary. The paper presents a practical pipeline for converting text corpora into quantitative semantic signals. Each document receives a full embedding from the Qwen model, is scored via log-probabilities against a configurable positional dictionary (instantiated here with six semantic dimensions), and is projected via UMAP onto a noise-reduced manifold; a three-stage anomaly-detection step then supports structural interpretation. The workflow is demonstrated on a corpus of 11,922 Portuguese news articles about artificial intelligence, yielding document-level positioning and aggregated corpus profiles intended for AI-engineering tasks such as inspection, monitoring, and downstream analysis. The framework is explicitly configurable rather than tied to a universal schema.

Significance. If the logprob-derived scores prove stable and semantically meaningful after projection, the approach supplies a reusable, training-free workflow that combines off-the-shelf embeddings, dictionary-based scoring, and standard dimensionality reduction for domain-specific corpus analysis. Its configurability is a practical strength for adapting the same pipeline to different analytical needs without retraining models.

major comments (3)

[Case Study] Case Study section: the manuscript demonstrates the pipeline on a single corpus but supplies no quantitative validation metrics (e.g., correlation with human semantic judgments, inter-annotator agreement, or downstream task performance), error analysis of the logprob scores, or comparisons against baselines such as direct embedding clustering or TF-IDF. This absence is load-bearing for the central claim of operational utility.
[Methodology] Methodology (pipeline description): the three-stage anomaly-detection procedure is described at a high level without specifying the exact statistical criteria, thresholds, or decision rules applied in each stage. These details are required to assess whether the procedure genuinely reduces noise or merely filters according to arbitrary cut-offs.
[Abstract and Methodology] Abstract and Methodology: the claim that logprob scoring against the positional dictionary produces 'semantically meaningful and stable quantitative signals' is asserted without any empirical test for systematic bias introduced by the underlying language model or for stability across different dictionary instantiations.

minor comments (2)

[Methodology] The notation for the six semantic dimensions and the exact construction of the positional dictionary should be formalized (e.g., as a table or explicit list) to improve reproducibility.
Standard references for UMAP and the Qwen model family are missing from the bibliography.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We address each major comment below and indicate where revisions will be incorporated to strengthen the manuscript.

read point-by-point responses

Referee: [Case Study] Case Study section: the manuscript demonstrates the pipeline on a single corpus but supplies no quantitative validation metrics (e.g., correlation with human semantic judgments, inter-annotator agreement, or downstream task performance), error analysis of the logprob scores, or comparisons against baselines such as direct embedding clustering or TF-IDF. This absence is load-bearing for the central claim of operational utility.

Authors: We acknowledge that the case study is primarily illustrative. To address the concern for quantitative support of operational utility, the revised manuscript will add baseline comparisons using TF-IDF and direct embedding clustering with metrics such as silhouette scores, plus a basic error analysis of logprob outliers. Full human judgment correlation is noted as a limitation for future work given resource constraints. revision: yes
Referee: [Methodology] Methodology (pipeline description): the three-stage anomaly-detection procedure is described at a high level without specifying the exact statistical criteria, thresholds, or decision rules applied in each stage. These details are required to assess whether the procedure genuinely reduces noise or merely filters according to arbitrary cut-offs.

Authors: We agree the description requires more precision. The revised Methodology section will specify the exact statistical criteria, thresholds (e.g., z-score thresholds and density parameters), and decision rules for each of the three stages to allow replication and evaluation of noise reduction. revision: yes
Referee: [Abstract and Methodology] Abstract and Methodology: the claim that logprob scoring against the positional dictionary produces 'semantically meaningful and stable quantitative signals' is asserted without any empirical test for systematic bias introduced by the underlying language model or for stability across different dictionary instantiations.

Authors: The claim follows from the direct use of model log-probabilities for alignment quantification. To address bias and stability concerns, the revision will add analysis of score stability across dictionary variants and qualitative discussion of potential LM biases with corpus examples. The abstract wording will be qualified to reflect the demonstrated case-study results. revision: partial

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper presents an operational pipeline that applies off-the-shelf components (Qwen embeddings, UMAP projection, logprob scoring over a user-supplied positional dictionary, and standard anomaly detection) to a corpus without any formal derivation, optimality proof, or self-referential fitting step. No equation reduces a claimed prediction to a fitted parameter by construction, no uniqueness theorem is imported from self-citation, and the dictionary itself is explicitly configurable rather than derived from the target data. The central claim is therefore a practical workflow whose validity rests on external model behavior and user-defined inputs, not on internal redefinition.

Axiom & Free-Parameter Ledger

1 free parameters · 2 axioms · 0 invented entities

The central claim rests on the assumption that standard embedding models and logprob outputs can be repurposed as semantic signals once a dictionary is supplied; the six dimensions are chosen for the specific corpus.

free parameters (1)

Number of semantic dimensions
Dictionary instantiated as six dimensions for the Portuguese AI news case study.

axioms (2)

domain assumption Qwen embeddings capture document-level semantics sufficiently for downstream scoring
Used as the representation step for every news item.
domain assumption Log probabilities from the model can be meaningfully aggregated against a positional dictionary
Core mechanism for producing quantitative semantic scores.

pith-pipeline@v0.9.0 · 5448 in / 1367 out tokens · 51231 ms · 2026-05-15T09:26:51.940563+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

8 extracted references · 8 canonical work pages · 3 internal anchors

[1]

BERTopic: Neural topic modeling with a class-based TF-IDF procedure

Maarten Grootendorst. “BERTopic: Neural topic modeling with a class-based TF-IDF procedure”. In:arXiv preprint arXiv:2203.05794(2022)

work page internal anchor Pith review Pith/arXiv arXiv 2022
[2]

UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction

Leland McInnes, John Healy, and James Melville. “UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction”. In:arXiv preprint arXiv:1802.03426(2018).URL:https://arxiv.org/abs/1802.03426

work page internal anchor Pith review Pith/arXiv arXiv 2018
[3]

The Mean-Difference: A Simple and Effective Method for Zero-Shot Classification

Eric Wallace et al. “The Mean-Difference: A Simple and Effective Method for Zero-Shot Classification”. In:arXiv preprint arXiv:2403.14859(2024).URL: https://arxiv.org/abs/2403.14859

work page arXiv 2024
[4]

MTEB: Massive Text Embedding Benchmark

Niklas Muennighoff et al. “MTEB: Massive Text Embedding Benchmark”. In: arXiv preprint arXiv:2210.07316(2023).URL: https : / / arxiv . org / abs / 2210.07316

work page internal anchor Pith review arXiv 2023
[5]

Hugging Face.URL: https://huggingface.co/ spaces/mteb/leaderboard(visited on 03/07/2026)

MTEB.MTEB Leaderboard. Hugging Face.URL: https://huggingface.co/ spaces/mteb/leaderboard(visited on 03/07/2026)

work page 2026
[6]

Estimating the intrinsic dimension of datasets by a minimal neighborhood information

Elena Facco et al. “Estimating the intrinsic dimension of datasets by a minimal neighborhood information”. In:Scientific Reports7.1 (2017).ISSN: 2045-2322. DOI: 10.1038/s41598- 017- 11873- y.URL: http://dx.doi.org/10. 1038/s41598-017-11873-y. 13

work page doi:10.1038/s41598- 2017
[7]

Deep one-class classification

Lukas Ruff et al. “Deep one-class classification”. In:International conference on machine learning. PMLR. 2018, pp. 4393–4402

work page 2018
[8]

Scan: a structural clustering algorithm for networks

Xiaowei Xu et al. “Scan: a structural clustering algorithm for networks”. In:Pro- ceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining. 2007, pp. 824–833. 14

work page 2007