REVIEW 4 major objections 3 minor 12 references
TinySearch -- Semantics based Search Engine using Bert Embeddings
T0 review · 4 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A BERT-based search engine ranks documents by meaning, not keywords
desk verdict A clearly written course project whose one new-ish idea—a learned similarity head over BERT embeddings—is never actually validated, because the evaluation has no baseline, self-assigned labels, and impossible F1 scores. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the pair of BERT embeddings—768-dimensional vectors from the [CLS] token of a fine-tuned BERT model—and the similarity network: a feedforward network that concatenates the two vectors, passes them through dense layers of 1024, 256, and 64 units with ReLU activation and dropout 0.5, and outputs a sigmoid score. This network, trained on 100,000 Quora question pairs, is what converts embedding distance into a relevance judgment. The BERT server provides fixed-length semantic encodings of arbitrary text, so the system avoids averaging word vectors and lets the [CLS] token carry the whole sentence's meaning.
What would settle it
Run TinySearch on a corpus of at least 100 complex natural-language queries with human relevance judgments, and compare its top-5 precision against a BM25 baseline; if the neural similarity scores do not rank relevant documents above irrelevant ones, or BM25 matches or beats it on long queries, the claimed semantic advantage is not real.
Extended reading notes
Core claim
The paper claims that TinySearch—a pipeline of a BERT encoding server, a neural similarity network, and a small GUI—can rank documents by semantic relevance. The query and each document are mapped to 768-dimensional embeddings using the [CLS] token of a BERT model fine-tuned on MRPC; a dense neural network trained on 100,000 Quora question pairs takes the concatenation of two embeddings and outputs a score between 0 and 1. Documents are sorted by that score and the top five shown. On three test queries with self-assigned relevance labels, the reported precision and recall are 0.8 for a long complex query and 0.4 for two short queries, leading the paper to conclude that longer queries yield more meaningful results. The claimed improvement over an existing search engine rests on an observed example where keyword-based engines missed synonyms such as 'deep learning' versus 'neural networks.'
Load-bearing premise
The load-bearing premise is that a network trained to detect paraphrase similarity on Quora question pairs will score query-document relevance correctly, so that a high output means the document answers the query.
Editorial extensions
If this is right
- If the reported evaluation is representative, a semantic search engine built on BERT embeddings can rank long natural-language queries more accurately than keyword-based engines.
- Longer, more specific queries should produce higher precision and recall than short ambiguous ones, as the author concludes from the three test queries.
- The same QQP-trained similarity network can be reused as a general query-document relevance scorer without task-specific retraining.
- A working search engine can be assembled from a precomputed document-embedding index, a runtime query encoder, and a neural similarity chip.
- Synonyms and paraphrases, such as 'deep learning' and 'neural networks', are captured by the semantic representation rather than requiring explicit synonym expansion.
Reading between the lines
- The 67% validation accuracy on Quora question pairs suggests the similarity network is a weak relevance scorer; a direct test of the paper's premise would compare its scores against human relevance judgments on a held-out set of complex queries.
- Because BERT embeddings are computed offline for documents, the architecture naturally scales to larger corpora by inverting the index: only the query needs to be encoded at runtime.
- A cross-encoder that lets the query and document attend to each other jointly would likely outperform the concatenation network, since the current design compresses both texts independently before comparison.
- The author's observation that shorter queries underperform could be framed as an embedding-space phenomenon: short queries sit in a sparse region far from document clusters, so similarity scores become uninformative; this is testable by measuring embedding distances as a function of query length.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops TinySearch, a semantic search engine that computes query-document similarity using BERT sentence embeddings and a feed-forward neural network trained on the Quora Question Pairs dataset. The system ranks a fixed set of 14 documents for a user query and displays the top five. The abstract claims that TinySearch shows improvement over an existing search engine for complex queries. The evaluation reports precision, recall, and F1 for three queries using relevance labels assigned by the author, along with the neural network's validation accuracy on QQP.
Significance. If the claimed improvement were supported by the evidence, the paper would provide a useful demonstration that BERT-based semantic similarity can outperform keyword-based search engines on long natural-language queries. The system architecture is described clearly, and the author provides a demo link and a public GitHub repository. The main weakness is that the empirical evaluation cannot substantiate the central claim: there is no baseline comparison, the relevance labels are self-assigned without external validation, the number of test queries is extremely small, and the reported F1 scores are arithmetically impossible. Because the core contribution is an empirical claim, these flaws are load-bearing.
major comments (4)
- [Abstract, Section V.C] The abstract's claim of improvement over an existing search engine is never tested. Section V.C states 'there is no baseline to compare the model,' and Section VII lists the addition of BM25 as future work. No existing search engine is evaluated on the same 14-document corpus, so no quantitative or qualitative comparison is made; the claimed improvement is therefore unsupported by the paper's own evidence.
- [Section V.C] The reported precision/recall/F1 numbers are internally inconsistent. Query1 reports TP=4, FP=1, FN=1, which gives precision 0.8, recall 0.8, and F1=0.8, not the printed F1=8; Queries 2 and 3 print F1=4 instead of 0.4. Moreover, Query2's confusion matrix (TP=2, FP=3, TN=8, FN=3) sums to 16, whereas the corpus contains only 14 documents. As reported, the evaluation table cannot be used to support any conclusion about system performance.
- [Section V.C] The relevance labels, called 'gold score', are assigned by the author alone, as stated in the text: 'I have assigned ... myself'. There is no labeling protocol, no second annotator, and no external ground truth. All precision and recall values are computed against these self-assigned labels, providing no independent verification of relevance judgments.
- [Sections IV.B and V.B] The neural network is trained on the Quora Question Pairs dataset and achieves only about 67% validation accuracy on that paraphrase-detection task, yet it is applied without further validation to score query-document relevance for web-like documents. The paper provides no evidence that paraphrase similarity learned from QQP transfers to document relevance, which is the core assumption underlying the design. This transfer premise is essential to the validity of the evaluation.
minor comments (3)
- [Section V.C] The third query is mislabeled as 'Query1' in the text; it should be labeled 'Query3'.
- [Section IV.B] The word 'Dropout' is misspelled as 'Droput' in the two dropout layers in the code snippet.
- [References] Several references are URLs to blog posts and theses rather than peer-reviewed sources, and reference [2] does not appear to be cited in the body of the paper.
Circularity Check
No circular derivation: ranking is computed from external BERT embeddings and a QQP-trained network; the unsupported baseline claim is an evidence gap, not circularity.
full rationale
The paper's derivation chain is not circular. TinySearch's ranking is produced by a forward pass: query and documents are encoded by a BERT server (fine-tuned on MRPC) and scored by a feed-forward network trained on Quora question pairs, then sorted. The network's parameters are learned from an external dataset (QQP), not from the 14 evaluation documents or from the queries used in Section V; hence the ranking is not fitted to the evaluation labels. The evaluation in Section V.C uses author-assigned gold scores, but the paper explicitly states these were assigned 'without being influenced by scores of search engines or results of GUI itself,' so the precision/F1 numbers are not definitionally implied by the model. The real defect is an evidence gap: the abstract claims 'improvement over one existing search engine,' yet Section V.C admits 'there is no baseline to compare the model' and Section VII lists adding BM25 as future work, so no baseline is scored on the same corpus. That makes the headline claim unsupported, and the F1 values (e.g., 8, 4) are arithmetically impossible, but neither issue is a case of a prediction reducing by construction to an input. There are no load-bearing self-citations or uniqueness theorems invoked to force a choice. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (2)
- Gold relevance labels =
Binary relevant/not-relevant for each of 14 documents per query (3 queries)
- Neural network hyperparameters =
epochs=30, batch_size=200, dropout=0.5, dense layers 1024, 256, 64, optimizer='rmsprop'
assumptions (3)
- domain assumption BERT [CLS] embedding captures the semantic content of a document
- ad hoc to paper Semantic similarity learned from Quora question pairs transfers to query-document relevance
- ad hoc to paper Author-assigned gold labels are a valid relevance ground truth
Cite this review
Pith. "Pith review of TinySearch -- Semantics based Search Engine using Bert Embeddings." pith.science (2026). https://pith.science/paper/YP7CEIQI
@misc{pith2026190802451,
author = {Pith},
title = {Pith review of: TinySearch -- Semantics based Search Engine using Bert Embeddings},
year = {2026},
howpublished = {\url{https://pith.science/paper/YP7CEIQI}},
note = {Machine review of arXiv:1908.02451}
}
read the original abstract
Existing search engines use keyword matching or tf-idf based matching to map the query to the web-documents and rank them. They also consider other factors such as page rank, hubs-and-authority scores, knowledge graphs to make the results more meaningful. However, the existing search engines fail to capture the meaning of query when it becomes large and complex. BERT, introduced by Google in 2018, provides embeddings for words as well as sentences. In this paper, I have developed a semantics-oriented search engine using neural networks and BERT embeddings that can search for query and rank the documents in the order of the most meaningful to least meaningful. The results shows improvement over one existing search engine for complex queries for given set of documents.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
https://www.google.com/search/howsearchworks/crawling-indexing/
-
[2]
http://ad-publications.informatik.uni-freiburg.de/theses/Bachelor Eneko Pinzolas 2017.pdf
work page 2017
-
[3]
Kassim, J. M., & Rahmany, M. (2009, August). Introduction to semantic search engine. In 2009 International Conference on Electrical Engineer- ing and Informatics (V ol. 2, pp. 380-386). IEEE
work page 2009
-
[4]
https://www.en.pms.ifi.lmu.de/publications/projektarbeiten/Jaderson.Webler/ PA Jaderson.Webler.pdf
-
[5]
https://towardsdatascience.com/elmo-contextual-language-embedding- 335de2268604
-
[6]
https://github.com/hanxiao/bert-as-service
-
[7]
Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre- training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
arXiv 2018
-
[8]
https://www.kaggle.com/c/quora-question-pairs/overview
Show all 12 references
-
[9]
https://www.microsoft.com/en-us/download/details.aspx?id=52398
-
[10]
https://www.internetlivestats.com/total-number-of-websites/
-
[11]
Demo : https://www.youtube.com/watch?v=wqx0EQX3XM4
-
[12]
Github: https://github.com/manishpatel005/tinysearch
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.