pith. sign in

arxiv: 1907.10943 · v1 · pith:BBBPM7EInew · submitted 2019-07-25 · 💻 cs.IR

Modelling Dynamic Interactions between Relevance Dimensions

Sagar Uprety , Shahram Dehdashti , Lauren Fell , Peter Bruza , Dawei Song This is my paper

Pith reviewed 2026-05-24 16:09 UTC · model grok-4.3

classification 💻 cs.IR
keywords relevance dimensionsquantum theoryvector space modelcognitive stateinterferenceincompatibilityinformation retrievaluser study
0
0 comments X

The pith

A quantum-inspired complex vector space model explains incompatibility and interference between relevance dimensions in user judgments.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that relevance is not a simple sum of independent dimensions but involves dynamic interactions that can be modeled using the mathematics of quantum theory. The authors conduct a user study modeled after a famous quantum experiment to gather data on how people judge documents across multiple criteria like topical relevance and credibility. They then build a complex-valued vector space representing the user's cognitive state to account for observed effects where dimensions interfere rather than combine linearly. If correct, this would mean information retrieval systems need to consider these cognitive interactions to better match how users actually decide what is relevant.

Core claim

The paper claims that by constructing a cognitive analogue of a quantum experiment in a user study, the resulting data supports a complex-valued vector space model of the cognitive state that explains the incompatibility between relevance dimensions and the interference effects arising from their interaction during the inference of relevance.

What carries the argument

The complex-valued vector space model of the user's cognitive state, which uses quantum theory to represent superposition and interference between relevance dimensions.

Load-bearing premise

That the vector space model built from the user study data truly reflects the underlying cognitive interactions between relevance dimensions instead of merely fitting the collected response data.

What would settle it

Conducting a follow-up experiment where users make sequential judgments on relevance dimensions and checking if the predicted interference effects from the model match the observed changes in judgment probabilities.

Figures

Figures reproduced from arXiv: 1907.10943 by Dawei Song, Lauren Fell, Peter Bruza, Sagar Uprety, Shahram Dehdashti.

Figure 1
Figure 1. Figure 1: Stern-Gerlach Experiment 4 COGNITIVE ANALOGUE OF THE S-G EXPERIMENT 4.1 Experiment Description in the Context of Relevance The cognitive analogue to the S-G experiment was originally dis￾cussed in [15]. In order to draw an analogy of the electron spin states in terms of human judgements, we consider the two-valued spin data to be equivalent to the yes/no answer data. The measure￾ment along the different ax… view at source ↗
Figure 2
Figure 2. Figure 2: Snapshot of a document Parameter Query 1 Query 2 Query 3 P(T+) 0.7622 0.6736 0.8993 P(U +,T+) 0.4405 0.5416 0.8724 P(R+,T+) 0.4609 0.4857 0.5616 P(R+,U +,T+) 0.2587 0.4513 0.6442 P(R+,U −,T+) 0.1188 0.0694 0.0000 P(U +, R+,T+) 0.2765 0.4285 0.5410 P(U +, R−,T+) 0.1560 0.0857 0.2739 t 2 0.7622 0.6736 0.8993 u 2 0.5779 0.8041 0.9701 r 2 0.5462 0.7311 0.6456 θr 80.62 deg 56.79 deg 51.43 deg [PITH_FULL_IMAGE:… view at source ↗
read the original abstract

Relevance is an underlying concept in the field of Information Science and Retrieval. It is a cognitive notion consisting of several different criteria or dimensions. Theoretical models of relevance allude to interdependence between these dimensions, where their interaction and fusion leads to the final inference of relevance. We study the interaction between the relevance dimensions using the mathematical framework of Quantum Theory. It is considered a generalised framework to model decision making under uncertainty, involving multiple perspectives and influenced by context. Specifically, we conduct a user study by constructing the cognitive analogue of a famous experiment in Quantum Physics. The data is used to construct a complex-valued vector space model of the user's cognitive state, which is used to explain incompatibility and interference between relevance dimensions. The implications of our findings to inform the design of Information Retrieval systems are also discussed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper claims that relevance dimensions interact dynamically and can be modeled using the quantum theory framework. It constructs a cognitive analogue of a quantum experiment via a user study, maps the resulting response data to a complex-valued vector space representing the user's cognitive state, and uses this model to demonstrate incompatibility and interference effects between relevance dimensions, with discussion of implications for IR system design.

Significance. If the mapping from user responses to the complex vector space is shown to be non-circular and predictive rather than post-hoc, the work would supply a concrete, falsifiable quantum-inspired model for context-dependent relevance judgment. This could strengthen theoretical accounts of relevance fusion and suggest new evaluation metrics or ranking features that explicitly incorporate interference terms.

major comments (2)
  1. [Abstract] Abstract: the central claim that the constructed complex vector space 'explains' incompatibility and interference rests on an unspecified mapping from user-study responses to amplitudes and phases; without the explicit construction (e.g., how the Hilbert-space basis is chosen or how the interference term is computed from the data), it is impossible to assess whether the model adds explanatory power beyond a classical multivariate fit.
  2. [Abstract] Abstract: the user-study design is described only as 'the cognitive analogue of a famous experiment in Quantum Physics'; the precise experimental protocol, choice of relevance dimensions, and statistical tests used to detect interference are not stated, making it impossible to evaluate whether the observed effects are robust or could be reproduced under a classical probability model.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the comments. Both major comments correctly note that the abstract is high-level and omits explicit construction details. The full manuscript supplies these in the methods and results, but we will revise the abstract to address the concerns directly.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that the constructed complex vector space 'explains' incompatibility and interference rests on an unspecified mapping from user-study responses to amplitudes and phases; without the explicit construction (e.g., how the Hilbert-space basis is chosen or how the interference term is computed from the data), it is impossible to assess whether the model adds explanatory power beyond a classical multivariate fit.

    Authors: The abstract is concise by design. The manuscript details the mapping: user response frequencies are converted to probability amplitudes via the Born-rule analogue, with the Hilbert-space basis vectors aligned to the chosen relevance dimensions. The interference term is obtained as the signed deviation between observed joint probabilities and the classical law of total probability. We will revise the abstract to include a one-sentence description of this construction so that the explanatory claim can be evaluated from the abstract alone. revision: yes

  2. Referee: [Abstract] Abstract: the user-study design is described only as 'the cognitive analogue of a famous experiment in Quantum Physics'; the precise experimental protocol, choice of relevance dimensions, and statistical tests used to detect interference are not stated, making it impossible to evaluate whether the observed effects are robust or could be reproduced under a classical probability model.

    Authors: The abstract summarises at a high level. The full paper specifies the protocol (sequential relevance judgments mirroring the quantum double-slit setup), the two relevance dimensions employed, and the statistical comparison of observed probabilities against classical predictions. We will revise the abstract to name the dimensions and note the use of classical-probability baseline tests. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The paper reports a user study that constructs a cognitive analogue of a quantum experiment and maps the resulting response data into a complex vector space to model interference between relevance dimensions. This is an empirical construction using an externally established quantum framework rather than any self-definitional loop, fitted parameter renamed as prediction, or load-bearing self-citation. No equations or derivation steps in the abstract or described claims reduce the model to its own inputs by construction; the mapping is presented as an explanatory application of the data.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only view yields no identifiable free parameters, axioms, or invented entities; the quantum framework itself is imported from prior literature.

pith-pipeline@v0.9.0 · 5665 in / 941 out tokens · 22015 ms · 2026-05-24T16:09:12.426380+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

54 extracted references · 54 canonical work pages

  1. [1]

    Carol L Barry. 1994. User-defined relevance criteria: An exploratory study.Journal of the American Society for Information Science 45, 3 (1994), 149–159

    work page 1994

  2. [2]

    Carol L Barry and Linda Schamber. 1998. Users’ criteria for relevance evaluation: a cross-situational comparison. Information processing & management 34, 2-3 (1998), 219–236

    work page 1998

  3. [3]

    Pia Borlund. 2003. The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Inf. Res. 8 (2003)

    work page 2003

  4. [4]

    Peter Bruza and Vivien Chang. 2014. Perceptions of document relevance.Frontiers in Psychology 5 (2014), 612. https://doi.org/10.3389/fpsyg.2014.00612 ICTIR ’19, October 2–5, 2019, Santa Clara, CA, USA Sagar Uprety, Shahram Dehdashti, Lauren Fell, Peter Bruza, and Dawei Song

    work page doi:10.3389/fpsyg.2014.00612 2014

  5. [5]

    Busemeyer and Peter D

    Jerome R. Busemeyer and Peter D. Bruza. 2012. Quantum Models of Cognition and Decision (1st ed.). Cambridge University Press, New York, NY, USA

    work page 2012

  6. [6]

    Busemeyer, Emmanuel M

    Jerome R. Busemeyer, Emmanuel M. Pothos, Riccardo Franco, and Jennifer S. Trueblood. 2011. A quantum theoretical explanation for probability judgment errors. Psychological Review 118, 2 (2011), 193–218. https://doi.org/10.1037/ a0022542

    work page 2011

  7. [7]

    Jaime Carbonell and Jade Goldstein. 1998. The Use of MMR, Diversity-based Reranking for Reordering Documents and Producing Summaries. InProceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’98). ACM, New York, NY, USA, 335–336. https: //doi.org/10.1145/290941.291025

    work page doi:10.1145/290941.291025 1998

  8. [8]

    Melanie Landvad Clemmensen and Pia Borlund. 2016. Order effect in interactive information retrieval evaluation: an empirical study. Journal of Documentation 72, 2 (2016), 194–213. https://doi.org/10.1108/JD-04-2015-0051

    work page doi:10.1108/jd-04-2015-0051 2016

  9. [9]

    System Development Corporation. 1967. Experimental Studies of Relevance Judg- ments: Final Report. System Development Corporation. https://books.google.co. uk/books?id=80GRRwAACAAJ

    work page 1967

  10. [10]

    Erica Cosijn. 2009. Relevance Judgments and Measurements. In Encyclopedia of Library and Information Sciences . CRC Press, 4512–4519

    work page 2009

  11. [11]

    Erica Cosijn and Peter Ingwersen. 2000. Dimensions of relevance. Informa- tion Processing & Management 36, 4 (2000), 533 – 550. https://doi.org/10.1016/ S0306-4573(99)00072-2

    work page 2000

  12. [12]

    Célia da Costa Pereira, Mauro Dragoni, and Gabriella Pasi. 2009. Multidimensional relevance: A new aggregation criterion. In European Conference on Information Retrieval. Springer, 264–275

    work page 2009

  13. [13]

    Célia da Costa Pereira, Mauro Dragoni, and Gabriella Pasi. 2012. Multidi- mensional Relevance: Prioritized Aggregation in a Personalized Information Retrieval Setting. Inf. Process. Manage. 48, 2 (March 2012), 340–357. https: //doi.org/10.1016/j.ipm.2011.07.001

    work page doi:10.1016/j.ipm.2011.07.001 2012

  14. [14]

    Michael Eisenberg and Carol Barry. 1988. Order effects: A study of the possible influence of presentation order on user judgments of document relevance.Journal of the American Society for Information Science 39, 5 (Sept. 1988), 293–300. https: //doi.org/10.1002/(sici)1097-4571(198809)39:5<293::aid-asi1>3.0.co;2-i

    work page doi:10.1002/(sici)1097-4571(198809)39:5 1988

  15. [15]

    L. Fell, S. Dehdashti, P.D. Bruza, and C. Moreira. 2019. An Experimental Protocol to Derive and Validate a Quantum Model of Decision-Making. In Proceedings of the 41st Annual Meeting of the Cognitive Science Society (COGSCI’19)

    work page 2019

  16. [16]

    Norbert Fuhr, Anastasia Giachanou, Gregory Grefenstette, Iryna Gurevych, An- dreas Hanselowski, Kalervo Jarvelin, Rosie Jones, YiquN Liu, Josiane Mothe, Wolfgang Nejdl, Isabella Peters, and Benno Stein. 2018. An Information Nu- tritional Label for Online Documents. SIGIR Forum 51, 3 (Feb. 2018), 46–66. https://doi.org/10.1145/3190580.3190588

    work page doi:10.1145/3190580.3190588 2018

  17. [17]

    Ernesto F. Galvão. 2005. Discrete Wigner functions and quantum computational speedup. Physical Review A 71, 4 (April 2005). https://doi.org/10.1103/physreva. 71.042302

    work page doi:10.1103/physreva 2005

  18. [18]

    Robin M Hogarth and Hillel J Einhorn. 1992. Order effects in belief updating: The belief-adjustment model. Cognitive Psychology 24, 1 (jan 1992), 1–55. https: //doi.org/10.1016/0010-0285(92)90002-j

    work page doi:10.1016/0010-0285(92)90002-j 1992

  19. [19]

    Mu hsuan Huang and Hui yu Wang. 2004. The influence of document presentation order and number of documents judged on users' judgments of relevance. Journal of the American Society for Information Science and Technology 55, 11 (2004), 970–

    work page 2004

  20. [20]

    https://doi.org/10.1002/asi.20047

    work page doi:10.1002/asi.20047

  21. [21]

    Jiang and J

    J. Jiang and J. Allan. 2016. Adaptive effort for search evaluation metrics. In ECIR ’16 (2016), 187–199

    work page 2016

  22. [22]

    Jiepu Jiang, Daqing He, and James Allan. 2017. Comparing In Situ and Multidi- mensional Relevance Judgments. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR '17. ACM Press. https://doi.org/10.1145/3077136.3080840

    work page doi:10.1145/3077136.3080840 2017

  23. [23]

    Jingfei Li, Peng Zhang, Dawei Song, and Yue Wu. 2017. Understanding an enriched multidimensional user relevance model by analyzing query logs.Journal of the Association for Information Science and Technology 68, 12 (2017), 2743–2754. https://doi.org/10.1002/asi.23868

    work page doi:10.1002/asi.23868 2017

  24. [24]

    Nilan and Patricia T

    Michael S. Nilan and Patricia T. Fletcher. 1987. INFORMATION BEHAVIORS IN THE PREPARATION OF RESEARCH PROPOSALS: A USER STUDY.. InPro- ceedings of the ASIS Annual Meeting , Ching chih Chen (Ed.), Vol. 24. Learned Information Inc, 186–192

    work page 1987

  25. [25]

    M. S. Nilan, R. P. Peek, and H. W. Snyder. 1988. A Methodology for Tapping User Evaluation Behaviors: An Exploration of Users'Strategy, Source and Information Evaluating. In Proceedings of the American Society for Information Science (ASIS) 51st Annual Meeting, C. L. Borgman and E. Y. H. Pai (Eds.). Learned Information, Medford, NJ, 152–159

    work page 1988

  26. [26]

    Olteanu, S

    A. Olteanu, S. Peshterliev, X. Liu, and K. Aberer. 2013. Web credibility: Features exploration and credibility prediction. In ECIR ’ 13 (2013), 557–568

    work page 2013

  27. [27]

    Joao Palotti, Lorraine Goeuriot, Guido Zuccon, and Allan Hanbury. 2016. Rank- ing Health Web Pages with Relevance and Understandability. InProceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’16) . ACM, New York, NY, USA, 965–968. https: //doi.org/10.1145/2911451.2914741

    work page doi:10.1145/2911451.2914741 2016

  28. [28]

    Taemin Kim Park. 1993. The nature of relevance in information retrieval: An empirical study. The library quarterly 63, 3 (1993), 318–351

    work page 1993

  29. [29]

    E. M. Pothos and J. R. Busemeyer. 2009. A quantum probability explanation for violations of 'rational' decision theory. Proceedings of the Royal Society B: Biological Sciences 276, 1665 (mar 2009), 2171–2178. https://doi.org/10.1098/rspb.2009.0121

    work page doi:10.1098/rspb.2009.0121 2009

  30. [30]

    J. J. Sakurai and Jim Napolitano. 2017. Modern Quantum Mechanics. Cambridge University Press

    work page 2017

  31. [31]

    Tefko Saracevic. 1996. Modeling Interaction in Information Retrieval (IR): A Review and Proposal.. In Proceedings of the ASIS annual meeting , Vol. 33. ERIC, 3–9

    work page 1996

  32. [32]

    Tefko Saracevic. 1997. The stratified model of information retrieval interaction: Extension and applications

    work page 1997

  33. [33]

    Tefko Saracevic. 2007. Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifesta- tions of relevance. Journal of the American Society for Information Science and Technology 58, 13 (2007), 1915–1933. https://doi.org/10.1002/asi.20682

    work page doi:10.1002/asi.20682 2007

  34. [34]

    Tefko Saracevic. 2016. The Notion of Relevance in Information Science: Ev- erybody knows what relevance is. But, what is it really? Synthesis Lectures on Information Concepts, Retrieval, and Services 8, 3 (Sept. 2016), i–109. https: //doi.org/10.2200/s00723ed1v01y201607icr050

    work page doi:10.2200/s00723ed1v01y201607icr050 2016

  35. [35]

    Schwarz and M

    J. Schwarz and M. Morris. 2011. Augmenting web pages and search results to support credibility assessment. In CHI ’ 11 (2011), 1245–1254

    work page 2011

  36. [36]

    Paul Slovic. 1995. The construction of preference. American Psychologist 50, 5 (1995), 364–371. https://doi.org/10.1037/0003-066x.50.5.364

    work page doi:10.1037/0003-066x.50.5.364 1995

  37. [37]

    Trueblood and Jerome R

    Jennifer S. Trueblood and Jerome R. Busemeyer. 2011. A Quantum Probability Account of Order Effects in Inference. Cognitive Science 35, 8 (sep 2011), 1518–

    work page 2011

  38. [38]

    https://doi.org/10.1111/j.1551-6709.2011.01197.x

    work page doi:10.1111/j.1551-6709.2011.01197.x 2011

  39. [39]

    A., Mathur, S., Salabert, D., Ballot, J., R´egulo, C., Metcalfe, T

    A. Tversky and D. Kahneman. 1974. Judgment under Uncertainty: Heuristics and Biases. Science 185, 4157 (Sept. 1974), 1124–1131. https://doi.org/10.1126/science. 185.4157.1124

    work page doi:10.1126/science 1974

  40. [40]

    Sagar Uprety and Dawei Song. 2018. Investigating Order Effects in Multidimen- sional Relevance Judgment Using Query Logs. In Proceedings of the 2018 ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR ’18) . ACM, New York, NY, USA, 191–194. https://doi.org/10.1145/3234944.3234972

    work page doi:10.1145/3234944.3234972 2018

  41. [41]

    Sagar Uprety, Yi Su, Dawei Song, and Jingfei Li. 2018. Modeling Multidimensional User Relevance in IR Using Vector Spaces. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR ’18) . ACM, New York, NY, USA, 993–996. https://doi.org/10.1145/3209978.3210130

    work page doi:10.1145/3209978.3210130 2018

  42. [42]

    Verma, E

    M. Verma, E. Yilmaz, and N. Craswell. 2016. On obtaining effort based judgements for information retrieval. In WSDM ’ 16 (2016), 277–286

    work page 2016

  43. [43]

    Benyou Wang, Peng Zhang, Jingfei Li, Dawei Song, Yuexian Hou, and Zhenguo Shang. 2016. Exploration of Quantum Interference in Document Relevance Judgement Discrepancy. Entropy 18, 12 (Apr 2016), 144. https://doi.org/10.3390/ e18040144

    work page 2016

  44. [44]

    Busemeyer

    Zheng Wang and Jerome R. Busemeyer. 2013. A Quantum Question Order Model Supported by Empirical Tests of anA Prioriand Precise Prediction. Topics in Cognitive Science (sep 2013). https://doi.org/10.1111/tops.12040

    work page doi:10.1111/tops.12040 2013

  45. [45]

    Wawer, R

    A. Wawer, R. Nielek, and A. Wierzbicki. 2014. Predicting webpage credibility using linguistic features. In WWW ’ 14 (2014), 1135–1140

    work page 2014

  46. [46]

    William K Wootters. 1987. A Wigner-function formulation of finite-state quantum mechanics. Annals of Physics 176, 1 (May 1987), 1–21. https://doi.org/10.1016/ 0003-4916(87)90176-x

    work page 1987

  47. [47]

    Yunjie Xu and Dong Wang. 2008. Order effect in relevance judgment. Journal of the American Society for Information Science and Technology 59, 8 (2008), 1264–

    work page 2008

  48. [48]

    https://doi.org/10.1002/asi.20826

    work page doi:10.1002/asi.20826

  49. [49]

    Yunjie (Calvin) Xu and Zhiwei Chen. 2006. Relevance judgment: What do in- formation users consider beyond topicality? Journal of the American Society for Information Science and Technology 57, 7 (2006), 961–973. https://doi.org/10.1002/ asi.20361

    work page 2006

  50. [50]

    Yamamoto and K

    Y. Yamamoto and K. Tanaka. 2011. Enhancing credibility judgment of web search results. In CHI ’ 11 (2011), 1235–1244

    work page 2011

  51. [51]

    Yilmaz, M

    E. Yilmaz, M. Verma, N. Craswell, F. Radlinski, and P. Bailey. 2014. Relevance and effort: An analysis of document utility. In CIKM ’ 14 (2014), 91–100

    work page 2014

  52. [52]

    Cheng Xiang Zhai, William W Cohen, and John Lafferty. 2003. Beyond inde- pendent relevance: methods and evaluation metrics for subtopic retrieval. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval . ACM, 10–17

    work page 2003

  53. [53]

    Yinglong Zhang, Jin Zhang, Matthew Lease, and Jacek Gwizdka. 2014. Mul- tidimensional Relevance Modeling via Psychometrics and Crowdsourcing. In Proceedings of the 37th International ACM SIGIR Conference on Research & Devel- opment in Information Retrieval (SIGIR ’14). ACM, New York, NY, USA, 435–444. https://doi.org/10.1145/2600428.2609577

    work page doi:10.1145/2600428.2609577 2014

  54. [54]

    Guido Zuccon. 2016. Understandability Biased Evaluation for Information Re- trieval. In Advances in Information Retrieval , Nicola Ferro, Fabio Crestani, Marie- Francine Moens, Josiane Mothe, Fabrizio Silvestri, Giorgio Maria Di Nunzio, Claudia Hauff, and Gianmaria Silvello (Eds.). Springer International Publishing, Cham, 280–292

    work page 2016