Pith. sign in

REVIEW 3 major objections 5 minor 30 references

Replacing the denominator in standard attention with an Lp norm lets repeated evidence show up in the representation, improving recovery of accumulated counts across five learning domains.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 05:34 UTC pith:HXLERHJG

load-bearing objection Modest but real: a parameter-free Lp attention normalization with an honest evaluation, though the headline recovery claim is partly p-selection and partly an amplitude that downstream normalization can erase. the 3 major comments →

arxiv 2607.22781 v1 pith:HXLERHJG submitted 2026-07-24 cs.LG cs.AI

What Softmax Throws Away: Mass-Aware Attention for Evidence Accumulation

Minwoo Yu , Young-guk Ha This is my paper
classification cs.LG cs.AI
keywords Mass-Aware Attentionattention normalizationrepresentation probingevidence accumulationtemporal graph learningrepetition invarianceLp normlayer normalization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that standard softmax attention's weighted-average denominator (L1 norm) is repetition-invariant: repeating an evidence pattern multiplies numerator and denominator equally, so the aggregate cannot tell 'one strong piece of evidence' from 'many copies of it.' The authors propose Mass-Aware Attention (MAA), which swaps only the denominator for an Lp norm (p>1), making the output scale as r^{1-1/p} under r repetitions. Across four temporal-graph models and three datasets, MAA improves recovery of graph statistics—especially preferential attachment—from the predictor-facing representation, and improves future-link AUC in 11 of 12 model-dataset cells. The same normalization principle yields recovery gains in marked point processes, temporal knowledge graphs, retrieval-augmented generation, and spatio-temporal point processes, with the caveat that a downstream LayerNorm can erase the amplitude signal.

Core claim

MAA claims that replacing only the denominator of standard attention, ||m||_1, with ||m||_p for p>1 makes the representation magnitude encode the effective number of contributing inputs, so evidence-accumulation information that standard attention throws away becomes linearly recoverable from the same hidden representation. The paper shows this through the identity A_p(X) = g_p(α) A_1(X), where g_p(α)=1/||α||_p is an input-dependent scale between 1 and n^{1-1/p}, and establishes empirically that linear recovery of preferential attachment improves in all 12 CTDG cells, candidate-mark count recovery improves by 25% in MTPP, and retained-passage recovery AUC jumps by 35% in RAG, while task util

What carries the argument

Lp normalization of the attention denominator. Standard softmax divides by the L1 norm of the weight vector; MAA divides by the Lp norm, producing a scale g_p(α)=1/||α||_p that grows with the effective support (equivalently, Rényi entropy) of the attention distribution. Under r repetitions of a weighted-value motif, MAA output scales as r^{1-1/p}, making the amplitude of the aggregate a continuous, parameter-free readout of accumulated evidence.

Load-bearing premise

MAA's representational signal lives in the amplitude of the attention output, so any scale-removing normalization (such as LayerNorm) placed between the modified attention and the predictor-facing representation erases the benefit; the paper's own STPP trace shows recovery turning negative after such normalization.

What would settle it

Take one CTDG model (e.g., TGAT) with MAA p=1.1 and insert a LayerNorm immediately after the attention aggregation that forms the predictor-facing representation, keeping everything else identical; the paper's mechanism predicts linear recovery of preferential attachment (mean +0.091 on the original path) should drop to near zero or negative while AUC stays similar. A second test: train the same baseline with p=1 but append the observed pair count as an auxiliary input feature; if recovery R2 rises by the same amount as MAA's gain, the effect reduces to explicit count injection rather than a s

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Future-link AUC improves in 11 of 12 CTDG model-dataset cells, and preferential-attachment recovery improves in all 12 cells after family-wise correction.
  • Linear recovery of graph statistics from the predictor-facing representation increases by 4.49% on average, without adding parameters, hidden dimensions, or explicit count features.
  • The same denominator change improves recovery of candidate-mark counts in MTPP, pair recurrence in TKG, retained-passage condition in RAG, and local event counts in STPP.
  • MAA's effect is not a uniform scale increase: removing the output norm attenuates but does not eliminate gains, and fixed powers p=1.1–1.2 mildly relaxing exact repetition invariance give the consistent improvements.
  • MAA recovers standard attention at p=1, so the method exposes the repetition invariance of standard attention as a continuous tunable axis rather than a binary architectural change.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If MAA is correct, representation auditing can be done without changing task heads: count-like statistics that the attention weights encode become accessible to linear probes, giving a cheap diagnostic for whether a model is counting evidence versus merely ranking it.
  • The LayerNorm sensitivity implies that many existing Transformer stacks with post-attention normalization are structurally blind to accumulated-evidence amplitude, so MAA's benefit in such architectures would require moving normalization before attention or using a scale-preserving variant—a testable architectural prescription.
  • MAA's power-selection rule could be transferred as a regularizer for tasks where evidence accumulation matters (e.g., multi-hop QA with many supporting passages), though the paper only sketches this; one could test whether p>1 improves answer selection when more passages are present.
  • The RAG result, where recovery rises but diagnostic AUC stays flat, suggests accessible information does not automatically flow to a task head; a natural next step is to train a cheap linear readout on the MAA representation and check whether the recovered evidence condition is actually usable for downstream decision rules.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper identifies a form of repetition invariance in standard softmax attention: because the numerator and L1 denominator grow at the same rate when an evidence pattern is repeated, different amounts of accumulated evidence can produce the same aggregate output. It proposes Mass-Aware Attention (MAA), which replaces the L1 denominator with an Lp norm. For p>1 the numerator and denominator scale at different rates under repetition, so the output magnitude retains an input-dependent factor gp(α)=1/‖α‖p (Eq. 4). The paper proves the core identity Ap(X^(r)) = r^{1−1/p}Ap(X) (Eq. 6), evaluates the method across CTDG, MTPP, TKG, RAG, and STPP, and reports improved recovery of count-like statistics from predictor-facing representations, with some task utility gains and an explicit demonstration that downstream LayerNorm can erase the signal.

Significance. If the central claim holds, MAA is an attractive, minimal intervention: it adds no parameters, no auxiliary supervision, and no hidden dimensions, and it is mathematically well founded. The paper ships code and data links, uses multi-model, multi-seed CTDG experiments with Holm-corrected inference, and is unusually candid about limitations, including the STPP LayerNorm boundary and the fact that RAG information access does not translate into task-head gains. The strongest empirical evidence is preferential-attachment recovery in CTDG (12/12 cells, pHolm=0.003906), which is a meaningful demonstration that amplitude-based evidence accumulation can be made linearly recoverable. However, because the proposed mechanism is carried by representation amplitude, the positive results in the other domains depend on whether the attention output actually reaches the probed representation without scale-removing normalization. This is not just a technical caveat: the paper itself shows that post-attention LayerNorm completely removes the effect in STPP. That issue determines whether the headline 'predictor-facing representation' claim is valid, and it is not yet resolved for the main CTDG results

major comments (3)
  1. [§6.4, Fig. 7, and §A.2] The mechanism's signal is the amplitude gp(α) from Eq. (4). The paper shows in STPP that this signal is +0.03165 at the attention output, but becomes -0.00349 after LayerNorm and -0.01467 at the predictor-facing Z. For CTDG, MTPP, TKG, and RAG, the positive recovery results assume that the modified attention boundary is the immediate source of Z, yet the paper never verifies that no LayerNorm, RMSNorm, or L2 normalization exists between that boundary and Z in the released implementations. Section A.2 only states where MAA is applied; it does not document the normalization path. Section 7.2's L2-normalization ablation shows amplitude is a principal transmission path, but that is a matched control, not an audit of the actual architectures. The central claim requires either an explicit per-model statement that no amplitude-removing operation follows the MAA boundary, or pre-norm variants fo
  2. [Table 3 and §5.3] The aggregate CTDG recovery endpoint 'shared linear R2' improves by 4.49%, but the Holm-corrected p-value is 0.137; 'beyond-posthoc linear R2' and 'context margin' also fail to reach significance after correction (pHolm=0.157 and 0.134, respectively). The confirmatory representation claim therefore rests almost entirely on preferential-attachment recovery. This is legitimate, but the abstract's statement that 'linear recovery from the same hidden representation increases by 4.49%' overstates the evidence unless PA is pre-specified as the primary recovery endpoint. The paper should either make that pre-specification explicit or temper the headline to reflect that the family-wise-significant recovery effect is statistic-specific.
  3. [Algorithm 2 and §4.2] Although the test split is held out, p is selected per domain using the recovery endpoint as the selection objective. For CTDG the leave-one-model-out design mitigates per-cell selection, but p=1.1 was still chosen as the power that maximizes recovery on the selection folds. For RAG, MTPP, and TKG, the reported recovery gains are the result of a domain-specific p choice. This is standard validation, but it means the reported gains are partly the output of an optimization over p, not a fixed-operator comparison. The paper should report the main recovery results for a default p (e.g., p=1.1) in addition to the selected p, or show that the conclusions survive a sensitivity analysis across the candidate grid in each domain rather than only in the four representative CTDG conditions of Table 8.
minor comments (5)
  1. [Fig. 3 and §4.1] The text says the beyond-posthoc comparison supplies 'the same type of auxiliary coordinate c' to both probes, but c is never precisely defined. If c is gp(α), then for the p=1 baseline it is identically 1 and the control probe receives no information, making the comparison asymmetric. Please define c explicitly (e.g., c = ||m||p or c = gp(α)) and state what value it takes for the baseline.
  2. [§3.2 and Algorithm 1] The paper calls MAA 'parameter-free' while p is a hyperparameter selected per domain. This is not an error, but it should be stated more carefully to avoid confusion with hyperparameter-free methods. The term 'no learned parameters' is clearer.
  3. [§4.3] The crossed hierarchical bootstrap is described as resampling models and datasets independently, but the implementation details (number of resamples per model-dataset pair, whether model and dataset are crossed) are not specified. A short algorithmic description would aid reproducibility.
  4. [Appendix A.2] For CTDG, the paper says MAA is applied to 'CRAFT's target-aware cross-attention,' 'TGAT's query-to-neighbor attention,' etc., but it does not say whether each of these boundaries is the exact input to the final predictor. A one-line diagram or table listing attention boundary → subsequent normalization → Z for each model would remove ambiguity.
  5. [Table 7] The RAG diagnostic AUC change of -0.73% has p=1.0 with 5 runs. Reporting a two-sided p=1.0 is uninformative; please also report the number of positive/negative runs and the effect-size confidence interval.

Circularity Check

0 steps flagged

No significant circularity: the Lp-normalization derivation is explicit algebra and the reported recovery gains are measured on held-out splits.

full rationale

The paper's derivation chain is self-contained. Equation (1) defines standard attention, Equation (3) introduces the Lp denominator, and Equation (6) is direct algebra under replication; no step defines one quantity in terms of the result it is supposed to predict. Algorithm 2 does select p using the recovery endpoint on a selection split, but the paper evaluates final recovery on held-out conditions—leave-one-model-out in CTDG, separate pilots in MTPP/TKG, an independent validation split in RAG—so the reported recovery is an empirical outcome, not the selection objective reported as a prediction. There are no self-citations, so no self-citation chain is load-bearing. The STPP boundary in Section 6.4 is an explicitly acknowledged architectural limitation: the paper reports that post-attention LayerNorm erases the signal and that a matched pre-norm architecture is required. This weakens cross-domain generalization and is a correctness/assumption concern, but it is not circularity. The decomposition Ap = gp(alpha) A1 in Equation (4) is by construction, but the paper uses it as a design statement rather than as evidence for its empirical claims. The central claim retains independent empirical content and can fail (e.g., ExtraTrees density decreases, ICEWS18 MRR slightly drops).

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central claim rests on the Lp norm replacement and the empirical transfer of the resulting amplitude through architectures. The main free parameter is p, chosen per domain on validation/recovery data; no new physical or architectural entities are postulated.

free parameters (5)
  • p (CTDG) = 1.1
    Selected by leave-one-model-out on three models; all four folds chose 1.1. The recovery endpoint is the selection objective, so CTDG improvements partly reflect choosing p for recovery.
  • p (MTPP) = 1.3
    Fixed from an earlier pilot before new endpoints; selection protocol not identical to CTDG (Section 6.1).
  • p (TKG) = 1.2 (ICEWS14), 1.02 (ICEWS18)
    Chosen on ICEWS14/ICEWS18 validation pilots before test evaluation (Section 6.2).
  • p (RAG) = selected by task-preserving recovery rule on validation split
    Algorithm 2 on an independent validation split; the recovery improvement is the selection objective (Section 6.3).
  • p (STPP) = 1.05
    Evaluated in a matched pre-norm architecture after diagnosing the post-norm boundary; the original architecture does not show gain at Z (Section 6.4).
axioms (4)
  • standard math Lp norm monotonicity: for a probability vector alpha, ||alpha||_p <= ||alpha||_1 = 1, so gp = 1/||alpha||_p satisfies 1 <= gp <= n^{1-1/p}.
    Used for Eq (5) and the claim that scale grows with attention spread.
  • domain assumption The attention boundary of the form sum_i alpha_i v_i supplies the predictor-facing representation, and its amplitude is not removed by downstream normalization in the target models.
    Needed for MAA's scale to reach Z; contradicted in the original STPP architecture (Section 6.4).
  • domain assumption Linear probes on frozen representations measure information accessibility relevant to the predictor.
    Probing methodology; the authors acknowledge probe-capacity caveats (Section 2.2).
  • domain assumption The replication thought experiment X^{(r)} (r copies of a weighted-value motif) represents how evidence accumulation appears in real attention inputs.
    Eq (2) is described as a 'mathematical thought experiment'; actual softmax inputs are not literal duplicates (Section 3.1).

pith-pipeline@v1.3.0-alltime-deepseek · 15278 in / 12081 out tokens · 119959 ms · 2026-08-01T05:34:26.169933+00:00 · methodology

0 comments
read the original abstract

High task performance does not show whether a model retains prediction-relevant structural information in its internal representation. Temporal graph models, for example, can achieve high future-link AUC while basic graph statistics remain difficult to recover from the same representation. We identify one source of this gap in the weighted averaging used by standard attention: when an evidence pattern is repeated, the numerator and denominator grow at the same rate, so inputs with different amounts of accumulated evidence can produce the same aggregate. We propose Mass-Aware Attention (MAA), which generalizes standard L1 normalization to an Lp family. Under repetition, MAA makes the numerator and denominator scale at different rates, retaining the effective number of contributing inputs in the representation magnitude. It adds no supervision, parameters, hidden dimensions, or explicit count features, and recovers standard attention at p=1. Across four continuous-time dynamic graph models and three datasets, MAA improves future-link AUC in 11 of 12 model-dataset cells. Linear recovery from the same hidden representation increases by 4.49% on average, and preferential-attachment recovery improves in all 12 cells after family-wise correction. We also observe consistent evidence in marked temporal point processes, temporal knowledge graphs, retrieval-augmented generation, and spatio-temporal point processes. Information accessibility and task utility remain distinct: NLL improves in MTPP, ranking is largely preserved in TKG, additional information in RAG does not improve the diagnostic head, and downstream LayerNorm can erase the signal in STPP. These results position MAA as a general normalization principle for improving predictor-facing representation informativeness by controlling repetition invariance in standard attention.

Figures

Figures reproduced from arXiv: 2607.22781 by Minwoo Yu, Young-guk Ha.

Figure 1
Figure 1. Figure 1: Main CTDG results. Linear recovery and context sensitivity increase on average. Confirmatory evidence is strongest for preferential-attachment [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Intuition behind Mass-Aware Attention. Standard attention can map repetitions of the same weighted-value pattern to an identical aggregate. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: 2 × 2 audit separating changes in the hidden representation Z learned by MAAfrom gains obtained by exposing a normalization-derived coordinate to the probe. The shared comparison directly contrasts ZB and ZMAA, while the post-hoc comparison supplies the same type of coordinate c to both sides. The four conditions in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Changes in predictor-facing representation recovery across domains. Each value is an equally weighted average over the model–dataset [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Relationship between representation accessibility and task utility. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Cross-domain boundaries. In RAG, increased information accessibility is separated from task utilization. In STPP, post-attention LayerNorm [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Fixed-power sensitivity. Mild normalization relaxation improves recovery and context margin while maintaining or improving AUC. Larger [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Mechanism validation. Removing the L2 norm of the MAAoutput attenuates, but does not eliminate, the effect. Input-dependent amplitude is a principal signal path, accompanied by a smaller change in the learned hidden direction. CRAFT, AttNHP, and xERTE. MAAinstead reflects the effective support of the learned attention distribution without a separate count, producing a more stable representation– task trade… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 3 linked inside Pith

  1. [1]

    Towards better evaluation for dynamic link prediction,

    F. Poursafaei, S. Huang, K. Pelrine, and R. Rabbany, “Towards better evaluation for dynamic link prediction,” inAdvances in Neural Information Processing Systems, vol. 35, 2022, datasets and Benchmarks Track

    2022

  2. [2]

    On the power of heuristics in temporal graphs,

    F. Cornell, O. Smirnov, G. Z. Gandler, and L. Cao, “On the power of heuristics in temporal graphs,” inProceedings on “I Can’t Believe It’s Not Better: Challenges in Applied Deep Learning” at ICLR 2025 Workshops, ser. Proceedings of Machine Learning Research, vol. 296. PMLR, 2025, pp. 37–46

    2025

  3. [3]

    What do temporal graph learning models learn?

    A. J. Hayes, T. Schumacher, and M. Strohmaier, “What do temporal graph learning models learn?”arXiv preprint arXiv:2510.09416, 2025. 11

    Pith/arXiv arXiv 2025

  4. [4]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” inAdvances in Neural Information Processing Systems, vol. 30, 2017

    2017

  5. [5]

    Inductive representation learning on temporal graphs,

    D. Xu, C. Ruan, E. Korpeoglu, S. Kumar, and K. Achan, “Inductive representation learning on temporal graphs,” inInternational Conference on Learning Representations, 2020

    2020

  6. [6]

    Do we really need complicated model architectures for temporal networks?

    W. Cong, S. Zhang, J. Kang, B. Yuan, H. Wu, X. Zhou, H. Tong, and M. Mahdavi, “Do we really need complicated model architectures for temporal networks?” inInternational Conference on Learning Representations, 2023

    2023

  7. [7]

    Towards better dynamic graph learning: New architecture and unified library,

    L. Yu, L. Sun, B. Du, and W. Lv, “Towards better dynamic graph learning: New architecture and unified library,” inAdvances in Neural Information Processing Systems, vol. 36, 2023

    2023

  8. [8]

    Future link prediction without memory or aggregation,

    L. Yi, R. Lei, F. Mo, Y. Zheng, Z. Wei, and Y. Ye, “Future link prediction without memory or aggregation,” inAdvances in Neural Information Processing Systems, vol. 38, 2025

    2025

  9. [9]

    Understanding intermediate layers using linear classifier probes,

    G. Alain and Y. Bengio, “Understanding intermediate layers using linear classifier probes,” 2017. [Online]. Available: https://openreview.net/forum?id=ryF7rTqgl

    2017

  10. [10]

    What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties,

    A. Conneau, G. Kruszewski, G. Lample, L. Barrault, and M. Baroni, “What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties,” inProceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018, pp. 2126–2136

    2018

  11. [11]

    Designing and interpreting probes with control tasks,

    J. Hewitt and P . Liang, “Designing and interpreting probes with control tasks,” inProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019, pp. 2733–2743

    2019

  12. [12]

    Classifier probes may just learn from linear context features,

    J. Kunz and M. Kuhlmann, “Classifier probes may just learn from linear context features,” inProceedings of the 28th International Conference on Computational Linguistics, 2020, pp. 5136–5146

    2020

  13. [13]

    Improving attention mechanism in graph neural networks via cardinality preservation,

    S. Zhang and L. Xie, “Improving attention mechanism in graph neural networks via cardinality preservation,” inInternational Joint Conference on Artificial Intelligence (IJCAI), 2020, pp. 1395–1402

    2020

  14. [14]

    Normalized attention without probability cage,

    O. Richter and R. Wattenhofer, “Normalized attention without probability cage,”arXiv preprint arXiv:2005.09561, 2020

    Pith/arXiv arXiv 2005

  15. [15]

    Affine-scaled attention: Towards flexible and stable transformer attention,

    J. Bae, B. Park, G. Park, M. Kim, J. Lee, J. Yoo, S. Woo, J. Ryu, S. J. Kwon, and D. Lee, “Affine-scaled attention: Towards flexible and stable transformer attention,”arXiv preprint arXiv:2602.23057, 2026

    arXiv 2026

  16. [16]

    Recurrent marked temporal point processes: Embedding event history to vector,

    N. Du, H. Dai, R. Trivedi, U. Upadhyay, M. Gomez-Rodriguez, and L. Song, “Recurrent marked temporal point processes: Embedding event history to vector,” inProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 1555–1564

    2016

  17. [17]

    The neural hawkes process: A neurally self-modulating multivariate point process,

    H. Mei and J. Eisner, “The neural hawkes process: A neurally self-modulating multivariate point process,” inAdvances in Neural Information Processing Systems, vol. 30, 2017

    2017

  18. [18]

    Transformer hawkes process,

    S. Zuo, H. Jiang, Z. Li, T. Zhao, and H. Zha, “Transformer hawkes process,” inProceedings of the 37th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 119, 2020, pp. 11 692–11 702

    2020

  19. [19]

    Transformer embeddings of irregularly spaced events and their participants,

    C. Yang, H. Mei, and J. Eisner, “Transformer embeddings of irregularly spaced events and their participants,” inInternational Conference on Learning Representations, 2022

    2022

  20. [20]

    Recurrent event network: Au- toregressive structure inference over temporal knowledge graphs,

    W. Jin, M. Qu, X. Jin, and X. Ren, “Recurrent event network: Au- toregressive structure inference over temporal knowledge graphs,” inProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020, pp. 6669–6683

    2020

  21. [21]

    Explainable subgraph reasoning for forecasting on temporal knowledge graphs,

    Z. Han, P . Chen, Y. Ma, and V . Tresp, “Explainable subgraph reasoning for forecasting on temporal knowledge graphs,” in International Conference on Learning Representations, 2021

    2021

  22. [22]

    Learning from history: Modeling temporal knowledge graphs with sequential copy-generation networks,

    C. Zhu, M. Chen, C. Fan, G. Cheng, and Y. Zhang, “Learning from history: Modeling temporal knowledge graphs with sequential copy-generation networks,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 5, 2021, pp. 4732–4740

    2021

  23. [23]

    HotpotQA: A dataset for diverse, explainable multi-hop question answering,

    Z. Yang, P . Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning, “HotpotQA: A dataset for diverse, explainable multi-hop question answering,” inProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 2369– 2380

    2018

  24. [24]

    Leveraging passage retrieval with generative models for open domain question answering,

    G. Izacard and E. Grave, “Leveraging passage retrieval with generative models for open domain question answering,” in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, 2021, pp. 874–880

    2021

  25. [25]

    Neural spatio-temporal point processes,

    R. T. Q. Chen, B. Amos, and M. Nickel, “Neural spatio-temporal point processes,” inInternational Conference on Learning Representa- tions, 2021

    2021

  26. [26]

    Neural point process for learning spatiotemporal event dynamics,

    Z. Zhou, X. Yang, R. Rossi, H. Zhao, and R. Yu, “Neural point process for learning spatiotemporal event dynamics,” inProceedings of the 4th Annual Learning for Dynamics and Control Conference, ser. Proceedings of Machine Learning Research, vol. 168, 2022, pp. 777–789

    2022

  27. [27]

    Layer normalization,

    J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,”arXiv preprint arXiv:1607.06450, 2016

    Pith/arXiv arXiv 2016

  28. [28]

    On layer normalization in the transformer architecture,

    R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu, “On layer normalization in the transformer architecture,” inProceedings of the 37th International Conference on Machine Learning, ser. Proceedings of Machine Learn- ing Research, vol. 119, 2020, pp. 10 524–10 533

    2020

  29. [29]

    Predicting dynamic embed- ding trajectory in temporal interaction networks,

    S. Kumar, X. Zhang, and J. Leskovec, “Predicting dynamic embed- ding trajectory in temporal interaction networks,” inProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019, pp. 1269–1278

    2019

  30. [30]

    Emergence of scaling in random networks,

    A.-L. Barab ´asi and R. Albert, “Emergence of scaling in random networks,”Science, vol. 286, no. 5439, pp. 509–512, 1999. APPENDIXA DEFINITIONS ANDAPPLICATIONDETAILS This appendix specifies the temporal scope of the recovery targets and the attention boundaries at which MAAis applied. A.1 Recovery Targets All recovery targets are computed solely from obse...

    1999