{"id":"efcae0c1-d3cd-45e5-a9ee-4ecbd4653cf2","arxiv_id":"1908.03109","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FAIRY ranks graph paths connecting a user's own social media actions to a feed item as explanations, and predicts which paths users find relevant or surprising better than three relationship-mining baselines.","lead":"FAIRY is a user-side framework that explains why items appear in a person's social media feed by finding paths between the user's own actions and the feed item in a graph of visible interactions. It ranks these explanation paths for relevance and surprise using learning-to-rank, and tests them with small user studies on Quora and Last.fm.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The train/test split in §5 likely leaks through shared paths and shared (u,f) queries, so the reported ranking accuracy gains may not survive a clean, query-disjoint evaluation.","rationale":"The paper's central claim has two parts: that paths in a user-visible interaction graph are pertinent explanations for feed items, and that a simple LTR model can rank such paths by user-judged relevance and surprisal. The first part is explicitly framed as a user-side proxy, not as reverse-engineering the platform's true feed algorithm (Section 1: 'a proxy that the users could find plausible'), so the hidden-signal objection, while real, is not the most load-bearing threat to the paper's stated contribution. The decisive support for the ranking claim is the accuracy in Table 2, and that support depends entirely on the validity of the train/test split. With each path appearing in roughly 1.9 pairs and each (u,f) query generating many pairs, a random pair split makes train/test leakage almost certain: the same path feature vector and the same query context appear on both sides of the split. This would let the supervised LTR model exploit within-query feature correlations and path-level memorization, whereas the unsupervised baselines would not benefit equally, so the reported significant gains could be an artifact. The manuscript does not describe any grouped split, and the judgment data are not released, so the concern cannot be resolved from the paper alone. The availability of code is a positive, but code alone does not fix the missing split specification. A query-disjoint, path-disjoint re-evaluation is a single concrete check that would settle the matter: if accuracy survives, the central claim is supported; if it collapses, the paper's main evidence fails. Until that check is run, the CONDITIONAL verdict remains appropriate.","tokens_in":15535,"tokens_out":4698,"duration_ms":55291,"concrete_test":"Re-run the LTR evaluation of §5 with a strict split that places all pairs for the same (u,f) query in the same fold and also ensures no path occurs in both training and test (e.g., assign paths to folds, then include all pairs whose two paths are both in the training path set, with held-out queries evaluated only on pairs whose paths are both unseen). Report pairwise accuracy on the held-out (u,f) queries, and additionally run a leave-one-user-out variant. If FAIRY's accuracy falls from the reported ~60%/~56% to near the unsupervised baselines, the headline gains are an artifact of train/test overlap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that FAIRY's LTR model ranks explanation paths according to user-judged relevance and surprisal, with Table 2 reporting 60.33/60.38 (Quora) and 56.24/54.21 (Last.fm) pairwise accuracy. Section 5 states only that 'we divided each dataset into 80% training, 10% development, and 10% test sets', without specifying that the split is by (u,f) query or that no explanation path appears in both train and test. The reported corpus statistics make this omission consequential: each path appears in about 1.9 pairs on average, and the 11,677 Quora pairs cover only 459 distinct (u,f) pairs, so a random pair-level split will place the same path and the same query in both training and test. Since the LTR features are properties of a path in the context of a specific feed item, test instances are near-duplicates of training instances. The model can then memorize query- or path-level feature patterns, inflating pairwise accuracy relative to the unsupervised baselines, which do not train on labels and therefore do not benefit from this leakage. The paper does not report a grouped split, leave-one-query-out, or leave-one-user-out evaluation, and the judgment data are not released, so the overlap cannot be checked externally. If a clean split drops accuracy toward the ~50% baselines, the central ranking claim is unsupported despite the reported p-values.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FAIRY, a user-side framework that constructs a heterogeneous interaction graph from a user's visible platform actions and treats paths from the user to a feed item as candidate explanations. It defines timestamp-constrained explanation paths (Definition 2.2), represents them with simple interpretable features (Section 3.1), and ranks them with SVMRank trained on pairwise user judgments of relevance and surprisal. User studies on Quora and Last.fm with 20 paid users per platform yield pairwise ranking accuracies of about 60% and 54-56% (Table 2), which are claimed to significantly outperform adapted ESPRESSO, REX, and PRA baselines. The paper also reports user-specific models, ablations, perturbation analyses, transitivity checks, and anecdotal examples.","tokens_in":15836,"tokens_out":6106,"duration_ms":67629,"significance":"If the reported accuracies are trustworthy, FAIRY is a useful step toward user-side feed transparency: the idea of grounding explanations in a user-visible interaction graph is well motivated, the features are deliberately interpretable, and the two-platform user study is a genuine attempt to collect gold judgments. I credit the authors for releasing code, for the transitivity sanity check, and for the perturbation analysis that probes what drives judgments. The main reservation is that the central empirical claim currently rests on a fragile evaluation protocol; without a query-disjoint split, the significance of the numbers in Table 2 cannot be assessed.","major_comments":[{"comment":"The train/test split is described only as '80% training, 10% development, and 10% test sets' (Section 5), without specifying the split unit. This is consequential because the judgment data are heavily overlapping: 11,677 Quora pairs are drawn from only 459 distinct (u,f) queries, and each explanation path appears in about 1.9 pairs on average; the corresponding Last.fm numbers are 4,791 pairs, 235 queries, and 1.7 pairs per path. A random pair-level split therefore places the same path and the same (u,f) query in both training and test, and since the LTR features are properties of a path in the context of a specific feed item, the test instances are near-duplicates of training instances. The unsupervised baselines in Table 2 do not train on labels and so do not benefit from this leakage, which can inflate FAIRY's reported 5-10 point gains. Please re-run the evaluation with grouped splits (e.g., leave-one-query-out, leave-one-user-out, or a split with no shared paths between train and test) and report whether the accuracies in Table 2 survive; the current p-values from a paired t-test do not address this form of non-independence.","section":"Section 5 (Datasets and LTR)"},{"comment":"The evaluation measures only pairwise preference accuracy on randomly sampled path pairs, not the quality of the final top-k explanation lists a user would see. The perturbation results in Table 4 show that when pairs differ only in the feed item, accuracy falls to about 50-52% on both platforms, which suggests that the model is strongly tied to the specific path-pair distribution used for training and may not transfer to the realistic setting of ranking all candidate paths for a new feed item. Please add an evaluation of full ranking quality (e.g., nDCG or precision at k on the complete candidate path set per (u,f) query) or otherwise justify that pairwise accuracy on random pairs supports the claimed usefulness of the ranked list.","section":"Section 4 (User Studies) and Section 6.2 (Perturbation analysis)"}],"minor_comments":[{"comment":"The abstract and introduction say FAIRY 'explains' why items were shown, but the framework computes plausible user-side paths and the evaluation measures user preference, not agreement with the platform's actual feed algorithm. Please consistently use 'plausible explanation' or 'user-side explanation' to avoid overclaiming, as the Introduction itself acknowledges these relationships are 'a proxy that the users could find plausible'.","section":"Abstract and Section 1"},{"comment":"The text states that removing path pattern features 'does not affect accuracies of models on Last.fm', but Table 3 shows a drop from 56.24 to 54.32 for relevance and from 54.21 to 53.65 for surprisal; please adjust the wording to say that the effect is smaller than on Quora rather than absent.","section":"Section 6.2 (Ablation study)"},{"comment":"Several path examples contain text-encoding artifacts (e.g., 'Cookinд', 'Enдekbert', 'sin дs') that obscure the examples; please ensure the final PDF uses correct glyphs.","section":"Table 5"},{"comment":"The user-specific model accuracies are reported without error bars or confidence intervals; with only 20 users per platform, please state how many judgments per user were used and whether the observed differences are stable across split seeds.","section":"Figures 3-6"},{"comment":"The paired t-test should state the pairing unit (pair of paths, query, or user) and whether it accounts for the non-independence of pairs sharing paths or queries; as written, the significance claim is not verifiable from the reported information.","section":"Table 2 (statistical test)"}],"recommendation":"major_revision","confidential_remarks":"The leakage concern is the main reason I cannot recommend acceptance. If the authors can produce a clean grouped-split evaluation and the gains persist, the paper would be a solid contribution. I also note that the judgments and interaction graphs are not released, so external verification is impossible; an artifact appendix with anonymized data or an independent replication would materially strengthen the paper. The novelty relative to prior relationship-discovery work is incremental but sufficient for WSDM if the evaluation is clean."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: this paper gives the field something it didn't quite have before—a user-centric formulation of feed explanation as ranking paths through the user's own visible interaction graph. The path definition with temporal constraints is clean, the features are deliberately interpretable, and the authors went out and ran a real two-month, 20-user study on each of two platforms. That's more than most papers in this area do. The ablation and the discussion of transitivity and surprisal-vs-complexity are also honest and useful.\n\nThe soft spot is the one the stress test flags, and it's load-bearing. The split into 80/10/10 is described as a random split over path pairs, not over (u,f) queries. With 11,677 Quora pairs covering only 459 distinct (u,f) pairs and each path appearing in ~1.9 pairs on average, a pair-level split very likely places the same path and the same query in both training and test. The LTR features are properties of a path in the context of a feed item, so a test instance can be a near-duplicate of a training instance. That means the reported 5–10 point gains over unsupervised baselines may be partly memorization, not ranking ability. The paper doesn't report a query-disjoint split or leave-one-user-out evaluation, and the judgment data aren't released, so the overlap can't be checked externally. That's the main reason I'd be wary of the central empirical claim.\n\nThe user pool is also narrow—20 paid graduate students on fresh accounts with prescribed interaction schedules—so the generality of the user judgments is limited. The accuracy numbers themselves are modest (54–60%) compared to the 50% chance level; the gains over the strongest baseline on Last.fm surprisal are only about three points. The paper is candid about not modeling sensitive topics or user background, which is a point in its favor, but those omissions matter for real-world transparency.\n\nWho is this for? Researchers working on feed explainability, filter-bubble transparency, and HIN-based relationship mining. The framework and the task formulation are worth citing. I would not take the reported accuracy as established fact until a clean split and ideally the annotation data are provided.\n\nMy recommendation: send to peer review, but with major revisions. Ask for a query-disjoint split, a leave-one-user-out analysis, and a toned-down claim about user benefits. The paper deserves referee time and revision, not a desk reject.","headline":"A genuinely new framing for feed explainability, but the evaluation's train/test split likely leaks and the data aren't released, so the accuracy claims need a clean re-run before they can be trusted.","tokens_in":16369,"tokens_out":2088,"would_cite":true,"duration_ms":24670,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Paths through a user's own actions can explain why feed items appear","keywords":["social feed explanation","interaction graph","heterogeneous information network","learning-to-rank","relevance and surprisal","explainable recommendation","user study","feed transparency"],"falsifier":"On a platform with an open recommender or logged ranking reasons, collect the top-ranked FAIRY paths for a sample of feed items and compare them with the logged reasons; if the majority of top paths do not match or subsume those reasons, or if randomly re-timestamped paths perform equally well in user preference tests, the central claim is refuted.","tokens_in":15328,"feed_emoji":"🔍","tokens_out":7700,"duration_ms":72815,"temperature":0.7,"pith_summary":"This paper tries to show that a user's own on-platform actions, visible to her alone, are enough to explain why specific items appear in her social feed. Its claim is that each meaningful explanation is a path through a user-specific interaction graph connecting the user to the feed item, and that a ranking model over simple, interpretable features can order these paths by how relevant and how surprising real users find them. In controlled studies with twenty users on each of two platforms, the resulting framework, called FAIRY, picked the user-preferred explanation in roughly 54 to 60 percent of pairwise comparisons, outperforming three established relationship-mining baselines. If the claim holds, users can audit their own feeds, rediscover forgotten actions that steer content, and take informed corrective steps, all without any help from the platform's internal algorithm.","feed_headline":"Paths through your own actions can explain why feed items appear","feed_subtitle":"FAIRY ranks paths linking your activity to each feed item, and users rate the top picks as relevant and surprising.","key_machinery":"The carrying mechanism is the interaction graph, a heterogeneous information network built only from information a user can see: nodes represent users, content items, and categories; edges are directed, weighted, timestamped actions such as follows, asks, upvotes, scrobbles, loves, and belongs-to relations, with inverse edges added so traversal can go both ways. An explanation path is defined as any path from the user to a feed item for which every edge timestamp is strictly earlier than the time the item was seen, guaranteeing that the explanation only cites actions already on record; when the graph is not connected, the platform's topic taxonomy is overlaid to route paths through category ancestors. The ranking engine is a pairwise learning-to-rank model trained with ordinal regression on user preference judgments, using five feature groups—user, category, item, path instance, and path pattern—chosen deliberately so that, in principle, a user could look at a path and see why it was ranked. This machinery converts a massive, hidden feed-generation process into a small set of readable human-scale explanations.","core_discovery":"The paper's central discovery is that explanation paths—sequences of timestamped edges connecting the user to the feed item—are a valid operationalization of why a feed item appeared, and that user judgments of relevance and surprisal on these paths are learnable from lightweight, user-visible features such as user influence, category specificity, item engagement, path length and recency, pattern frequency, and edge-type counts. The paper demonstrates this by collecting thousands of pairwise preference judgments from real users and training ordinal-regression ranking models that predict the preferred explanation. On one platform the model reaches 60.33% accuracy for relevance and 60.38% for surprisal; on the other it reaches 56.24% and 54.21%, each time statistically above the strongest of three baseline relationship-discovery methods. The paper also finds that path-pattern features carry much of the signal on one platform, user-specific models often beat one global model, and users are consistent in their judgments for 74–80% of transitive triplets.","pith_inferences":["If platforms ever open their ranking logs, FAIRY-style paths could serve as a user-side audit that checks whether visible explanations match the real algorithmic reasons; mismatches would flag hidden personalization.","The surprisal ranking could double as a filter-bubble detector: paths that reveal unexpected connections between the user and incoming content expose associations the user did not consciously make.","A natural extension is to turn the ranking model around: instead of explaining a given item, find the user action whose removal most changes the set of top explanation paths, giving a prescriptive 'do this to see less of X' recommendation.","The pairwise preference protocol itself could be reused to collect explanation judgments at scale via crowd platforms, since users found the path-pair task cognitively manageable."],"forward_implications":["A user-side tool can surface the top few explanation paths for any feed item, letting users audit their feeds without any cooperation from the platform.","Because the features are user-visible and interpretable, a user can trace why one path was ranked above another and can see which of her own actions contributed.","The same interaction-graph construction and ranking pipeline transfers to other platforms whose data can be laid out as nodes and timestamped action edges, not just the two studied here.","Pairwise preference judgments from users are a workable gold standard for explanation quality: users were consistent in 80% of transitive triplets on one platform and 74% on the other.","Embedding-based similarity between feed items and path nodes improves ranking over taxonomy-distance alone, so content signals belong in explanation ranking."],"supporting_citations":[{"why":"Supplies the dense-subgraph relationship baseline whose ranking FAIRY must beat.","marker":"[36]"},{"why":"Supplies the relationship-ranking baseline and the distributional interestingness measures adapted for comparison.","marker":"[14]"},{"why":"Supplies the path-constrained random-walk baseline that scores paths by reachability.","marker":"[24]"},{"why":"Provides the ordinal-regression pairwise learning-to-rank method used to train the relevance and surprisal models.","marker":"[21]"},{"why":"Establishes the motivating user problem: people reason about invisible feed algorithms and want explanations.","marker":"[13]"}],"fun_headline_variants":["FAIRY: trace your actions to every feed item's reason","Why this item? FAIRY ranks paths from your actions","Your feed decoded: FAIRY links actions to items","FAIRY explains feeds via your action paths","See why: FAIRY maps your actions to feed items"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a feed item's appearance is always traceable to paths in the user's own visible interaction graph, so if the platform uses hidden signals, global trends, or actions outside the user's view, FAIRY's explanations are plausible-sounding but not true reasons.","fun_headline_variants_meta":{"raw":{"variants":["FAIRY: trace your actions to every feed item's reason","Why this item? FAIRY ranks paths from your actions","Your feed decoded: FAIRY links actions to items","FAIRY explains feeds via your action paths","See why: FAIRY maps your actions to feed items"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000727,"raw_usage":{"total_tokens":3266,"prompt_tokens":962,"completion_tokens":2304,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":2221}},"tokens_in":578,"tokens_out":2304,"duration_ms":17469,"temperature":1.0,"reasoning_tokens":2221,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:23:25.475857+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a platform with an open recommender or logged ranking reasons, collect the top-ranked FAIRY paths for a sample of feed items and compare them with the logged reasons; if the majority of top paths do not match or subsume those reasons, or if randomly re-timestamped paths perform equally well in user preference tests, the central claim is refuted.","supporting_citations":[{"cited_title":"Bedathur, Sarath Kumar Kondreddi, Patrick Ernst, and Gerhard Weikum","cited_arxiv_id":null,"evidence_quote":"Supplies the dense-subgraph relationship baseline whose ranking FAIRY must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the relationship-ranking baseline and the distributional interestingness measures adapted for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the path-constrained random-walk baseline that scores paths by reachability."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ordinal-regression pairwise learning-to-rank method used to train the relevance and surprisal models."},{"cited_title":"I always assumed that I wasn’t really that close to [her]","cited_arxiv_id":null,"evidence_quote":"Establishes the motivating user problem: people reason about invisible feed algorithms and want explanations."}],"review_version":1}