PARQR: Augmenting the Piazza Online Forum to Better Support Degree Seeking Online Masters Students

Abhishek Mangal; Chirag Tailor; Girish Murali; India Irish; Niklas Gustafsson; Noah Bilgrien; Rosa Arriaga; Roy Finkelberg; Sumedha Raman; Thad Starner

REVIEW 4 major objections 4 minor 13 references

PARQR: Augmenting the Piazza Online Forum to Better Support Degree Seeking Online Masters Students

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PARQR, a browser extension that suggests similar old forum posts while a student types, is reported to cut duplicate questions in a large online master's course by 40%.

desk verdict A well-built system paper with a credible retrieval evaluation, but the headline duplicate-reduction claim is not causally identified and the '40%' figure overstates the reported effect. read the letter →

arxiv 1909.02043 v1 pith:IUMPJH2D submitted 2019-09-04 cs.HC

Noah Bilgrien , Roy Finkelberg , Chirag Tailor , India Irish , Girish Murali , Abhishek Mangal , Niklas Gustafsson , Sumedha Raman

show 2 more authors

Thad Starner Rosa Arriaga

This is my paper · ORCID

classification cs.HC

keywords OnlineForumsDegreesRecommenderSystemsComputer-AssistedInstructionDistanceLearningDuplicatePostReductionPiazzaTF-IDF

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces PARQR, a browser extension for the Piazza forum that watches a question as a student writes it and continuously suggests existing posts that may already answer it. The authors claim that using PARQR in an Introduction to AI course in an online degree-seeking master's program reduced duplicate posts from 25.6% to 17.8% during a comparable assignment, a 40% relative drop, and that the recommender surfaced a relevant post 73.5% of the time when one existed. The motivation is that large online courses generate so many forum posts that neither students nor instructors can direct attention well, and standard keyword search happens only after a student commits to asking. If the reduction is real, PARQR points to a design principle: catch the duplicate intent at composition time rather than after the fact.

What carries the argument

The mechanism is PARQR itself: a browser extension that reads the in-progress post (title, body, and tags) and calls a retrieval service based on term frequency–inverse document frequency (TF-IDF). Each class has four vector-space models over (1) the question text, (2) instructor answers, (3) student answers, and (4) follow-up discussions; cosine similarity to all existing posts is computed, a hand-tuned weighted average combines the four scores, and the five nearest posts are shown. The home-page suggestion uses a separate importance formula $I_n = \frac{v_n f_n}{1+e^{(a_n-\theta)}}$, where $v_n$ is normalized views, $f_n$ normalized follow-ups, $a_n$ age in days, and $\theta=7$ days. The backend is a collection of containerized microservices that fetch posts through an unofficial application programming interface for Piazza, store them in a document database, refresh the models every 15 minutes, and serve recommendations over a RESTful endpoint. The key design choice is the Remembrance-Agent style continuous retrieval: the suggestion appears before submission, so a student's cost of checking an old answer is lower than the cost of posting a duplicate.

What would settle it

Run a randomized crossover: in the same semester and same course, give one half of the students the composer-suggestion extension and the other half a placebo that shows unrelated posts, then compare duplicate-post proportions; if the difference is not replicated or is near zero, the paper's central claim is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that composing-time recommendation—injecting related posts into the composer instead of relying on a separate search step—can materially reduce duplicate questions in a real degree-seeking online course. In the comparison of Assignment 2 of an Introduction to AI course, the proportion of duplicate posts fell from 25.6% (50 of 195 posts) in Spring 2017 to 17.8% (30 of 168 posts) in Spring 2019, with a one-sided Z-test giving p = 0.039; 98% of Piazza users used PARQR in the later semester. The same recommendation engine, validated walk-forward on a labeled gold-standard set, retrieved at least one relevant post in 73.5% of cases where one existed. The authors also report that teaching assistants found the instructor-facing suggestions a useful triage tool, though they could not themselves perceive the duplicate-rate change.

Load-bearing premise

The two semesters are assumed to be equivalent in everything except PARQR; but they differed in enrollment (390 vs 590 students), cohort, teaching staff, and possibly course materials, so the drop in duplicate posts cannot be causally isolated to the tool.

Editorial extensions

If this is right

If adopted broadly, PARQR-style composing-time recommendation could lower the attention load on instructors by shrinking the pool of duplicate questions they have to read.
Students can get answers faster, since relevant previous answers surface while the question is still being written.
The same architecture (TF-IDF over four post components with weighted cosine similarity) can be reused for other large courses or other forum software with modest adaptation.
Instructor-facing attention scoring (unanswered posts with many views and follow-ups) may help teaching staff triage during office hours.
The 73.5% retrieval rate suggests that a substantial portion of duplicate questions are predictable from existing content, which could inform feature design in learning management systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The 40% drop may overstate the tool's effect because the comparison is between different semesters with different enrollment (390 vs 590) and no randomization; a within-semester randomized assignment would give a cleaner estimate.
We infer that the mechanism is not just retrieval accuracy but the cost of checking: showing suggestions during composition may convert 'ask anyway' into 'read first,' which a post-hoc search prompt might not do.
A testable extension would be to measure not only duplicate posts but also time-to-first-answer and student satisfaction, and to see whether the duplicate reduction persists beyond the first assignment.
The importance-scoring home page may be doing unseen work; the two features (composer recommendation and attention ranking) are bundled, so an ablation would separate their contribution.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a well-designed system with one solid evaluation and one shaky one. The walk-forward retrieval test is credible; the headline "40% duplicate reduction" is not backed by the study design.

PARQR is a browser extension that watches a student compose a Piazza post and recommends similar existing posts, plus dashboards for instructors. That's a sensible and genuinely new application of remembrance-agent ideas to degree-seeking online master's programs, which are understudied relative to MOOCs. The system is deployed at scale (1000 users) and the design choices are reasonable.

The retrieval evaluation is a real plus. On 179 chronologically ordered Fall 2017 posts, the ensemble found a relevant existing post 73.5% of the time when one existed. It's walk-forward and held-out, so that number is informative. The inter-rater reliability work also looks careful, with 93% pairwise agreement and a negotiated gold standard.

The soft spot is the duplicate-reduction claim. The paper compares Assignment 2 in Spring 2017 (no PARQR) with Spring 2019 (98% PARQR use). The two semesters differ on enrollment (390 vs 590), active students (385 vs 590), and total posts (195 vs 168). Any number of unmeasured changes—course materials, TA staffing, platform behavior, cohort composition—could explain the drop. The one-sided Z-test p=0.0392 is marginal and not corrected for the fact that they picked one assignment after seeing the data. And the headline number is misleading: 25.6% to 17.8% is a 30.5% relative reduction in the rate; the 40% figure is the reduction in raw duplicate count (50 to 30). That's an overstatement in the abstract.

There's also no direct evidence that PARQR's recommendations caused the drop. The extension logs recommendation clicks and submissions, but the paper doesn't link exposure to behavior. The instructor interviews are anecdotal, and the TAs themselves were unsure whether PARQR reduced duplicates.

This deserves a serious referee. The system is real, the retrieval evaluation is a contribution, and the question of whether proactive recommendations cut duplicates in online degree forums is worth asking. But the authors need to either reframe the duplicate-reduction claim as a preliminary observation or bring a design that controls for confounds—randomized assignment, matched cohorts, or at least covariate adjustment with exposure analysis. As is, the abstract overclaims.

Who is this for? People building tools for online degree forums and researchers studying forum behavior in higher ed. It's worth a reading group discussion, mainly to talk about why before/after comparisons without controls can't carry causal weight.

Recommendation: engage with the paper, but the causal claim should be heavily qualified or require stronger evidence before being accepted as stated.

Referee Report

4 major / 4 minor

Summary. The paper presents PARQR, a browser-extension recommender for Piazza-style class forums. While a student composes a question, PARQR suggests similar past posts; on the Piazza home page it highlights high-attention posts for students and unanswered-but-active posts for instructors. The authors report that in a degree-seeking online masters course, the duplicate-post rate during Assignment 2 fell from 25.6% in Spring 2017 (no PARQR) to 17.8% in Spring 2019 (98% PARQR adoption), with a one-sided Z-test p=0.0392, and they describe a walk-forward retrieval experiment in which PARQR retrieves at least one relevant prior post 73.5% of the time when one exists. The paper also reports inter-rater reliability for duplicate clustering and interviews with five teaching assistants.

Significance. If the causal claim were supported, this would be a valuable contribution to the CSCW/L@S community: a deployed, at-scale system for reducing redundant questions in online degree forums, with a practical browser-extension design and a temporal retrieval evaluation. The strengths are real: the walk-forward evaluation on chronologically ordered posts is a genuine held-out design and not forced by construction; the duplicate-labeling protocol includes a measured inter-rater agreement above chance; and the qualitative instructor interviews give useful context for deployment viability. However, the paper's central quantitative claim -- that PARQR reduces duplicate posts by 40% -- rests on a non-randomized comparison of two semesters that differ in enrollment, cohort composition, teaching staff, and platform behavior, and the reported effect size conflates raw count reduction with rate reduction. The current evidence supports, at most, an observational association, not a causal reduction.

major comments (4)

[Analysis of Reduction of Duplicate Posts; Table 1] The central claim that PARQR reduces duplicate posts is not causally identified. The comparison is between Spring 2017 and Spring 2019 of the same course, but these semesters differ in enrollment (390 vs 590), number of active students (385 vs 590), total posts (195 vs 168), and posts per active student (0.506 vs 0.291). No randomization, blocking, matching, or covariate adjustment is reported, and the one-sided Z-test (p=0.0392) only establishes that the duplicate proportions differ between the two offerings. The fact that Assignment 2 is 'identical' does not make the cohorts, teaching staff, or Piazza interface exchangeable. The authors should either reframe the result as an observational association or add an exposure-response analysis (for example, within Spring 2019, compare duplicate rates for students who did and did not interact with PARQR recommendations) and discuss plausible confounders explicitly.
[Abstract, Introduction, and Table 1] The headline '40% reduction' is arithmetically misleading. The duplicate-post rate fell from 25.6% to 17.8%, which is a relative reduction of 30.5% (1 - 17.8/25.6), not 40%. The 40% figure is the reduction in the raw duplicate count (50 to 30). Because the rate is the quantity used for the significance test and is the more meaningful metric for duplicate suppression, the abstract should report a 30.5% relative reduction, or state clearly that 40% refers to the count reduction rather than the rate reduction.
[Evaluating the Model] The 73.5% retrieval rate is measured on Fall 2017 Assignment 1 posts, not on the Spring 2019 deployment, and it measures whether any relevant prior post is retrieved while composing, not whether the student refrains from submitting a duplicate. The logged events (New Post clicks, recommendation clicks, Submit Post clicks) are described in the architecture section but are not analyzed. Thus the paper does not establish the mechanism by which recommendations reduce duplicate posts. The authors should either present an event-log analysis linking recommendation exposure to non-duplicate submission or explicitly temper the claim to retrieval quality rather than duplicate suppression.
[Duplicate Post Inter-Rater Reliability] The reliability evidence is computed on 200 pairs sampled from the Spring 2017 Assignment 1 gold standard, but the main comparison dataset (the combined Spring 2017 and Spring 2019 Assignment 2 posts) was clustered by eight researchers with no reported inter-rater reliability. Because the outcome variable is based on human duplicate judgments, the paper should report agreement on the actual dataset used for the main analysis, or use verified labelers for that dataset, before treating the duplicate counts in Table 1 as a measured outcome.

minor comments (4)

[Background and Related Work] The phrase 'non-technical approachs' should be 'non-technical approaches'.
[References] Reference [9] is incomplete; it lacks the publication venue, year, and page numbers for the Remembrance Agent paper.
[PARQR, Equation (1)] The importance score uses min-max normalized views and followups, but the normalization window is not specified; please clarify whether normalization is per-class, per-time-period, or global.
[Instructor Interviews] The interview section reports qualitative themes but does not describe the interview protocol or whether responses were systematically coded; a brief description of the coding method would strengthen reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation uses held-out human labels and sequential walk-forward retrieval, and the only self-citation is inspirational.

full rationale

The paper's central claims are not circular. The duplicate-reduction claim is based on human clustering of posts from the two semesters that was performed blind to semester and independent of PARQR's retrieval model; the duplicate proportion is therefore not forced by the tool's construction. The retrieval claim is evaluated by a walk-forward validation on Fall 2017 Assignment 1, where the model is trained only on posts prior to each target post and checked against gold-standard duplicate clusters; a correct recommendation is not definitionally guaranteed, so the 73.5% figure is an empirical outcome rather than an identity. The hand-tuned weights and the sigmoid offset (theta = 7 days) are fitted design parameters for the recommendation UI and do not determine the outcome labels used for measuring duplicate rates or retrieval success. The paper's one self-citation, to the Remembrance Agent (Rhodes and Starner 2002, where Starner is a co-author), is explicitly motivational ('PARQR is inspired by a class of recommender systems called Remembrance Agents') and is not load-bearing for either the duplicate-reduction or retrieval conclusions. The main weaknesses of the paper are causal identification concerns, not circularity: the Spring 2017 versus Spring 2019 comparison is confounded by enrollment, cohort composition, and staffing, and the '40% reduction' headline refers to the raw duplicate count (50 to 30) rather than the relative rate reduction (25.6% to 17.8%, a 30.5% relative reduction). These are important validity issues but do not constitute the paper reducing its predictions to its inputs by construction. No circular step is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The system adds no new theoretical entities. The main assumptions are standard text-retrieval premises plus the comparability of two semesters, which is the fragile load-bearing assumption.

free parameters (4)

theta (importance score offset) = 7 days
Hand-set in Eq 1 to prioritize newer posts; not learned from data and not varied in sensitivity analysis.
ensemble model weights = not reported
Weights for the four TF-IDF models were hand tuned, with highest weight on question content; the paper does not give the values.
recommendation list length (top 5) = 5
Number of related posts shown while composing; chosen by designers, not evaluated across values.
home page list length and age cutoff = 6 and 3 weeks
Instructor and student home pages show six posts; posts older than three weeks are never shown. Hard-coded design choices.

assumptions (5)

standard math TF-IDF and cosine similarity are adequate measures of semantic relevance between question drafts and existing posts.
Used throughout the ranking pipeline; standard information retrieval assumptions.
domain assumption Post views and follow-up counts are proxies for usefulness or attention-worthiness.
Eq 1 uses normalized views and follow-ups to rank home page suggestions; no validation that these correlate with learning outcomes.
domain assumption A duplicate is defined as a post whose author would stop if shown the answers to another post, and researcher clustering reliably approximates this.
Inter-rater reliability is reported, but the gold standard itself rests on subjective judgment.
domain assumption Spring 2017 and Spring 2019 offerings of the same assignment are otherwise comparable, so the duplicate-rate difference can be attributed to PARQR.
This is the causal identification assumption; it is not tested.
domain assumption PARQR adoption in Spring 2019 (98%) did not coincide with other forum or course changes.
No data on other changes is reported; same assumption as the comparability assumption but specifically about adoption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of PARQR: Augmenting the Piazza Online Forum to Better Support Degree Seeking Online Masters Students." pith.science (2026). https://pith.science/paper/IUMPJH2D

@misc{pith2026190902043,
  author       = {Pith},
  title        = {Pith review of: PARQR: Augmenting the Piazza Online Forum to Better Support Degree Seeking Online Masters Students},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IUMPJH2D}},
  note         = {Machine review of arXiv:1909.02043}
}

read the original abstract

We introduce PARQR, a tool for online education forums that reduces duplicate posts by 40\% in a degree seeking online masters program at a top university. Instead of performing a standard keyword search, PARQR monitors questions as students compose them and continuously suggests relevant posts. In testing, PARQR correctly recommends a relevant post, if one exists, 73.5\% of the time. We discuss PARQR's design, initial experimental results comparing different semesters with and without PARQR, and interviews we conducted with teaching instructors regarding their experience with PARQR.

Figures

Figures reproduced from arXiv: 1909.02043 by the authors.

**Figure 2.** What a PARQR user sees when authoring a new question. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 9 canonical work pages

[1]

Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O’Reilly Media

work page 2009
[2]

C. G. Brinton, M. Chiang, S. Jain, H. Lam, Z. Liu, and F. M. F. Wong. 2014. Learning about Social Learning in MOOCs: From Statistical Analysis to Generative Model. IEEE Transactions on Learning Technologies 7, 4 (Oct 2014), 346–359

work page 2014
[3]

Anat Cohen, Udi Shimony, Raﬁ Nachmias, and Tal Soffer. 2019. Active learners characterization in MOOC forums and their generated knowledge. British Journal of Educational Technology 50, 1 (2019), 177–198

work page 2019
[4]

Hamza Faran. 2018. piazza-api. https://github.com/hfaran/piazza-api/. (2018)

work page 2018
[5]

Goel and Lalith Polepeddi

Ashok K. Goel and Lalith Polepeddi. 2016. Jill Watson: A Virtual Teaching Assistant for Online Education. (2016)

work page 2016
[6]

David Joyner. 2018. Squeezing the Limeade: Policies and Workﬂows for Scalable Online Degrees. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale (L@S ’18). ACM, New York, NY , USA, Article 53, 10 pages. DOI:http://dx.doi.org/10.1145/3231644.3231649

arXiv 2018
[7]

Kizilcec, Chris Piech, and Emily Schneider

René F. Kizilcec, Chris Piech, and Emily Schneider

work page
[8]

Pedregosa, G

F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830

2011

Show all 13 references

[9]

Rhodes and Thad Starner

Bradley J. Rhodes and Thad Starner. 2002. Remembrance Agent : A continuously running automated information retrieval system

2002
[10]

Dhawal Shah. 2018. By The Numbers: MOOCs in 2018

2018
[11]

Diyi Yang, Mario Piergallini, Iris Howley, and Carolyn Rose. 2014. Forum thread recommendation for massive open online courses. In Educational Data Mining 2014 . Citeseer

2014
[12]

Saijing Zheng, Pamela Wisniewski, Mary Beth Rosson, and John M. Carroll. 2016. Ask the Instructors: 4 Motivations and Challenges of Teaching Massive Open Online Courses. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW ’...

2016
[2013]

In Proceedings of the Third International Conference on Learning Analytics and Knowledge (LAK ’13)

Deconstructing Disengagement: Analyzing Learner Subpopulations in Massive Open Online Courses. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (LAK ’13). ACM, New York, NY , USA, 170–179.DOI: http://dx.doi.org/10.1145/2460296.2460330

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O’Reilly Media

work page 2009

[2] [2]

C. G. Brinton, M. Chiang, S. Jain, H. Lam, Z. Liu, and F. M. F. Wong. 2014. Learning about Social Learning in MOOCs: From Statistical Analysis to Generative Model. IEEE Transactions on Learning Technologies 7, 4 (Oct 2014), 346–359

work page 2014

[3] [3]

Anat Cohen, Udi Shimony, Raﬁ Nachmias, and Tal Soffer. 2019. Active learners characterization in MOOC forums and their generated knowledge. British Journal of Educational Technology 50, 1 (2019), 177–198

work page 2019

[4] [4]

Hamza Faran. 2018. piazza-api. https://github.com/hfaran/piazza-api/. (2018)

work page 2018

[5] [5]

Goel and Lalith Polepeddi

Ashok K. Goel and Lalith Polepeddi. 2016. Jill Watson: A Virtual Teaching Assistant for Online Education. (2016)

work page 2016

[6] [6]

David Joyner. 2018. Squeezing the Limeade: Policies and Workﬂows for Scalable Online Degrees. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale (L@S ’18). ACM, New York, NY , USA, Article 53, 10 pages. DOI:http://dx.doi.org/10.1145/3231644.3231649

arXiv 2018

[7] [7]

Kizilcec, Chris Piech, and Emily Schneider

René F. Kizilcec, Chris Piech, and Emily Schneider

work page

[8] [8]

Pedregosa, G

F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830

2011