{"id":"2d43792b-07e9-404b-a29d-bc73c0d377fa","arxiv_id":"2508.02592","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A field report claims that a scavenger-hunt structured graduate course on critical visualization successfully developed critical thinking, communication, and theory knowledge in six students, though evidence is anecdotal.","lead":"This paper describes a graduate computer science course on critical visualization, taught as a self-directed scavenger hunt where students choose assignments to earn points. It reports that the course broadly met its goals of building critical thinking, communication, and theory knowledge, based on instructor reflection and student work.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Success claim rests on instructor self-report and co-authored student artifacts, with no baseline, rubric, or external evaluation.","rationale":"The reader's weakest-assumption analysis identifies the same central issue: the report assumes instructor observations and selected student submissions are valid evidence that learning objectives were met. My read agrees with that assessment and with the CONDITIONAL verdict. The paper is transparent about its nature as a field report, and the pedagogical content and candid reflection on the scavenger-hunt difficulties are valuable. But the abstract's definitive claim that the course 'achieved the learning objectives' goes beyond what the evidence supports. The concern is not that the course was ineffective; it is that no method is in place to distinguish between an effective course and a pleasant but unmeasured one. A blind rubric-based evaluation of artifacts, with a baseline, would directly test the central claim. Since the reader already recommended CONDITIONAL, and my concern does not shift that recommendation, UNCHANGED is appropriate.","tokens_in":4162,"tokens_out":1681,"duration_ms":21030,"concrete_test":"Run a blind external evaluation of the student artifacts from the course. Collect all submitted works (or de-identified excerpts), have two evaluators who were not involved in the course score them on a pre-defined rubric with the three learning objectives (critical thinking, communication, theory knowledge) and compare those scores against a pre-course baseline task of comparable difficulty. If inter-rater agreement is poor, or if post-course scores do not clearly exceed baseline, then the claimed achievement of learning objectives is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the Abstract and Section 2 is that the course 'generally was successful' and achieved the three learning objectives: critical thinking, communication, and theory knowledge. The evidence for this is (a) the instructor's statement that the class 'felt successful' and that students produced 'cogent syntheses' (Section 2), and (b) selected student submissions shown in Figure 1. There is no pre-course baseline, no scoring rubric, no external evaluator, and no comparison group. The six enrolled students are also listed as co-authors on this report, so the student perspective is not independent. Section 2.2 further documents that at the two-thirds mark no submissions had been made, that the instructor had to intervene with one-on-one planning sessions, and that final work was 'rushed' and 'traditional'. That evidence complicates, but does not directly contradict, the success claim; however, it highlights how thin the positive evidence is. Since the central claim is an empirical effectiveness claim, the absence of any systematic outcome measurement is the load-bearing weakness: if the measurement is absent, the headline assertion of achieving learning objectives is unsupported rather than merely understated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This field report describes a 2025 graduate computer science course on critical visualization (CS6967) at the University of Utah. The course used an experimental 'scavenger hunt' structure in which students assembled a portfolio of optional, self-designed submissions to earn points, and content focused on feminist and critical visualization supplemented by foundational theory. The report claims that the course was generally successful in meeting three learning objectives: critical thinking, communication about complex ideas, and knowledge of theories in the area. It also reflects on content selection, the scavenger hunt's strengths and failures, and possible future improvements. The six graduate students are listed as co-authors, and the first author is both instructor and report author.","tokens_in":4473,"tokens_out":2655,"duration_ms":31267,"significance":"If the central claim were well-supported, the paper would offer a useful template for teaching critical visualization in graduate CS programs, a topic with little published pedagogy. The report candidly documents both successes and difficulties, including the two-thirds-mark failure to turn in work and the need for instructor intervention. Its value lies in the practical details and honest reflection about course structure and content. However, the significance is seriously limited by the fact that the success claim rests on the instructor's self-assessment and selected student work, with no systematic outcome measurement, independent evaluation, or anonymous student perspectives. The paper is best regarded as a reflective experience report, not as evidence that the learning objectives were achieved.","major_comments":[{"comment":"The central claim that the course 'generally was successful, achieving the learning objectives' is not supported by the evidence presented. The evidence consists of the instructor's own impressions ('this class felt successful to me as an educator') and selected student submissions (Figure 1). There is no pre-course baseline, no scoring rubric, no external evaluator, and no comparison group. Because the six students are co-authors of this report, their participation cannot serve as an independent validation of the claimed learning outcomes. The paper should either add a systematic evaluation component or explicitly reframe its contribution as a reflective field report without an effectiveness claim.","section":"Abstract and Section 2 (Reflections)"},{"comment":"The report itself documents observations that complicate the success narrative: at the two-thirds mark no student had turned in any work, the instructor had to hold one-on-one planning sessions, and the resulting submissions were 'rushed' and 'traditional.' These facts are acknowledged but are not reconciled with the abstract's strong claim of achieving the learning objectives. The paper needs to either integrate these negative observations into a more nuanced conclusion or weaken the overall success claim.","section":"Section 2.2 (The Scavenger Hunt)"},{"comment":"The claim that 'students were able to express cogent syntheses of their learning' is illustrated only by a single figure of zines, with no textual analysis of student work against the stated learning objectives. For a paper whose central claim is about learning outcomes, the evidence chain from course activities to claimed outcomes is missing. If the authors wish to retain the success claim, they should present a more systematic analysis of the submissions (e.g., how each artifact demonstrates critical thinking, communication, or theory knowledge) and acknowledge the absence of external assessment.","section":"Section 2.1 (Utility) and Figure 1"}],"minor_comments":[{"comment":"The point values for the scavenger hunt activities are mentioned ('6 points' or '10 points') but the full list of activities and point values is not given in the paper; it is only available via a URL. Adding a brief table of the available activities and associated points would make the structure more concrete for readers.","section":"Section 2.2"},{"comment":"The figure caption describes the zines as an example of student submissions but does not explain what the zines contain or how they demonstrate learning. A short description of the content or a reference to the archived course materials would help the reader interpret the figure.","section":"Figure 1"},{"comment":"The phrase 'critical moment' is used with a footnote distinguishing from the academic field of critical cartography; this is clear, but the paper would benefit from a brief statement of how the course relates to existing critical visualization pedagogy beyond the cited 'Critical InfoVis' and the CG&A special issue.","section":"Section 1"},{"comment":"Some references lack complete page numbers or access dates for URLs (e.g., references [3], [6], [12], [14]). These should be completed for a archival version.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is framed as a 'field report' rather than a controlled study, which is an appropriate genre for sharing course design experiences. However, the abstract and Section 2 make strong empirical claims about achieving learning objectives that the report cannot support. In a revision, the authors should either substantially weaken the claims to match the reflective, anecdotal evidence, or add a credible evaluation (e.g., pre/post self-efficacy surveys, external review of anonymized artifacts, or anonymous end-of-course reflections). The co-authorship of the students is a related concern: while giving students credit for their work in the course is generous, it also undermines the independence of any student perspective. The paper's candid discussion of structural failures is a strength and should be preserved; it already contains the seeds of a more balanced conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading if you teach anything like critical visualization, but don't send it to press as-is. The headline claim that the course \"generally was successful\" and achieved its three learning objectives is not supported by the evidence, and the paper's own narrative gives good reasons for skepticism: at the two-thirds mark no one had submitted anything, the instructor had to push students through one-on-one planning, and the final work was rushed and traditional. There is no baseline, no rubric, no external evaluation, and the six students are co-authors, so their perspective is not independent. The stress-test note is right: the success claim is load-bearing and unmeasured.\n\nThat said, the report has real value. It is a candid, readable account of an experimental course, and the instructor does not hide the problems. The scavenger-hunt format was tried before, so the format itself is not new, but pulling it together with a critical-visualization syllabus and reflecting on how it played out is a modest extension that other instructors can learn from. The reflections on content restriction (the CRT episode), on how theoretical \"characters\" like Latour and Wittgenstein recurred across student work, and on the trade-off between free-form exploration and problem-set practice are genuinely useful. The course materials are archived, which gives the paper reproducible grounding of the practical kind.\n\nI largely agree with the reader's conditional verdict. The paper's central claim should be recalibrated: instead of asserting success, it should say \"we tried this structure and here is what we observed, including the missteps.\" That would make the report internally consistent and still useful. The citation pattern is clean; the self-citation to the earlier scavenger-hunt pilot is appropriate and not a flaw.\n\nWho is this for? Instructors designing critical or humanistic courses in CS/visualization, and people thinking about alternative course formats. It deserves a serious referee, because it is a legitimate experience report with practical value, but I would not accept it until the claims are aligned with the evidence. The writing is clear, the self-awareness is real, and the substance is honest—it just overclaims.\n\nRecommendation: send to peer review, but ask the authors to reframe the central assertion as a reflective field study, not an evaluation of success.","headline":"A useful, honest field report whose success claim in the abstract outruns the evidence the paper itself presents.","tokens_in":4871,"tokens_out":1493,"would_cite":false,"duration_ms":19190,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This field report argues that a scavenger-hunt-structured graduate course taught critical visualization and met its learning objectives.","keywords":["critical visualization","visualization pedagogy","scavenger hunt course design","critical theory","graduate education","data feminism","humanistic computing","experience report"],"falsifier":"Have independent raters blindly score pre-course and post-course student work from this class, alongside a comparison group, using a critical-thinking rubric; if post-course scores show no improvement over baseline, the claim that the course developed critical thinking would be unsupported.","tokens_in":3979,"feed_emoji":"🎓","tokens_out":5299,"duration_ms":57010,"temperature":0.7,"pith_summary":"Few academic courses teach critical visualization, so this field report tries out one and asks whether it can work. The authors describe a graduate computer science seminar of six students run as a point-based scavenger hunt: students assembled submissions toward a total of 20 points with only an end-of-term deadline. The report claims the course generally succeeded, meeting the learning objectives of developing critical thinking, improving communication about complex ideas, and building knowledge of critical theories. It also candidly records a near-failure: at two-thirds of the term nothing had been submitted, one-on-one planning sessions were needed, and final works were often rushed. The sympathetic reading is that this design is a workable, improvable template for teaching critical visualization, not a finished recipe.","feed_headline":"Scavenger-hunt course teaches critical visualization to CS grads","feed_subtitle":"A six-student CS seminar met its critical thinking goals, though late submissions forced a rescue; here is the template.","key_machinery":"The central mechanism is the 'course as scavenger hunt' structure: an a la carte menu of activities each with a point value, where an A requires assembling 20 points and the only fixed deadline is the end of the term. This replaces sequential readings plus a final project with student-selected, personally meaningful submissions, and it is the experiment claimed to carry the learning objectives. A second mechanism is the critical-visualization reading corpus—contemporary feminist and critical visualization works supplemented by original theory readings—used to develop criticality, with visualization treated as the subject of study rather than the means.","core_discovery":"The central claim is that critical visualization can be taught to graduate computer science students using a course structured as a scavenger hunt. Six students engaged with humanistic readings in feminist and critical visualization plus foundational theory, and their submitted works—zines, projects, and syntheses—were judged by the instructor to show coherent understanding; one example applied the notion of vanguardism to data feminism. The report argues that the learning objectives of critical thinking, communication, and theory knowledge were met, and that students from diverse backgrounds grew into more critical thinkers. It also reports an accidental success: the works 'talked to each other' through recurring figures and resonances across readings, giving the course a connective texture. The report's stated position is that the format has value worth further exploration, with improvements such as enforced mid-way deadlines explicitly suggested.","pith_inferences":["Because the evidence is the instructor's own judgment plus selected student work, the robust lesson is feasibility rather than measured efficacy; an independent pre/post evaluation with a critical-thinking rubric is the natural next test.","The two-thirds no-submission episode suggests that the format's freedom is its main risk, and mandatory checkpoints could convert a happy accident into a designed feature.","The same scavenger-hunt container could carry other cross-disciplinary humanistic topics inside computer science, not just visualization, provided the instructor curates a similarly strong reading list.","The course's recurring named figures from the readings hint that narrative threads could be designed in advance to help students connect theories across sessions."],"forward_implications":["Graduate computer science students can engage with humanistic critical-theory material and apply it in their own work, so critical visualization need not be confined to design or humanities programs.","The scavenger-hunt structure can elicit creative submissions such as zines, but it needs scaffolding like mid-way deadlines to prevent end-of-term rushes.","A course of this kind offers computer science curricula a route to critical thinking that complements standard ethics modules.","Recurring figures and cross-talk among readings can be deliberately used as a connective device in multidisciplinary courses.","If the format is repeated, other instructors can build on this syllabus as a shared curriculum, making critical visualization teaching more standardized and comparable."],"supporting_citations":[{"why":"Supplies the named conceptual starting point, Critical InfoVis, that defines the course's topic.","marker":"[5]"},{"why":"Anchors the contemporary feminist reading list and recurs in student syntheses and class discussion.","marker":"[4]"},{"why":"Provides a core reading on visualization conventions that students discuss and apply.","marker":"[11]"},{"why":"Foundational reading on situated knowledges that frames the course's critical-theory content.","marker":"[9]"},{"why":"Introduces critical theory as an original source for the course's theoretical grounding.","marker":"[10]"},{"why":"Offers a recurring analytic frame for theory that the class drew on when analyzing works.","marker":"[7]"},{"why":"The earlier assignment experiment that shaped the expectation that the scavenger-hunt structure would elicit lively engagement.","marker":"[12]"},{"why":"Explains the zine medium that several students adopted in their submissions.","marker":"[13]"}],"fun_headline_variants":["Scavenger hunt course teaches critical thinking to CS grads","Scavenger hunt course: critical visualization works for 6 CS grads","Six CS grads learn critical visualization via scavenger hunt","Tiny CS course: scavenger hunt teaches critical visualization","Scavenger hunt for critical visualization: a field report"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the instructor's observations of class discussions and his evaluation of students' submitted works are sufficient evidence that the course met its learning objectives.","fun_headline_variants_meta":{"raw":{"variants":["Scavenger hunt course teaches critical thinking to CS grads","Scavenger hunt course: critical visualization works for 6 CS grads","Six CS grads learn critical visualization via scavenger hunt","Tiny CS course: scavenger hunt teaches critical visualization","Scavenger hunt for critical visualization: a field report"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000524,"raw_usage":{"total_tokens":2454,"prompt_tokens":788,"completion_tokens":1666,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":404,"completion_tokens_details":{"reasoning_tokens":1580}},"tokens_in":404,"tokens_out":1666,"duration_ms":14442,"temperature":1.0,"reasoning_tokens":1580,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:37:22.260410+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent raters blindly score pre-course and post-course student work from this class, alongside a comparison group, using a critical-thinking rubric; if post-course scores show no improvement over baseline, the claim that the course developed critical thinking would be unsupported.","supporting_citations":[{"cited_title":"D’ignazio and L","cited_arxiv_id":null,"evidence_quote":"Anchors the contemporary feminist reading list and recurs in student syntheses and class discussion."},{"cited_title":"D ¨ork, P","cited_arxiv_id":null,"evidence_quote":"Supplies the named conceptual starting point, Critical InfoVis, that defines the course's topic."},{"cited_title":"Kennedy, R","cited_arxiv_id":null,"evidence_quote":"Provides a core reading on visualization conventions that students discuss and apply."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Foundational reading on situated knowledges that frames the course's critical-theory content."},{"cited_title":"Horkheimer","cited_arxiv_id":null,"evidence_quote":"Introduces critical theory as an original source for the course's theoretical grounding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Offers a recurring analytic frame for theory that the class drew on when analyzing works."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The earlier assignment experiment that shaped the expectation that the scavenger-hunt structure would elicit lively engagement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Explains the zine medium that several students adopted in their submissions."}],"review_version":2}