Pith. sign in

REVIEW 2 cited by

Web Scraping for Research: Legal, Ethical, Institutional, and Scientific Considerations

Not yet reviewed by Pith; the record is open.

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.23432 v2 pith:BPRIN4VQ submitted 2024-10-30 cs.CY cs.SI

classification cs.CYcs.SI
keywords researchersscrapingdataresearchaccessethicalcollectconduct
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Scientists across disciplines often use data from the internet to conduct research, generating valuable insights about human behavior. However, as generative AI relying on massive text corpora becomes increasingly valuable, platforms have greatly restricted access to data through official channels. As a result, researchers will likely engage in more web scraping to collect data, introducing new challenges and concerns for researchers. This paper proposes a comprehensive framework for web scraping in social science research for U.S.-based researchers, examining the legal, ethical, institutional, and scientific factors that researchers should consider when scraping the web. We present an overview of the current regulatory environment impacting when and how researchers can access, collect, store, and share data via scraping. We then provide researchers with recommendations to conduct scraping in a scientifically legitimate and ethical manner. We aim to equip researchers with the relevant information to mitigate risks and maximize the impact of their research amidst this evolving data access landscape.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Patent Expiry to Business Pathways: AI Workflows for Activating Innovation Archives

    cs.IR 2026-07 conditional novelty 5.0 of 10

    An AI-enabled design-science framework discovers expired and near-expiry patents and converts disclosures into ranked, uncertainty-aware business pathway review packets, demonstrated on a full CIPO weekly archive.

  2. Building a Media Ecosystem Observatory from Scratch: Infrastructure, Methodology, and Insights

    cs.DL 2025-06 conditional novelty 4.0 of 10

    A national-scale observatory for cross-platform Canadian media monitoring is presented, with an infrastructure blueprint and preliminary analyses of the Meta news ban and the 2025 federal election.

Pith tools