OpenAI, the creator of ChatGPT, quietly released a new website crawling bot recently. This bot is designed to scan website content for the purpose of training its large language models (LLMs). However, as news of the bot spread, discontent among website owners and creators grew rapidly. Many began sharing tips on how to block GPTBot from scraping their site’s data.
OpenAI responded to the backlash by adding a support page for GPTBot, which included instructions on how to block the bot from accessing a website. By making a small modification to a website’s robots.txt file, website owners could prevent their content from being shared with OpenAI. However, there is uncertainty about whether this method is sufficient to completely stop the inclusion of content in LLM training data, considering the extensive web scraping that already occurs.
Websites like The Verge have taken precautions by adding the robots.txt flag to prevent the OpenAI model from grabbing their content. Casey Newton, a writer for Platformer, asked his readers if he should take similar actions. Neil Clarke, the editor of sci-fi magazine Clarkesworld, even announced on social media that his publication would block GPTBot.
Partnership with New York University
After GPTBot’s launch became public, OpenAI surprised the public by announcing a $395,000 grant and a partnership with New York University’s Arthur L. Carter Journalism Institute. This collaboration aims to assist students in developing responsible ways to leverage AI in the news industry, led by former Reuters editor-in-chief Stephen Adler.
OpenAI’s chief of intellectual property and content, Tom Rubin, expressed excitement about the initiative but failed to address the issue of public web scraping or the controversy surrounding it. The announcement seemed incongruous given the ongoing concerns about website scraping.
While blocking GPTBot may provide some control over open internet content usage, its effectiveness in preventing LLMs from consuming non-paywall content remains uncertain. LLMs and other generative AI platforms have already utilized vast collections of public data for training. Google’s Colossal Clean Crawled Corpus (C4) dataset and Common Crawl, a nonprofit organization, are well-known sources of training data.
Experts suggest that if any data or content has been captured in previous scraping efforts, it is likely already incorporated into the training datasets used by OpenAI’s ChatGPT and other platforms. Although services like CommonCrawl allow for robots.txt blocks, website owners would have needed to implement such changes before the data was collected.
Even reputable outlets like VentureBeat find their information included in the C4 training data and available through the Common Crawl datasets.
The Legal and Ethical Challenge
Web scraping of publicly accessible data is generally considered a legal activity, as confirmed by the U.S. Ninth Circuit of Appeals. This ruling stated that web scraping does not violate the Computer Fraud and Abuse Act (CFAA).
However, the practice of data scraping for AI training purposes has faced criticism and legal challenges in recent times. OpenAI faced two lawsuits in July, one accusing them of unlawful copying of book text without consent or appropriate credit and compensation, and another alleging the collection of personal data in violation of privacy laws.
Furthermore, authors Sarah Silverman, Christopher Golden, and Richard Kadrey have filed lawsuits claiming that their published works were used to train LLMs without their consent. Other platforms like X and Reddit have also been involved in similar controversies, taking measures to restrict access and protect their datasets from web scraping attempts.
As the prevalence of AI data scraping increases, it becomes imperative to find ethical solutions that balance data accessibility and privacy concerns. The temporary restrictions implemented by X and Reddit, limiting access to content and raising API prices, reflect the challenges platforms face in combatting web scraping.
OpenAI’s partnership with NYU’s Ethics and Journalism Initiative is a step in the right direction. It highlights the importance of addressing the ethical challenges faced by journalists and content creators when utilizing AI technologies. However, it is crucial for OpenAI and other organizations to proactively engage in public discussions and adopt transparent policies regarding data scraping and the training of language models.
Finding common ground between the advancement of AI technology and respecting the rights of content creators is essential for fostering a more ethical and responsible AI ecosystem.
Leave a Reply