Why the Internet Is Starting to Feel Small to AI

Modern AI models improve with more and better training data. Scaling laws suggest that beyond a certain point, performance gains come from feeding models many more high-quality tokens, not just adding parameters. DeepMind’s Chinchilla paper showed that compute-optimal models should be trained on far more tokens than many early large language models received—on the order of tens of trillions of tokens for trillion-parameter models (DeepMind).

Meanwhile, researchers estimate that the supply of high-quality, diverse, human-written text on the public web is limited and could be effectively exhausted for frontier training in the near future without new sources. Analyses from Epoch AI suggest that high-quality language data is finite, and that the industry will need licensing, curation, and synthetic data to keep scaling responsibly (Epoch AI; MIT Technology Review).

It’s not just about volume. The open web contains duplicates, spam, low-quality content, and content with unclear rights. As models expand beyond text into images, audio, code, and video, the standards for quality and lawful use rise even higher.

Where AI Companies Are Getting Fresh Data

To keep improving, leading developers are lining up new pipelines that go beyond general web crawl datasets. The most visible shift is toward licensing and partnerships.

Licensing Deals and Partnerships

• News and Publishing: OpenAI has struck paid deals with outlets, including the Associated Press (AP), Axel Springer (Axel Springer), the Financial Times (FT), and News Corp (News Corp), allowing use of archives for training and real-time summaries in products.
• Community Platforms: Reddit and Stack Overflow now license content for training and safety research, with API-based access and brand attribution in AI experiences (Reddit; Stack Overflow).
• Media Libraries: Shutterstock provides licensed image, video, and metadata to multiple AI developers, part of a broader push to source lawful multimodal data (Shutterstock).

These deals aim to secure high-quality, rights-cleared content while compensating creators. They also help models stay current and minimize the risk of training on copyrighted or sensitive material without permission.

Curated and Domain-Specific Corpora

Another trend is the investment in curated datasets focused on code, scientific literature, legal and financial filings, and technical manuals. For many enterprise and safety-critical uses, a smaller, cleaner, and more relevant dataset can outperform a larger but noisier one. Techniques like retrieval-augmented generation help models leverage proprietary knowledge bases without incorporating all that data directly into pretraining.

The Legal and Ethical Crosswinds

As AI companies seek more data, they navigate unsettled legal territory regarding copyright, fair use, privacy, and platform rules.

• Copyright Lawsuits: The New York Times sued OpenAI and Microsoft over alleged unauthorized use of Times content in training and outputs, a case that could shape the boundaries of fair use for AI training (The New York Times). Visual model developers have also faced lawsuits from image libraries like Getty Images over training on copyrighted photos (Reuters).
• Robots.txt and Opt-Out: Some AI crawlers now honor publisher opt-outs via robots.txt and meta tags, including OpenAI’s GPTBot (OpenAI). But norms are evolving and not all scrapers comply.
• Regional Rules: In 2024, Meta paused plans to train generative AI on public posts in Europe after Ireland’s data protection regulator raised concerns, highlighting stricter EU privacy expectations (Reuters).
• Platform Terms: Training on content from platforms with restrictive terms remains contentious. Reports suggested Google used YouTube transcripts in training at least some systems, raising questions about compliance with YouTube’s policies; Google has disputed improper use (The Verge).

Regulators are weighing transparency requirements for training data and obligations to respect copyright. The EU is advancing the AI Act with rules on data governance and transparency for general-purpose models (European Parliament).

Will Synthetic Data Solve the Shortage?

Synthetic data is artificially generated content that can augment or replace human-created data for training. In language models, that includes model-generated explanations, reasoning traces, and new problem sets. For images and audio, it includes procedurally generated scenes, voice clones, and labeled variations.

When done well, synthetic data can boost performance, reduce bias, and help cover edge cases that are rare in the wild. It’s increasingly combined with human feedback and expert review, such as reinforcement learning from human or AI feedback and techniques like Constitutional AI (Anthropic).

However, there are limits. If models are trained primarily on their outputs, quality can degrade over generations, a risk sometimes called model collapse. Research has shown that repeatedly training on model-generated data can amplify errors and reduce diversity unless carefully managed with deduplication, grounding, and periodic infusions of human data (Shumailov et al.).

Takeaway: synthetic data helps, but it’s not a free replacement for diverse, high-quality human data. The winning recipe mixes licensed, curated, and synthetic sources with strong quality controls.

What This Shift Means for Businesses and Creators

If You Build or Buy AI Systems

• Prioritize Data Governance: Track provenance, rights, and consent. Ask vendors how datasets were sourced and filtered.
• Invest in Quality Over Sheer Size: Curated, domain-specific corpora and retrieval-augmented generation often outperform indiscriminate web crawls for enterprise use.
• Blend Real and Synthetic: Use synthetic data to cover gaps and stress-test models, but maintain a backbone of high-quality human data and rigorous evaluation sets.
• Monitor Drift and Contamination: Deduplicate, diversify sources, and refresh datasets to avoid training on repeated model outputs.

If You Publish or Create Content

• Know Your Options: Consider licensing content to AI providers or using platform tools to opt-out of scraping where available.
• Use Machine-Readable Signals: Robots.txt, meta tags, and emerging frameworks like C2PA can signal permissions and provenance (C2PA).
• Negotiate Value: Collective bargaining through publishers, labels, or creator groups is becoming a standard route to fair compensation.

What to Watch Next

• Data Markets: Expect more licensing deals, APIs, and brokered access to archives, forum data, and high-quality niche corpora.
• Transparency Rules: Policymakers may require clearer disclosures for training mixtures, data provenance, and opt-out mechanisms.
• Trust Signals: The adoption of content credentials and provenance standards could help trace what models were trained on and how outputs were generated.
• Multimodal Shift: As models lean more on audio and video, look for new partnerships with broadcasters, stock media libraries, and platforms.

Conclusion

The internet unlocked the last wave of AI breakthroughs, but it won’t single-handedly power the next. High-quality data is scarce, valuable, and increasingly governed by rights and norms. The frontier now is not just bigger models or more compute, but smarter data strategy: license what matters, curate ruthlessly, generate synthetic data responsibly, and be transparent about it. That’s how the industry keeps advancing while respecting creators and users.

FAQs

Are AI Companies Really Running Out of Training Data?

They are not running out of all data, but they are constrained by a shortage of high-quality, diverse, rights-cleared data at the necessary scale for frontier models. Analyses suggest the best public web text is limited, so companies are turning to licensing, curation, and synthetic data.

What Counts as High-Quality Data?

Human-written or expertly labeled content with low duplication, clear rights, diverse viewpoints, and strong signal-to-noise ratio. Examples include edited news archives, peer-reviewed papers, high-quality code, and well-moderated forums.

Is Synthetic Data Safe to Use for Training?

Yes, when used carefully. It works best as a complement to real data, with deduplication, grounding, and frequent evaluation on human-built test sets. Overreliance on model-generated data can degrade quality over time.

Can I Stop AI Companies from Using My Site or Content?

You may be able to signal restrictions with robots.txt and meta tags, use platform opt-out tools, or negotiate licenses. Effectiveness varies by crawler and jurisdiction, and legal norms are still evolving.

How Will Regulation Change Training Data Practices?

Expect more transparency and consent requirements, especially in the EU. The proposed rules for general-purpose AI models emphasize data governance and accountability, pushing companies toward licensed and documented datasets.

Sources

1. DeepMind – Training Compute-Optimal Large Language Models (Chinchilla)
2. Epoch AI – Will We Run Out of Data?
3. MIT Technology Review – AI’s Next Bottleneck Will Be Data, Not Compute
4. Associated Press – AP and OpenAI Announcement
5. Axel Springer – Strategic Partnership with OpenAI
6. Financial Times – FT and OpenAI Licensing Deal
7. News Corp – News Corp and OpenAI Partnership
8. Reddit – Partnership with OpenAI
9. Stack Overflow – Partnership with OpenAI
10. Shutterstock – AI Partnerships and Licensing
11. The New York Times – Lawsuit Against OpenAI and Microsoft
12. Reuters – Getty Images Lawsuit Against Stability AI
13. OpenAI – GPTBot and Robots.txt Guidance
14. Reuters – Meta Pauses AI Training in EU
15. The Verge – Did Google Train on YouTube Transcripts?
16. Anthropic – Constitutional AI
17. Shumailov et al. – The Curse of Recursion: Training on Generated Data
18. European Parliament – EU AI Act Overview
19. C2PA – Content Provenance and Authenticity