Library Guides: Text Mining & Computational Text Analysis: Sources

Books

HathiTrust Research Center (HTRC)
Nearly 14 million books from the HathiTrust Library are currently available for analysis, offering various levels of immediate access. Check the HTRC tab on this guide for more information to help you get started.
Project Gutenberg (Mirror sites)
Project Gutenberg hosts over 50k ebooks, most of which are older books in the public domain. If you want to download more than about 100 books/day, use one of the mirror sites listed from the link above.

Newspapers & Magazines

Chronicling America (Library of Congress, 1836-1922)
The Chronicling America API provides access to information about historic U.S. newspapers and millions of digitized newspaper pages and their OCR data is available for bulk download. See the full list of digitized newspaper titles (1836-1922) for more information.
Godey's Lady's Book (1830-1865)
19MB zip file containing an XML document for every full text article from Godey's Lady Book, Parts I-III (Accessible Archives). The magazine was intended to entertain, inform and educate the women of America and covers fashion, biographical sketches, articles about mineralogy, handcrafts, female costume, the dance, equestrienne procedures, health and hygiene, recipes, remedies, and the like.
Los Angeles Sentinel (1934-2005)
ProQuest Historical Newspaper data for the los Angeles Sentinel 1934 - 2005, OCR'ed content (results from automated Optical Character Recognition - quality varies).
New York Times Annotated Corpus (1987-2007)
Linguistic Data Consortium's NY Times Corpus contains over 1.8 million articles from the New York Times between January 1, 1987 and June 19, 2007. The corpus includes: over 1.8 million articles (excluding wire service articles); over 650,000 article summaries; human- and algorithm-assigned tags drawn from a normalized indexing vocabulary of people, organizations, locations and topic descriptors; Java tools for parsing corpus documents from .xml into a memory resident object.
NY Times APIs (1851-present)
The Article Search API provides access to headlines, abstracts, lead paragraphs and more (but NOT full-text articles) from the New York Times, 1851 to present.
Pennsylvania Gazette (1728-1800)
105MB zip file containing an XML document for every full text article from the Pennsylvania Gazette (Accessible Archives). This paper provides a first-hand view of colonial America, the American Revolution and the New Republic, offering important social, political and cultural perspectives of each of these periods.
San Francisco Chronicle Archive (1865 - 1922)
Request access from the D-Lab for ProQuest Historical Newspaper data for the San Francisco Chronicle (1865-1922). Note that the quality of the OCR (results from automated Optical Character Recognition) is quite low and varies from paper to paper.
Times of India (1838 - 2005)
ProQuest Historical Newspaper data for the Times of India from 1838-2005. OCR content quality varies.

Available by request: ProQuest Historical Newspapers

Researchers may request OCR full text from any of the following specific newspapers for a specific time period, though requests will require significant processing time:

Chicago Defender (1910-1975)‎ - requested Spring 2018
The Baltimore Afro-American (1893-1998)

Scholarly journals

CORE: Open Access Research Papers
CORE provides a central API to access full content from tens of thousands of openly available scientific publications from thousands of OA repositories. They also provide full datasets by request.
Elsevier (ScienceDirect, Scopus)
Researchers can text mine UCB-subscribed journals and books published by Elsevier on the ScienceDirect full-text platform. Sign up for a developer account to use the Elsevier APIs for non-commercial purposes, and make sure to query the API from UC Berkeley IPs to ensure full access. You can also use the APIs to access citation data and abstracts from scholarly journals indexed by Scopus. For more information, see their Text Mining documentation.
PLOS (Public Library of Science)
Python tool for downloading/updating/maintaining a repository of all PLOS XML article files. Use this program to download all PLOS XML article files instead of doing web scraping. See also: PLOS APIs to query content from the seven open-access peer-reviewed journals from the Public Library of Science using any of the twenty-three terms in the PLOS Search.
ProQuest databases
ProQuest provides limited access to an "XML Gateway/Z39.50 Federated Search interface," that allows individual researchers to program searches across some, but not all, PQ databases, and get the result sets back as XML. Login and password access is only available by request at tdm-access@berkeley.edu.
Springer's Digital Content
"Individual researchers are encouraged to download subscription and open access content for TDM purposes directly from the SpringerLink platform. No registration or API key is required. Full-text content can be accessed easily and programmatically at friendly URLs based on the content’s Digital Object Identifier (DOI)." via Springer's Text and Data Mining Policy.
JSTOR Data for Research
Data for Research allows you to download word frequencies, citations, key terms, and ngrams for up to 25,000 JSTOR articles at a time, or to easily submit requests for larger sets of articles. See also:
- JSTORr, a package of simple functions in R to work with DFR output.
- JSTOR's Text Analyzer, a reverse search engine that analyzes documents that you upload (your own, or other articles) to find related materials in JSTOR.
- Public domain and OA datasets include full OCR text from early journals and current academic press open access titles

Citations & Metadata

Elsevier (ScienceDirect, Scopus)
Researchers can text mine UCB-subscribed journals and books published by Elsevier on the ScienceDirect full-text platform. Sign up for a developer account to use the Elsevier APIs for non-commercial purposes, and make sure to query the API from UC Berkeley IPs to ensure full access. You can also use the APIs to access citation data and abstracts from scholarly journals indexed by Scopus. For more information, see their Text Mining documentation.
Library of Congress: 25 million bibliographic metadata records
The LOC release of 25 million MARC records for free bulk download. MARC (Machine Readable Cataloging Records) is a international metadata standard for the representation and communication of bibliographic and related information.
NY Times APIs
The Article Search API provides access to headlines, abstracts, lead paragraphs and more (but NOT full-text articles) from the New York Times, 1851 to present.
Open Academic Graph
Downloadable datasets for citations drawn from two large academic graphs: Microsoft Academic Graph (MAG) and AMiner.
PubMed and NLM: Data Guide
A guide to using this API, called E-Utilities, to access citation data for medical journal literature in PubMed and other NCBI databases, including the National Library of Medicine Catalog, MeSH, Gene, and PMC (PubMed Central).

Government Documents

Congress and Government Information APIs (Sunlight Foundation)
Sunlight Foundation provides APIs to a variety government information sources including: Congress API v3: information on legislators, districts, committees, bills, votes, as well as real-time notice of hearings, floor activity and upcoming bills. Open States API: Information on the legislators and activities of all 50 state legislatures, Washington, D.C. and Puerto Rico. Real-Time Federal Campaign Finance API: A JSON and CSV API that delivers up-to-the-minute campaign finance information on federal candidates, committees, PACs and other groups that file electronically with the Federal Election Commission.
CourtListener APIs and Bulk Legal Data
Opinions, docket files, and more from 420 courts.
ProQuest Congressional Record Dataset
Full text of verbatim debates of the US Senate and House of Representatives, with subject heading, speaker, bill and public law number search facilities available. Contains both the Congressional Record (1873 - the late 1990s) and previous titles including the Congressional Globe (1833-1873), the Register of Debates in Congress (1824-1837), and the Annals of Congress (1789-1824).

Linguistic Corpora

BYU Corpus Data
UC Berkeley has licensed access to the full-text corpus data from BYU's COCA: Corpus of Contemporary American English, COHA: Corpus of Historical American English, and GloWbE: Global Web-based English.
Linguistic Data Consortium Corpora
The LDC collects language data from both written texts and transcriptions of speech, in various languages, to support corpus linguistics. The Library subscription begins from 2016, and the Library is currently working to migrate legacy collections from the Berkeley Language Center. If you don't see an LDC dataset in OskiCat, search the LDC catalog and email tdm-access@berkeley.edu with any questions.
New York Times Annotated Corpus (1987-2007)
Linguistic Data Consortium's NY Times Corpus contains over 1.8 million articles from the New York Times between January 1, 1987 and June 19, 2007. The corpus includes: over 1.8 million articles (excluding wire service articles); over 650,000 article summaries; human- and algorithm-assigned tags drawn from a normalized indexing vocabulary of people, organizations, locations and topic descriptors; Java tools for parsing corpus documents from .xml into a memory resident object.
Penn parsed corpora of historical English
Available in the Library Data Lab (189 Doe Annex).
The Penn Historical Corpora, including the Penn-Helsinki Parsed Corpus of Middle English, second edition (PPCME2), the Penn-Helsinki Parsed Corpus of Early Modern English (PPCEME), and the Penn Parsed Corpus of Modern British English (PPCMBE), are syntactically annotated corpora of prose text samples of English from the indicated time periods.

Literature

Beckett Digital Manuscript Project
Digital "genetic" editions of the original manuscripts for five major works by Samuel Beckett, featuring full TEI/XML downloads.
more...

The purpose of the Beckett Digital Manuscript Project is to reunite the manuscripts of Samuel Beckett's works in a digital way, and to facilitate genetic research: the project brings together digital facsimiles of documents that are now preserved in different holding libraries, and adds transcriptions of Beckett's manuscripts, tools for bilingual and genetic version comparison, a search engine, and an analysis of the textual genesis of his works.

Delpher (Dutch language resources)
Dutch newspapers, books, journals and radio bulletins available in full-text, along with rich datasets, APIs and other digital humanities tools for interaction.
HathiTrust Research Center (HTRC)
Nearly 14 million books from the HathiTrust Library are currently available for analysis, offering various levels of immediate access. Check the HTRC tab on this guide for more information to help you get started.
Project Gutenberg (Mirror sites)
Project Gutenberg hosts over 50k ebooks, most of which are older books in the public domain. If you want to download more than about 100 books/day, use one of the mirror sites listed from the link above.

Social Media

Blogger Corpus (2004)
The collected posts of 19,320 bloggers gathered from blogger.com in August 2004. The corpus incorporates a total of 681,288 posts.
reddit APIs
Access data from posts, threads, comments, users and more from reddit and subreddits.
Historical Reddit data has been collected at http://files.pushshift.io/reddit/ as monthly CSV downloads.
Twitter Streaming APIs
Public streams provide access to public data flowing through Twitter. Suitable for following specific users or topics, and data mining. You can also access single-user streams, containing roughly all of the data corresponding with a single user’s view of Twitter.
Yelp API
Access to business data, including location, photos, Yelp rating, price levels, hours of operation, and types of transactions. Also includes a Review API, which returns up to 3 review excerpts for a business. See also: the Yelp Dataset Academic Challenge.

Historical & Archival Collections

Beckett Digital Manuscript Project
Digital "genetic" editions of the original manuscripts for five major works by Samuel Beckett, featuring full TEI/XML downloads.

The purpose of the Beckett Digital Manuscript Project is to reunite the manuscripts of Samuel Beckett's works in a digital way, and to facilitate genetic research: the project brings together digital facsimiles of documents that are now preserved in different holding libraries, and adds transcriptions of Beckett's manuscripts, tools for bilingual and genetic version comparison, a search engine, and an analysis of the textual genesis of his works.

Chronicling America (Library of Congress)
The Chronicling America API provides access to information about historic U.S. newspapers and millions of digitized newspaper pages and their OCR data is available for bulk download. See the full list of digitized newspaper titles (1836-1922) for more information.
Godey's Lady's Book (1830-1865)
19MB zip file containing an XML document for every full text article from Godey's Lady Book, Parts I-III (Accessible Archives). The magazine was intended to entertain, inform and educate the women of America and covers fashion, biographical sketches, articles about mineralogy, handcrafts, female costume, the dance, equestrienne procedures, health and hygiene, recipes, remedies, and the like.
HathiTrust Research Center (HTRC)
Nearly 14 million books from the HathiTrust Library are currently available for analysis, offering various levels of immediate access. Check the HTRC tab on this guide for more information to help you get started.
Los Angeles Sentinel (1934-2005)
ProQuest Historical Newspaper data for the los Angeles Sentinel 1934 - 2005, OCR'ed content (results from automated Optical Character Recognition - quality varies).
Pennsylvania Gazette (1728-1800)
105MB zip file containing an XML document for every full text article from the Pennsylvania Gazette (Accessible Archives). This paper provides a first-hand view of colonial America, the American Revolution and the New Republic, offering important social, political and cultural perspectives of each of these periods.
Project Gutenberg (Mirror sites)
Project Gutenberg hosts over 50k ebooks, most of which are older books in the public domain. If you want to download more than about 100 books/day, use one of the mirror sites listed from the link above.
San Francisco Chronicle Archive (1865 - 1922)
Request access from the D-Lab for ProQuest Historical Newspaper data for the San Francisco Chronicle (1865-1922). Note that the quality of the OCR (results from automated Optical Character Recognition) is quite low and varies from paper to paper.

Available by Request:

Adam Matthew Digital
Contact History Librarian Jennifer Dorner (dorner@berkeley.edu) to request access to OCR text and full metadata from any of Adam Matthew Digital's primary source databases.

Gale Digital Collections
Request access to Gale content for text analysis purposes, including access to OCR text from databases like the Eighteenth Century and Nineteenth Century Collections Online, as well as content from Gale’s newspaper archives. See Gale's FAQ (pdf) or brief description for more information. Email tdm-access@berkeley.edu.

Data Repositories

Awesome Public Datasets
Check out the Natural Language category for a list of text corpora and ngrams for text analysis.
Data.gov
Catalog of hundreds of thousands of public data sets created at the city, state and federal levels.

Inter-university Consortium for Political and Social Research (ICPSR)
Some datasets from the ICPSR include corpora assembled to support data analyses, and include sources such as survey text, text messages, the Congressional Record, political speeches and more.

Direct download access to data sets requires the creation of a personal account. In addition, analysis of ICPSR data sets requires the use of specialized software. For more information on this process, please consult the ICPSR Get Help page or schedule an appointment with the Library Data Lab.

Programmable Web API Directory
Search over 15,000 APIs, or browse by categories, such as:
reddit Datasets
A subreddit for sharing and discussing datasets.

Please scrape responsibly

Copyright and Text Mining Guide
Using Python, Selenium, or other programmatic tools to scrape database search results (even cleverly) can result in access being shut down for the entire campus. Check this Scholarly Communications guide to ensure that what we're doing with potentially copyrighted texts complies with legal standards when we publish our results.
Access Flowchart
If you're unsure how to legally access large quantities of text from particular publication or platform, check this handy flowchart!

For help

tdm-access@berkeley.edu
Send questions about text and data mining access to library resources to this shared email above, which brings together librarians and campus partners with subject, copyright, technical, and licensing expertise.

For help with text mining tools and software, check out the D-Lab.
Questions and suggestions related to this guide can go to Cody Hennesy.

What is text mining?

"Text Mining is the discovery by computer of new, previously unknown information, by automatically extracting information from different written resources... The difference between regular data mining and text mining is that in text mining the patterns are extracted from natural language text rather than from structured databases of facts."

- from What is Text Mining? by Marti Hearst (2003)

Secondary menu

Text Mining & Computational Text Analysis: Sources