Library Guides: Text Mining & Computational Text Analysis: Home

What Is Text Mining?

"Text Mining is the discovery by computer of new, previously unknown information, by automatically extracting information from different written resources... The difference between regular data mining and text mining is that in text mining the patterns are extracted from natural language text rather than from structured databases of facts."

- from What is Text Mining? by Marti Hearst (2003)

Where Can I Find Data & Texts for Analysis?

TDM platforms like TDM Studio and Gale Digital Scholar Lab
Library datasets (listed in this guide)
Web sources (we've listed some of them on this guide)
To suggest or request data not listed on this guide, please email tdm-access@berkeley.edu.

Please Scrape Responsibly

Help us keep library databases available for everyone.
Before you programatically scrape a library database or website, check its terms of service, APIs, or contact the library for help.
Using Python, Selenium, or other programmatic tools to scrape database search results, no matter how carefully, can result in access being shut down for the entire campus.

What Should I Do? Webscraping Flowchart
If you're unsure how to legally access large quantities of text from particular publication or platform, check this handy flowchart!
Copyright and Text Mining Guide
Check this Scholarly Communications guide to ensure that what we're doing with potentially copyrighted texts complies with legal standards when we publish our results.

Software & Tools

Python
Python is a free open source and general-purpose programming language that often serves as a foundation for text analysis projects.
R
R is a free software environment for statistical computing and graphics. RStudio is a development interface for R featuring a console, syntax-highlighting editor that supports direct code execution, as well as tools for plotting, history, debugging and workspace management.
Mallet
MALLET is a Java-based package for statistical natural language processing, document classification, clustering, topic modeling, information extraction, and other machine learning applications to text.
Natural Language Toolkit (Python)
NLTK is a free open source platform for building Python programs to work with human language data.
the @unitedstates project
A shared commons of data and tools for accessing data from the United States government.

Digital Scholar Lab (Gale)
The Digital Scholar Lab has cloud-based tools for automatically performing common computational queries on and creating visualizations from content sets built with Gale primary source collections.

Primary source collections include: American Fiction
17th and 18th Century Burney Collection, American Civil Liberties Union Papers, 1912-1990, American Fiction, Archives Unbound, Archives of Sexuality & Gender, British Library Newspapers, The Economist Historical Archive, Eighteenth Century Collections Online, Indigenous Peoples: North America, The Making of Modern Law, The Making of the Modern World, Nineteenth Century Collections Online, Nineteenth Century U.S. Newspapers, Sabin Americana, 1500-1926, The Times Digital Archive, The Times Literary Supplement Historical Archive, U.S. Declassified Documents Online

Voyant
A cloud-based tool for performing some automated computational text analysis on documents that you upload.

OCR: Tools for Making PDFs and Images of Text Usable

Simple Jobs:

Library scanners will output OCR for most English language documents.

Complex Jobs: OCR Desktop

ABBYY FineReader: accurate, supports 190 languages
Tesseract OCR: process large corpora in bulk
Supported by Research-IT

Why OCR?

As a humanities researcher, you may have a large number of PDFs, photos of archival documents, or other images of text that are not yet machine readable.
Optical Character Recognition (OCR) converts images of scanned text into machine-readable text, so you can copy and paste, search, or edit.
Research-level Optical Character Recognition requires accuracy, multiple language support, and bulk processing.

Learn more

Natural Language Processing with Python by Steven Bird; Ewan Klein; Edward Loper
This book offers a highly accessible introduction to natural language processing, the field that supports a variety of language technologies, from predictive text and email filtering to automatic summarization and translation.
Text Analysis with R for Students of Literature by Matthew L. Jockers
Text Analysis with R for Students of Literature is written with students and scholars of literature in mind but will be applicable to other humanists and social scientists wishing to extend their methodological tool kit to include quantitative and computational approaches to the study of text.

Introduction to the tm Package: Text Mining in R
A short 2015 introduction to text mining in R by Ingo Feinerer. Includes data import, corpus handling, preprocessing, metadata management, and creation of term-document matrices.
Extracting Data from the Internet in Python
Course materials from Rochelle Terman's May 2016 D-Lab workshop.
Text Analysis in Python Intensive for Digital Humanities
Course materials from Teddy Roland's May 2016 D-Lab workshop.

Campus resources

Computational Text Analysis Working Group (D-Lab)
This group of graduate students, faculty and researchers meet twice a month during the semester to share research, provide technical workshops, and offer space to support ongoing analysis.
D-Lab Trainings
D-Lab helps Berkeley faculty, staff, and graduate students move forward with world-class research in data intensive social science.

Secondary menu

Text Mining & Computational Text Analysis

For help