Skip to Main Content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

Update: Moffitt Library is closed for seismic work, but most other libraries are open. Learn more.

Text Mining & Computational Text Analysis

Where Can I Find Data & Texts?

Right here! The Library offers a wealth of texts and data for your TDM research. We have also included datasets available freely on the web. Use the navigation menu to browse by type of data. Data include:

To suggest or request data not listed on this guide, please email tdm-access at berkeley.edu.

What about Copyright?

Can I Web Scrape?

  • Help us keep library databases available for everyone.
  • Using Python, Selenium, or other programmatic tools to scrape database search results, no matter how carefully, can result in access being shut down for the entire campus. That’s because license agreements and other laws affect what and how you can mine.
  • Before you programmatically scrape a library database or website, make sure you’re following the tips in this guide, which walks you through everything you need to know. And if you’re wondering whether you should use an API, you can check out this flowchart.
  • Have more questions? Contact tdm-access at berkeley.edu for help. 

Can I Do TDM on Material under Copyright?

Our Copyright and Text Mining Guide explains everything you need to know about doing TDM on material under copyright or with licensing restrictions, both when you are running your analyses and when you are publishing your results. 

Can I Do TDM on eBooks & DVDs Protected by DRM?

Some materials may have an added technological protection layer of "digital rights management" (DRM). There are some situations in which it’s permitted to “break” eBook and DVD DRM to conduct TDM, but there are very specific rules you must follow. Check out the DRM parameters in our TDM law and policy guide. And if you have any questions, please get in touch at tdm-access at berkeley.edu.

Software & Tools

Programming

Cloud-Based Tools

OCR: Tools for Making PDFs and Images of Text Usable

Simple Jobs:

  • Library scanners will output OCR for most English language documents. Results with other languages may vary.

Complex Jobs

  • ABBYY FineReader: accurate, supports 190 languages
  • Tesseract OCR: process large corpora in bulk

Why OCR?

  • As a humanities researcher, you may have a large number of PDFs, photos of archival documents, or other images of text that are not yet machine readable.
  • Optical Character Recognition (OCR) converts images of scanned text into machine-readable text, so you can copy and paste, search, or edit.
  • Research-level Optical Character Recognition requires accuracy, multiple language support, and bulk processing.

Learn about TDM

What Is Text Data Mining (TDM)?

"Text Mining is the discovery by computer of new, previously unknown information, by automatically extracting information from different written resources... The difference between regular data mining and text mining is that in text mining the patterns are extracted from natural language text rather than from structured databases of facts."

- from What is Text Mining? by Marti Hearst

Guide Books and Online Tutorials

Workshops and Training On Campus

DH on Campus