Text mining centers on identifying patterns and trends in unstructured texts. This often involves using a program or software to “read” text files and provide data about them, including data on word frequencies, common word patterns, tone indicators, and more. It is sometimes referred to as a "distant reading" method, in which you take a step back to identify patterns in language across a large group of texts. 

Many research questions and methods fall within the scope of text and data mining, including: 

• Identifying word frequencies
• Concordance (what passages mention specific key terms)
• Keyness (how often key terms appear in certain texts when compared to others) 
• Topic modelling (grouping key terms together to identify common themes and topics) 
• Named entity recognition (identifying names of people, places, things across texts)
• Sentiment analysis (identifying positive or negative tone)

Why do text mining?

Text mining helps researchers detect patterns and connections in large volumes of textual material.

According to researcher Marti Hearst, "In text mining, the goal is to discover heretofore unknown information, something that no one yet knows and so could not have yet written down." Text mining enables researchers to draw conclusions from large volumes of material they would not be able to otherwise read, synthesize, and incorporate into their scholarship.

Researchers in fields ranging from biological sciences to the humanities have begun using text mining to detect patterns and discover unknown information. 

If you have questions about text mining, reach out to Scholarly Communications and Publishing (scp@library.illinois.edu). ​

Except where otherwise indicated, original content in this guide is licensed under a Creative Commons Attribution (CC BY) 4.0 license. You are free to share, adopt, or adapt the materials. We encourage broad adoption of these materials for teaching and other professional development purposes, and invite you to customize them for your own needs.