Skip to Main Content

University Library

LibGuides

Finding Text Data Sets

Research From Library Databases

The University of Illinois has contract agreements with several scholarly publishing vendors to conduct text mining. These vendors are listed below, along with instructions on how to get started. If you have a question about a vendor not listed below, or otherwise need assistance accessing the data, please contact the Scholarly Communications and Publishing Department at scpub@library.illinois.edu.

The following pages link directly to the scholarly publishing databases, and do not automatically sign you in via the University of Illinois' institutional login. To login to these databases, you can either find and log in via the "Institutional Login" pages for these databases, or you can find the database in the library's A-Z Databases page. 

ACM  

  • The Association for Computing Machinery’s Digital Library provides research articles, books, conference proceedings, and magazine articles on topics in computer and data science and technology.
  • Data Mining Instructions: Data mining requests are approved on a case-by-case basis. Contact permissions@acm.org describing your specific use case and they will make a determination.
  • Please note that, per ACM policies the following uses are not permitted under any circumstances:
    • Sharing access to ACM DL materials with individuals or organizations outside the subscribing institution
    • Using scripts, spiders or other robotic activity to automatically download articles or harvest metadata form the ACM Dl. Such activity is a serious violation of the ACM DL's Terms of Use and will result in the temporary or permanent termination of downloading rights for the subscribing institution. 
    • The posting of ACM DL materials to any 3rd party server or peer-to-peer network without the prior written permission of the ACM.    

Elsevier 

  • Elsevier provides research articles and books focused on fields of science and technology, including engineering, medicine, social science, and GIS. Notable databases include INSPEC, ScienceDirect, Scopus, and Engineering Village.
  • Data Mining Instructions: Access the Elsevier API, and review their data mining policy. Each researcher must create an Elsevier account and register for their own API key. Data delivered in XML format.

JSTOR 

  • JSTOR is a collection of research articles and books dating back to the earliest publications in humanities fields, especially language, literature, history, and philosophy. 
  • Data Mining Instructions: JSTOR Data for Research provides an API that can be used to retrieve metadata and reference information for up to 25,000 documents. For researchers needing to conduct full-text analysis OR retrieve more than 25,000 documents, contact JSTOR directly to request the data set. Data delivered in XML or plain-text format.

MathSciNet

  • MathSciNet® is an electronic publication of the American Mathematical Society (AMS) offering access to a  searchable database of reviews, abstracts and bibliographic information for much of the mathematical sciences literature.
  • Please note that, per AMS policies the following uses are not permitted under any circumstances:
    • Download substantial portions of the database.
    • Conduct automated searching or downloading, by use of scripted searches, robots, spiders, crawlers, or otherwise;
    • Make MathSciNet available to any third party, whether by telephone link, password sharing, permitting access through your computer, or by other similar or dissimilar means or arrangements.
  • Any use not expressly permitted by AMS may be made only with specific written permission from the AMS; inquiries should be addressed to Executive Editor, Mathematical Reviews, mrexec@ams.org.

Springer

  • Springer is the provider of BioMed Central, an open access database of academic articles related to science, technology, and medicine. Social scientists may be interested in researching topics like public health, substance abuse, and health care policy using this source.
  • Data Mining Instructions: Springer provides multiple APIs depending on the researcher's needs. Access the Springer APIs. Alternatively, you may download articles directly from the website. See their full policy and how-to guide. Data delivered in PDF, HTML (if available), XML, or JSON format.

Wiley

  • Wiley provides a variety of databases, notably Anthrosource for current and archived issues of publications from the American Anthropological Association, and the International Studies Encyclopedia. They also provide AGU Digital Library for earth science, and Organic Reactions related to chemistry.
  • Data Mining Instructions: Use the CrossRef data mining service, which includes thousands of publishers, including Wiley. See more of Wiley's terms and conditions here. Data delivered in JSON format.

Research From Open Access Publishers and Indexes

Listed below are some Open Access Journals that publish research publicly. Most articles from the libraries below are related to science and technology.

arXiv

  • arXiv is an electronic archive for research articles in various STEM fields, including physics, mathematics, computer science, biology, finance, engineering, and economics.
  • Data Mining Instructions: Use the arXiv API to access arXiv data, search, and linking facilities. The API can only be used to download metadata, not full-text. No key is required. To access full-text articles in bulk, the researcher must purchase a license from Amazon S3. Data delivered in Atom XML format.

Public Library of Science

  • Public Library of Science (PLoS) is a publisher of research articles in various fields of science, such as biology, medicine, computational biology, genetics, and disease.
  • Data Mining Instructions: PLoS offers two APIs for data retrieval. The Article-Level Metrics API retrieves data regarding an article’s usage statistics to demonstrate its reach. The Search API provides the ability to query PLoS content across their journals. Data delivered in XML or JSON format. An API key is required to access either API. 

PubMed

  • PubMed contains 27 million citations for research articles on biomedicine from various research journals and online books. Full-text of each article may be available from its original publisher, but not through this database itself.
  • Data Mining Instructions: Text and data mining is limited due to copyright restrictions, but some collections are available for mining. The largest collection is the Open Access Subset. Data delivered in XML format. 
  • PubMed has also published a list of tools that can be used for data mining.

CORE

  • CORE is an initiative in the UK to harvest and maintain metadata and full-text content from Open Access journals and repositories across the world.
  • Data Mining Instructions: Access the CORE API. Data delivered in JSON format.

Research From Digital Libraries

Biodiversity Heritage Library

  • The Biodiversity Heritage Library is an online collection of scientific texts focused on natural history, biology, botany, and other natural sciences. It contains both scholarly journal articles and books.
  • Data Mining Instructions: Request an API key to access the BHL API. Data delivered in JSON or XML format.