Skip to main content

University Library, University of Illinois at Urbana-Champaign

Open Educational Resources Promotions Guide: Measuring Impact

OER Promotions Guide by Janet Swatscheno as part of the SPARC Open Education Leadership Program

Measuring Impact

There are several ways to measure the impact of textbooks. Authors and publishers of traditional textbooks track certain metrics like books sales, royalties, course adoptions, and e-book downloads. Creators of open educational resources often have different goals in mind than commercial publishers and may want to consider alternative types of metrics. For example, the author of an open textbook might be more concerned with the amount of savings to students rather than book sales. Deciding which metrics are important to track should not be a last minute decision. Instead, it is important to consider metrics closer to the beginning of a project for a few reasons. One reason to consider metrics early is because authors and publishers might not be interested in the same metrics. For example, an author is concerned with the success and adoption of their individual textbook, while a publisher is more concerned with the overall sustainability and success of their program. Also, authors are better positioned to collect certain metrics while publishers and libraries are better positioned to collect others.

Metrics Resources

Metrics Toolkit

The Metrics Toolkit was developed for researchers and evaluators to provide guidance for demonstrating and evaluating claims of research impact. While it was primarily designed to address metrics for original research, it includes useful information about different types of altmetrics that could be useful to OER creators and publishers. The Toolkit also identifies which services can provide particular metrics such as Google Analytics, Altmetrics, and others.

Open Syllabus Project

Open Syllabus Project is an online database of university course syllabi. The website scrapes university websites for syllabi and lets instructors submit syllabi. While it is a new tool, it may prove to be a convenient way to track and measure course adoption.  

36 Indicators of OER Impact

List of the following 36 indicators of OER Impact: Peer review, physical book sales, student savings, downloads, website hits, adoptions, adaptions, media mentions, quotes in media, course enrollment, course completion, rabble rousing, social networking contacts, increased diversity, grant $, student success, mention by policy maker, angry letters from important people, meetings with important people, protests/demonstrations/sit-ins, trending in social media, student surveys, student testimonials, trust/reputation, blog mentions, faculty recommendations, faculty award/prize, influencing curriculum creation, influencing textbook market, participating in public education programs, invitations to present, invitations to consult, invitations to evaluate, change in policy, change in processes or behavior, and thank you cards from students and parents

36 Indicators of OER Impact by Janet Swatscheno, licensed CC BY 4.0. This graphic is adapted from 56 Indicators of Impact by Holbrook, J Britt, Kelli R. Barr, and Keith Wayne Brown. 2013. Figshare. https://doi.org/10.6084/m9.figshare.707081.v3. Licensed under CC BY 4.0

In the article “We Need Negative Metrics Too”, J. Britt Holbrook, Kelli R. Barr and Keith Wayne Brown identify a number of indicators of impact. Many of the indicators are standard measures of success such as book sales, but the list also takes into account alternatives like policy change and student success. Library publishers should consider the types of quantitative and qualitative data to collect about the resources they publish.

 

Metrics to consider for open educational resources:

  • Downloads
  • Page visits
  • User engagement (e.g. web page visit duration)
  • Sales of physical copies
  • Course adoptions
  • Adaptations and remixes of the resource
  • Reviews
  • Peer review
  • Student surveys and testimonials
  • Overall cost savings for students
  • Effect on the textbook market

The BCcampus Open Textbook statistics page is a good example of an OER publishing program collecting and sharing metrics about the books they’ve published. It includes information about student savings, known adoptions, and the number of reviews for each book.