Precision, accuracy and sensitivity

Searches can be evaluated based on their recall [aka sensitivity] (number of items retrieved), precision [aka specificity] (the relevancy of the items retrieved) and accuracy [aka precision] (the proportion of all results that are within scope). You can illustrate these terms by thinking of a target: 

Image credit: "Precision illustration." https://byjus.com/maths/precision/. Downloaded 9/17/2024

In this illustration, accuracy is shown as proximity to the center of the target, which precision is shown as the consistency of clustering of the darts. In the left hand illustration, both the accuracy and precision lead to a highly clustered, centered result. In the middle illustration, all results are clustered but they do not contain the desired outcome. In the right illustration, the darts are inconsistently spaced across the target, and do not contain the desired outcome.

In expert searching, we desire to create a search set that is sensitive or has high recall (meaning that we retrieve all relevant results), with good accuracy (results are on topic), and good precision (results have similar desired metadata). There are a number of factors that can lead to the desired outcome of sensitive, precise and accurate searches, including critical evaluation of keywords, appropriate use of special characters and Boolean logic, and matching selection of the database to the search that you are conducting. First and foremost, an in-depth search should balance recall and precision to maximize accuracy.

Planning a basic search

• Decide what your topic is and what results you would like to see.
  • What is the research question?
  • How do I know what a relevant result may look like?
• Determine the individual concepts that are related to this search.
  • Most topics contain 2 to 4 concept clusters of synonyms.
• Mine your keywords.
  • What words must  a record contain in order to be relevant?
  • Is the jargon or vocabulary appropriate for the search content? (Is it medical terminology? Patent legalese? etc.)
  • Does the jargon or vocabulary match the types of information you are searching for? Example: "Kidney cancer" may return sufficient precision in a general academic database, but "renal neoplasm" OR "kidney neoplasms" OR "renal carcinoma" OR "mesoblastic nephroma" OR "Wilms Tumor" may be needed in medical databases like PubMed for appropriate accuracy.
• Perform the search in concept blocks.
  • Use a Boolean "OR" to combine all synonyms and search that string in the database. Do this for each concept block.  When all concept blocks are searched, combine each of the concept blocks using Boolean AND ensuring that each of the concept blocks has a set of parentheses around it to maintain order of operations. Hint: think mathematical order of operations. The database will compute what's inside the parentheses first. Then move to on.

A visual tool to support creation of search strings

Worksheets can be used to work through the preparatory steps prior to beginning your search. This planning process allows the searcher to further clarify concepts, keywords/controlled vocabulary and search syntax in one interface.

Input synonyms of terms for one single concept in the same column. Combine all synonyms within the single concept cluster with OR.  Combine concepts with OR first.  Add parentheses around the synonyms of the single concept. Then AND concept clusters together.

Precision vs sensitivity in designing searches

You can design your search to target either precision or sensitivity in the initial phases of your search.

To design for precision (highly targeted results), conduct a search of the concepts you have brainstormed. Open articles that are aligned with your topic and look for the controlled vocabulary that is used assigned to that record. 

Add relevant controlled vocabulary to your concept clusters and re-run the search.

OR

Go to the reference list at the end of the article and gather citations to other articles that are closely related to your concepts. Search those citations, look at the assigned vocabulary, and add new terms to your concept cluster. Re-run the search in order to increase precision.

To design for sensitivity (eliminate volume and increase precision gradually), begin your search generally with your known concept clusters. Then use faceted searching to eliminate irrelevant results with the goal of increasing precision as you progress through the search. Useful facets include those for controlled vocabulary, date limiters, author, author affiliation, and classification codes such as NAICS or SIC codes or Patent classification system codes.

Increase database search precision

Precision (the quality of retrieving as many relevant resources as possible without retrieving so many that the search results are untenable) is an important factor in planning your searches. Databases do not automatically understand a phrase correctly.

Basic search

Databases read search string prompts from left to right, unless explicitly told to do otherwise. In expert searching, we use parentheses () to tell a database the correct order in which the database should read the search string.

Automatic term mapping

Many databases also use Automatic Term Mapping to understand / interpret your search string. So, the above search is interpreted by PubMed as follows:

Advanced search

Using the advanced search interface, a search grid helps walks users through the creation of a logical order for the search string, using Boolean logic, parentheses, and more. In the advanced search interface, the searcher creates a search string in a similar manner to computer code, with the syntax and search string nested within parentheses to accurately reflect the intent of the searcher.  

Example

Picture it: 4th grade

You are in your classroom and your teacher writes the following on the board:

{3 x [10/(6-4]} + 2

This is functionally the same type of expression as {defibrillat* AND [arrythmia NOT (torsade OR "atrial fibrillat*")]} AND sensor*

The first rule of evaluating expressions is start from the inside out. The logic within the innermost brackets is carried out first, then move outwards until all brackets have been resolved, and then complete the logic as remains from left to right.

In the case of our two examples, 6-4 is resolved first, yielding 2. Similarly, a set containing all articles with the terms "torsade" OR "atrial fibrillat* is created first.

Next, 10/2 is calculated, yielding 5.  Similarly the set of articles with the term arrythmia, excluding the contents of our torsade/afib set is found.

Then, 3*2 is calculated, yielding 6. For our keywords, we combine the concept of defibrillator or defibrillation with the concept of arrythmias.

Finally, 6+2 resolves our expression. We arrive at the answer of 8.  Similarly, the defibrillator AND arrythmia set is combined through a logical AND with sensor*, resulting in a search set that contains all aspects of our concept clusters in the order specified. 

Some databases recognize all three bracket symbols. Others do not.  You will need to adjust your search syntax based on the capabilities of the given database that you are in.

Some databases have syntax checkers, similar to spell checking, to ensure that your search strings have correctly used logic. PubMed signifies problems with a red exclamation point to the left the search results in the Advanced search page. Other databases do not check your search syntax at all.

Always count your brackets.  There should always be an even number of brackets.

You can also use a notepad program such as Notepad++ to type out your search string, indenting in your search in every place a new bracket is introduced. This allows you to visually account for all brackets with greater clarity.