accuracy reliability and validity in scientific experiments

Validity, Accuracy and Reliability Explained with Examples

This is part of the NSW HSC science curriculum part of the Working Scientifically skills.

Part 1 – Validity

Part 2 – Accuracy

Part 3 – Reliability

Science experiments are an essential part of high school education, helping students understand key concepts and develop critical thinking skills. However, the value of an experiment lies in its validity, accuracy, and reliability. Let's break down these terms and explore how they can be improved and reduced, using simple experiments as examples.

Target Analogy to Understand Accuracy and Reliability

The target analogy is a classic way to understand the concepts of accuracy and reliability in scientific measurements and experiments. 

Accuracy refers to how close a measurement is to the true or accepted value. In the analogy, it's how close the arrows come to hitting the bullseye (represents the true or accepted value).

Reliability  refers to the consistency of a set of measurements. Reliable data can be reproduced under the same conditions. In the analogy, it's represented by how tightly the arrows are grouped together, regardless of whether they hit the bullseye. Therefore, we can have scientific results that are reliable but inaccurate.

• Validity  refers to how well an experiment investigates the aim or tests the underlying hypothesis. While validity is not represented in this target analogy, the validity of an experiment can sometimes be assessed by using the accuracy of results as a proxy. Experiments that produce accurate results are likely to be valid as invalid experiments usually do not yield accurate result.

Validity refers to how well an experiment measures what it is supposed to measure and investigates the aim.

Ask yourself the questions:

• "Is my experimental method and design suitable?"
• "Is my experiment testing or investigating what it's suppose to?"

For example, if you're investigating the effect of the volume of water (independent variable) on plant growth, your experiment would be valid if you measure growth factors like height or leaf size (these would be your dependent variables).

However, validity entails more than just what's being measured. When assessing validity, you should also examine how well the experimental methodology investigates the aim of the experiment.

Assessing Validity

An experiment’s procedure, the subsequent methods of analysis of the data, the data itself, and the conclusion you draw from the data, all have their own associated validities. It is important to understand this division because there are different factors to consider when assessing the validity of any single one of them. The validity of an experiment as a whole , depends on the individual validities of these components.

When assessing the validity of the procedure , consider the following:

• Does the procedure control all necessary variables except for the dependent and independent variables? That is, have you isolated the effect of the independent variable on the dependent variable?
• Does this effect you have isolated actually address the aim and/or hypothesis?
• Does your method include enough repetitions for a reliable result? (Read more about reliability below)

When assessing the validity of the method of analysis of the data , consider the following:

• Does the analysis extrapolate or interpolate the experimental data? Generally, interpolation is valid, but extrapolation is invalid. This because by extrapolating, you are ‘peering out into the darkness’ – just because your data showed a certain trend for a certain range it does not mean that this trend will hold for all.
• Does the analysis use accepted laws and mathematical relationships? That is, do the equations used for analysis have scientific or mathematical base? For example, `F = ma` is an accepted law in physics, but if in the analysis you made up a relationship like `F = ma^2` that has no scientific or mathematical backing, the method of analysis is invalid.
• Is the most appropriate method of analysis used? Consider the differences between using a table and a graph. In a graph, you can use the gradient to minimise the effects of systematic errors and can also reduce the effect of random errors. The visual nature of a graph also allows you to easily identify outliers and potentially exclude them from analysis. This is why graphical analysis is generally more valid than using values from tables.

When assessing the validity of your results , consider the following: 

• Is your primary data (data you collected from your own experiment) BOTH accurate and reliable? If not, it is invalid.
• Are the secondary sources you may have used BOTH reliable and accurate?

When assessing the validity of your conclusion , consider the following:

• Does your conclusion relate directly to the aim or the hypothesis?

How to Improve Validity

Ways of improving validity will differ across experiments. You must first identify what area(s) of the experiment’s validity is lacking (is it the procedure, analysis, results, or conclusion?). Then, you must come up with ways of overcoming the particular weakness. 

Below are some examples of this.

Example – Validity in Chemistry Experiment 

Let's say we want to measure the mass of carbon dioxide in a can of soft drink.

The following steps are followed:

• Weigh an unopened can of soft drink on an electronic balance.
• Open the can.
• Place the can on a hot plate until it begins to boil.
• When cool, re-weigh the can to determine the mass loss.

To ensure this experiment is valid, we must establish controlled variables:

• type of soft drink used
• temperature at which this experiment is conducted
• period of time before soft drink is re-weighed

Despite these controlled variables, this experiment is invalid because it actually doesn't help us measure the mass of carbon dioxide in the soft drink. This is because by heating the soft drink until it boils, we are also losing water due to evaporation. As a result, the mass loss measured is not only due to the loss of carbon dioxide, but also water. A simple way to improve the validity of this experiment is to not heat it; by simply opening the can of soft drink, carbon dioxide in the can will escape without loss of water.

Example – Validity in Physics Experiment

Let's say we want to measure the value of gravitational acceleration `g` using a simple pendulum system, and the following equation:

$$T = 2\pi \sqrt{\frac{l}{g}}$$

• `T` is the period of oscillation
• `l` is the length of string attached to the mass
• `g` is the acceleration due to gravity

• Cut a piece of a string or dental floss so that it is 1.0 m long.
• Attach a 500.0 g mass of high density to the end of the string.
• Attach the other end of the string to the retort stand using a clamp.
• Starting at an angle of less than 10º, allow the pendulum to swing and measure the pendulum’s period for 10 oscillations using a stopwatch.
• Repeat the experiment with 1.2 m, 1.5 m and 1.8 m strings.

The controlled variables we must established in this experiment include:

• mass used in the pendulum
• location at which the experiment is conducted

The validity of this experiment depends on the starting angle of oscillation. The above equation (method of analysis) is only true for small angles (`\theta < 15^{\circ}`) such that `\sin \theta = \theta`. We also want to make sure the pendulum system has a small enough surface area to minimise the effect of air resistance on its oscillation.

In this instance, it would be invalid to use a pair of values (length and period) to calculate the value of gravitational acceleration. A more appropriate method of analysis would be to plot the length and period squared to obtain a linear relationship, then use the value of the gradient of the line of best fit to determine the value of `g`. 

Accuracy refers to how close the experimental measurements are to the true value.

Accuracy depends on

• the validity of the experiment
• the degree of error:
• systematic errors are those that are systemic in your experiment. That is, they effect every single one of your data points consistently, meaning that the cause of the error is always present. For example, it could be a badly calibrated temperature gauge that reports every reading 5 °C above the true value.
• random errors are errors that occur inconsistently. For example, the temperature gauge readings might be affected by random fluctuations in room temperature. Some readings might be above the true value, some might then be below the true value.
• sensitivity of equipment used.

Assessing Accuracy 

The effect of errors and insensitive equipment can both be captured by calculating the percentage error:

$$\text{% error} = \frac{\text{|experimental value – true value|}}{\text{true value}} \times 100%$$

Generally, measurements are considered accurate when the percentage error is less than 5%. You should always take the context of the experimental into account when assessing accuracy. 

While accuracy and validity have different definitions, the two are closely related. Accurate results often suggest that the underlying experiment is valid, as invalid experiments are unlikely to produce accurate results.

In a simple pendulum experiment, if your measurements of the pendulum's period are close to the calculated value, your experiment is accurate. A table showing sample experimental measurements vs accepted values from using the equation above is shown below. 

All experimental values in the table above are within 5% of accepted (theoretical) values, they are therefore considered as accurate. 

How to Improve Accuracy

• Remove systematic errors : for example, if the experiment’s measuring instruments are poorly calibrated, then you should correctly calibrate it before doing the experiment again.
• Reduce the influence of random errors : this can be done by having more repetitions in the experiment and reporting the average values. This is because if you have enough of these random errors – some above the true value and some below the true value – then averaging them will make them cancel each other out This brings your average value closer and closer to the true value.
• Use More Sensitive Equipments: For example, use a recording to measure time by analysing motion of an object frame by frame, instead of using a stopwatch. The sensitivity of an equipment can be measured by the limit of reading . For example, stopwatches may only measure to the nearest millisecond – that is their limit of reading. But recordings can be analysed to the frame. And, depending on the frame rate of the camera, this could mean measuring to the nearest microsecond.
• Obtain More Measurements and Over a Wider Range:  In some cases, the relationship between two variables can be more accurately determined by testing over a wider range. For example, in the pendulum experiment, periods when strings of various lengths are used can be measured. In this instance, repeating the experiment does not relate to reliability because we have changed the value of the independent variable tested.

Reliability

Reliability involves the consistency of your results over multiple trials.

Assessing Reliability

The reliability of an experiment can be broken down into the reliability of the procedure and the reliability of the final results.

The reliability of the procedure refers to how consistently the steps of your experiment produce similar results. For example, if an experiment produces the same values every time it is repeated, then it is highly reliable. This can be assessed quantitatively by looking at the spread of measurements, using statistical tests such as greatest deviation from the mean, standard deviations, or z-scores.

Ask yourself: "Is my result reproducible?"

The reliability of results cannot be assessed if there is only one data point or measurement obtained in the experiment. There must be at least 3. When you're repeating the experiment to assess the reliability of its results, you must follow the  same steps , use the  same value  for the independent variable. Results obtained from methods with different steps cannot be assessed for their reliability.

Obtaining only one measurement in an experiment is not enough because it could be affected by errors and have been produced due to pure chance. Repeating the experiment and obtaining the same or similar results will increase your confidence that the results are reproducible (therefore reliable).

In the soft drink experiment, reliability can be assessed by repeating the steps at least three times:

The mass loss measured in all three trials are fairly consistent, suggesting that the reliability of the underly method is high.

The reliability of the final results refers to how consistently your final data points (e.g. average value of repeated trials) point towards the same trend. That is, how close are they all to the trend line? This can be assessed quantitatively using the `R^2` value. `R^2` value ranges between 0 and 1, a value of 0 suggests there is no correlation between data points, and a value of 1 suggests a perfect correlation with no variance from trend line.

In the pendulum experiment, we can calculate the `R^2` value (done in Excel) by using the final average period values measured for each pendulum length.

Here, a `R^2` value of 0.9758 suggests the four average values are fairly close to the overall linear trend line (low variance from trend line). Thus, the results are fairly reliable. 

How to Improve Reliability

A common misconception is that increasing the number of trials increases the reliability of the procedure . This is not true. The only way to increase the reliability of the procedure is to revise it. This could mean using instruments that are less susceptible to random errors, which cause measurements to be more variable.

Increasing the number of trials actually increases the reliability of the final results . This is because having more repetitions reduces the influence of random errors and brings the average values closer to the true values. Generally, the closer experimental values are to true values, the closer they are to the true trend. That is, accurate data points are generally reliable and all point towards the same trend.

Reliable but Inaccurate / Invalid

It is important to understand that results from an experiment can be reliable (consistent), but inaccurate (deviate greatly from theoretical values) and/or invalid. In this case, your procedure  is reliable, but your final results likely are not.

Examples of Reliability

Using the soft drink example again, if the mass losses measured for three soft drinks (same brand and type of drink) are consistent, then it's reliable. 

Using the pendulum example again, if you get similar period measurements every time you repeat the experiment, it’s reliable.  

However, in both cases, if the underlying methods are invalid, the consistent results would be invalid and inaccurate (despite being reliable).

Do you have trouble understanding validity, accuracy or reliability in your science experiment or depth study?

Consider getting personalised help from our 1-on-1 mentoring program !

RETURN TO WORKING SCIENTIFICALLY

• choosing a selection results in a full page refresh
• press the space key then arrow keys to make a selection

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

Reliability vs. Validity in Research | Difference, Types and Examples

Published on July 3, 2019 by Fiona Middleton . Revised on June 22, 2023.

Reliability and validity are concepts used to evaluate the quality of research. They indicate how well a method , technique. or test measures something. Reliability is about the consistency of a measure, and validity is about the accuracy of a measure.opt

It’s important to consider reliability and validity when you are creating your research design , planning your methods, and writing up your results, especially in quantitative research . Failing to do so can lead to several types of research bias and seriously affect your work.

Table of contents

Understanding reliability vs validity, how are reliability and validity assessed, how to ensure validity and reliability in your research, where to write about reliability and validity in a thesis, other interesting articles.

Reliability and validity are closely related, but they mean different things. A measurement can be reliable without being valid. However, if a measurement is valid, it is usually also reliable.

What is reliability?

Reliability refers to how consistently a method measures something. If the same result can be consistently achieved by using the same methods under the same circumstances, the measurement is considered reliable.

What is validity?

Validity refers to how accurately a method measures what it is intended to measure. If research has high validity, that means it produces results that correspond to real properties, characteristics, and variations in the physical or social world.

High reliability is one indicator that a measurement is valid. If a method is not reliable, it probably isn’t valid.

If the thermometer shows different temperatures each time, even though you have carefully controlled conditions to ensure the sample’s temperature stays the same, the thermometer is probably malfunctioning, and therefore its measurements are not valid.

However, reliability on its own is not enough to ensure validity. Even if a test is reliable, it may not accurately reflect the real situation.

Validity is harder to assess than reliability, but it is even more important. To obtain useful results, the methods you use to collect data must be valid: the research must be measuring what it claims to measure. This ensures that your discussion of the data and the conclusions you draw are also valid.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Reliability can be estimated by comparing different versions of the same measurement. Validity is harder to assess, but it can be estimated by comparing the results to other relevant data or theory. Methods of estimating reliability and validity are usually split up into different types.

Types of reliability

Different types of reliability can be estimated through various statistical methods.

Types of validity

The validity of a measurement can be estimated based on three main types of evidence. Each type can be evaluated through expert judgement or statistical methods.

To assess the validity of a cause-and-effect relationship, you also need to consider internal validity (the design of the experiment ) and external validity (the generalizability of the results).

The reliability and validity of your results depends on creating a strong research design , choosing appropriate methods and samples, and conducting the research carefully and consistently.

Ensuring validity

If you use scores or ratings to measure variations in something (such as psychological traits, levels of ability or physical properties), it’s important that your results reflect the real variations as accurately as possible. Validity should be considered in the very earliest stages of your research, when you decide how you will collect your data.

• Choose appropriate methods of measurement

Ensure that your method and measurement technique are high quality and targeted to measure exactly what you want to know. They should be thoroughly researched and based on existing knowledge.

For example, to collect data on a personality trait, you could use a standardized questionnaire that is considered reliable and valid. If you develop your own questionnaire, it should be based on established theory or findings of previous studies, and the questions should be carefully and precisely worded.

• Use appropriate sampling methods to select your subjects

To produce valid and generalizable results, clearly define the population you are researching (e.g., people from a specific age range, geographical location, or profession).  Ensure that you have enough participants and that they are representative of the population. Failing to do so can lead to sampling bias and selection bias .

Ensuring reliability

Reliability should be considered throughout the data collection process. When you use a tool or technique to collect data, it’s important that the results are precise, stable, and reproducible .

• Apply your methods consistently

Plan your method carefully to make sure you carry out the same steps in the same way for each measurement. This is especially important if multiple researchers are involved.

For example, if you are conducting interviews or observations , clearly define how specific behaviors or responses will be counted, and make sure questions are phrased the same way each time. Failing to do so can lead to errors such as omitted variable bias or information bias .

• Standardize the conditions of your research

When you collect your data, keep the circumstances as consistent as possible to reduce the influence of external factors that might create variation in the results.

For example, in an experimental setup, make sure all participants are given the same information and tested under the same conditions, preferably in a properly randomized setting. Failing to do so can lead to a placebo effect , Hawthorne effect , or other demand characteristics . If participants can guess the aims or objectives of a study, they may attempt to act in more socially desirable ways.

It’s appropriate to discuss reliability and validity in various sections of your thesis or dissertation or research paper . Showing that you have taken them into account in planning your research and interpreting the results makes your work more credible and trustworthy.

Prevent plagiarism. Run a free check.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Normal distribution
• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Mixed methods research
• Non-probability sampling
• Quantitative research
• Ecological validity

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Middleton, F. (2023, June 22). Reliability vs. Validity in Research | Difference, Types and Examples. Scribbr. Retrieved June 24, 2024, from https://www.scribbr.com/methodology/reliability-vs-validity/

Is this article helpful?

Fiona Middleton

Other students also liked, what is quantitative research | definition, uses & methods, data collection | definition, methods & examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

Measurement: Accuracy and Precision, Reliability and Validity

• Reference work entry
• Cite this reference work entry

• Goran Trajković 2  

2643 Accesses

5 Citations

Assessment; Judgment; Mensuration; Metage; Rating; Quantification

Measurement is the process in which numbers or other symbols are assigned to the characteristics that are being observed.

Basic Characteristics

Measurement is the process in which numbers or other symbols are assigned to the characteristics of the units that are observed, in such a way that the relation between numbers or symbols reflects the relation between characteristics that are the subject of the research. Measurement is the process in which the numbers or other symbols are assigned to the characteristics of the units that are observed, in a way where the relation between numbers or symbols refelcts the realtion between characteristics that are the subject of the research. Figure  1 shows relation of measurement to unit of observation, variable and data, some of key terms in biostatistics (see synopsis Biostatistics).

Relation of measurement, unit of observation, variable and data

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Blacker D (2000) Psychiatric Rating Scales. In: Sadock BJ, Sadock VA (eds) Kaplan & Sadock's comprehensive textbook of psychiatry, on CD-ROM, 7th edn. Lippincott Williams & Wilkins, Philadelphia

Google Scholar  

Chernick MR, Friis RH (2003) Introductory biostatistics for the health sciences. Wiley‐Interscience, Hoboken, NJ

Feinstein AR (2002) Principles of medical statistics. Chapman & Hall/CRC, Boca Raton, FL

Fleiss JL (1981) Statistical methods for rates and proportions, 2nd edn. John Wiley & Sons Inc, New York

Harlow R, Dotson C, Thompson RL (2002) Fundamentals of dimensional metrology, 4th edn. Delmar Publishers, Albany, NY

Katz DL (1997) Epidemiology, biostatistics, and preventive medicine review. W.B. Saunders, Philadelphia

Krishnamurty GB, Kasovia‐Schmitt P, Ostroff DJ (1995) Statistics: an interactive text for the health and life sciences. Jones and Bartlett, Boston

Nunnally JC, Bernstein IH (1994) Psychometric theory, 3rd edn. McGraw-Hill, New York

Webb P, Bain C, Pirozzo S (2005) Essential epidemiology: an introduction for students and health professionals. Cambridge University Press, New York

Download references

Author information

Authors and affiliations.

Medical Statistics and Informatics, School of Medicine, University of Pristina, Kosovska Mitrovica, Serbia

Goran Trajković

You can also search for this author in PubMed   Google Scholar

Editor information

Editors and affiliations.

Network EUROlifestyle Research Association Public Health Saxony-Saxony Anhalt e.V. Medical Faculty, University of Technology, Fiedlerstr. 27, 01307, Dresden, Germany

Wilhelm Kirch ( Professor Dr. Dr. ) ( Professor Dr. Dr. )

Rights and permissions

Reprints and permissions

Copyright information

© 2008 Springer-Verlag

About this entry

Cite this entry.

Trajković, G. (2008). Measurement: Accuracy and Precision, Reliability and Validity . In: Kirch, W. (eds) Encyclopedia of Public Health. Springer, Dordrecht. https://doi.org/10.1007/978-1-4020-5614-7_2081

Download citation

DOI : https://doi.org/10.1007/978-1-4020-5614-7_2081

Publisher Name : Springer, Dordrecht

Print ISBN : 978-1-4020-5613-0

Online ISBN : 978-1-4020-5614-7

eBook Packages : Medicine Reference Module Medicine

Share this entry

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Reliability, Validity, and Accuracy of Experiments

Introduction

In your VCE Physics practical, you will be tested on your ability to conduct experiments and analyse results. To do so, it is important to understand the concepts of  validity, reliability, accuracy, and precision .

Each term has a distinct definition, though they are often conflated or confused with one another. Let us clarify them so you can better design experiments, analyze results, and judge the soundness of findings.

Validity refers specifically to whether an experiment measures what it purports to measure. For example, a test designed to assess math skills would have high validity if it focuses only on math questions and not unrelated subject areas. Validity ensures that experimental results reflect the phenomenon under study rather than extraneous variables.

Reliability describes the consistency and replicability of research findings and experimental outcomes. A reliable experimental protocol will yield the same results time after time when repeated by multiple researchers. Reliability provides assurance that results are not due to random chance.

Accuracy concerns how close experimental measurements or data are to the true or accepted value. High accuracy means that any errors are minimal and small in magnitude. Improving accuracy involves reducing both random error and systematic biases.

Precision relates to the level of detail or resolution in an experimental measurement. For instance, a scale that measures weight to the nearest tenth of a gram has higher precision than one that measures only whole grams. Precision is independent from accuracy.

What is Validity?

Validity focuses on whether an experiment actually measures what it is intended to measure. It ensures that the results obtained from the experiment are meaningful and significant.

In other words “Does my experiment actually measure what I intent to in the aim”?

Internal Validity

Internal validity looks at whether the cause and effect relationship between the independent and dependent variables is a strong one. In an experiment we are investigating whether X (independent variable) causes Y (dependent variable), all other variables being equal (control variables). Confounding variables that provide alternative explanations for results must be controlled.

External Validity

External validity concerns whether results can be generalized beyond the specific study conditions and sample group. The participants, environment and methods should represent the broader population and real-world situations.

How to Test & Improve it?

To ensure high validity, you must:

• Clearly define variables and constructs being measured
• Demonstrate logical relationships between measures used and outcomes
• Control confounding variables
• Replicate studies under different conditions with different samples
• Check that the test or experiment actually measures the intended concept
• Address & address assumptions

Following strong methodology and research design principles enhances validity. Validity ensures the results are robust and meaningful.

What is Reliability?

Reliability refers to the consistency of measurements or experiment outcomes when repeated under the same conditions. It helps determine whether results are reproducible and not due to chance.

There are several types of reliability:

Test-Retest Reliability

This involves administering the same test twice over a period of time to the same group of individuals. High test-retest reliability means the scores are similar between the two tests. This helps evaluate the stability of results over time.

Inter-Rater Reliability

This evaluates the degree to which different raters or observers give consistent estimates or ratings of the same behavior or phenomenon. High inter-rater reliability indicates the ratings are consistent across raters.

To ensure high reliability:

• Use standardized instructions and procedures for administering tests or experiments. This reduces variability.
• Ensure observers and raters are properly trained so ratings are calibrated across people.
• Increase sample sizes, which reduces the variability of mean values between tests.
• Carefully design experiments and tests to isolate the effect and relationship you want to measure.

Reliability is essential for valid interpretation of results. Methods and metrics with high reliability give researchers confidence the results reflect true effects rather than random fluctuations.

What is Accuracy?

Accuracy refers to how close the results of a measurement or experiment are to the true or accepted value. It is different from validity, which focuses on whether the experiment actually measures what it is intended to measure. Accuracy is also separate from reliability, which refers to consistency of results when measurements are repeated.

While high validity and reliability are important, a measurement can be valid and reliable but inaccurate. For example, a scale may consistently report someone's weight as 10 pounds higher than their true weight. The measurement would have high reliability but low accuracy.

There are a few key ways to improve accuracy in experiments and measurements:

• Use equipment and methods that are known to produce accurate, error-free results when used properly. For example, use a properly calibrated scale rather than one that constantly misreports values.
• Take multiple measurements and use statistical techniques to minimize random errors. Averaging multiple measurements reduces the effect of random fluctuations.
• Identify and minimize systematic errors - errors that shift all measurements consistently higher or lower than the true value. This may involve adjustments like accounting for equipment calibration drift.
• Use the most precise measurement tools available to capture the highest level of detail. This avoids rounding or estimation errors.
• Ensure the experiment and equipment used are appropriate for measuring the desired variable, rather than inadvertently measuring something else. This improves accuracy by measuring the intended target.

Paying attention to accuracy, in addition to validity and reliability, helps ensure research results closely reflect the true, accepted values in the real world. Accurate measurements instill greater confidence in the conclusions drawn from an experiment.

What is Precision?

What is it.

Precision refers to the level of detail in measurements. It is determined by the unit of measurement being used. For example, a measurement of 5 meters is less precise than one of 5.2 meters. Precision relates to reproducibility - how close repeated measurements are to each other.

Precision is different from accuracy. A measurement can be extremely precise but not accurate. For example, if a scale consistently measures an object at 5.00000 kg but its true weight is 5.2 kg, the measurement is very precise but not accurate.

Precision depends on the instrumentation being used and the care taken when recording measurements. Ways to increase precision include:

• Using instruments capable of measuring in smaller units, like millimeters instead of centimeters. This provides more decimal places and detail.
• Making multiple measurements and reporting results to a consistent number of decimal places. Don't round early in calculations.
• Using the proper techniques when taking measurements and reading instruments to get consistent, repeatable results.
• Calibrating instruments regularly to manufacturer specifications.
• Reporting measurements with clear units and to an appropriate number of significant figures based on the precision of the instrument.

While accuracy is about trueness to the real value, precision relates to the level of reproducibility and detail in measurements. Increasing instrumentation precision allows more definitive comparisons between measurements and minimizes uncertainty.

What is Systematic Errors?

Systematic errors, also known as experimental bias, are errors that consistently occur due to inaccuracies in the equipment or methodology used in an experiment. Unlike random errors, systematic errors follow identifiable patterns or trends that influence the results in one direction. They affect the accuracy of a measurement by shifting all the results in a systematic way.

Some examples of systematic errors include:

• Instrument calibration errors : When the equipment used to take measurements is not properly calibrated, it can consistently give inaccurate readings. For example, a miscalibrated analytical balance may overestimate or underestimate the true mass of a sample.
• Observer bias : If the person conducting the experiment unintentionally influences the results in one direction, this is a systematic error. They may inadvertently affect the observation or recording of data based on subjective expectations.
• Environmental factors : Changes in environmental conditions like temperature or humidity can cause systematic deviations if they are not controlled for. For example, increased ambient temperature could expand the scale being used to take measurements.
• Instrument drift : If the performance of equipment deteriorates over time, such as a sensor becoming less sensitive, this can induce a directional bias in the measurements.
• Sampling errors : Taking non-random samples or disproportionate samples can skew results in a particular direction away from the true population value.

Systematic errors reduces Accuracy

The effect of systematic errors is an overall shift in the accuracy of results that leads to consistently over- or underestimated values. Unlike random errors which can cancel out, systematic errors accumulate and affect the accuracy in a defined direction. Identifying and minimizing sources of systematic error is crucial for improving the accuracy of experimental results. Careful calibration, eliminating observer bias, controlling environmental factors, and random sampling help reduce systematic errors.

What is Random Errors?

Random errors, also known as random variation, refer to inaccuracies that occur unpredictably and inconsistently in measurements. They are caused by factors that vary randomly between measurements, making the results scatter randomly above or below the true value.

Random errors cannot be corrected as their causes are unknown. However, they can be reduced by taking multiple measurements and calculating an average. The more measurements that are taken, the lower the random error. Taking an average also has a smoothing effect that cancels out the random fluctuation in readings.

There are several common causes of random errors:

• Small irregularities in instruments that affect each measurement differently
• Tiny environmental changes like temperature fluctuations
• Imperfections in experimental techniques between trials
• Biological variance when measuring living subjects

For example, when using a scale to weigh an object multiple times, the readings may be 4.2 lbs, 4.1 lbs, 4.3 lbs, 4.25 lbs due to slight variations in the instrument, air currents, and handling between each measurement.

Random errors reduces Reliability & Precision

Random errors reduce reliability and precision. But they do not affect accuracy systematically because the variation is symmetric around the true value. Reducing random errors improves the repeatability and reproducibility of experimental results. Careful experimental design is key, by controlling variables, standardizing procedures, calibrating equipment, and averaging repeated measurements.

In summary, validity, reliability, accuracy, and precision are important concepts in experimental design and measurement. Though often used interchangeably, there are key differences between these terms:

• Validity refers to whether an experiment measures what it is intended to measure. Validity can be improved through careful selection of appropriate equipment and controls.
• Reliability relates to the consistency of results when measurements are repeated under identical conditions. Reliability can be enhanced by minimizing random errors in the experimental procedure.
• Accuracy refers to how close a measurement is to the true value. Accuracy is affected by systematic errors which can be minimized through proper calibration and technique.
• Precision relates to the exactness of measurements and is expressed through the number of significant figures. Precision is not the same as accuracy.

Carefully considering and optimizing for validity and reliability in experimental design is crucial for generating meaningful results. Valid, reliable data allows researchers to draw accurate conclusions from their work. Paying attention to these factors demonstrates scientific rigor and helps advance knowledge within a field.

Subscribe for free VCE Physics study resources!

Attend Free Trial lesson Claim 15% off First Term

• God (or some type of Gods) did it.
• Nature works with "an unseen hand".
• There are "rational laws" to be discovered (and people are capable of discovering these).
• Causal relations are an illusion; the universe is random and chaotic, and runs on entropy.
• Controlled experiments in which purported causal factors are manipulated systematically.
• Citing recognized authorities, such as Biblical or Quran scripture-or Sigmund Freud.
• Marshalling one's reasonable arguments as in a court of law or journalism.
• Precedent as in a court of law.
• Intuition...feelings...one just "knows" (in love?).
• Reading traces in the environment (Sherlock Holmes stories).
• Devine revelation in dreams, visions, bones, tea leaves, etc.
• Statistically controlling various purported causal variables.

1.15: Reliability and Validity

Chapter 1: research methods, chapter 2: the social self, chapter 3: social judgement and decision-making, chapter 4: understanding and influencing others, chapter 5: attitudes and persuasion, chapter 6: close relationships, chapter 7: stereotypes, prejudice, and discrimination, chapter 8: helping and hurting, chapter 9: group dynamics.

The JoVE video player is compatible with HTML5 and Adobe Flash. Older browsers that do not support HTML5 and the H.264 video codec will still use a Flash-based video player. We recommend downloading the newest version of Flash here, but we support all versions 10 and above.

In a scientific setting, suppose a researcher wants to create a test to measure the compatibility of potential partners on an online dating website. They must consider two important factors to generate successful outcomes.

One is reliability , which refers to the ability of a test—or other research instrument—to provide consistent, and thus, reproducible, results under similar circumstances.

In this context, the compatibility test would be considered reliable if the same people take the test twice and perform similarly each time. This situation is also known as test-retest reliability .

However, getting consistent results does not ensure that a test is accurate. Now, the second factor, validity —the extent to which a test accurately measures or predicts what it set out to measure—must be reflected.

Will they actually enjoy spending time together on a date?

If the pair scored low in dating compatibility and still went out to dinner together, perhaps they had a miserable time. In this case, the test did have high predictive validity —it forecasted the behavior.

In the end, researchers strive for reproducibility and accuracy: here, the test is successful if other incompatible couples continue in misery, while those who share common interests enjoy a fantastic time together.

Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.

Unfortunately, being consistent in measurement does not necessarily mean that you have measured something correctly. To illustrate this concept, consider a kitchen scale that would be used to measure the weight of cereal that you eat in the morning. If the scale is not properly calibrated, it may consistently under- or overestimate the amount of cereal that’s being measured. While the scale is highly reliable in producing consistent results ( e.g. , the same amount of cereal poured onto the scale produces the same reading each time), those results are incorrect. This is where validity comes into play. Validity refers to the extent to which a given instrument or tool accurately measures what it’s supposed to measure. While any valid measure is by necessity reliable, the reverse is not necessarily true. Researchers strive to use instruments that are both highly reliable and valid.

How Valid Is the SAT?

Standardized tests like the SAT are supposed to measure an individual’s aptitude for a college education, but how reliable and valid are such tests? Research conducted by the College Board suggests that scores on the SAT have high predictive validity for first-year college students’ GPA (Kobrin, Patterson, Shaw, Mattern, & Barbuti, 2008). In this context, predictive validity refers to the test’s ability to effectively predict the GPA of college freshmen. Given that many institutions of higher education require the SAT for admission, this high degree of predictive validity might be comforting.

However, the emphasis placed on SAT scores in college admissions has generated some controversy on a number of fronts. For one, some researchers assert that the SAT is a biased test that places minority students at a disadvantage and unfairly reduces the likelihood of being admitted into a college (Santelices & Wilson, 2010). Additionally, some research has suggested that the predictive validity of the SAT is grossly exaggerated in how well it is able to predict the GPA of first-year college students. In fact, it has been suggested that the SAT’s predictive validity may be overestimated by as much as 150% (Rothstein, 2004). Many institutions of higher education are beginning to consider de-emphasizing the significance of SAT scores in making admission decisions (Rimer, 2008).

In 2014, College Board president David Coleman expressed his awareness of these problems, recognizing that college success is more accurately predicted by high school grades than by SAT scores. To address these concerns, he has called for significant changes to the SAT exam (Lewin, 2014).

This text is adapted from OpenStax, Psychology. OpenStax CNX.

Get cutting-edge science videos from J o VE sent straight to your inbox every month.

mktb-description

We use cookies to enhance your experience on our website.

By continuing to use our website or clicking “Continue”, you are agreeing to accept our cookies.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• ADVERTISEMENT FEATURE Advertiser retains sole responsibility for the content of this article

Six factors affecting reproducibility in life science research and how to handle them

Produced by

There are several reasons why an experiment cannot be replicated.

Independent verification of data is a fundamental principle of scientific research across the disciplines. The self-correcting mechanisms of the scientific method depend on the ability of researchers to reproduce the findings of published studies in order to strengthen evidence and build upon existing work. Stanford University medical researcher, John Ioannidis, a prominent scholar on reproducibility in science, has pointed out that the importance of reproducibility does not have to do with ensuring the ‘correctness’ of results, but rather with ensuring the transparency of exactly what was done in a given line of research 1 .

In theory, researchers should be able to re-create experiments, generate the same results, and arrive at the same conclusions, thus helping to validate and strengthen the original work. However, reality does not always meet these expectations. Too often, scientific findings in biomedical research cannot be reproduced 2 ; consequently, resources and time are wasted, and the credibility of scientific findings are put at risk. Furthermore, despite recent heightened awareness, there remains a significant need to better educate students and research trainees about the lack of reproducibility in life science research and actions that can be taken to improve it. Here, we review predominant factors affecting reproducibility and outline efforts to improve the situation.

What is reproducibility?

The phrase ‘lack of reproducibility’ is understood in the scientific community, but it is a rather broad expression that incorporates several aspects. Though a standardized definition has not been fully established, the American Society for Cell Biology ® (ASCB ® ) has attempted a multi-tiered approach to defining the term reproducibility by identifying the subtle differences in how the term is perceived throughout the scientific community.

ACSB 4 has discussed these differences with the following terms: direct replication , which are efforts to reproduce a previously observed result by using the same experimental design and conditions as the original study; analytic replication , which aims to reproduce a series of scientific findings through a reanalysis of the original data set; systemic replication , which is an attempt to reproduce a published finding under different experimental conditions (e.g., in a different culture system or animal model); and conceptual replication , where the validity of a phenomenon is evaluated using a different set of experimental conditions or methods.

It is generally thought that the improvement of direct replication and analytic replication is most readily addressed through training, policy modifications, and other interventions, while failures in systematic and conceptual replication are more difficult to connect to problems with how research was performed as there is more natural variability at play.

The reproducibility problem

Many studies claim a significant result, but their findings cannot be reproduced. This problem has attracted increased attention in recent years, with several studies providing evidence that research is often not reproducible. A 2016 Nature survey 3 , for example, revealed that in the field of biology alone, over 70% of researchers were unable to reproduce the findings of other scientists and approximately 60% of researchers could not reproduce their own findings.

The lack of reproducibility in scientific research has negative impacts on health, lower scientific output efficiency, slower 6 , 7 scientific progress, wasted time and money, and erodes the public’s trust in scientific research. Though many of these problems are difficult to quantify, there have been attempts to calculate financial losses. A 2015 meta-analysis 5 of past studies regarding the cost of non-reproducible research estimated that $28 billion per year is spent on preclinical research that is not reproducible. Looking at avoidable waste in biomedical research on the whole, it is estimated that as much as 85% of expenditure may be wasted due to factors that similarly contribute to non-reproducible research such as inappropriate study design, failure to adequately address biases, non-publication of studies with disappointing results, and insufficient descriptions of interventions and methods.

Factors contributing to the lack of reproducibility

Failures of reproducibility cannot be traced to a single cause, but there are several categories of shortcomings that can explain many of the cases where research cannot be reproduced. Here are some of the most significant categories.

A lack of access to methodological details, raw data, and research materials.

For scientists to be able to reproduce published work, they must be able to access the original data, protocols, and key research materials. Without these, reproduction is greatly hindered and researchers are forced to reinvent the wheel as they attempt to repeat previous work. The mechanisms and systems for sharing raw unpublished data and research materials, such as data repositories and biorepositories, need to be made robust so that sharing is not an impediment to reproducibility.

Use of misidentified, cross-contaminated, or over-passaged cell lines and microorganisms.

Reproducibility can be complicated and/or invalidated by biological materials that cannot be traced back to their original source, are not thoroughly authenticated, or are not properly maintained. For example, if a cell line is not identified correctly, or is contaminated with mycoplasma or another cell type, results can be affected significantly and their likelihood of replication diminished. There are many cases of studies conducted with misidentified or cross-contaminated cell lines, so results rendered questionable, and conclusions drawn from them are potentially invalid 8 . Improper maintenance of biological materials via long-term serial passaging can also seriously affect genotype and phenotype, which can make reproducing data difficult. Several studies have demonstrated that serial passaging can lead to variations in gene expression, growth rate, spreading, and migration in cell lines 9 , 10 ; and changes in physiology, virulence factor production, and antibiotic resistance in microorganisms 11 , 12 , 13 .

Inability to manage complex datasets

Advancements in technology have enabled the generation of extensive, complex data sets; however, many researchers do not have the knowledge or tools needed for analyzing, interpreting and storing the data correctly. Further, new technologies or methodologies may not yet have established or standardized protocols, so variations and biases can be easily introduced, which in turn can affect the ability to analytically replicate the data.

Poor research practices and experimental design

Among the findings from scholarly efforts examining non-reproducibility is that, in a significant portion of cases, the cause could be traced to poor practices in reporting research results, and poor experimental design 14 , 15 . Poorly designed studies without a core set of experimental parameters, whose methodology is not reported clearly, are less likely to be reproducible. If a study is designed without a thorough review of existing evidence, or if the efforts to minimize biases are insufficient, reproducibility becomes more problematic.

Cognitive bias

These refer to the ways that judgement and decision-making are affected by the individual subjective social context that each person builds around them. They are errors made in cognitive processes that are due to personal beliefs or perceptions. Researchers strive for impartiality and try to avoid cognitive bias, but it is often difficult to completely shut out the subtle, subconscious ways that cognitive bias can affect the conduct of research 16 , 17 . Scientists have identified dozens of different types of cognitive biases, including confirmation bias, selection bias, the bandwagon effect, cluster illusion, and reporting bias 17 . Confirmation bias is the unconscious act of interpreting new evidence in ways that confirm one’s existing belief system or theories; this type of bias impacts how information is gathered, interpreted, and recalled. Selection bias sees researchers choose subjects or data for analysis that is not properly randomized; here, the sample obtained is not truly representative of the whole population. The bandwagon effect is the tendency to agree with a position too easily, without sufficient evaluation in order to maintain group harmony; this form of bias may lead to the acceptance of unproven ideas that have gained popularity. Cluster illusion is when patterns are perceived in a pool of random data in which no actual pattern exists; a bias based on the tendency of the brain to seek out patterns. Reporting bias is when study participants selectively reveal or suppress information in a study according to their own subconscious drivers; this form of bias may lead to underreporting of negative or undesirable experimental results.

A competitive culture that rewards novel findings and undervalues negative results

The academic research system encourages the rapid publication of novel results. Researchers are rewarded more for publishing novel findings, and not for publishing negative results (e.g., where a correlation was not found) 15 . Indeed, there are limited arenas for publishing negative results, which could hone researchers’ efforts and avoid repeating work that may be difficult to replicate. Overall, reproducibility in research is hindered by under-reporting of studies that yield results deemed disappointing or insignificant. University hiring and promotion criteria often emphasize publishing in high-impact journals and do not generally reward negative results. Also, a competitive environment for research grants may incentivize researchers to limit reporting of details learned through experience that make experiments work better.

Recommended best practices

A number of significant efforts have been aimed at addressing the lack of reproducibility in scientific research. Individual researchers, journal publishers, funding agencies, and universities have all made substantial efforts toward identifying potential policy changes aimed at improving reproducibility 16 , 18 , 19 , 20 , 21 . What has emerged from these efforts is a set of recommended practices and policy prescriptions that are expected to have a large impact.

Training on statistical methods and study design is essential for reproducible research.

Robust sharing of data, materials, software, and other tools.

All of the raw data that underlies any published conclusions should be readily available to fellow researchers and reviewers of the published article. Depositing the raw data in a publicly available database would reduce the likelihood that researchers would select only those results that support a prevailing attitude or confirms previous work. Such sharing would accelerate scientific discoveries, and enable scientists to interact and collaborate at a meaningful level.

Use of authenticated biomaterials

Data integrity and assay reproducibility can be greatly improved by using authenticated, low-passage reference materials. Cell lines and microorganisms verified by a multifaceted approach that confirms phenotypic and genotypic traits, and a lack of contaminants, are essential tools for research. By starting a set of experiments with traceable and authenticated reference materials, and routinely evaluating biomaterials throughout the research workflow, the resulting data will be more reliable, and more likely to be reproducible.

Training on statistical methods and study design

Experimental reproducibility could be considerably improved if researchers were trained how to properly structure experiments and perform statistical analyses of results. By strictly adhering to a set of best practices in statistical methodology and experimental design, researchers could boost the validity and reproducibility of their work.

Pre-registration of scientific studies

If scientists pre-register proposed scientific studies (including the approach) prior to initiation of the study, it would allow careful scrutiny of all parts of the research process and would discourage the suppression of negative results.

Publish negative data

Many times, ‘negative’ data that do not support a hypothesis typically go unpublished as they are not considered high impact or innovative. By publishing negative data, it helps to interpret positive results from related studies and can help researchers adjust their experimental design so that further resources and funding are not wasted 22 .

Thorough description of methods

It is important that research methodology is thoroughly described to help improve reproducibility. Researchers should clearly report key experimental parameters, such as whether experiments were blinded, which standards and instruments were used, how many replicates were made, how the results were interpreted, how the statistical analysis was performed, how the randomization was done, and what criteria were used to include or exclude any data.

Ongoing efforts to improve reproducibility

There is a varied and influential group of organizations that are already working to improve the reproducibility of scientific research. The following is a list of initiatives aimed at supporting one or more aspects of the research reproducibility issue.

American Society for Cell Biology (ASCB) - The ASCB Report on Reproducibility

ASCB continues to identify methods and best practices that would enhance reproducibility in basic research. From its original analysis, the ASCB task force identified and published several recommendations focused on supporting existing efforts and initiating new activities on better training, reducing competition, sharing data, improving peer reviews, and providing cell authentication guidelines.

American Type Culture Collection (ATCC) - Cell and Microbial Authentication Services and Programs

Biological resource centers, such as ATCC, provide the research community with standardized, traceable, fully authenticated cell lines and microorganisms to aid in assay reproducibility. At ATCC, microbial strains are authenticated and characterized through genotypic, phenotypic, and functional analyses to confirm identity, purity, virulence, and antibiotic resistance. ATCC has also taken a lead in cell line authentication by publishing the voluntary consensus standard, ANSI/ATCC ASN-0002: Authentication of Human Cell Lines: Standardization of STR Profiling , and by performing STR profiling on all human cell lines managed among its holdings.

Furthermore, ATCC offers online cell line authentication training in partnership with Global Biological Standards Institute, NIH (R25GM116155-03), and Susan G. Komen (SPP160007), which focuses on the best practices for receiving, managing, authenticating, culturing, and preserving cell cultures. To further support cell authentication and reproducibility in the life sciences, ATCC also provides STR profiling and mycoplasma detection testing as services to researchers.

National Institutes of Health (NIH) - Rigor and Reproducibility

To help improve rigor, reproducibility, and transparency in scientific research, the NIH issued a notice in 2015 that informed scientists of revised grant application instructions that focused on improving experimental design, authenticating biological and chemical resources, analyzing and interpreting results, and accurately reporting research findings. These efforts have led to the adoption of similar guidelines by journals across numerous scientific disciplines and has resulted in cell line authentication becoming a prerequisite for publication.

Science Exchange & the Center for Open Science - The Reproducibility Project: Cancer Biology

This initiative was designed to provide evidence of reproducibility in cancer research and to identify possible factors that may affect reproducibility. Here, selected results from high-profile articles are independently replicated by unbiased third parties to evaluate if data could be consistently reproduced. For each evaluated study, a registered report delineating the experimental workflow is reviewed and published before experimentation is initiated; after data collection and analysis, the results are published as a replication study.

Author Policies for Publication

Many peer-reviewed journals have updated their reporting requirements to help improve the reproducibility of published results. The Nature Research journals, for example, have implemented new editorial policies that help ensure the availability of data, key research materials, computer codes and algorithms, and experimental protocols to other scientists. Researchers must now complete an editorial policy checklist to ensure compliance with these policies before their manuscript can be considered for review and publication.

Most people familiar with the issue of reproducibility agree that these efforts are gaining traction. However, progress will require sustained attention on the issue, as well as cooperation and involvement from stakeholders across various fields.

The academic research system encourages the rapid publication of novel results.

Moving forward

Accuracy and reproducibility are essential for fostering robust and credible research and for promoting scientific advancement. There are predominant factors that have contributed to the lack of reproducibility in life science research. This issue has come to light in recent years and a number of guidelines and recommendations on achieving reproducibility in the life sciences have emerged, but the practical implementation of these practices may be challenging. It is essential that the scientific community are objective when designing experiments, take responsibility for depicting their results accurately, and thoroughly and precisely describe all methodologies used. Further, funders, publishers, and policy-makers should continue to raise awareness about the lack of reproducibility and use their position to promote better research practices throughout the life sciences. By taking action and seeking opportunities for improvement, researchers and key stakeholders can help improve research practices and the credibility of scientific data.

For more information on how you can improve the reproducibility of your research, visit ATCC online.

Ioannidis JP. PLoS Medicine 11 : e1001747, 2014.

Article   PubMed   Google Scholar  

Feilden T. Science & Environment, BBC News, February 22, 2017.

Baker M. Nature News Feature , May 25 , 2016.

ASCB. ASCB, 2014.

Freedman LP, Cockburn IM, Simcoe TS. PLoS Biology 13 : e1002165, 2015.

Chalmers I, Glasziou P. Lancet 374 : 86-89, 2009.

Macleod MR, et al. Lancet 383 : 101-104, 2014.

Horbach S, Halffman W. PLoS One 12 : e0186281, 2017.

Mouriaux F, et al. Invest Ophthalmol Vis Sci 57 (13): 5288-5301, 2016.

Liao H, et al. Cytotechnology 66 : 229-238, 2014.

Somerville GA, et al. J Bacteriol 184 : 1430-1437, 2002.

Grimm D, et al. Infect Immun 71 (16): 3138-3145, 2003.

Lee JY, et al. Scientific Reports 6: 25543, 2016.

Resnik DB, Shamoo AE. Account Res 24 (2): 116-123, 2017.

The Academy of Medical Sciences, BBSRC, Medial Research Council, Wellcome Trust. Symposium report, October 2015.

Munafò MR, et al. Nature Human Behaviour 1: 0021, 2017.

Article   Google Scholar  

Cherry K. Cognitive Psychology, Very Well Mind. October 8, 2018.

Google Scholar  

Stodden V, Leisch F, Peng RD. 1st Edition, Chapman and Hall/CRC, 2014.

Landis SC, et al. Nature 490 : 187-191, 2012.

NIH https://www.nih.gov/research-training/rigor-reproducibility 2015.

Davies, EW, Edwards, DD. A Report from the American Academy of Microbiology: Promoting Responsible Scientific Research, 2016.

Weintraub PG. J Insect Sci 16(1): 109, 2016.

Download references

Related Articles

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Meet the Mentors
• Get Involved
• Get the T-Shirt
• Life Science Marketing
• Community Marketing
• Custom Marketing

Join Us Sign up for our feature-packed newsletter today to ensure you get the latest expert help and advice to level up your lab work.

• Genomics & Epigenetics
• DNA / RNA Manipulation and Analysis
• Protein Expression & Analysis
• PCR & Real-time PCR
• Flow Cytometry
• Microscopy & Imaging
• Cells and Model Organisms
• Analytical Chemistry and Chromatography Techniques
• Chemistry for Biologists
• Basic Lab Skills & Know-how
• Equipment Mastery & Hacks
• Managing the Scientific Literature
• Career Development and Networking
• Dealing with Fellow Scientists
• Getting Funded
• Lab Statistics & Math
• Organization & Productivity
• Personal Development
• PhD Survival
• Soft Skills & Tools
• Software & Online Tools
• Survive & Thrive
• Taming the Literature
• Writing, Publishing & Presenting
• Basic Lab Skills and Know-how

How to Measure and Improve Lab Accuracy and Precision

Accuracy and precision are critical for achieving reliable and reproducible results. Read on to discover what these terms mean and how to improve your accuracy and precision.

Published October 11, 2021

I am a results-oriented biochemist with over a decade of experience performing research and process development spanning microbiology, protein chemistry, and formulation development. My background includes extensive work in high-throughput assays, analytical chemistry, microbiology, project coordination, and lab management.

Listen to one of our scientific editorial team members read this article. Click  here  to access more audio articles or subscribe.

One of the first things any scientist should consider when measuring any attribute, whether it is the concentration of a solution, the quantity of DNA in a sample, or fluorescence intensity, is how accurate is this measurement?

Understanding accuracy (and limitations) in the lab is of the utmost importance to forming sound conclusions about experiments. This article discusses accuracy and precision and provides concrete examples of ways to understand method limitations and improve measurements in your lab

What Do We Mean by Accuracy and Precision?

You likely have a good understanding of the difference between accuracy and precision . Since accuracy and precision are fundamental to almost all the sciences, they are typically one of the first subjects covered in introductory STEM courses. With that said, let’s get a quick refresher before we proceed to the consequences of misunderstanding accuracy and precision in the lab.

For every measurement we make, there is an actual value that we are trying to obtain. Furthermore, whenever we prepare a material by weighing or dispensing it, there is a target value we are trying to reach. Simply put, accuracy is how close a measurement is to the actual true value, whereas precision is how close those measurements tend to be to each other.

For example, consider that you have four pipettes that you are using to dispense 30 µL of water. If you took 10 measurements from each pipette and knew the actual volume dispensed, you could determine whether each pipette was accurate and precise (see Figure 1 for an example).

Let’s analyze the chart in Figure 1 more closely. • Pipette 1 definitely outperforms the other three in terms of accuracy and precision —it most consistently dispenses volumes closest to our 30 µL target. We’ll give a gold star to this pipette. • Pipette 2 is relatively accurate compared to pipettes 3 and 4, but it’s not nearly as precise as pipette 1. That is, it does not consistently pipette similar volumes when used multiple times and has greater variability. • Pipette 3 is extremely precise, and pipettes nearly identical volumes each time it is used. However, it’s not anywhere near the target! Therefore, it has lower accuracy than pipettes 1 or 2. • Finally, pipette 4 is the least accurate and precise of the bunch. Not only are the volumes dispensed all over the place ( low precision ), they’re often nowhere near our target volume ( not accurate)! This is a pipette in need of some serious TLC.

For more on this topic, check out our article on checking pipette accuracy .

Why Is It Important to Be Aware of the Trueness and Precision of Our Measurements?

Understanding the concepts of accuracy (also known as trueness) and precision in a simple pipetting example is one matter, but it’s trickier to identify and monitor ALL the factors that may affect accuracy and precision, as well as the resulting impacts on your results.

Consider a situation where you are preparing custom cell culture media.

Perhaps you plan to weigh out 200 g of dextrose for your weekly experiments. You might go ahead and add dextrose with a scoop until the scale reads 200.0 g. How often do you think about how close the scale is reporting to the TRUE mass? It could be the case that the scale is out of calibration, and the actual mass you are adding is closer to 196 g. In other words, how accurate is your measurement?

Let’s assume that your scale is accurate. You still must consider the precision of the measurements! What if the scale is on a wobbly table, or a vent that turns on and off affects the measurement? In this case, the finished culture might turn out slightly differently from week to week.

Although this is a simple example, the effects can be pretty profound. In our example above, the composition of cell culture media directly affects the health and growth of cultures, which further influences other measurements we are taking and could change conclusions drawn from the entire experiment.

Simple issues like these can quickly compound and cascade into a plethora of issues , like increased variability and invalid results. Therefore, determining if your system measures a characteristic without bias and repeatably is clearly of crucial importance.

Is Any Measurement Ever Truly 100% Accurate and Precise?

Here’s the bad news—while being 100% accurate and precise is clearly ideal, it is impossible in reality. There is always some non-zero variability from factors outside our control, such as the instruments, environmental conditions, and lab personnel.

With that said, there are many things we can do to maximize accuracy and precision when conducting experiments. Drum roll, please…!

8 Ways to Improve Your Accuracy and Precision in the Lab

1. keep everything calibrated.

Calibration is the number one item on this list for a very important reason: it is the MOST critical means of ensuring your data and measurements are accurate.

Calibration involves adjusting or standardizing lab equipment so that it is more accurate AND precise.

Calibration typically requires comparing a standard to what your instrument is measuring and adjusting the instrument or software accordingly.

The complexity of calibrating instruments or equipment varies widely, but, typically, user manuals have recommended recalibration recommendations. Bitesize Bio has several articles on routine calibration, including routine calibration of pipettes and calibrating your lab scales .

2. Conduct Routine Maintenance

Even if all instruments in your lab are calibrated , odds are they need regular care to operate at their maximum accuracy and precision.

For instance, pH meters need routine maintenance that can be performed by novice scientists, while more sensitive instrumentation may require shipment of parts to vendors or even on-site visits.

Again, check your user manuals and call equipment manufacturers to ensure you take appropriate measures to keep lab equipment running under conditions optimal for accuracy.

3. Operate in the Appropriate Range with Correct Parameters

Always use tools that are designed and calibrated to work in the range you are measuring or dispensing. For example, don’t try to measure OD 600 beyond an absorption of >1.0 since optical density (OD) readings this high are beyond the dynamic range of most spectrophotometers! If you are ever unsure about using an instrument to measure accurately at an extreme value, reach out to a trusted peer or mentor for advice.

What if you are choosing between two tools that are both calibrated for use at a given target? You might have two pipettes that are both designed to dispense 100 µL (e.g., 20–100 µL or 100–1000 µL pipettes). When in doubt, choose the tool with more precision —in this case, the 20–100 µL pipette.

Watch our on-demand webinar on improving your pipetting technique for more information.

4. Understand Significant Figures (and Record Them Correctly!)

The number of significant figures (“sig figs”) you use and record is critical. Specifically, sig figs provide the degree of uncertainty associated with values.

Keep sig figs consistent when measuring items repeatedly, and ensure the number of sig figs you are using is appropriate for each measurement.

5. Take Multiple Measurements

The more samples you take for a given attribute, the more precise the representation of your measurement. In situations where sampling is destructive, or you can’t take multiple measurements (e.g., growth rates in a culture), you can increase the number of replicates to compensate.

However, for measurements like OD readings or cell counting , it’s reasonably easy to measure multiple parts of a single sample.

6. Detect Shifts Over Time

Some systems are prone to drift over time. For instance, background absorption in high-performance liquid chromatography (HPLC) may be indicative of column failure.

If you notice that measurements drift in a single direction over weeks or months, address the issue immediately by recalibration or preventative maintenance.

7. Consider the “Human Factor”

We don’t often talk about how a technique in the lab may vary from person to person, resulting in differences in the measurements of a single property.

To minimize the inherent variability between scientists, ensure that procedures are kept up to date and are as descriptive as possible.

In some cases, it may be easiest to have only one person responsible for a given measurement, but this may not always be possible. Ensure that all lab personnel are trained, especially on highly manual techniques like pipetting, to maximize accuracy and precision.

8. Perform a Measurement Systems Analysis (MSA)

While this is a relatively complicated method to gauge accuracy and precision, measurement systems analysis (or gage repeatability and reproducibility analysis) is the most comprehensive and statistically sound way to get a complete picture of the accuracy and precision of your measurement. This technique mathematically determines the amount of variation that exists when taking measurements multiple times.

To conduct an MSA, you’ll need to design a study that incorporates known and unknown sources of variation. There are various analysis methods available, but if your measurement is absolutely critical, it may be worth exploring. Stay tuned for a future article explaining various ways to conduct an MSA!

Final Thoughts

There are a wealth of resources on these topics if you want to learn more. For a more statistics-based primer on accuracy, precision, and trueness, check out Artel’s resource library on these topics . You might also consider reading about accuracy and precision through the International Organization of Standardization , which is a global organization that works to align scientists and engineers in every field when it comes to these topics.

Do you have more ideas on how to keep your lab measurements accurate and precise? Let us know in the comments below!

Share this article:

More 'Basic Lab Skills and Know-how' articles

Reproducibility 101: How to Make Media

Same is dull and boring, right? Not when it comes to making media batches – learn how to be boring and get great results every time.

The Dos and Don’ts of Weighing Dangerous Chemicals

A lot of chemical reagents are relatively non-hazardous.But there are just as many that are extremely hazardous, which means you’ll want to take precautions to reduce any risk of exposure, repeated exposure, and of course, accidental contamination of anything – or anyone – that walks out of the lab at the end of the day….

Tips for Choosing Your Lab Notebook Pen (and Why You Need to Choose Carefully)

Keeping a meticulous lab record of your experiments is a necessity. And it’s drilled into us to back up our computers, including backups stored in different locations to ensure vital records don’t get lost. But how do we protect the hard copy information in our lab books? You may not have given much thought previously…

6 Laboratory Sterilization Methods

Effective laboratory sterilization methods are essential for working with isolated cell lines. Read our guide to the top 6 sterilization techniques to banish those bugs.

Buying a Secondary Antibody: Why all the Choices?

So you grab a quick 5 minutes in between lectures to sit down and tackle an item on your to-do list: order a secondary antibody for an upcoming experiment.  But when you start to search your favorite secondary antibody provider’s website, you realize it is not going to be a 5 minute job.  Conjugated, F(ab’)2…

Four Free and Easy-To-Use Online Primer Design Tools

Designing and running PCR reactions in the lab has become so commonplace that the number of primer design tools available can be a bit overwhelming for a beginner (or even an experienced molecular biologist!). Below are four of my favorite online programs available to make primer design quick, easy, and effective. A quick note before…

Leave a Reply Cancel reply

You must be logged in to post a comment.

This site uses Akismet to reduce spam. Learn how your comment data is processed .

Raise your Research Game with Bitesize Bio

Sign up for our feature-packed newsletter today to ensure you get the latest expert help and advice to level up your lab work.

You’ll stay up-to-date with our podcasts, webinars, workshops, downloadables, and more, delivered to your inbox every fortnight.

Don’t delay! Sign up now

Newsletters

All emails contain an unsubscribe link. You can review our privacy policy , cookie policy and terms and conditions online.

• Technical Skills
• More Skills

Bitesize Bio Powered

• Microscopy Focus
• Privacy Policy
• Cookie Policy
• Terms of Use

Copyright © Science Squared – all rights reserved

• Assessing Primary and Secondary Sources
• Edit on GitHub

Assessing Primary and Secondary Sources ¶

Students identify trends, patterns and relationships; recognise error, uncertainty and limitations in data; and interpret scientific and media texts. They evaluate the relevance, accuracy, validity and reliability of the primary or secondary-sourced data in relation to investigations. –Chemistry Stage 6 Syllabus, NESA

The definitions of accuracy, reliability, and validity are taken from Resources for science instruction (NSW Education)

Primary Sources ¶

The extent to which a measured value agrees with the true value. Requires prior knowledge about the value to be measured.

An experiment which is accurate should show a value that is close to the true value.

Reliability ¶

The extent to which the findings of repeated experiments, conducted under identical or similar conditions, agree with each other. Repeating the experiment minimises the effect of outliers, etc.

The extent to which an experiment addresses the question under investigation. Requires the experiment to be reliable, accurate, and precise. In addition, you must be only changing ONE independent variable , and controlling all the other variables. It must also be using the correct equipment, and addressing the aim. Validity can be assesed with:

mention variables being controlled

only ONE variable should be changed, and its effect on ONE dependent variable

state how errors (e.g. friction in a pendulum swing gravity experiment) have been minimised (or not)

Precision ¶

This is not part of the chemistry syllabus. However, it is important to know this as it is a distinct category to accuracy or reliability.

The extent to which multiple measurements, made under identical or similar conditions, agree with each other (i.e. variations within a dataset).

This is referring to the uncertainty of measurements, i.e. how close the measured the value is to the value being measured. Can be quantified with the range of values, written as: \(5\pm1\) .

See Mainpulating Uncertainties in Resources for science instruction (NSW Education) for how to combine uncertainties.

Secondary Sources ¶

The consistency of information between sources. Can be evaluated through showing that various sources all gave the same information.

The appropriateness of the information. Needs to consider:

the author’s credentials (are they qualified in that field)

the purpose of the article - is it biased?

is it current (not outdated - this is not “recent”)

is the publisher reputable?

The information needs to be both valid and reliable.

In order to continue enjoying our site, we ask that you confirm your identity as a human. Thank you very much for your cooperation.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• HHS Author Manuscripts

Accuracy, Precision, and Reliability of Chemical Measurements in Natural Products Research

Joseph m. betz.

a Office of Dietary Supplements, U.S. National Institutes of Health, Bethesda, MD 20892, USA

Paula N. Brown

b Centre for Applied Research & Innovation, British Columbia Institute of Technology, Burnaby, BC, V5G 3H2, Canada

Mark C. Roman

c Tampa Bay Analytical Research, Inc., Largo, FL 33777

Associated Data

Natural products chemistry is the discipline that lies at the heart of modern pharmacognosy. The field encompasses qualitative and quantitative analytical tools that range from spectroscopy and spectrometry to chromatography. Among other things, modern research on crude botanicals is engaged in the discovery of the phytochemical constituents necessary for therapeutic efficacy, including the synergistic effects of components of complex mixtures in the botanical matrix. In the phytomedicine field, these botanicals and their contained mixtures are considered the active pharmaceutical ingredient (API), and pharmacognosists are increasingly called upon to supplement their molecular discovery work by assisting in the development and utilization of analytical tools for assessing the quality and safety of these products. Unlike single-chemical entity APIs, botanical raw materials and their derived products are highly variable because their chemistry and morphology depend on the genotypic and phenotypic variation, geographical origin and weather exposure, harvesting practices, and processing conditions of the source material. Unless controlled, this inherent variability in the raw material stream can result in inconsistent finished products that are under-potent, over-potent, and/or contaminated. Over the decades, natural products chemists have routinely developed quantitative analytical methods for phytochemicals of interest. Quantitative methods for the determination of product quality bear the weight of regulatory scrutiny. These methods must be accurate, precise, and reproducible. Accordingly, this review discusses the principles of accuracy (relationship between experimental and true value), precision (distribution of data values), and reliability in the quantitation of phytochemicals in natural products.

1. INTRODUCTION

The word “pharmacognosy” was coined in the early 19th century to designate the discipline related to the study of medicinal plants [ 1 ]. The science of pharmacognosy became aligned with botany and plant chemistry, and until the early 20 th century, dealt mostly with physical description and identification of whole and powdered plant drugs including their history, commerce, collection, preparation, and storage. Advances in organic chemistry added a new dimension to the description and quality control of these drugs, and the discipline has since expanded to include discovery of novel chemical therapeutic agents from the natural world.

While discovery of new chemical entities has become the modern focus of much natural products work, identification and quality control remain important for pharmacopoeial identification and quality control of goods traded as crude botanicals or extracts [ 2 ]. Books and courses on analytical chemistry often do not fully describe the overall process of analytical method design, development, optimization, and validation [ 3 ]. As a result, the chemical literature is rich in procedures that have been developed with variable rigor and conclusions that imply, rather than prove, correctness and validity of reported results. Peer-review of publications that report quantitative results but are not primarily analytical papers may not address method validity and the methods may not be useful for actual samples. The role of reliable measurements in regulatory settings has obvious public health implications; tight control over active ingredients, nutrients and other constituents of foods and supplements (including deleterious substances such as pesticides and toxic elements) are necessary for safety and efficacy.

While this review cannot capture the breadth of all existing rules surrounding measurements made on commercial goods, two excerpts from U.S. Good Manufacturing Practice (GMP) regulations for drugs and dietary supplements shall highlight the importance that the U.S. government places on the integrity of data. For drugs, 21 CFR Part 211.194 (a)(2) requires a “statement of each method used. … statement shall indicate the location of data that establish that the methods used in the testing… meet proper standards of accuracy and reliability…” [ 4 ]. For dietary supplements, 21 CFR Part 111.75 requires manufacturers to “ensure that the tests and examinations that you use to determine whether the specifications are met are appropriate, scientifically valid methods”, and notes that “a scientifically valid method is one that is accurate, precise, and specific for its intended purpose” [ 5 ]. The International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH) [ 6 ] defines fitness for purpose as the “degree to which data produced by a measurement process enables a user to make technically and administratively correct decisions for a stated purpose.” This relates to scope and applicability. In order for a method to be of use, it needs to be tailored to specific analytes, matrices and expected concentration ranges.

However, method development and validation can be challenging when dealing with poorly defined analytes, such as antioxidants, flavonoids and phenolics, as well as the complex matrices of botanical raw materials and finished products. Defining analytes and matrices in the fitness for purpose statement is important for developing a successful method.

2. PARAMETERS OF VALIDATION

Various organizations are involved with analytical method validation: (a) the International Union of Pure and Applied Chemistry (IUPAC) publishes chemical data and standard methods for analytical, clinical, quality control and research laboratories, while ICH has developed validation guidelines [ 6 ]; (b) FDA s “Guidance for Industry: Analytical Procedures and Methods Validation” provides recommendations on submitting analytical procedures, validation data and samples to support the documentation of the identity, strength, quality, purity and potency of drug substances and drug products [ 7 ]. A more specific guidance document focuses on the “what” and “how” of chromatographic method validation [ 8 ]; (c) AOAC International (AOACI) produces rigorous, well recognized validation guidelines that range from single laboratory validation (SLV) guidelines [ 9 ] complete with acceptance criteria [ 10 ] and sample protocol [ 11 ] to guidelines for the conduct of interlaboratory collaborative studies [ 12 ].

While there are numerous approaches to quantitative chemical analysis of natural products, space is limited and this review will focus on validation of chromatographic methods since they are the most widely used for determination of phytochemicals in raw materials and finished products. Analytical methods are not universal; characteristics, techniques, scope and applicability can differ substantially. Thus, it is impossible to have a single set of instructions that can be used to validate all methods. However they do share basic commonalities that can be addressed to ensure confidence in their use and the measurements obtained. Beyond the health implications of inaccurate measurements made on commercial products, practitioners should be aware that inaccurate quantitative measurements can cause significant bias when they are published.

2.1 ACCURACY AND PRECISION

A good starting point for basic definitions and descriptions of the key terms and concepts pertaining to the assurance of the quality of quantitative chemical measurements is the U.S. Food and Drug Administration s (FDA) Reviewer Guidance [ 8 ]. The two most important elements of a chromatographic test method are accuracy and precision. Accuracy is a measure of the closeness of the experimental value to the actual amount of the substance in the matrix. Precision measures of how close individual measurements are to each other.

2.1.1 Accuracy and Recovery

The purpose of analysis of botanicals and other natural products is quantitation of target compounds in the matrix in which the compounds occur. The most common technique for determining accuracy in natural product studies is the spike recovery method, in which the amount of a target compound is determined as a percentage of the theoretical amount present in the matrix. In a spike recovery experiment, a measured amount of the constituent of interest is added to a matrix (spiked) and then the analysis is performed on the spiked material, from the sample preparation through chromatographic determination. A comparison of the amount found versus the amount added provides the recovery of the method, which is an estimate of the accuracy of the method. In an ideal situation, such as the determination of a synthetic pesticide in food, the matrix will be devoid of the target analyte(s). However, this is seldom the case in phytochemical studies where the target analyte occurs naturally in the matrix. Therefore, analysts will frequently perform parallel analyses of spiked and un-spiked materials. The theoretical recovery of the target analyte from the spiked material is the sum of the amount of added analyte plus the amount of naturally occurring analyte (as determined in the parallel analysis of unspiked material). The difference between the theoretical amount and the amount analytically determined in the spiked matrix provides an estimate of accuracy. Other approaches to spike recovery studies include adding the target analyte to a similar matrix that does not contain the target and spiking the target analyte into natural matrix from which the target has been exhaustively extracted and then dried. Recovery is frequently concentration dependent; the FDA guidance for drugs [ 8 ] suggests that matrices be spiked at 80, 100, and 120% of the expected value, and that the experiment be performed in triplicate. For botanical materials and dietary supplements; where the analyte may be present over a large concentration range, recovery should be determined over the entire analytical range of interest for the method.

While analyte addition has both pros and cons, it is one commonly practiced in the natural products community. Other techniques such as exhaustive extraction can be used to help verify the accuracy of the method. In some cases a certified reference material may be available that contains the substance(s) of interest. These materials contain a known amount of the analyte with a given uncertainty and can be used in lieu of and/or in addition to analyte spiking. If available, certified reference materials can be obtained from national metrological laboratories such as the U.S. National Institute for Standards and Technology (NIST), the Environmental Protection Agency (EPA), or commercial suppliers.

Various factors affect the accuracy of an analytical method. These range from extraction efficiency to stability of the analyte to adequacy of the chromatographic separation and can generally be optimized during the method development and optimization phase of a study.

Important but frequently overlooked factors that affect accuracy are assumptions made in setting up and performing the assays. The first assumption involves the purity of the reference materials used to establish the identity of the analyte, create the calibration curve, and arrive at a quantitative analytical result. Available in milligram to gram quantities, these materials are usually accompanied by a label declaration of purity and/or a certificate of analysis that includes a purity declaration. Depending on their stability and the technique(s) used to determine their purity, the actual purity of these materials may differ from the claimed value, and investigators should take steps to assure identity and purity before using them.

The second assumption also involves calibration standards. There are many compounds that are not commercially available or that are prohibitively expensive. As a result, some analyses are designed to use a single compound that is nominally similar to all of the analytical targets, and quantitative results for the other compounds are expressed in terms of the one compound at hand (normalization). In UV detection, this may be appropriate if the specific extinction coefficients of the target compounds are similar; the less similar they are, the more inaccurate are the results.

2.1.2 Accuracy Case Study

An HPLC investigation [ 13 ] of cranberry ( Vaccinium macrocarpon Aiton) was performed using two different means of constructing the calibration curve for the major cranberry anthocyanins. The first set of experiments was modeled after previous approaches [ 14 ] and compared results of the quantitation of individual anthocyanins in cranberry fruit using cyanidin-3-glucoside as calibrant for all compounds. The underlying assumption was that detector response at a wavelentgth of 520 nm would be the same for all of the anthocyanins. In the second experiment, the major anthocyanins were obtained and used to construct individual calibration curves for each. When individual calibration curves were used, the amounts of individual compounds were found to be different from those reported using normalization ( Figure 1 , Table 1 ).

Graphical comparison of anthocyanin content in cranberry fruit when determined by normalization using cyanidin 3-O-glucoside as the external calibrant as compared to quantitation using calibration curves generated for each individual anthocyanin [ 7 ]. C3Ga=Cyanidin-3- O -galactoside, C3Gl=Cyanidin-3- O -glucoside, C3Ar= Cyanidin-3- O -Arabinoside, P3Ga=Peonidin-3- O -galactoside, P3Ar=Peonidin-3- O -galactoside.

Anthocyanin content of cranberries determined by HPLC using normalization to cyanidin 3-O-glucoside [ 13 , 14 ] vs. anthocyanin content determined using individual anthocyanins as calibrants [ 13 ]

Purity of reference materials can also affect accuracy. An illustration of the importance of verifying the purity of chemicals used as calibrants is provided in Table 2 . In the HPLC investigation of cranberry anthocyanins described above [ 13 ], calibration standards for the five major cranberry anthocyanins were purchased from a commercial supplier. In preparation for the analysis, the investigator determined the purity of the purchased standards using a standard approach [ 15 ]. While the manufacturer’s certificates of analysis declared that all five compounds were > 97% pure (as determined by HPLC), the investigators found that their actual purity ranged from 66–97%. Calculation of individual anthocyanin content of cranberry using the declared purity of the calibration standards would have resulted in inaccurate results for several of the compounds. In addition, actual purities were different for different lots of the same material.

Claimed and actual purity of commercial cranberry anthocyanins [ 13 ]

2.1.3 Precision

The FDA guidance document on validation of chromatographic methods [ 8 ] breaks the overall concept of precision into three components: repeatability , intermediate precision , and reproducibility. Repeatability is a measure of the within-laboratory uncertainty. It takes into account the reproducibility of injections and other aspects of the analysis such as weighing, fluid dispensing and handling, serial dilution, and adequacy of extraction. Among other factors, calibration of balances and glassware can increase repeatability. The guidance recommends that a validation package include data from a minimum of 10 injections that show a relative standard deviation of less than one percent. Intermediate precision is a measure of the ruggedness of the method, i.e., reliability when performed in different environments. Demonstration of intermediate precision requires that the method be run on multiple days by different analysts and on different instruments. At a minimum, such studies should be run on at least two separate occasions. Reproducibility is an indication of the precision that can be achieved between different laboratories and is evaluated using multi-laboratory collaborative studies.

As with accuracy, precision can be affected by a number of factors. Use of inappropriate or uncalibrated equipment such as pipets or analytical balances, failure to control light or moisture when required, or inadequately trained analysts can all reduce precision. Inadequate chromatographic resolution, tailing peaks, and attempts to measure different analytes across an excessive dynamic range can also decrease precision as data handling systems struggle to perform integrations against unstable baselines. The problem is especially acute when simultaneously determining low and high levels of analytes in complex natural products. Finally, the lack of homogeneity between test portions in multi-laboratory studies can result in apparent imprecision.

2.1.4 Precision-Case Study

Decoctions of Má Huáng or ephedra ( Ephedra sinica Stapf., E. equisetina Bunge, E. intermedia var. tibetica Stapf., or E. distachya L.) are used in Traditional Chinese Medicine to expel cold wind. In western allopathic medicine, ephedrine and pseudoephedrine, first isolated from Ephedra spp. [ 16 ], are used for treatment of asthma and as a decongestant. Until banned from use as a dietary supplement ingredient by FDA in 2004 [ 17 ], ephedra plants and their extracts were used as ingredients in dietary supplements intended for weight loss and to “increase energy” [ 16 ]. Early FDA attempts to analyze ephedra-containing products for alkaloid content met with mixed success as the available published analytical methods were designed primarily for ephedrine and/or pseudoephedrine in finished pharmaceutical dosage forms or for a single plant species. Ephedra products marketed in the US as dietary supplements were almost always sold as mixtures of several plant species and often included caffeine and other alkaloids. Figure 2A is typical a HPLC chromatogram [ 18 ] of a multi-botanical ephedra product using a published method for separation of ephedrine alkaloids in ephedra herb [ 19 ]. The sample was run as part of an FDA investigation [ 18 ], and sample preparation involved a solvent extraction without additional cleanup. Note the complexity of the chromatogram and the incomplete resolution of the pseudoephedrine (P) and N-methylephedrine (N-ME) peaks from non-ephedra botanical constituents. The separation was sufficient to allow identification of the major alkaloids, but repeat injections of the same sample yielded different area under the curve values due to difficulties in integration.

Comparison LC-UV chromatograms of dietary supplement products containing ephedra, caffeine, and other botanical ingredients using three different analytical methods. A: Extraction with no cleanup [ 18 , 19 ]. B: Solid-phase extraction [ 20 , 21 ]. C: AOAC Official Method of Analysis [ 22 – 24 ]. E=Ephedrine, P=Pseudoephedrine, N-ME=N-methylephedrine, NE=Norephedrine, NPE=Norpseudoephedrine, N-MPE=N-methylpseudoephedrine, Ph=Phentermine.

Figure 2B shows a chromatogram of a multi-herb ephedra product obtained [ 20 ] using a method [ 21 ] that included a solid-phase extraction cleanup step and phentermine (Ph) as internal standard. It provides for near-baseline separation of the six ephedra alkaloids in the complex multi-botanical product because the sample cleanup has removed most of the interfering substances. This method gave good precision for ephedrine (E) and pseudoephedrine (P) measurements, but norpseudoephedrine (NPE) was present in small quantities relative to E and was not well resolved from a small inflection in the baseline at about the same retention time. Thus, unreliable integration of the peak reduced precision for NPE. In addition, column performance and mobile-phase composition had to be carefully monitored for this separation. The peak eluting at 11.219 minutes in Figure 2B (just after pseudoephedrine) was identified by LC/MS as a phthalate that was leached from the solid-phase extraction (SPE) column used for cleanup. Consequently, small deviations in the organic content of the mobile-phase or column aging caused loss of resolution and imprecise integration of the pseudoephedrine peak.

Finally, Figure 2C shows a typical HPLC chromatogram of a multi-botanical ephedra product [ 22 ] obtained using the AOAC Official Method of Analysis [ 23 ]. This method yields much improved resolution and lack of interference for NPE, E, PE, and N-ME. A small interference with an unknown constituent remains with the NE peak. In the validation study that led to the approval of the official method, overall precision was deemed adequate only for E and PE [ 24 ]. Quantitative determination of the other four compounds was not sufficiently precise due to a lack of homogeneity in the blind duplicate test articles sent to the individual investigators in the collaborative study rather than to any fault of the method itself [ 24 ].

3. ADDITIONAL VALIDATION PARAMETERS

Additional parameters to be evaluated when demonstrating accuracy and precision are part of the method development and optimization process, or are performed during the validation process when demonstrating acceptable method performance. These parameters include limits of detection and quantification, linearity of the method, range, recovery, robustness and selectivity.

3.1 LIMITS OF DETECTION AND QUANTIFICATION

The Limit of Detection (LOD) is defined as the smallest amount or concentration of an analyte that can be reliably detected in a given type of sample or medium by a specific measurement process [ 25 ]. The United States Pharmacopeia defines the LOD as 2 or 3 times the baseline noise [ 26 ]. This is derived from the assumption that 3 times the noise will contain approximately 100% of the data from a normal distribution. Alternatively, the AOAC [ 9 ] and IUPAC [ 27 ] calculate limits from the variability of a blank matrix. With this methodology, the LOD is based on a minimum of 6 independent determinations of a matrix blank, where the LOD will equal the sum of the mean of blank measures and the product of the standard deviation of the blank measures and a numerical factor chosen according to the confidence level desired. The confidence level should be the Student t statistic with α = 0.05 [ 28 ], Alternatively, a value of 3 can also be used according to AOAC [ 9 ] and IUPAC [ 27 ].

The FDA chromatography guidance document notes that simply using instrument noise to estimate the limits is not adequate [ 8 ]. According to FDA, the value obtained from the chromatogram can be considered as an instrument detection limit rather than a method detection limit because the baseline noise technique does not take into consideration errors that occur during sample preparation. Although a blank that has gone through the entire sample preparation procedure may account for some of these errors, it is important to consider analyte specific effects, such as the UV extinction coefficient, which may contribute to the detection limit. Therefore, it is recommended that the LODs be calculated from the analysis of samples containing the analyte of interest [ 8 , 27 , 28 ]. The U.S. Environment Protection Agency (EPA) defines the Method Detection Limit (MDL) to be product of the standard deviation and Student t value calculated from the analysis of at least seven samples containing a low level of analyte that is near the actual detection limit [ 29 ]. All of the described methods are statistical estimates of the limit of detection and the levels should be verified under actual conditions of use.

Another limit to consider for an analytical method is the Limit of Quantification (LOQ). The LOQ is the amount of substance that can reliably be assigned a quantitative value. This limit is usually defined as 10% RSD [ 27 ] or as a fixed multiple (typically 10) of the noise [ 26 ] or standard deviation [ 29 ] used to calculate the detection limit.

3.2 LINEARITY & RANGE

In a validated method, the detector response should be linear over the anticipated range of analyte concentrations. Linearity is determined by creating a minimum 5 level calibration curve using the analyte(s) of interest. The resulting plot of detector response versus analyte concentration should have a regression coefficient of at least 0.999, and should be visually inspected for areas of non-linearity. Figures 3A and 3B [ 8 ] show plots of area under the curve versus concentration for two different analytes. Figure 3A shows an acceptable linearity over the entire range of concentrations evaluated, while Figure 3B does not. Figure 3C is a gas chromatogram of an extract of an ephedra product [ 16 ] obtained using a nitrogen/phosphorous detector. The chromatogram is enlarged to allow visualization of the minor alkaloid peaks (N-MPE, PE, N-ME, NE), and the ephedrine peak was truncated in this view. Truncation can result in integration errors, and in fact the calibration curve across the entire range of analytes was not linear. In this case, the sample had to be analyzed twice: the first analysis was performed on an undiluted sample, and the second on a diluted sample in order to bring the detector response for the ephedrine peak into the linear portion of the calibration curve. Both analyses were necessary, because the dilution step dropped the minor alkaloid concentrations below their limits of detection. Knowing the working range, (i.e., the interval between the high and low levels of analytes to be determined) of a method prevents erroneous interpretation of results.

Linearity and dynamic range. A, B: Calibration curves plotting generic detector response versus concentration [ 8 ]. C: Gas chromatogram of an ephedra product extract using a Nitrogen/Phosphorous Detector [ 16 ]. E=Ephedrine, P=Pseudoephedrine, N-ME=N-methylephedrine, NPE=Norephedrine, N-MPE=N-methylpseudoephedrine, Ph=Phentermine, ?=Unknown.

3.3 ROBUSTNESS

Robustness is typically evaluated during method development/optimization, but can have a pronounced effect on the validation of a method. Robustness experiments measure a method s ability to remain unaffected by small but deliberate variations in method parameters. Examples of potentially sensitive processes include extraction time, extraction temperature, and extraction process (soxhlet, wrist shaker, orbital shaker). Column oven temperature, the percent organic phase, pH, or buffer concentration of mobile phase may also be important for chromatographic separations. Figure 4 provides a graphic comparison between chromatography outcomes in LC/MS analyses of ephedrine alkaloids and shows the differences in baseline noise, chromatographic resolution, peak shape, and analysis time achieved when HPLC columns with different carbon loading (4A) or ion-pairing reagents (4B) were used [ 30 ]. Impact of ion-pairing reagents and other factors on detector response is not addressed, but may be important to overall method performance.

Representative total ion chromatogram of mixtures of six ephedra alkaloid standards plus ephedrine-d 5 . A: Separation using 3 different ion-pairing reagents. B: Separation on LC columns with different amounts of carbon loading. 1=norephedrine, 2=norpseudoephedrine, 3=ephedrine-d 5 , 4=ephedrine, 5=pseudoephedrine, 6=N-methylephedrine, and 7=methylpseudoephedrine [ 23 ].

Although the parameters affecting the method can be explored using an approach that tests one variable at a time, the use of factorial studies can be much more efficient when facing a large number of factors [ 9 , 31 ]. For instance, the AOAC International recommends the use of a Youden Ruggedness Trial that permits the examination of up to 7 factors in a single experiment requiring only 8 determinations [ 9 ].

3.4 SPECIFICITY (SELECTIVITY)

It is vital to ensure the identity of the chromatographic peak that will be measured. When evaluating the previously mentioned HPLC method for determination of ephedrine-alkaloids in botanical supplements [ 21 ], a matrix blank was run using Ephedra nevadensis as the test article. This North American species was once thought to contain pseudoephedrine [ 32 ], but this claim has been controversial. Analysis using the method shown in Figure 2B produced a chromatogram (not shown) that had a flat baseline except for a small, unexpected peak that the HPLC/UV data system erroneously identified as pseudoephedrine. As noted previously, LC/MS analysis found this peak to be a phthalate from the solid-phase extraction column. Instead of an confirming the presence of pseudoephedrine in E. nevadensis , this only showed that certain solvents are incompatible with certain brands of SPE columns. The claim that E. nevadensis contains ephedrine-types alkaloids was subsequently dismissed [ 33 ]. A classical technique for verifying, but not proving, analyte identity is standard addition to a natural matrix that contains the compound of interest. Other techniques for analyte verification include the use of a photodiode array detector or a mass spectrometer. An earlier technique collects the eluted peak and performs subsequent mass spectrometry or another identity analysis.

3.5 REFERENCE MATERIALS

Finally, identity, purity, and stability of reference compounds must be confirmed. While the case for reference material purity was already made above, the authors have experienced instances in which commercial chemicals intended for use as reference materials have been incorrectly identified. In one case, proton NMR was used to confirm the identity of purchased hydrastine when received from the supplier. The experiment demonstrated that the alkaloid dimer (hydrastine) had decomposed into hydrastinine, its constituent monomers. In a second case, the detergent, 3-[(3-cholamidopropyl)dimethylammonio]-1-propanesulfonate (CHAPS), had been shipped labeled as caffeine. These incidents typically do not make it into the peer-reviewed literature, but do occur.

In the age of reliable autosamplers, it is also important to assure the stability of analytical standards and target analytes in solution for the duration of the test-run. In the gas chromatogram seen in Figure 3C , the small peak eluting a few minutes before the N-MPE peak was not present when the extract was first made. As the solution aged, it turned from clear and colorless to yellow. As the color of the solution increased, so did the size of the unidentified peak. Solutions of the pure compound NE also turned yellow with time, even at refrigerator temperatures, and the size of the unknown peak increased as the intensity of the yellow color increased. More important, the size of the NE peak decreased as the size of the unknown peak increased.

In practice, it is often difficult or impossible to confirm the purity of reference materials due to their limited availability and cost. In these situations, certificates of analyses should be examined for accuracy and completeness. Determination of moisture, residual solvents, residue on ignition (inorganics), and chromatographic purity (preferably by two independent methods) are all needed to obtain an accurate assessment of material suitability. Moisture in particular can be problematic, and it is important to equilibrate the standards before use under the same conditions used prior to the moisture determination.

3.6 CHROMATOGRAPHIC PERFORMANCE

While extraction efficiency, analyte stability and purity, linearity, recovery, and selectivity are important to the final result, they must all lead to a viable separation. This is evaluated by determining system suitability. A typical approach involves development of an optimized method with adequate system suitability, prior to performing validation studies. The FDA reviewer guidance [ 8 ] suggests that the peak of interest should have a capacity factor (k′) greater than or equal to 2 and a resolution (R S ) greater than 2. Additional desirable characteristics are provided in detail in the FDA guidance [ 6 ] and in numerous other sources [ 3 , 9 , 12 , 26 , 27 , 34 – 37 ].

4. CONCLUSIONS

Systematic evaluation of analytical method performance is critical to the utility of analytical methods and to the integrity of scientific research. While accuracy, precision, and fitness for purpose are often assumed in published methods, this assumption does not bear close scrutiny in many cases. Accurate measurements are as important in clinical- and pre-clinical studies as they are in regulatory or manufacturing environments. While demonstration of performance should be a pre-requisite for any quantitative method used in a laboratory, the burden of proving that any measurements made are correct and reproducible depends on the intended use and pedigree of the method being evaluated.

There are a number of validation study designs available, and each is intended to accomplish certain pre-defined goals. In-house or single laboratory validation (SLV) studies can demonstrate applicability of the method to the analysis at hand, evaluate intra-laboratory performance, ruggedness, accuracy, and repeatability while identifying interferences and critical control points [ 9 ]. Inter-laboratory collaborative studies, including but not limited to studies for the purpose of creating AOAC Official Methods of Analysis, provide information on inter-laboratory reproducibility [ 12 ].

Finally, performing validation experiments is often viewed as “technician s work”. However, designing an appropriate validation protocol that will demonstrate the functional qualities required of the method, performing the appropriate statistics on the results, and drawing the correct conclusions from those statistics requires considerable knowledge and intellectual input. Knowledgeable senior scientists should be involved in assuring integrity of published quantitative chemical data of natural product analysis.

Supplementary Material

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.