Validity and reliability are measured using different statistical methods because they capture different qualities. Reliability tells you whether a measurement produces consistent results across repeated uses, while validity tells you whether it actually measures what it claims to measure. A tool can be perfectly reliable without being valid: an alarm clock that rings at 6:30 every morning when it’s set for 7:00 is consistent, but wrong. Measuring both requires a combination of statistical tests, expert judgment, and structured comparisons.
How Reliability Differs From Validity
The core distinction comes down to error type. Random error, the kind that fluctuates unpredictably from one measurement to the next, reduces reliability. Systematic error, a consistent bias that skews every result in the same direction, reduces validity. A bathroom scale that reads three pounds heavy every time is reliable (consistent) but not valid (not accurate). A scale that gives you a different number each time you step on it has a reliability problem.
This means you need to evaluate them separately, using different tools. Reliability testing focuses on whether scores stay stable across time, raters, or items. Validity testing focuses on whether the instrument connects meaningfully to the concept it’s supposed to capture.
Measuring Internal Consistency
Internal consistency checks whether the individual items on a test or questionnaire are measuring the same underlying concept. The most common metric is Cronbach’s alpha, a coefficient that ranges from 0 to 1. Values between 0.70 and 0.90 are generally considered acceptable. Below 0.70 often signals that items aren’t well related to each other or that there are too few questions. Above 0.90 can actually be a red flag, suggesting that some items are redundant and the instrument could be shortened without losing information.
A low alpha doesn’t always mean bad questions. It can also result from trying to measure a concept that’s genuinely multidimensional with a single scale. If your questionnaire covers several distinct subtopics, you’d want to calculate alpha for each subscale rather than for the whole instrument.
Measuring Test-Retest Reliability
Test-retest reliability checks whether the same person gets a similar score when measured at two different time points. You administer the instrument once, wait an appropriate interval, then administer it again to the same group. The key statistic here is the intraclass correlation coefficient (ICC) for continuous variables like pain scores or mood ratings. For yes/no or categorical outcomes, Cohen’s kappa or intraclass kappa is preferred.
ICC is favored over the more familiar Pearson correlation because it captures both how closely scores track together and how much they actually agree in absolute terms. Pearson’s correlation only tells you whether two sets of scores move in the same direction; it misses systematic shifts. If every person’s second score is exactly five points higher than their first, Pearson’s r would be perfect, but the ICC would flag the discrepancy.
Interpretation guidelines for ICC values, based on 95% confidence intervals: below 0.5 indicates poor reliability, 0.5 to 0.75 is moderate, 0.75 to 0.9 is good, and above 0.90 is excellent.
Measuring Inter-Rater Reliability
When your measurement depends on human judgment, such as clinicians rating symptom severity or coders categorizing survey responses, you need to verify that different raters produce similar results. For categorical judgments with two raters, Cohen’s kappa is the standard metric. It improves on simple percent agreement by accounting for the amount of agreement you’d expect by chance alone. For continuous ratings or situations with more than two raters, the ICC for multiple raters serves the same purpose.
The process is straightforward: have two or more raters independently evaluate the same set of cases, then calculate the appropriate statistic. Kappa values follow a similar interpretive logic to ICC, where higher values indicate better agreement, though the specific thresholds vary by field. Low inter-rater reliability usually points to vague scoring criteria or insufficient rater training, both of which can be addressed before a full study begins.
Measuring Content Validity
Content validity asks whether your instrument covers the full scope of the concept you’re trying to measure, without including irrelevant material. This is evaluated through expert judgment rather than statistics. You assemble a panel of people with expertise in the topic, and each panelist independently rates every item for relevance and representativeness.
The results can be quantified using Lawshe’s content validity ratio (CVR), which compares the number of panelists who agree an item is essential against those who don’t. The minimum acceptable inter-judge agreement is 50%. Items that fall below this threshold get revised or dropped. The content validity index (CVI) averages the CVR values across all items retained in the final instrument, giving you a single summary number for the whole tool.
Measuring Face Validity
Face validity is the simplest and most subjective form of validation. It asks whether the instrument looks like it measures what it’s supposed to measure, as judged by the people who will actually use it or complete it. Evaluators assess clarity, readability, formatting, and appropriateness for the intended audience. Unlike content validity, face validity doesn’t require subject-matter experts; it can be assessed by members of the target population.
Face validity is typically evaluated during pilot testing, before the instrument is finalized. A common approach is to have a small group (eight to ten people is typical) review the questionnaire and flag items that are confusing, poorly worded, or seemingly unrelated to the topic. This step often leads to revisions in wording and layout. It’s a necessary first pass, but it’s not sufficient on its own because something can appear valid on its surface while failing more rigorous validity tests.
Measuring Criterion Validity
Criterion validity compares your instrument against an established reference, often called a gold standard. It comes in two forms depending on timing. Concurrent validity measures how well your tool’s scores line up with the gold standard when both are assessed at the same time. Predictive validity measures how well your tool’s scores forecast a future outcome of clear importance, such as disease progression or treatment response.
Both are measured using correlation coefficients. You calculate the correlation between scores on your instrument and scores on the reference measure (for concurrent validity) or the outcome of interest (for predictive validity). Higher correlations indicate stronger criterion validity. This type of evidence is especially powerful because it ties your measurement directly to real-world outcomes rather than relying on theoretical arguments.
Measuring Construct Validity
Construct validity is the broadest and most theoretically demanding form. It tests whether your instrument behaves the way theory predicts it should. Two complementary approaches are used: convergent and discriminant validity.
Convergent validity checks that your instrument correlates strongly with other instruments measuring the same or closely related concepts. If you’ve built a new anxiety questionnaire, its scores should correlate highly with scores from an established anxiety measure. You calculate this using Pearson’s correlation coefficient, and a high value supports convergent validity.
Discriminant validity checks the opposite: your instrument should not correlate strongly with measures of unrelated concepts. That same anxiety questionnaire should show a low correlation with, say, a measure of physical fitness. A low correlation here is the good result, because it means your tool is capturing something distinct rather than just picking up general noise.
A technique called the multitrait-multimethod matrix, introduced by Campbell and Fiske in 1959, evaluates both convergent and discriminant validity simultaneously. You measure multiple traits using multiple methods, then examine the full pattern of correlations. High correlations between different methods measuring the same trait (convergent evidence) combined with low correlations between different traits measured by different methods (discriminant evidence) provide the strongest support for construct validity.
The Order of Operations
Validity and reliability testing follow a logical sequence during instrument development. You start with content and face validity while the tool is still being built, using expert panels and pilot testing to refine items. Once the instrument is in a stable form, you test reliability: internal consistency, test-retest stability, and inter-rater agreement as applicable. Reliability testing may prompt further revisions to reduce redundancy or improve consistency. Only after the tool is demonstrably reliable do you move to the more resource-intensive forms of validity testing, including criterion and construct validity, which require collecting data from larger samples and comparing against external measures.
Planning this sequence in advance matters. Each stage can trigger changes to the instrument, and testing validity on a version that later gets revised wastes effort. By the time the tool is deployed in actual research, the goal is to have accumulated evidence across multiple types of reliability and validity, building a cumulative case that the instrument measures its intended construct consistently and accurately.