What Makes a Test Valid and Reliable, Explained

A test is reliable when it gives the same answer each time it measures the same thing. It is valid when that answer actually means what the test says it means. The two are not equals: a test can be perfectly consistent and still be measuring the wrong thing, which makes validity the harder and more important question. Neither is a single yes-or-no stamp. Each is built from several kinds of checks, and validity in particular is judged for a specific use, such as hiring or diagnosis.1Research Matters. Tracing the evolution of validity in educational measurement: past issues and contemporary challenges

Why No Test Score Is Perfectly Accurate

Any score you get is your true level plus some random error that pushes it up or down.2Wikipedia. Classical test theory A bad night’s sleep, a noisy room, a lucky guess or a vaguely worded question can shift a score without any change in what you actually know. A reliable test is one where that random push is small, so the score sits close to your true level.

This is also why small differences between people can mean nothing. If error is large, two people a few points apart may truly be equal, and ranking them or cutting off at an exact score treats noise as if it were real.

One caution: error that pushes everyone the same way, like a test that is simply too hard, is not random. That is a validity problem, not a reliability one.

Three Ways Reliability Gets Checked

Test-retest reliability means giving the same test twice and seeing whether people keep roughly the same scores. It is reported on a 0 to 1 scale, and about 0.70 or higher is widely treated as good, 0.60 to 0.69 as satisfactory, and below 0.60 as weak.3PubMed Central. Test–retest reliability, practice effects and estimates of change: A study on the Mindmore digital cognitive assessment tool Results vary a lot even inside one family of tests. Within one set of digital thinking tests, some tasks reached good consistency and six fell below 0.60.3PubMed Central. Test–retest reliability, practice effects and estimates of change: A study on the Mindmore digital cognitive assessment tool In a large battery of brain-function tests most scored 0.70 to the low 0.90s, while memory tests were notably less consistent.4PubMed. Test-retest reliability and practice effects of expanded Halstead-Reitan Neuropsychological Test Battery

Retesting has a built-in complication: people often score higher the second time simply from practice, and that gain is larger in younger adults.4PubMed. Test-retest reliability and practice effects of expanded Halstead-Reitan Neuropsychological Test Battery5PubMed. Test-retest reliability and practice effects of the Wechsler Memory Scale-III A higher second score may reflect familiarity, not real improvement. That matters when you are tracking change, like recovery after an injury or learning over a semester. Testers reduce the problem by using different versions or longer gaps between sessions. When someone reports an improvement, the useful question is whether it exceeds the usual practice gain.

The second check looks inside a single sitting. Internal consistency asks whether the questions on one test move together, and it is usually reported as Cronbach’s alpha.6Communication Methods and Measures. Use Omega Rather than Cronbach’s Alpha for Estimating Reliability. But… Alpha can slightly underestimate consistency, and a related figure called omega is often preferred, though the two usually come out close.6Communication Methods and Measures. Use Omega Rather than Cronbach’s Alpha for Estimating Reliability. But…7Social Sciences and Humanities Open. Estimating reliability: A comparison of Cronbach’s α, McDonald’s ωt and the greatest lower bound

The third check applies when people grade answers. Inter-rater reliability checks whether different graders give the same marks.8PubMed. Interrater agreement and interrater reliability: key concepts, approaches, and applications Some agreement measures count only exact matches, while others give partial credit when graders are close rather than far apart.9PubMed Central. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial

Essay exams and job interviews depend heavily on human judgment, so grader agreement matters there. A clear scoring guide makes graders agree more. A teacher can test this by having a colleague blind-mark a few papers and comparing the results.

The Kinds of Evidence That Make a Test Valid

Content validity means experts judge whether each question is relevant to what the test is supposed to cover.10PubMed. Evaluation of methods used for estimating content validity Questions the experts disagree on get revised or dropped.11PubMed. Two quantitative approaches for estimating content validity Experts may also seem to agree by chance, so checks for chance agreement can flag additional weak questions.11PubMed. Two quantitative approaches for estimating content validity That is why building a test often means writing many more questions than survive. One instrument kept 133 of 216 drafted items.12Education in Medicine Journal. Critical Thinking Transfer Practice Instrument: A Content Validity Calculation Steps Based on Expert Panel Evaluation

An exam meant to cover a whole unit that asks only about one chapter misses content. A test blueprint avoids this by mapping questions to topics in proportion, and a teacher can check by listing each question’s topic and seeing what is missing or overrepresented.

Criterion validity compares test scores with a trusted outside measure, like a full clinical diagnosis or later job performance.13PubMed Central. Criterion validity, construct validity, and factor analysis: An introductory overview It is called concurrent when both are measured at the same time and predictive when the outside measure comes days to years later.13PubMed Central. Criterion validity, construct validity, and factor analysis: An introductory overview14PubMed. Investigating the criterion validity of psychiatric symptom scales using surrogate marker validation methodology In two large hiring studies, results from current employees matched results from real applicants, despite applicants trying to look good.15Applied Psychology. Does concurrent validity really estimate predictive validity in psychological testing? Two local studies

Construct validity asks whether the test acts like the trait it names.13PubMed Central. Criterion validity, construct validity, and factor analysis: An introductory overview Scores should agree with other measures of the same trait gathered a different way.16PubMed. Convergent and discriminant validation by the multitrait-multimethod matrix But they should not agree too closely with measures of different traits, or the test may be measuring something else.17PubMed. The Criteria for Convergent and Discriminant Validity in a Multitrait-Multimethod Matrix

Consider a personality test that is really a mood test. If scores track today’s mood more than lasting traits, the claim to measure a trait fails, and comparing the test with a measure of mood reveals this. No single correlation proves construct validity.

How Reliability and Validity Fit Together

A test riddled with random error cannot track anything well, so poor reliability puts a ceiling on validity. The relationship does not run the other way, though. Increasing reliability beyond an optimal point can reduce how well a test predicts real outcomes.18The Journal of Experimental Education. Reconsideration of the “Attenuation Paradox”—and Some New Paradoxes in Test Validity Making questions ever more alike raises consistency while narrowing what the test covers. Validity, in other words, does not rise steadily with reliability.19Rasch Measurement Transactions. What is The Attenuation Paradox?18The Journal of Experimental Education. Reconsideration of the “Attenuation Paradox”—and Some New Paradoxes in Test Validity Asking the same thing ten ways inflates consistency figures and can hide that the test ignores other parts of the skill, so a very high alpha is not automatically a good sign.

A bathroom scale makes the distinction concrete. One that always reads five pounds heavy is perfectly consistent, but it is consistently wrong, so it is reliable and not valid. A scale that jumps around randomly is neither. Only a steady, accurate scale is both.

Why Validity Depends on How Scores Are Used

Modern measurement treats validity as a judgment of how well evidence supports the inferences and actions based on scores.1Research Matters. Tracing the evolution of validity in educational measurement: past issues and contemporary challenges Early thinking reduced validity to a single correlation, which later proved too narrow.1Research Matters. Tracing the evolution of validity in educational measurement: past issues and contemporary challenges So validity is gathered from many sources over time rather than proved once.1Research Matters. Tracing the evolution of validity in educational measurement: past issues and contemporary challenges

That makes the same test valid for one purpose and shaky for another. A reading test that works well for placing students in classes may not justify holding a child back a grade. A screening quiz may flag risk without being able to diagnose. The higher the stakes, the more evidence a use needs, so the practical question is what decision the score will drive.

Consequences count too. Scores can cause unintended harm when they capture only part of the trait or pick up things unrelated to it.20PubMed. Social consequences of educational measures: A commentary A hiring test that screens out a group for unrelated reasons fails its use. Teaching to a narrow test can shrink what students learn. These effects count against the test’s validity for that purpose.

Is the Test Fair Across Groups and Languages

To find a biased question, testers match people on overall ability and see whether groups still differ on that one item.21PubMed Central. Checking Equity: Why Differential Item Functioning Analysis Should Be a Routine Part of Developing Conceptual Assessments A question that equally able groups answer differently may be measuring something else, like culture or vocabulary.21PubMed Central. Checking Equity: Why Differential Item Functioning Analysis Should Be a Routine Part of Developing Conceptual Assessments22PubMed. Modern psychometric methods for detection of differential item functioning: application to cognitive assessment measures These checks have been used on memory and thinking screens across education levels and racial and ethnic groups.22PubMed. Modern psychometric methods for detection of differential item functioning: application to cognitive assessment measures

A biased item might be a math word problem about a sport few groups know. Equally skilled people miss it for reasons unrelated to math.

Translated tests need their own evidence. Straight translation is not enough because idioms and examples may not carry over, and a symptom described one way in one culture may be described very differently elsewhere. International guidelines say a translated test needs fresh evidence of equivalence, reliability and validity in each language and culture.23International Test Commission. ITC Guidelines for Translating and Adapting Tests If a test does not measure the same thing in two groups, score differences between them cannot be taken as real differences, because they may come from the measurement itself.24Paidéia. Cross-cultural adaptation and validation of psychological instruments: some considerations

Hidden Limits That Can Fool Even a Good Test

A ceiling effect happens when many people hit the top score, and a floor effect when many hit the bottom.25PubMed Central. Floor and Ceiling Effects in the EORTC QLQ-C30 Physical Functioning Subscale Among Patients With Advanced or Metastatic Breast Cancer The test cannot show change beyond that limit.25PubMed Central. Floor and Ceiling Effects in the EORTC QLQ-C30 Physical Functioning Subscale Among Patients With Advanced or Metastatic Breast Cancer A high-functioning person could decline without the score moving, leading to a false “no change.”25PubMed Central. Floor and Ceiling Effects in the EORTC QLQ-C30 Physical Functioning Subscale Among Patients With Advanced or Metastatic Breast Cancer These effects can also bias comparisons between groups, and more so as they grow.26PubMed Central. Robustness of statistical methods when measure is affected by ceiling and/or floor effect

Many perfect scores on a quiz mean it cannot separate top students, and a too-hard test lumps strugglers together at zero. Adjusting the difficulty spreads scores out.

A second limit is relevance to daily life. Many thinking tests use abstract tasks that may not reflect everyday problems like cooking or shopping.27PubMed Central. The ‘Real-World Approach’ and Its Problems: A Critique of the Term Ecological Validity Such tests still predict everyday functioning moderately well, best when the test and the task use the same skill.28PubMed. The ecological validity of neuropsychological tests: a review of the literature on everyday cognitive skills Newer approaches use real-life tasks or virtual reality scenes to get closer to daily life.27PubMed Central. The ‘Real-World Approach’ and Its Problems: A Critique of the Term Ecological Validity29Journal of the International Neuropsychological Society. Validation of the Virtual Reality Everyday Assessment Lab (VR-EAL): An Immersive Virtual Reality Neuropsychological Battery with Enhanced Ecological Validity

Can Computer-Scored Tests Be Trusted

Automated scores are validated by checking how closely they agree with human experts.30Learning and Instruction. Validating automated assessments of teaching effectiveness using multimodal data31Educational Assessment. Validation of Automated Scoring for a Formative Assessment that Employs Scientific Argumentation Even when agreement is satisfactory, small discrepancies remain.31Educational Assessment. Validation of Automated Scoring for a Formative Assessment that Employs Scientific Argumentation Machines misscore most on answers whose rubric needs human interpretation and on more sophisticated reasoning.32Frontiers in Education. Validation of automated scoring for learning progression-aligned Next Generation Science Standards performance assessments Computer scoring is therefore safest for clear-cut answers and needs human checks for complex ones. Schools should spot-check machine marks against teachers’ marks, and agreement figures should be published before any high-stakes use.

How to Size Up a Test You Are Given

Start by looking for reported consistency figures, with about 0.70 as the usual bar described earlier.3PubMed Central. Test–retest reliability, practice effects and estimates of change: A study on the Mindmore digital cognitive assessment tool Then ask whether its validity was shown for the same purpose it is being used for now.1Research Matters. Tracing the evolution of validity in educational measurement: past issues and contemporary challenges Check whether it was tested on people of your age, language and background.23International Test Commission. ITC Guidelines for Translating and Adapting Tests For retests, ask whether practice gains were accounted for.4PubMed. Test-retest reliability and practice effects of expanded Halstead-Reitan Neuropsychological Test Battery

Online personality and IQ quizzes usually fail these checks. Most publish no consistency or validity evidence, and retaking one and getting a different type is a sign of low reliability. Treat the results as entertainment, not diagnosis. Established tests come with technical manuals that report the evidence.

If you are writing your own classroom or survey test, map questions to the topics you mean to cover. Pilot it with a few people and drop confusing items. Use a scoring guide and have a second marker check samples, and avoid making every question nearly identical.

For a job or clinical test you have taken, you can ask the employer or clinician what the test predicts. Ask whether a single score decides the outcome or is one piece of evidence. If needed, request accommodations or a translated, validated version.