Newcastle-Ottawa Scale: How It Works and Its Reliability

The Newcastle-Ottawa Scale (NOS) is one of the most widely used tools for judging the quality of observational studies included in systematic reviews and meta-analyses. Developed in the late 1990s by researchers at the Universities of Newcastle (Australia) and Ottawa (Canada), it gives reviewers a structured way to rate cohort and case-control studies on a simple star-based scoring system. Its popularity is enormous, but so is the criticism: multiple studies have found that two reviewers applying the NOS to the same paper often reach different conclusions, and the thresholds researchers use to label a study “high quality” or “low quality” are largely arbitrary.

How the Scale Works

The NOS evaluates observational studies across three broad areas. For cohort studies, these are the selection of the study groups, whether the groups were comparable on important factors, and how the outcome was assessed. For case-control studies, the structure is similar but swaps in questions about how cases and controls were defined and how exposure was measured. Each item on the scale can earn a star, and the maximum possible score is nine stars for a cohort study and nine for a case-control study. A study that scores all nine stars has, in theory, addressed the most common sources of bias in its design.

The comparability section is the one area where up to two stars can be awarded for a single item, because controlling for confounders is considered especially important. The other items each offer a single star. Reviewers work through the checklist item by item, awarding or withholding stars based on how the study handled each potential source of bias. The result is a score that researchers then use to decide whether a study should carry more or less weight in a meta-analysis, or whether it should be included at all.

Why Researchers Keep Choosing It

A big part of the NOS’s dominance comes down to speed. In a head-to-head comparison, a reviewer using the NOS spent roughly 30 minutes assessing a single study, while a reviewer using the more detailed ROBINS-I tool spent between three and seven hours on the same task.¹ When a systematic review includes dozens of observational studies, that time difference adds up fast. Research teams working under deadline pressure or with limited personnel gravitate toward the quicker option.

The scale is also recommended by the Cochrane Collaboration as one of several acceptable tools for assessing nonrandomized studies. That endorsement carries weight, since Cochrane reviews are considered among the most methodologically rigorous in evidence-based medicine. The NOS’s simplicity helps, too: it does not require extensive training or background reading to use, which makes it accessible to graduate students and early-career researchers who may be conducting their first systematic review. The tradeoff is that simplicity can become a liability when reviewers interpret vague items differently.

The Reliability Problem

The most persistent criticism of the NOS is that different reviewers applying it to the same studies often disagree, sometimes sharply. A study that tested the scale across multiple reviewers found that overall agreement was only fair, with some individual items performing far worse. Agreement on the length of follow-up item was substantial, but agreement on items like the selection of the unexposed group and whether the outcome was absent at the start of the study was essentially at chance level or below.² Reviewers in that study described the tool as difficult to use, with decision rules they found vague even after receiving additional guidance.

The problem is not limited to experienced reviewers disagreeing. When novice student raters used the NOS, agreement on cohort study items was poor across the board, and agreement on case-control items was only somewhat better.³ This suggests the reliability issues are baked into the tool’s design rather than being a function of who uses it.

A separate study took a different angle: instead of having two external reviewers rate the same studies, it compared ratings made by systematic review authors with independent reviewer ratings. Here, agreement on individual items ranged from slight to poor, and agreement on the overall score was essentially nonexistent.⁴ This is a striking result because it means even the researchers who chose to use the NOS in their own reviews could not reliably reproduce their ratings when an independent rater tried the same exercise.

Perhaps the most concerning finding across these reliability studies is that NOS scores did not appear to predict anything meaningful about the study results themselves. Neither individual items nor the overall score were associated with effect estimates in the studies being rated.⁵ A quality assessment tool that cannot distinguish studies likely to produce biased results from those that are more trustworthy has a fundamental validity problem, regardless of how convenient it is to use.

The Cutoff Score Problem

Even if two reviewers agree on the NOS score for a given study, the question of what score counts as “high quality” has no standardized answer. There is no officially validated threshold. Some systematic reviews call studies with seven or more stars “high quality.” Others draw the line at six, or at five. Some divide studies into three tiers. These cutoffs are chosen by individual review teams, and they vary widely across the literature.

This matters because the cutoff directly influences the conclusions of a meta-analysis. A review that sets the bar at five stars might include studies that another review, using a seven-star cutoff, would exclude entirely. The resulting pooled estimates can differ substantially depending on which studies make it through the quality filter. Critics have pointed out that the NOS’s original developers never published validated thresholds, and subsequent attempts to establish them have not produced consensus.⁶ The result is a situation where the same tool, applied to the same body of evidence, can yield different quality judgments depending on choices that are never formally justified.

The star-based scoring system itself contributes to this. Because the NOS produces a single number, it invites threshold-based thinking: above the line is good, below is bad. More nuanced approaches would treat each domain separately, recognizing that a study can be strong in participant selection but weak in outcome assessment. Collapsing these distinctions into one number obscures where the actual risks of bias lie.

How It Compares with ROBINS-I

The main alternative to the NOS for assessing nonrandomized studies is ROBINS-I (Risk Of Bias In Non-randomized Studies of Interventions), developed by Cochrane. ROBINS-I is a domain-based tool that does not produce a single summary score. Instead, it rates risk of bias across seven specific areas, including confounding, participant selection, classification of interventions, deviations from intended interventions, missing data, outcome measurement, and selection of reported results. Each domain gets its own judgment, and the overall risk-of-bias rating is driven by the worst-performing domain.

Both tools cover overlapping territory: selection bias, confounding, missing data, and how outcomes were measured all appear in some form in each tool.⁷ The key differences are in granularity and time investment. ROBINS-I asks more detailed questions, especially around how the intervention was classified and whether participants deviated from the intended treatment. The NOS covers these topics more broadly, which is partly why it is so much faster to complete.

The reliability comparison is interesting. One study found that inter-observer agreement was similar for both tools, with the NOS actually showing slightly higher average agreement than ROBINS-I.⁸ This is a surprising result given that ROBINS-I is supposed to be the more rigorous instrument, and it suggests that adding complexity does not automatically improve consistency. The same study noted that the agreement between the two tools themselves was only moderate, meaning a study rated as low risk of bias on one tool might not receive the same judgment on the other.

In practice, the choice between the tools often comes down to resources. ROBINS-I is better suited for reviews where precision about the source of bias matters, such as when a review is informing clinical guidelines. The NOS remains the go-to for teams that need to assess a large number of studies quickly and want a general sense of quality, even if that general sense is less informative.

Other Quality Assessment Tools Worth Knowing

The NOS and ROBINS-I are not the only options. The Downs and Black checklist is another commonly used instrument that covers both randomized and nonrandomized studies. In a head-to-head comparison of the NOS, Downs and Black, and the Scottish Intercollegiate Guidelines Network (SIGN) checklist applied to the same set of observational studies, the Downs and Black tool showed the highest inter-rater reliability, with the NOS in the middle and SIGN performing poorly on agreement between reviewers.⁹

That same comparison revealed something else worth noting: the NOS did not demonstrate statistical validity, meaning its scores were not significantly associated with study quality as judged by an external standard. The SIGN checklist, despite its poor reliability, did show statistical validity with medium to large effect sizes. This is an awkward combination for both tools. SIGN seems to measure something real but reviewers cannot agree on the measurement; the NOS is more consistent but may not be measuring what it claims to measure. The Downs and Black checklist performed best on both dimensions in that particular study, though it has its own limitations, including a more complex scoring system and items that can be difficult to interpret for certain study designs.

Agreement between different tools applied to the same studies was also weak. Two tools might both rate a study as “high quality,” but they were only moderately likely to agree, and sometimes barely better than chance. This raises a broader concern: quality assessment in systematic reviews is not as objective as the published tables might imply.

Adapting the NOS for Cross-Sectional Studies

One of the NOS’s original limitations is that it was designed specifically for cohort and case-control studies. Cross-sectional studies, which collect data at a single point in time rather than following participants forward or looking backward, do not fit neatly into either version. This is a real gap, because cross-sectional designs are common in many fields, from public health surveys to dental research.

A recently proposed adaptation called the NOS-xs attempts to fill this gap. Like the original NOS, it uses a star-based system and evaluates studies across three domains: how the study sample was selected, how exposures and outcomes were assessed, and how confounders were handled. The full NOS-xs awards up to nine stars across six items and is intended for analytical cross-sectional studies that examine associations between variables. A simplified version called NOS-xs2, designed for descriptive studies that mainly report how common something is in a population, uses a four-star system across three items.¹⁰

Whether the NOS-xs will gain traction the way the original NOS did remains to be seen. It has the advantage of familiarity: researchers already comfortable with the NOS format will find the adapted version intuitive. But it inherits the original’s structural issues, including the single-number scoring approach and the lack of validated cutoffs. The development team acknowledged these concerns and framed the tool as a starting point rather than a final product, anticipating that reliability and validity testing will follow.

When NOS Scores Appear in a Review You Are Reading

If you encounter a systematic review that uses NOS scores to judge study quality, there are a few things worth keeping in mind. First, look at whether the review reports scores at the domain level or only as a total. A review that says “all included studies scored 7 or above on the NOS” is telling you less than one that breaks down how studies performed on selection, comparability, and outcome assessment separately. A study can score highly overall while still having a serious weakness in one critical area, like failing to control for the most important confounder.

Second, check whether the review explains its quality cutoff and why that particular threshold was chosen. If no justification is given, the cutoff is essentially arbitrary, and a different cutoff might have led to different conclusions. Some reviews run sensitivity analyses to test whether their findings change when the quality threshold is raised or lowered. Reviews that do this are being more transparent about how much their results depend on which studies they consider trustworthy.

Third, consider how many reviewers rated each study and whether their agreement was measured. The best practice is for two independent reviewers to rate every study and then resolve disagreements through discussion or a third reviewer. If a review does not mention this process, you cannot be confident the scores reflect anything more than one person’s interpretation. Given how variable inter-rater agreement is on the NOS, single-rater assessments are particularly unreliable.

Finally, remember that the NOS is a tool for flagging potential sources of bias in study design. It does not, on its own, tell you whether the study’s findings are correct or incorrect. A low-scoring study might have reached the right answer despite its design flaws, and a high-scoring study might have gotten it wrong for reasons the NOS does not capture, such as selective outcome reporting or analytical flexibility. Quality scores are one piece of the puzzle, and treating them as the whole picture overstates what any checklist can deliver.