Cherry-picking in data analytics means selectively presenting only the data points or results that support a desired conclusion while ignoring or hiding evidence that contradicts it. It’s one of the most common forms of bias in both research and business analytics, and it can happen deliberately or without the analyst even realizing it. A study of randomized clinical trials found that selective outcome reporting appeared in nearly 50% of the studies examined, with only 31.7% of primary outcomes fully described in publications.
How Cherry-Picking Works
At its core, cherry-picking is about making a subset of data stand in for the whole picture. Imagine you run an A/B test on a new website design and measure ten different metrics. Eight of them show no improvement, but two show gains. If you report only those two winning metrics in your presentation to stakeholders, you’ve cherry-picked. The data you showed is technically real, but the story it tells is false.
The same principle applies at larger scales. An analyst building a case for a marketing campaign might choose a time window where results looked strong, exclude customer segments that performed poorly, or drop “outlier” data points that happen to weaken the trend. In research contexts, cherry-picking often takes the form of arbitrary inclusion and exclusion criteria: setting rules for which studies or data points count in a way that steers the final result toward a preferred outcome. The key problem is that once you’ve filtered data to match a narrative, the statistical measures you calculate (like averages, correlations, or significance levels) no longer reflect reality. As one research team put it, the resulting analysis “no longer controls the ratio of false findings.”
Intentional vs. Unconscious Cherry-Picking
Not all cherry-picking is deliberate manipulation. Analysts and researchers can cherry-pick without realizing it, driven by confirmation bias or pressure to deliver positive results. When you already believe a product launch was successful, you naturally pay more attention to metrics that confirm that belief and gloss over the ones that don’t. This unconscious version is arguably more dangerous because the analyst genuinely believes they’re being objective.
Intentional cherry-picking, on the other hand, involves knowingly selecting data to build a misleading case. This can look like a vendor choosing only their best-performing client results for a case study, or a team presenting quarter-over-quarter growth while hiding the year-over-year decline. Climate scientist Richard Somerville has described this practice as “a hallmark of poor science or pseudo-science,” and the same applies in business analytics: it produces decisions built on a distorted foundation.
Related Practices: P-Hacking and Data Dredging
Cherry-picking belongs to a family of questionable analytical practices that are worth distinguishing. P-hacking is the relentless reanalysis of data, trying different statistical tests, variable combinations, or subgroup splits until something comes back as statistically significant. Where cherry-picking is about selectively reporting results, p-hacking is about selectively running analyses until you find something worth reporting. In practice, the two often go hand in hand: an analyst p-hacks their way to a favorable finding, then cherry-picks that finding for the final report.
Data dredging (sometimes called fishing) is a broader term for searching through large datasets without a clear hypothesis, looking for any pattern that seems interesting. The problem isn’t exploration itself, which is a legitimate part of analytics. The problem is presenting a pattern you found through exploration as though you predicted it all along. This is sometimes called HARKing: hypothesizing after the results are known.
What It Costs in Business
Cherry-picked data leads to real financial consequences. Research from Wharton examined cherry-picking behavior in retail, where shoppers selectively buy only discounted items across multiple stores. That study found the practice had a “material effect on customer profitability” when compared against typical supermarket net margins of just 1.5% to 2%. Secondary stores were hit hardest: they sold less per shopper ($45 vs. $71 at primary stores) and made lower margins on those sales.
The parallel in analytics is direct. When a team cherry-picks data to justify a campaign, a product feature, or a pricing strategy, the organization makes resource allocation decisions based on inflated expectations. A marketing channel that looked profitable in a cherry-picked analysis may actually be losing money. A product feature that appeared popular among a carefully selected user segment may flop with the broader customer base. The costs compound over time because each biased analysis builds on previous biased analyses, creating a feedback loop of overconfidence.
How to Spot Cherry-Picked Analysis
There are several red flags that suggest an analysis may be cherry-picked:
- Narrow time windows. If results are shown for a suspiciously specific date range (say, March 3 through April 17) without a clear reason, the analyst may have searched for the most favorable window.
- Missing context. When only one or two metrics are highlighted from a project that surely generated many, ask what happened with the rest.
- Unexplained exclusions. Dropping data points or customer segments should always come with a documented reason established before the analysis, not after.
- Results that seem too clean. Real data is messy. If every number in a report points in the same direction with no caveats, some contradictory evidence may have been removed.
- No baseline or comparison. Presenting a 15% conversion rate means nothing without knowing what it was before, or what similar campaigns achieved.
In academic research, statistical tools like funnel plots and regression-based methods (such as the Egger test) can detect patterns consistent with selective reporting across multiple studies. These work by checking whether published results cluster in suspiciously favorable areas. While these specific tools are mostly used in formal research settings, the underlying principle applies to business analytics too: if the results look asymmetrically positive, something may be missing.
Data Cleaning vs. Cherry-Picking
Every analysis involves decisions about what data to include. Removing duplicate records, correcting obvious entry errors, and filtering out test transactions are all standard data cleaning steps. The line between legitimate cleaning and cherry-picking comes down to two things: when you set your criteria, and why.
Legitimate data cleaning follows rules established before you look at the results. You decide in advance that you’ll exclude transactions under $1 (likely test orders) or remove records with missing key fields. Cherry-picking works in the opposite direction: you look at the results first, then find reasons to exclude data points that don’t fit your story. The distinction isn’t always obvious from the outside, which is why documentation matters. If you can’t explain your filtering criteria without referencing the results they produced, you’ve likely crossed the line.
Preventing Cherry-Picking in Your Work
The most effective safeguard is preregistration, a practice borrowed from academic research that’s increasingly relevant in business analytics. Preregistration means writing down your hypothesis, the metrics you’ll measure, your inclusion criteria, and your analysis plan before you touch the data. This document gets timestamped (tools like the Open Science Framework at osf.io serve this purpose in research) so there’s a clear record of what you planned versus what you discovered along the way.
In a business context, this doesn’t need to be formal. Before running a campaign analysis, write a brief document stating which metrics define success, what time period you’ll evaluate, and which customer segments you’ll include. Share it with your team before anyone opens the dataset. This simple step makes it much harder to unconsciously shift the goalposts once results start coming in.
Preregistration doesn’t prevent exploratory analysis. You can still dig into unexpected patterns and report surprising findings. The key is transparency: labeling what was planned and what was discovered after the fact. As one research team noted, preregistration “does not exclude unplanned work. It merely makes the choices made by researchers more transparent.” Another useful practice is having someone independent review or replicate the analysis, since a second pair of eyes is far less likely to share the same confirmation bias. Even rotating who presents results within a team can reduce the temptation to frame data favorably.