Survivorship Bias: The Ghosts of Data
Intro
In today's post, we will examine the intriguing concept of Survivorship Bias and the remarkable story of statistician Abraham Wald, who coined the term. We will delve deeper into the essence of UX Research and Design, as well as the critical impact of missing data in shaping user-centered products. Being aware of biases, how they work, and the influence they have on our lives is important not just for our professional lives but also for our personal ones.
Biases
What are biases, and why do we need to discuss them? The human brain can do many amazing things, but it's not a perfect machine; there are flaws, which we call biases. These 'flaws' should not necessarily be looked at as bugs of our cognitive system; they help us navigate the complex world. In a way, they are both our weaknesses and strengths. For example, biases serve as information-processing shortcuts that don't always result in us making rational judgments or decisions. We can argue that biases have positive roles in our lives as well; for example, they make us more efficient and help us recognize patterns. Whatever the ratio of good and bad influence biases have on us, one thing is for sure: they are a big part of our lives and have a huge effect on our behavior. This is the reason biases have been studied for decades in the field of cognitive science.
Survivorship bias
Survivorship bias is a form of selection bias, also known as the selection effect. ‘Selection bias is the bias introduced by the selection of individuals, groups, or data for analysis in such a way that proper randomization is not achieved, thereby failing to ensure that the sample obtained is representative of the population intended to be analyzed.’
In simpler terms, survivorship bias occurs when people optimistically believe they know what happened in a scenario, focusing only on one or a few selected factors and overlooking others. It is basically a shortcut for our brain to form a belief, to find correlations or patterns. This helps us move on and form a belief instead of being stuck in a loop.
This next quotation is an excellent example of what survivorship bias is and what situations can be called “biased”: ‘Survivorship bias would involve thinking that an incident happened in a particular way because the only people who were involved in the incident and can speak about it are those who survived it. Even if one knew that some people are dead, they would not have their voice to add to the conversation, making it biased.’
We can all recognize ourselves in survivorship bias; how many times have we jumped to a conclusion without even thinking about the other factors? That's just how humans are built, and there is not much we can do about it. We can't be resilient to the influence of biases. However, there are a few things we should be aware of and a few things we should learn about ourselves that will help us be more rational in our decision-making process and career life. Before we jump into these valuable insights, let’s talk about a man who coined the term survivorship bias and his remarkable life story; his name is Abraham Wald.
Abraham Wald
Survivorship bias term was coined by Abraham Wald in WW2. Abraham was a Jewish Hungarian mathematician and statistician who worked on aircraft survivability research during WW2. He was a member of the Statistical Research Group (SRG) and worked on the problem of estimating the vulnerability of aircraft using data from the aircraft that survived the battle. In his work, he could only observe planes that returned from the fight; others that didn’t were out of reach, making the sample biased.
This is important because it points out the tendency to focus on successful subjects and overlook failures. His work is important for us, and we should learn more about survivability and missing data because survivorship bias doesn't only affect military research; it affects business as well.
Unveiling Holistic User Insights
As UX researchers, product managers, designers, developers, and product professionals in general, our goal is to comprehend user behavior, preferences, and pain points comprehensively. Survivorship Bias reminds us to avoid narrow perspectives based solely on successful outcomes. By considering failed experiences or unrepresented user segments, we unearth crucial insights, ensuring our research isn't skewed. This awareness enables us to construct more nuanced user personas, conduct comprehensive usability tests, and design with empathy, ultimately crafting experiences that resonate across diverse user groups.
Crafting Inclusive, Effective Designs
Designers play a pivotal role in translating research insights into tangible, user-centric solutions. Understanding Abraham Wald's approach prompts us to consider what's missing in our design context. By acknowledging the overlooked aspects and leveraging insights from unsuccessful attempts, designers can create interfaces that cater to a broader spectrum of user needs. This approach fosters more inclusive designs, reduces friction points, and enhances user satisfaction. At least we can give it a shot, right? Build or mock a feature that we think was overlooked and test it.
Informed Decision-Making
Product professionals steer the direction of products and services. Acknowledging the impact of Missing Data empowers them to make more informed decisions. By recognizing gaps in user data or feedback, product teams can strategize better user acquisition, engagement, and retention efforts. This holistic approach aids in refining product roadmaps, prioritizing features that cater to a wider user base, and ultimately driving overall product success.
Conclusion
Understanding these concepts equips product teams with a critical lens, enabling them to navigate complexities, challenge biases, and foster more robust, user-centered design practices. By embracing a holistic perspective that includes successes, failures, and what's missing, these professionals create products that resonate deeply with users and drive sustained success.
Summary
Bias is a brain mechanism that helps us humans understand and interpret information.
Survivorship bias is a form of selection bias, also known as selection effect.
Survivorship bias is when we focus only on the few selected factors and overlook the others.
Do not fall into the trap of thinking you know your users based on a few successfully selected research studies and data.
Consider what remains undiscovered in research and datasets. When you think you know, test it.
When you encounter incomplete data, strive to develop a product for the broader population rather than a specific segment, strategically reducing risk.
References
Desjardins, Jeff. "Every Single Cognitive Bias in One Infographic." Visual Capitalist, 2021, https://www.visualcapitalist.com/every-single-cognitive-bias/.
Mangel, Marc, et al. “Abraham Wald’s Work on Aircraft Survivability.” UC Santa Cruz, 2001, https://people.ucsc.edu/~msmangel/Wald.pdf
“Abraham Wald.” Wikipedia: The Free Encyclopedia, Wikimedia Foundation, 11 January 2024, https://en.wikipedia.org/wiki/Abraham_Wald
