Setting Up Your Data Analysis for Success | by Jordan Gomes | Oct, 2022
5 key questions to answer before starting to answer THE question
When you start learning about ‘how to do a data analysis,” it is usually in a very polished way. You have a precise question that leaves no room for interpretation (“is the mean of group A different than the mean of group B”), and a very nice, tidy, and well-documented dataset that you can use.
Data analysis feels simple and uncomplicated. It is similar to solving a mathematical problem. Just find X.
But in real life, data analyses are rarely done “in a vacuum,” independently of other events. Usually:
- An event happened that led some people to ask you to look into something (e.g. a deck was presented at a key meeting raising some questions and now your manager wants to know more)
- Those same people are expecting something to happen once you complete your analysis (e.g. if the previous findings are true, your team should change its strategy)
- This something might impact the way other people are currently doing their job (e.g. if the strategy change, the operational teams will be impacted)
Your study is not the starting point nor the finish line in itself: it is usually one key part of a larger journey your organization is undertaking. And to make your study as impactful as possible, you need to be aware of the key elements of this journey and understand the general dynamic of the fellowship surrounding this journey.
Below I propose 5 key questions to ask yourself before starting any work, to make sure your output is as impactful as possible.
“Nature hath given men one tongue but two ears, two that we may hear from others twice as much as we speak” — Epictetus
#1: Why is this question important today?
The first thing to understand, before working on a project, is the context of the ask. Why are you being asked to work on this project today?
- Understanding the events leading to this ask can give you a starting point. For instance — your process will be different if the ask is triggered by some numbers that were presented in an internal forum earlier, or if the ask is coming from an executive who got an idea from reading something on the internet.
- It will also be very important when you will be presenting your results. As Chip Heath said, “You can’t appreciate the solution until you appreciate the problem”. Clearly stating the problem will make your audience want to see a solution — but you can only clearly state the problem if the problem is clearly defined in your mind.
- Finally, really understanding the problem can help with sizing the actual issue, and this can be very helpful to prioritize its resolution vs other competing priorities (if any).
The above — while seemingly very straightforward — generally isn’t. Imagine the situation in which you are working for a mobile game company, and you are being asked to look into ways to reduce churn, as “it is too high” right now. The first step here should be to understand what “too high” means, and how much of a problem this is in terms of $$$ (and this, by itself, is a study inside the study). Imagine now that you find out that the churn rate is high because of free users’ churn, but on the paid users’ side, you are actually well-below industry rates. Should you still go ahead with the initial ask (i.e. finding ways to reduce churn)?
#2: Who is interested by this work?
Clarifying who the official and unofficial decision makers are and who the users of the work will be is extremely important before starting any work.
- You will need the green light from your decision makers to go ahead —so you need to know exactly what they will value and how you’ll be able to sell them the project.
- You will want your work to be used in real life so you need to make sure that your findings will be actionable by your users.
Having a good relationship with those two groups of stakeholders will allow you to make sure your study is taking into account their inputs and answering their questions. Failing to do so can result in very adverse effects.
This is especially true when you fail to listen to some of your potential users. As I was mentioning in a previous article, there is this great TED video by Ernesto Sirolli called “Want to help someone? Shut up and Listen”. In it, he explained how, when he was 21 years old, he worked for an Italian NGO which tried to help Zambian people by planting vegetables near the Zambezi river. They did manage to grow “magnificent tomatoes” but, as Sirolli puts it in his own’s words:
When the tomatoes were nice and ripe and red, overnight, some 200 hippos came out from the river and they ate everything. And we said to the Zambians, ‘My God, the hippos!’ And the Zambians said, ‘Yes, that’s why we have no agriculture here.’
Morel of the story: Just listen.
#3: What has already been done in the field?
It is always interesting to get a sense of what has been done regarding this project (whether it is internally, in the industry, in a similar field, or in the research world). Not only this can help you learn from past experiences and understand what could be some potential pitfalls, but it can also help give you some great ideas.
This step is often missed — because it is quite time-consuming, especially if your organization doesn’t have a good knowledge management system in place. When that happens, the knowledge still exists but is spread among multiple people, and finding those people can be a difficult task. This is when internal networking pays off.
#4: Who are some good SMEs you can start working with?
To simplify, there are 2 types of subject matter experts (SMEs): technical and non-technical.
- The technical SMEs can help you understand how the data has been collected, and what are some common pitfalls to avoid. Basically, the technical SME is the person that can tell you which data sources to use, which data transformation to make, or what’s the correct filter to use on the data — all those kinds of slightly important things that is better to uncover earlier in your study rather than at the end.
- Non-technical SMEs, on the other side, can help give a better understanding of the phenomenon you are studying. If you think about it, a database is a simplified way to document information, and only looking at a phenomenon through this simplified lens can only give a simpler vision of reality. If you want to make sure that your simplified vision is the right one, it is important to get qualitative data.
#5: How will you know if this project is a success?
This one is often overlooked — but it is always good to define some kind of success/exit criteria for your work. Is your work supposed to define a new metric? Should it end with the launch of a new dashboard? Is it supposed to change the way your company does things and bring an additional $Xk dollars per month/year?
The idea here is to come up with objective criteria that you’ll be able to use to determine when to stop and if your work is successful. Coming up with a way to assess the study independently from the bigger journey is not easy, but it is an interesting thing to do as it allows you to understand if you brought the value you were supposed to and can help you take a step back and uncover some learnings that you wouldn’t have uncovered otherwise.
The fellowship of the churn
Answering those questions should generally help make sure that you have the right level of context for your project, and that you are clear on who are the main protagonists and what are the expectations regarding your work.
Obviously, now you still have to do the work, but hopefully, you are better equipped to be successful.
“Frodo wouldn’t have gone that far without Sam, would he” Frodo Baggins
5 key questions to answer before starting to answer THE question
When you start learning about ‘how to do a data analysis,” it is usually in a very polished way. You have a precise question that leaves no room for interpretation (“is the mean of group A different than the mean of group B”), and a very nice, tidy, and well-documented dataset that you can use.
Data analysis feels simple and uncomplicated. It is similar to solving a mathematical problem. Just find X.
But in real life, data analyses are rarely done “in a vacuum,” independently of other events. Usually:
- An event happened that led some people to ask you to look into something (e.g. a deck was presented at a key meeting raising some questions and now your manager wants to know more)
- Those same people are expecting something to happen once you complete your analysis (e.g. if the previous findings are true, your team should change its strategy)
- This something might impact the way other people are currently doing their job (e.g. if the strategy change, the operational teams will be impacted)
Your study is not the starting point nor the finish line in itself: it is usually one key part of a larger journey your organization is undertaking. And to make your study as impactful as possible, you need to be aware of the key elements of this journey and understand the general dynamic of the fellowship surrounding this journey.
Below I propose 5 key questions to ask yourself before starting any work, to make sure your output is as impactful as possible.
“Nature hath given men one tongue but two ears, two that we may hear from others twice as much as we speak” — Epictetus
#1: Why is this question important today?
The first thing to understand, before working on a project, is the context of the ask. Why are you being asked to work on this project today?
- Understanding the events leading to this ask can give you a starting point. For instance — your process will be different if the ask is triggered by some numbers that were presented in an internal forum earlier, or if the ask is coming from an executive who got an idea from reading something on the internet.
- It will also be very important when you will be presenting your results. As Chip Heath said, “You can’t appreciate the solution until you appreciate the problem”. Clearly stating the problem will make your audience want to see a solution — but you can only clearly state the problem if the problem is clearly defined in your mind.
- Finally, really understanding the problem can help with sizing the actual issue, and this can be very helpful to prioritize its resolution vs other competing priorities (if any).
The above — while seemingly very straightforward — generally isn’t. Imagine the situation in which you are working for a mobile game company, and you are being asked to look into ways to reduce churn, as “it is too high” right now. The first step here should be to understand what “too high” means, and how much of a problem this is in terms of $$$ (and this, by itself, is a study inside the study). Imagine now that you find out that the churn rate is high because of free users’ churn, but on the paid users’ side, you are actually well-below industry rates. Should you still go ahead with the initial ask (i.e. finding ways to reduce churn)?
#2: Who is interested by this work?
Clarifying who the official and unofficial decision makers are and who the users of the work will be is extremely important before starting any work.
- You will need the green light from your decision makers to go ahead —so you need to know exactly what they will value and how you’ll be able to sell them the project.
- You will want your work to be used in real life so you need to make sure that your findings will be actionable by your users.
Having a good relationship with those two groups of stakeholders will allow you to make sure your study is taking into account their inputs and answering their questions. Failing to do so can result in very adverse effects.
This is especially true when you fail to listen to some of your potential users. As I was mentioning in a previous article, there is this great TED video by Ernesto Sirolli called “Want to help someone? Shut up and Listen”. In it, he explained how, when he was 21 years old, he worked for an Italian NGO which tried to help Zambian people by planting vegetables near the Zambezi river. They did manage to grow “magnificent tomatoes” but, as Sirolli puts it in his own’s words:
When the tomatoes were nice and ripe and red, overnight, some 200 hippos came out from the river and they ate everything. And we said to the Zambians, ‘My God, the hippos!’ And the Zambians said, ‘Yes, that’s why we have no agriculture here.’
Morel of the story: Just listen.
#3: What has already been done in the field?
It is always interesting to get a sense of what has been done regarding this project (whether it is internally, in the industry, in a similar field, or in the research world). Not only this can help you learn from past experiences and understand what could be some potential pitfalls, but it can also help give you some great ideas.
This step is often missed — because it is quite time-consuming, especially if your organization doesn’t have a good knowledge management system in place. When that happens, the knowledge still exists but is spread among multiple people, and finding those people can be a difficult task. This is when internal networking pays off.
#4: Who are some good SMEs you can start working with?
To simplify, there are 2 types of subject matter experts (SMEs): technical and non-technical.
- The technical SMEs can help you understand how the data has been collected, and what are some common pitfalls to avoid. Basically, the technical SME is the person that can tell you which data sources to use, which data transformation to make, or what’s the correct filter to use on the data — all those kinds of slightly important things that is better to uncover earlier in your study rather than at the end.
- Non-technical SMEs, on the other side, can help give a better understanding of the phenomenon you are studying. If you think about it, a database is a simplified way to document information, and only looking at a phenomenon through this simplified lens can only give a simpler vision of reality. If you want to make sure that your simplified vision is the right one, it is important to get qualitative data.
#5: How will you know if this project is a success?
This one is often overlooked — but it is always good to define some kind of success/exit criteria for your work. Is your work supposed to define a new metric? Should it end with the launch of a new dashboard? Is it supposed to change the way your company does things and bring an additional $Xk dollars per month/year?
The idea here is to come up with objective criteria that you’ll be able to use to determine when to stop and if your work is successful. Coming up with a way to assess the study independently from the bigger journey is not easy, but it is an interesting thing to do as it allows you to understand if you brought the value you were supposed to and can help you take a step back and uncover some learnings that you wouldn’t have uncovered otherwise.
The fellowship of the churn
Answering those questions should generally help make sure that you have the right level of context for your project, and that you are clear on who are the main protagonists and what are the expectations regarding your work.
Obviously, now you still have to do the work, but hopefully, you are better equipped to be successful.
“Frodo wouldn’t have gone that far without Sam, would he” Frodo Baggins
Denial of responsibility! Techno Blender is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – [email protected]. The content will be deleted within 24 hours.