How to Speed Up Data Science Deliveries | by Bárbara Barbosa | Jan, 2023
Bringing agile to data science and reestablishing the stakeholders’ trust in the team
When we start to work with data science, other people expect us to be as agile as the developers, but can we use the same techniques and rituals that they use to be agile?
I scaled a Data Science team from 1 to around 10 people over some years, and I want to share my experience with the multiple strategies that I used as a manager to be agile in a very uncertain area as is Data Science. If you are a tenured data scientist, it could happen that many of your projects sometimes don’t go to production. At the beginning of the project, the uncertainty is high and you don’t know if it will make it to production. If this is your case, I want to share some of my strategies to mitigate the uncertainty at the beginning of a project.
At the beginning of the team, everything was nice. We used a single Kanban because the team was pretty small and we were capable to manage everything with it. The problem started when the team grew and we were acting in multiple domains across the company. The stakeholders wanted to know the deadlines and what we were going to do to finish the projects. I talked to many companies back then and decided that we should try to move to Scrum.
We used Scrum for some years, but with time I realized that the team was unhappy. We had multiple problems:
- the open Pull Requests took forever to be revised, even in a very small story;
- never (or almost never) the scientists were capable of finishing the sprint;
- the refining meetings took too long, and after some alignment, with the stakeholders, everything always change and the whole refining meeting was pointless;
- the team was always superestimating the time and size of stories and, with that, stakeholders stopped trusting our deadlines and we had a reputation of being a slow team;
- it was super hard to bet on the size of an exploratory story (being an EDA — Exploratory Data Analysis, or even the discovery of the data itself), and after that step, if the data was not good, it could end in a bad model or an analysis that wasn’t what we actually need to improve the model performance.
Now, let me explain in more detail how the team worked at that time. The team was a Chapter, so we were inside some Tribes and we worked together with other teams to reach the goals of that Tribes. A data scientist normally would lead a project inside the Tribe and the other people on the Data Science team would help her, giving ideas during daily meetings or other ceremonies. Some projects would have two data scientists, but most of the time, we had one person per project and this person could cont with the support from the Data Science chapter.
At that time, I thought that we were agile because we used an agile framework. In fact, being agile is actually responding to change over following a plan, and the problems that I listed made us look for a new way to manage the projects of the Data Science team.
We identified that one of our biggest problems was trying to solve everything with the first launch of a product. This created a long feedback loop that generated a massive amount of code (between Jupyter notebooks and Python code), which jeopardizes reviews. That was especially harmful when there was a problem with the logic at the beginning of the project that would create a cascade event of change (and a project being completely invalidated sometimes).
So we stopped. We started working to validate a single hypothesis and with that in mind, we started creating production-ready versions for every hypothesis that we validated. The main changes were:
- Work with small versions that could go to production at the end of the iteration. The versions are prioritized based on the biggest problem the stakeholders are facing at the moment.
- Start using soft deadlines to increase the creativity of the team and bring more visibility to the stakeholders.
- The research review is super important, but it was one of the biggest time drainers of the project. We moved this step to the end.
I will detail those steps below:
Small versions
The first version of the project is the MVP and after that new versions are developed. The MVP is the baseline version, and this nomenclature helps the data scientists to think about how to work on the smallest iteration possible of a project and also with the stakeholders’ expectations since they will also help with the stakeholder’s expectations and the data scientists to do the smallest iteration possible of a project that helps to solve the problem.
One of the first things that we do is Understand the Problem, or to match the above image, the Business Understanding, and to make this step easier, we use the Machine Learning Canvas (ML Canvas). I also usually call this step Discovery, because here we actually understand if this is actually a Data Science problem and if we need data scientists to solve this issue. The canvas helps to assess that because we can align the main goals of the project with everyone impacted by it.
During the canvas section, we understand the stakeholder’s problems, and the main data we need and we can already define an MVP and reach an agreement on what we will deliver on the next versions. Although this seems pretty straightforward, during the MVP everything can change, and this is okay, because uncertainty is part of a data science project, and without having your hands dirty with data it is hard to know what to expect next and what the next steps will be.
In a full remote or hybrid environment you can use Miro or another similar tool to make brainstorm with everyone and this initial alignment allows us to understand different aspects of the project before the data team actually starts working on it.
For the next steps and versions, it is imperative to have someone that is an expert in the area the data scientist will be working on as a partner. This person should be available to have brainstorming sessions and help the data scientist acquire the business knowledge they need for the problem, help with a hypothesis, and version prioritization. This person, whom we named Business Partner, will help the data scientist to be more in touch with the daily problems the area is facing and will make sure that the deliveries of the data scientist are solving their problems. A weekly meeting with a Business Partner can do wonders for a project, making sure the data scientist is down to earth with the business and the business understand what is being done by the data scientist.
Soft deadlines
We used a Kanban with the phases from CRISP-DM, which has 6 steps. Some steps are grouped and they have a soft deadline and an alignment meeting with the project stakeholders to wrap it up. This enhanced the credibility that we had with the stakeholders as without it, we didn’t have a specific agenda, and sometimes the results were not enough consolidated or validated. Having some time to wrap the results up creates a more productive meeting and we can focus on the points that we want to discuss, not on finding possible errors for the analysis, enhancing the business value of that specific deliverable.
Often when I talk about this step the question of why it is “soft” commonly comes to people’s minds. I believe this is because everyone is used to the way that Software Development is made and Data Science is not Software Engineer and has much uncertainty at some steps of the project. So the word soft brings this idea of something that has a high degree of uncertainty, as an EDA (Exploratory Data Analysis) can be hard to actually measure without the business logic. After the Discovery and initial data preparation and exploration, it is already possible for the data scientist to schedule an alignment meeting and also define the next soft deadlines for the project.
The soft deadline also brings flexibility for the data scientist: she can do all that she thinks/wants it is necessary to solve the problem until that date and, in case of a planning error, she still has material to present at the checkpoint and collect feedback (assuming that she focused first on the simpler version and scaled from there with the rest of her time). This will allow her to realize different experiments, having in mind the deadline.
The image has an example of what was doable in a second/third version of a project (so a lot of the uncertainty about data, deploy, usage, etc. was already removed) and in a team that was already very familiar with the speed of their members. For MVP projects this might be totally different as the data preparation can be very complex. Projects that are beyond the first phase normally are much faster using this approach as most of the risk is already known and the stakeholders are already familiar with the way of working.
Having the alignment meeting with the stakeholders after the EDA is finished can guarantee that we are well aligned on the next steps and also increase the visibility of the Data Science team for them.
Review at the end
As stated at the beginning, one of our major problems was the research review time, and this was then moved to the end of the project. In a code sense, this is quite bad as it is not some atomic part that will create a quick code review, and in the code sense, this is perfectly correct, but for research, you don’t want just to see a single piece of code that trains a model, you want to also see the evaluation to understand what kind of things you can do to improve this model (is it data? hyperparameters? just looking at the train code is impossible to know). This allows the reviewer to understand the business problem and suggest modifications that were more relevant to the problem. Before that, one person could be reviewing the EDA and after 2 weeks start to review the model evaluation, or worse than that, different people would review each part, losing the context. In Data Science the code is just part of the whole picture, there are Jupyter notebooks with insights, graphics, and data. Seeing the model evaluation and missing the EDA did not allow the review to be complete, and reviewing the whole iteration also helps the data scientist be mindful that the iteration should be small.
Because this step takes quite some time, just one data scientist reviews each other’s research and it is optional for other people on the team to revise it. It is very important that after the Evaluation someone is responsible for the review task and there is also a deadline for it, otherwise, the revision will be a problem again and this step can block the entire project.
This step increases the team’s knowledge and also eases the transition of people to another area. If you want to know more about how to do Pull requests using Github, take a look at this article I wrote a while ago.
Challenges and other learnings
Because the review step is only the last one, the data scientists can have a hard time collaborating and asking questions. This can be solved by working with a feature branch, where the data scientist can still deliver small pieces of code and get help on small chunks, but the team still keeps the context analysis when the feature branch goes to production. This also reduces the review time for the project in the end, because minor issues were already addressed. This can help to create asynchronous conversations with the team about a specific question to be asked.
Keeping the retrospectives was very important at that time and it is a good opportunity for the team discusses what is working or not in a project and exchange the experience with the rest of the team. This allows the Chapter to be in continuous improvement, even if there are no deliveries in the period, I recommend scheduling the meeting with a good frequency, like every 15 days.
This methodology is my adaptation of what is presented in this article. In it, the author presents an approach that uses Kanban and a checklist with the next steps. The soft deadline forces the data scientist to think about the steps that she needs to follow and this creates a more structured work, but I never used a predefined checklist, the data scientist thought about what was important and discussed the tasks with the team (almost as a refining ceremony, if you like to think like that). The implementation of the approach that I explain here was a success. It brought flexibility and ownership for the data scientist and more visibility and transparency with the stakeholders. This approach works better in a team that can be more focused on a single project at a time, but of course, all teams will have their own particularities and will adapt better to different frameworks.
I hope this article brings some light to questions you might be having and if you tested it, please share your experience with me, I will be glad to hear about it.
This is a translated and revised version of my original article in Portuguese.
Bringing agile to data science and reestablishing the stakeholders’ trust in the team
When we start to work with data science, other people expect us to be as agile as the developers, but can we use the same techniques and rituals that they use to be agile?
I scaled a Data Science team from 1 to around 10 people over some years, and I want to share my experience with the multiple strategies that I used as a manager to be agile in a very uncertain area as is Data Science. If you are a tenured data scientist, it could happen that many of your projects sometimes don’t go to production. At the beginning of the project, the uncertainty is high and you don’t know if it will make it to production. If this is your case, I want to share some of my strategies to mitigate the uncertainty at the beginning of a project.
At the beginning of the team, everything was nice. We used a single Kanban because the team was pretty small and we were capable to manage everything with it. The problem started when the team grew and we were acting in multiple domains across the company. The stakeholders wanted to know the deadlines and what we were going to do to finish the projects. I talked to many companies back then and decided that we should try to move to Scrum.
We used Scrum for some years, but with time I realized that the team was unhappy. We had multiple problems:
- the open Pull Requests took forever to be revised, even in a very small story;
- never (or almost never) the scientists were capable of finishing the sprint;
- the refining meetings took too long, and after some alignment, with the stakeholders, everything always change and the whole refining meeting was pointless;
- the team was always superestimating the time and size of stories and, with that, stakeholders stopped trusting our deadlines and we had a reputation of being a slow team;
- it was super hard to bet on the size of an exploratory story (being an EDA — Exploratory Data Analysis, or even the discovery of the data itself), and after that step, if the data was not good, it could end in a bad model or an analysis that wasn’t what we actually need to improve the model performance.
Now, let me explain in more detail how the team worked at that time. The team was a Chapter, so we were inside some Tribes and we worked together with other teams to reach the goals of that Tribes. A data scientist normally would lead a project inside the Tribe and the other people on the Data Science team would help her, giving ideas during daily meetings or other ceremonies. Some projects would have two data scientists, but most of the time, we had one person per project and this person could cont with the support from the Data Science chapter.
At that time, I thought that we were agile because we used an agile framework. In fact, being agile is actually responding to change over following a plan, and the problems that I listed made us look for a new way to manage the projects of the Data Science team.
We identified that one of our biggest problems was trying to solve everything with the first launch of a product. This created a long feedback loop that generated a massive amount of code (between Jupyter notebooks and Python code), which jeopardizes reviews. That was especially harmful when there was a problem with the logic at the beginning of the project that would create a cascade event of change (and a project being completely invalidated sometimes).
So we stopped. We started working to validate a single hypothesis and with that in mind, we started creating production-ready versions for every hypothesis that we validated. The main changes were:
- Work with small versions that could go to production at the end of the iteration. The versions are prioritized based on the biggest problem the stakeholders are facing at the moment.
- Start using soft deadlines to increase the creativity of the team and bring more visibility to the stakeholders.
- The research review is super important, but it was one of the biggest time drainers of the project. We moved this step to the end.
I will detail those steps below:
Small versions
The first version of the project is the MVP and after that new versions are developed. The MVP is the baseline version, and this nomenclature helps the data scientists to think about how to work on the smallest iteration possible of a project and also with the stakeholders’ expectations since they will also help with the stakeholder’s expectations and the data scientists to do the smallest iteration possible of a project that helps to solve the problem.
One of the first things that we do is Understand the Problem, or to match the above image, the Business Understanding, and to make this step easier, we use the Machine Learning Canvas (ML Canvas). I also usually call this step Discovery, because here we actually understand if this is actually a Data Science problem and if we need data scientists to solve this issue. The canvas helps to assess that because we can align the main goals of the project with everyone impacted by it.
During the canvas section, we understand the stakeholder’s problems, and the main data we need and we can already define an MVP and reach an agreement on what we will deliver on the next versions. Although this seems pretty straightforward, during the MVP everything can change, and this is okay, because uncertainty is part of a data science project, and without having your hands dirty with data it is hard to know what to expect next and what the next steps will be.
In a full remote or hybrid environment you can use Miro or another similar tool to make brainstorm with everyone and this initial alignment allows us to understand different aspects of the project before the data team actually starts working on it.
For the next steps and versions, it is imperative to have someone that is an expert in the area the data scientist will be working on as a partner. This person should be available to have brainstorming sessions and help the data scientist acquire the business knowledge they need for the problem, help with a hypothesis, and version prioritization. This person, whom we named Business Partner, will help the data scientist to be more in touch with the daily problems the area is facing and will make sure that the deliveries of the data scientist are solving their problems. A weekly meeting with a Business Partner can do wonders for a project, making sure the data scientist is down to earth with the business and the business understand what is being done by the data scientist.
Soft deadlines
We used a Kanban with the phases from CRISP-DM, which has 6 steps. Some steps are grouped and they have a soft deadline and an alignment meeting with the project stakeholders to wrap it up. This enhanced the credibility that we had with the stakeholders as without it, we didn’t have a specific agenda, and sometimes the results were not enough consolidated or validated. Having some time to wrap the results up creates a more productive meeting and we can focus on the points that we want to discuss, not on finding possible errors for the analysis, enhancing the business value of that specific deliverable.
Often when I talk about this step the question of why it is “soft” commonly comes to people’s minds. I believe this is because everyone is used to the way that Software Development is made and Data Science is not Software Engineer and has much uncertainty at some steps of the project. So the word soft brings this idea of something that has a high degree of uncertainty, as an EDA (Exploratory Data Analysis) can be hard to actually measure without the business logic. After the Discovery and initial data preparation and exploration, it is already possible for the data scientist to schedule an alignment meeting and also define the next soft deadlines for the project.
The soft deadline also brings flexibility for the data scientist: she can do all that she thinks/wants it is necessary to solve the problem until that date and, in case of a planning error, she still has material to present at the checkpoint and collect feedback (assuming that she focused first on the simpler version and scaled from there with the rest of her time). This will allow her to realize different experiments, having in mind the deadline.
The image has an example of what was doable in a second/third version of a project (so a lot of the uncertainty about data, deploy, usage, etc. was already removed) and in a team that was already very familiar with the speed of their members. For MVP projects this might be totally different as the data preparation can be very complex. Projects that are beyond the first phase normally are much faster using this approach as most of the risk is already known and the stakeholders are already familiar with the way of working.
Having the alignment meeting with the stakeholders after the EDA is finished can guarantee that we are well aligned on the next steps and also increase the visibility of the Data Science team for them.
Review at the end
As stated at the beginning, one of our major problems was the research review time, and this was then moved to the end of the project. In a code sense, this is quite bad as it is not some atomic part that will create a quick code review, and in the code sense, this is perfectly correct, but for research, you don’t want just to see a single piece of code that trains a model, you want to also see the evaluation to understand what kind of things you can do to improve this model (is it data? hyperparameters? just looking at the train code is impossible to know). This allows the reviewer to understand the business problem and suggest modifications that were more relevant to the problem. Before that, one person could be reviewing the EDA and after 2 weeks start to review the model evaluation, or worse than that, different people would review each part, losing the context. In Data Science the code is just part of the whole picture, there are Jupyter notebooks with insights, graphics, and data. Seeing the model evaluation and missing the EDA did not allow the review to be complete, and reviewing the whole iteration also helps the data scientist be mindful that the iteration should be small.
Because this step takes quite some time, just one data scientist reviews each other’s research and it is optional for other people on the team to revise it. It is very important that after the Evaluation someone is responsible for the review task and there is also a deadline for it, otherwise, the revision will be a problem again and this step can block the entire project.
This step increases the team’s knowledge and also eases the transition of people to another area. If you want to know more about how to do Pull requests using Github, take a look at this article I wrote a while ago.
Challenges and other learnings
Because the review step is only the last one, the data scientists can have a hard time collaborating and asking questions. This can be solved by working with a feature branch, where the data scientist can still deliver small pieces of code and get help on small chunks, but the team still keeps the context analysis when the feature branch goes to production. This also reduces the review time for the project in the end, because minor issues were already addressed. This can help to create asynchronous conversations with the team about a specific question to be asked.
Keeping the retrospectives was very important at that time and it is a good opportunity for the team discusses what is working or not in a project and exchange the experience with the rest of the team. This allows the Chapter to be in continuous improvement, even if there are no deliveries in the period, I recommend scheduling the meeting with a good frequency, like every 15 days.
This methodology is my adaptation of what is presented in this article. In it, the author presents an approach that uses Kanban and a checklist with the next steps. The soft deadline forces the data scientist to think about the steps that she needs to follow and this creates a more structured work, but I never used a predefined checklist, the data scientist thought about what was important and discussed the tasks with the team (almost as a refining ceremony, if you like to think like that). The implementation of the approach that I explain here was a success. It brought flexibility and ownership for the data scientist and more visibility and transparency with the stakeholders. This approach works better in a team that can be more focused on a single project at a time, but of course, all teams will have their own particularities and will adapt better to different frameworks.
I hope this article brings some light to questions you might be having and if you tested it, please share your experience with me, I will be glad to hear about it.
This is a translated and revised version of my original article in Portuguese.
Denial of responsibility! Techno Blender is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – [email protected]. The content will be deleted within 24 hours.