5 Learnings from a 24-Hour Hackathon to Prevent ML Project Failures | by Isabelle Flückiger | Jun, 2022

By Jessie Hobb On Jun 2, 2022

Generating value from the project needs more than data, coding skills, and tools

More than 85% of data science projects fail, and only 20% of analytic insights deliver business outcomes, according to Gartner. I can confirm these rates from my experience when advising companies.

And no, it is not the data. It is neither missing data nor bad data.

The Data Days at ETH Zurich closed with a 24-hours Machine Learning hackathon on May 15, 2022, and I was a jury member in one of the four competitions.

Being a jury member of such a competition gives the privilege of gaining a lot of insights. Besides the solution per se, there are several learnings about how the different teams approach the problem-solving and their journey during the hackathon.

What was the challenge to solve?

In our competition, the teams had to build a travel insurance chatbot helping the travelers to guide through the different challenges in a COVID-19-driven world with several requirements per country, travel options, and insurance coverage and assistance in all these situations.

Are the costs of quarantine in a hotel covered if my son gets COVID-19?
What support can I get when a flight is canceled because of the pandemic?
In what countries do I have travel insurance coverage?

These nicely framed questions come to your mind when you think about a chatbot for travel insurance.

The reality looks different. Questions are not written out, and answers cannot be found just by having an ‘advanced search’ tool. The terms and conditions of an insurance contract are legal texts with space for interpretation. Personal assistance is sometimes needed, and incidents can be prevented.

Answers, assistance, and the user experience need to be personalized.

What did the teams get?

All teams got access to the AWS tools for building chatbots, NLP, and machine learning. There was also the annotation that this is only the ‘starter kit,’ and there are no limitations to using any tools and method independent of any provider.

Insurance policy data and a set of questions to answer have been given.

What had been the criteria to evaluate a solution?

We evaluated the solutions on the following five criteria

(1) A quantitative score for the answered (given) set of questions
(2) A qualitative assessment of additional tested tricky questions
(3) Creativity of the solution
(4) Range and depth of the solution
(5) The pitch incl. storytelling

While (1) and (2) assessed the technical accuracy of the solution, (3) linked the technical methods with the business side. (4) covered the value creation for the different stakeholders through this solution, and (5) showed the skills how to present and sell the solution to management or business

It is a usual value chain assessment that drives management decisions to implement a solution or not.

We finally got a broad range of solutions with several different approaches. Some worked, some not.

So, the question is:

What separated the best-performing teams from the rest?

1. Clarify the business problem first, then define how to translate it into a technical solution.

Start by becoming clear in any detail about the problem to solve, the available data, and how to translate it into a machine learning approach before you start anything technically.

In this first step, become clear about

The business problem. Summarize it in a concise business problem statement that focuses on answering a specific question. It contains problem — need — impact. This aligns the team to the goal and lays the basis for communication with the stakeholders (“the pitch”).

Identify all direct and indirect stakeholders affected by the project and solutions. It is not only the insurer’s customers for whom you build this solution.

Analyzing whether the business problem is amenable to an analytics solution, tools, and what data is available.

Refine the business problem statement based on the antecedent analyses and, if necessary, depict known or possible constraints.

Determine the business benefits. Business benefits are quantitative (e.g., ROI, NPV) AND qualitative (e.g., in-house skills development). It guides the priority setting during a project as you will always have time and resources constraints.

Obtain stakeholder agreement on the business problem statement, resources, time frame, performance measures, and budget. In our case, utilize the available experts and participants to get an agreement about evaluation criteria and interpretations of the problem, tools, and constraints.

The best-performing teams invested quite some time in this step.

2. Think from the different stakeholder perspectives.

Even though the solution should address a traveler’s issue, think about other stakeholders.

What are the costs and benefits of your company introducing such a solution? What is the situation of the people who maintain the solution, e.g., in IT? What skills and resources are available? What other stakeholders are affected? Travel agencies? Hotels? Family members of the insurer’s customer? Lawyers and regulators? (data protection, or legal disputes if wrong advice is given).

Identify interests, needs, constraints, and potential positive/negative impact of all stakeholders

Clients / customers
Potential customers
Own company
The company’s IT
The decision-maker in the business

The winning teams did that and considered their insights into the solutions. Some of these teams built different personas to mirror and test the design of the solution. They justified all decisions made during the hackathon based on the stakeholder analysis.

3. UX matters.

Customer-centricity is key. The best technical solution will not have any use and demand when the user experience is not given.

Evaluate the solution against the users and your own behavior. Whenever possible, test it with clients and customers. Let them test your solution and observe the behavior of the user.

How do they operate the tool? How are the results used?

This approach created various features. One team implemented paraphrasing when the request was ambiguous. Another one did a cross-check with publicly available information to give warnings of potential changing situations and additional links to information. While the ones implemented a fallback, a possible manual interaction in the loop, when the solution cannot address a particular question, others directly gave the option for manual assistance. And the last one re-routed users automatically to personal service when certain trigger expressions appeared.

All these teams did it because they evaluated the interaction of users with the solution. One group stated during the pitch, “we could have gone further on with technical accuracy, but we decided to focus on user experience when we tried out our own solution after having a longer break and a fresh view on it.”

4. Be tool (and approach) agnostic.

All started with the provided ‘starter kit.’ They found flaws when they designed the solution and tested the various tools. While half of the teams tried to fix them within the given ecosystem, the other half started to experiment with integrating other available tools: AWS with Azure tools, GPT-3, TensorFlow to integrate the latest research on similarity analyses, and so on.

Both approaches will fix the flaws. The difference is prioritization. Instead of reinventing the wheel, the best-performing teams focused on their strategic and unique USP. Just because you can develop it, too, that does not mean you have to re-build it.

So, use what is available and do not reinvent the wheel. You should not limit yourself to only one provider and invest time in developing already available things. Focus on your unique USP, strategic differentiator, and value-added knowledge. Not the technology makes the differentiation, but the smart combination with your unique differentiator. This will make you succeed in the market.

5. Always test the outcomes against real-world cases.

In every machine learning course, we learn how to train and test a model on data. Then we have some quantitative measures, choose one or two, and select the model. But this is not enough.

The winning team did it extensively. They have taken many different single cases and tested them against the real world. They have not been deep insurance experts, so they have contacted these experts and assessed with them case-by-case many data points, the result, and what should be the right outcome. They built up an immense knowledge about the real world and how their solution reacts to it and adjusted algorithms accordingly. The accuracy of the results has been impressive.

You need to go into detail and pick single observations and results. Start with simple cases and add complexity step by step. Compare each observation and outcome case-by-case to the real world. This will dramatically enhance your understanding of how to improve hyper-tuning, parameter setting, and calibration.

Connecting the dots

These five steps should apply to any data science project. They clarify essential topics which are primarily a source of project failure. From my experience across different industries, I would state that 90% of all data science projects that did not make it into deployment had an issue with at least one of these five.

My advice for your next data science project:

Understand the business problem in all facets
Think about all different direct and indirect stakeholders and their needs and pains
User experience makes or breaks your solution
Do not reinvent the wheel but focus on your strategic USP
Test against the real-world

This increases the success rate tremendously.