Top 10 Most Powerful Lessons I Learned by Delivering Data Projects | by Hanzala Qureshi | Aug, 2022

Core Lessons That Will Ultimately Help You in Your Learning Journey

At the start of any new project/venture, you should reflect on the past and the lessons learned. Having delivered multiple complex data warehouses, data lakes and analytics projects, I learned a thing or two about what to do and what to avoid.

The key to any successful project is the people contributing to it. But the top reasons for failure are also people, along with unrealistic timelines, changing goalposts, and tired stakeholders.

Today, I will summarise the top 10 lessons I learned whilst delivering complex data projects. I am sure these will apply to other niches too.

Let’s dive in.

1. There are no unicorns

Data is complex by nature; it has many moving parts, challenges — difficult to comprehend and solutions — hard to digest. And I have yet to meet one individual who has answers to ALL the problems in a project. That person doesn’t exist.

The infrastructure, application, data engineering and analytics team have skills you must capitalise on to achieve a successful outcome. Some people are flexible enough to wear a few hats, but there is no substitute for specialisation. Use it to your advantage and ensure all the relevant skills are present on the project.

2. End users don’t know what they want

This is in no way a flippant comment — end users have a faint idea of what they want from the project. However, once they see the first iteration, they realise quickly that the outcome is unaligned with their first vision. Treating early requirements with a fistful of salt is not a bad idea.

Agile deliveries have to some extent, resolved this issue; however, in deep technical deliveries, the quick first iteration is sometimes impossible. Wireframes are usually crucial in this scenario to show what you expect the end product, whether a report, metric or model output, to look like.

3. Assume that the data quality is poor

So this should be no surprise, but most organisations struggle with poor quality data. When planning the data project delivery, it is imperative to include data profiling and remediation time. 99% of the time, the data will be of a sub-standard quality requiring some human or machine intervention to resolve.

4. No one cares about risks until they become an issue

Humans are naturally not risk averse. If they see a risk, even with flashing red lights, they’ll see it as a future event that may not materialise. For example, saying data quality could be poor and the resolution could take some time generally gets translated to “we are probably going to be fine.”

Be generous about what you class as risk and issue. Don’t worry about sounding negative by highlighting areas where you see potential problems early and putting mitigation steps before they become an issue. If you keep mundanely reporting on a risk weekly, the people will adjust to it; take action to show the threat of an impending problem.

5. Someone’s technical debt is your bankruptcy

Almost all projects get delivered with some technical debt. You know that IOU, the project sponsor signs off to get the bloody thing over the line. Unlike banking IOUs, there are no ways for someone to hold the project accountable for this debt.

Much like country debts, it gets passed to the next generation. In our case, it is given to the next project that needs this data. Ensure at the time of project planning you have a clear sight of these technical debts. You will need a plan to resolve the debt without the original project team being around.

6. Soft skills are equally (if not more) as important as technical skills

Dealing with challenging stakeholders, clearly managing key messages, and delivering bad news in a controlled fashion; are some things expected in this line of work. Of course, technical skills will set you aside as a dependable resource, but having the soft skills to go with it will make you exceptional.

Understanding complex technical concepts, translating them into business speak, and then persuading the business users whilst maintaining your integrity and reputation is the recipe for successful project delivery and, in fact, a successful career.

If you can’t explain it simply, you don’t understand it well enough — Albert Einstein

7. Too many cooks spoil the broth

Unless the delivery is highly complex, it will make sense to stick to 1 or 2 suppliers (consultants, system integrators etc.) and the same amount of vendors (software companies). Everyone has an end goal and agenda; spending too much time aligning people to your end vision is not easy.

Try and keep the team structure simple, reporting lines transparent, bureaucracy to a minimum and escalation paths well-communicated. Ensure problems are quickly managed, and misalignments are rectified. Data projects already have many moving pieces, don’t make people and teams another piece that could cause failure.

8. 20% of the work will deliver 80% of the benefits

The Pareto principle nicely applies to data projects; a limited amount of work will derive the most benefits. The key is to identify this 20% quickly and ensure the risks on that part are mitigated promptly. For example, improving data quality will reduce manual re-work, data pipeline intervention, and cost and risk reduction. Focus on that first, and this will lead to quick benefits realisation.

Keep using this method for ongoing prioritisation; soon, you will end up with a delivery backlog that benefits the end users quickly and more efficiently.

9. Culture of accountability doesn’t always exist

Complex programmes mean you are working with a diverse team of individuals. I was most surprised to see that not everyone considers accountability the same way you do. When the going gets tough and a project deadline is looming, you want a culture of accountability in the team where everyone is equally pulling their weight.

But letting minor misdemeanours go unchallenged such as delays in code development, testing outputs, user engagement etc., without a robust reason, can breed a culture of laxity. Set accountability as a critical principle since day one of the project, ensure everyone aligns to it and lives and breaths it.

10. Regular feedback makes a happy team

If I asked anyone which projects they most enjoyed? The answer will not be the easiest/challenging ones but the ones with the best team. The project in which you know your team members have your back and are collectively working towards a common goal feels exhilarating.

But a good team is only possible when you share regular feedback on what works and what could be improved. We are wired to avoid confrontational situations, but this kind of conversation often leads to the biggest improvement.

Conclusion

These are my top ten lessons, I am sure there are many more that I could document, and maybe I will leave that for a sequel. If you have encountered any more, please share them by leaving a comment below.

If you would like to learn about Data Architecture trends, check out this article:

If you are not subscribed to Medium, consider subscribing using my referral link. It’s cheaper than Netflix and objectively a much better use of your time. If you use my link, I earn a small commission, and you get access to unlimited stories on Medium.

I also write regularly on Twitter; follow me here.

Core Lessons That Will Ultimately Help You in Your Learning Journey

At the start of any new project/venture, you should reflect on the past and the lessons learned. Having delivered multiple complex data warehouses, data lakes and analytics projects, I learned a thing or two about what to do and what to avoid.

The key to any successful project is the people contributing to it. But the top reasons for failure are also people, along with unrealistic timelines, changing goalposts, and tired stakeholders.

Today, I will summarise the top 10 lessons I learned whilst delivering complex data projects. I am sure these will apply to other niches too.

Let’s dive in.

1. There are no unicorns

Data is complex by nature; it has many moving parts, challenges — difficult to comprehend and solutions — hard to digest. And I have yet to meet one individual who has answers to ALL the problems in a project. That person doesn’t exist.

The infrastructure, application, data engineering and analytics team have skills you must capitalise on to achieve a successful outcome. Some people are flexible enough to wear a few hats, but there is no substitute for specialisation. Use it to your advantage and ensure all the relevant skills are present on the project.

2. End users don’t know what they want

This is in no way a flippant comment — end users have a faint idea of what they want from the project. However, once they see the first iteration, they realise quickly that the outcome is unaligned with their first vision. Treating early requirements with a fistful of salt is not a bad idea.

Agile deliveries have to some extent, resolved this issue; however, in deep technical deliveries, the quick first iteration is sometimes impossible. Wireframes are usually crucial in this scenario to show what you expect the end product, whether a report, metric or model output, to look like.

3. Assume that the data quality is poor

So this should be no surprise, but most organisations struggle with poor quality data. When planning the data project delivery, it is imperative to include data profiling and remediation time. 99% of the time, the data will be of a sub-standard quality requiring some human or machine intervention to resolve.

4. No one cares about risks until they become an issue

Humans are naturally not risk averse. If they see a risk, even with flashing red lights, they’ll see it as a future event that may not materialise. For example, saying data quality could be poor and the resolution could take some time generally gets translated to “we are probably going to be fine.”

Be generous about what you class as risk and issue. Don’t worry about sounding negative by highlighting areas where you see potential problems early and putting mitigation steps before they become an issue. If you keep mundanely reporting on a risk weekly, the people will adjust to it; take action to show the threat of an impending problem.

5. Someone’s technical debt is your bankruptcy

Almost all projects get delivered with some technical debt. You know that IOU, the project sponsor signs off to get the bloody thing over the line. Unlike banking IOUs, there are no ways for someone to hold the project accountable for this debt.

Much like country debts, it gets passed to the next generation. In our case, it is given to the next project that needs this data. Ensure at the time of project planning you have a clear sight of these technical debts. You will need a plan to resolve the debt without the original project team being around.

6. Soft skills are equally (if not more) as important as technical skills

Dealing with challenging stakeholders, clearly managing key messages, and delivering bad news in a controlled fashion; are some things expected in this line of work. Of course, technical skills will set you aside as a dependable resource, but having the soft skills to go with it will make you exceptional.

Understanding complex technical concepts, translating them into business speak, and then persuading the business users whilst maintaining your integrity and reputation is the recipe for successful project delivery and, in fact, a successful career.

If you can’t explain it simply, you don’t understand it well enough — Albert Einstein

7. Too many cooks spoil the broth

Unless the delivery is highly complex, it will make sense to stick to 1 or 2 suppliers (consultants, system integrators etc.) and the same amount of vendors (software companies). Everyone has an end goal and agenda; spending too much time aligning people to your end vision is not easy.

Try and keep the team structure simple, reporting lines transparent, bureaucracy to a minimum and escalation paths well-communicated. Ensure problems are quickly managed, and misalignments are rectified. Data projects already have many moving pieces, don’t make people and teams another piece that could cause failure.

8. 20% of the work will deliver 80% of the benefits

The Pareto principle nicely applies to data projects; a limited amount of work will derive the most benefits. The key is to identify this 20% quickly and ensure the risks on that part are mitigated promptly. For example, improving data quality will reduce manual re-work, data pipeline intervention, and cost and risk reduction. Focus on that first, and this will lead to quick benefits realisation.

Keep using this method for ongoing prioritisation; soon, you will end up with a delivery backlog that benefits the end users quickly and more efficiently.

9. Culture of accountability doesn’t always exist

Complex programmes mean you are working with a diverse team of individuals. I was most surprised to see that not everyone considers accountability the same way you do. When the going gets tough and a project deadline is looming, you want a culture of accountability in the team where everyone is equally pulling their weight.

But letting minor misdemeanours go unchallenged such as delays in code development, testing outputs, user engagement etc., without a robust reason, can breed a culture of laxity. Set accountability as a critical principle since day one of the project, ensure everyone aligns to it and lives and breaths it.

10. Regular feedback makes a happy team

If I asked anyone which projects they most enjoyed? The answer will not be the easiest/challenging ones but the ones with the best team. The project in which you know your team members have your back and are collectively working towards a common goal feels exhilarating.

But a good team is only possible when you share regular feedback on what works and what could be improved. We are wired to avoid confrontational situations, but this kind of conversation often leads to the biggest improvement.

Conclusion

These are my top ten lessons, I am sure there are many more that I could document, and maybe I will leave that for a sequel. If you have encountered any more, please share them by leaving a comment below.

If you would like to learn about Data Architecture trends, check out this article:

If you are not subscribed to Medium, consider subscribing using my referral link. It’s cheaper than Netflix and objectively a much better use of your time. If you use my link, I earn a small commission, and you get access to unlimited stories on Medium.

I also write regularly on Twitter; follow me here.