pipelines

Code implementations for ML pipelines: from raw data to predictionsPhoto by Rodion Kutsaiev on UnsplashReal-life machine learning involves a series of tasks to prepare the data before the magic predictions take place. Filling the missing values, one hot encoding for the categorical features, standardization and scaling for the numeric ones, feature extraction, and model fitting are just some of the stages that take part during a machine learning project before making any predictions. When working with NLP applications it…

Scaling the use of AI/ML by building Continuous Integration (CI) / Continuous Delivery (CD) / Continuous Training (CT) pipelines for ML based applicationsBackgroundIn my previous article:MLOps in Practice — De-constructing an ML Solution Architecture into 10 componentsI talked about the importance of building CI/CT/CD solutions to automate the ML pipelines. The aim of MLOps automation is to continuously test and integrate code changes, continuously train new models with new data, upgrade model performance when required,…

Beginners guide: Why unit and integration tests are so important for your data platformPhoto by Simon Wilkes on UnsplashThis story is for those who would like to learn how to code and how to run tests, automate CI/CD checks and run them in any environment including locally.Unit testing is an essential must-have skill for machine learning engineers these days. It looks great on your CV and increases the chances of getting employed.I’m a Data Engineer and very often I need to create microservices to process the data (ETL).…

Photo by Luca Bravo on UnsplashAutomating ML pipelines using Azure ML CLI(v2) & github actionsMachine learning models affect our interaction with the world as much as software products we use on a regular basis. Just like DevOps is required for seamless CI/CD, MLOps has become imperative for continuously building up-to-date models and utilising their predictions.In this article, we are going to build end-end MLOps using Azure ML CLI(v2) and Github Actions. This article hopes to serve as the starting point for your…

PYTHON PROGRAMMINGComprehension pipelines are a Python-specific idea for building pipelinesComprehension pipelines take you straight to the goal. Photo by Anika Huizinga on UnsplashGenerator pipelines offer a Pythonic way to create software pipelines, that is, chains of operations in which each operation but the first one takes the output of the previous operation as its input:They enable you to apply transforming programming as described by Thomas and Hunt in their great book The Pragmatic Programmer:A typical generator…

Definitive tutorial for advanced use of MLflowPhoto by Jeswin Thomas on UnsplashMLflow is a powerful tool that is often talked about for its experiment tracking capabilities. And it’s easy to see why — it’s a user-friendly platform for logging all the important details of your machine learning experiments, from hyper-parameters to models. But did you know that MLflow has more to offer than just experiment tracking? This versatile framework also includes features such as MLflow Projects, the Model Registry, and built-in…

OpinionMLOps in Practice — as a data scientist, are you handing over a notebook or an ML pipeline to your ML engineers or DevOps engineers for the ML model to be deployed in a production environment?BackgroundIn my previous articles :I talked about the importance of building ML pipelines. In today’s article, I will deep dive into the topic of ML pipelines and explain in detail:Why is it necessary and important to build ML pipelinesWhat are the key components of a ML pipelineWhy and how data scientists should adopt ML…

A step-by-step guide on tracking carbon emissions using Vertex AIimage generated by the Author with Midjourney.Machine learning has become a regular part of our daily lives, therefore it is time to consider its potential impacts on the environment. Otherwise, Mother Nature might just give us an ‘I told you so’ in the form of natural disasters leading to severe human suffering. One way we can help combat climate change is by starting to measure and reduce the carbon footprint of our machine-learning models. The carbon…

Forget about the new data trends in 2023! This fundamental data engineering challenge is still not solved.It is 2023! New data paradigms (or buzz words) like ELT, reverse ETL, EtLT, Data mesh, Data contracts, FinOps and modern data stack found their way into mainstream data conversations. Our data teams are still figuring out what is hype and what is not.There may be 10 new paradigms tomorrow but some of the fundamental challenges in data engineering — like data quality — are still relevant and not solved completely (I…

Photo by Jez Timms on UnsplashI am a big fan of unit-testing.Reading two books — The Pragmatic Programmer and Refactoring — completely changed the way I viewed unit-testing.“Testing is not about finding bugs.We believe that the major benefits of testing happen when you think about and write the tests, not when you run them.”— The Pragmatic Programmer, David Thomas and Andrew HuntInstead of seeing testing as a chore to complete after I have finished my data pipelines, I see it as a powerful tool to improve the design of my…