pipelines

How to find more relevant search results by combining traditional keyword-based search with modern vector searchSearch bar with hybrid search capabilitiesWith the recent interest in Retrieval-Augmented Generation (RAG) pipelines, developers have started discussing challenges in building RAG pipelines with production-ready performance. Just like in many aspects of life, the Pareto Principle also comes into play with RAG pipelines, where achieving the initial 80% is relatively straightforward, but attaining the remaining…

Databricks is a cloud-based platform designed to simplify the process of building data engineering pipelines and developing machine learning models. It offers a collaborative workspace that enables users to work with data effortlessly, process it at scale, and derive insights rapidly using machine learning and advanced analytics. On the other hand, Visual Studio Code (VSCode) is a free, open-source editor by Microsoft, loaded with extensions for virtually every programming language and framework, making it a favorite…

Prepare CredentialsFirst of all, you have to create a .env file where you will add all our credentials.I already showed you in Lesson 1 how to set up your .env file. Also, I explained in Lesson 1 how the variables from the .env file are loaded from your ML_PIPELINE_ROOT_DIR directory into a SETTINGS Python dictionary to be used throughout your code.Thus, if you want to replicate what I have done, I strongly recommend checking out Lesson 1.If you only want a light read, you can completely skip the "Prepare Credentials"…

Building the Forecasting ModelBaseline modelFirstly, you will create a naive baseline model to use as a reference. This model predicts the last value based on a given seasonal periodicity.For example, if seasonal_periodicity = 24 hours, it will return the value from "present - 24 hours".Using a baseline is a healthy practice that helps you compare your fancy ML model to something simpler. The ML model is useless if you can't beat the baseline model with your fancy model.Fancy ML modelWe will build the model using Sktime…

ETLs don’t have to be complex. If that’s the case, use GitHub Actions.Photo by Roman Synkevych 🇺🇦 on UnsplashIf you’re into software development, you’d know what GitHub actions are. It’s a utility by GitHub to automate dev tasks. Or, in popular language, a DevOps tool.But people hardly use it for building ETL pipelines.The first thing that comes to mind when discussing ETLs is Airflow, Prefect, or related tools. They are, without a doubt, the best in the market for task orchestration. But many ETLs we build are simple,…

PYTHON PROGRAMMINGSee the power of dictcomp pipelinesPipelines process tasks one after another. Photo by Daniel Schludi on UnsplashThis article is motivated by a task I contributed to in a real life project a couple of years ago. After proposing the concept of comprehension pipelines, I noticed the solution could be nicely implemented using a dictcomp pipeline, with additional help of the OptionalBool data structure I proposed in yet another article.This article aims to show you how we can implement such a pipeline. I…

Automate Machine Learning Workflow with Continuous IntegrationAs a data scientist, you are responsible for improving the model currently in production. After spending months fine-tuning the model, you discover one with greater accuracy than the original.Excited by your breakthrough, you create a pull request to merge your model into the main branch.Image by AuthorUnfortunately, because of the numerous changes, your team takes over a week to evaluate and analyze them, which ultimately impedes project progress.Furthermore,…

One of the most popular Python libraries for dealing with machine learning tasks is scikit-learn. It went public in 2010 and has since been essential for implementing popular supervised ML algorithms like logistic regression, random forests, and support vector machines.When writing code in scikit-learn, you can use a feature called pipelines. This feature allows you to bundle up several of the steps in the machine learning process into a single component. The use of pipelines is one of the single most determining factors…

Applying MLOps best practices to advanced serving optionsImage from Unsplash by GrowtikaMLOps is an essential practice to productionizing your Machine Learning workflows. With MLOps you can establish workflows that are catered for the ML lifecycle. These make it easier to centrally maintain resources, update/track models, and in general simplify the process as your ML experimentation scales up.A key MLOps tool within the Amazon SageMaker ecosystem is SageMaker Pipelines. With SageMaker Pipelines you can define workflows…

A quick and simple demo for running DAGs on AirflowPhoto by Kelly Sikkema on UnsplashApache Airflow is quite popular in the data science and data engineering space. It boasts many features that enable users to programmatically create, manage, and monitor complex workflows.However, the platform’s range of features may inadvertently become a detriment to beginners. New users that explore Apache Airflow’s documentation and tutorials can easily become inundated by new terminology, tools, and concepts.With the aim of creating…