DAGs

Two graphical criteria for estimating causal effectsThis is the 4th article in a series on causal effects. In the last article of this series, we explored the question of identifiability. In other words, can the causal effect be evaluated from the given data? There we saw a systematic 3-step process to express any causal effect given a causal model where all variables are observed. The problem, however, becomes much more interesting when we have unmeasured confounders. In this article, I discuss two quick-and-easy…

Authoring Apache Airflow DAGs and Tasks with Python decoratorsPhoto by Chaitanya Tvs on UnsplashIntroductionThe most common way for writing pipelines in Airflows is by using the DAG context managers to automatically assign new operators to that DAG. As of Airflow 2, you can now use decorators in order to author Airflow DAGs and Tasks.In today’s tutorial, we will introduce these decorators and showcase how one can use them in order to write cleaner code. Additionally, we will also demonstrate how the same DAG would have…

Data EngineeringThe two most important properties you need to know when designing a workflowPhoto by Campaign Creators on UnsplashLast week, we learned how to quickly spin up a development environment for Apache Airflow.This is awesome!However, we have yet to learn how to design an efficient workflow. Simply having a great tool at our fingertips won’t cut the deal alone — unfortunately.Although Apache Airflow does a pretty good job at doing most of the heavy lifting for us, we still need to ensure certain key properties…