Iannone

Photo from Annie Spratt on UnsplashWhen you choose the platforms and systems that comprise your modern data infrastructure, the pricing model is a major factor to consider.Your data warehouse and data pipelines handle a ton of data every day, and they can quickly make or break your budget. Evaluating exactly how and why you’ll be charged is a crucial planning step.The pricing conversation arguably doesn’t get the attention it deserves. Instead, the rhetoric often gets fixated on a system’s performance, shiny new…

From agile to abstraction, thinking about data the way we think about software can save us lots of grief.Photo by Headway on UnsplashLet’s start by getting something important out of the way: there are some major differences between data engineering and software engineering.At the same time, they’re similar enough that many of the best practices that originated for software engineering are extremely helpful for data engineering, as long as you frame them correctly.In this article, I’ll walk through several software…

OpinionImproving your data infrastructure and data culture takes more than a flashy pipelineImage by Hatice EROL from PixabayHistory likes to repeat itself.To put it another way: we humans have a talent for creating the same problems for ourselves over and over, disguised as something new.Data stacks and data pipelines are no exception to this rule.Today’s data infrastructure landscape is dominated by the “modern data stack” (ie, a data stack centered around a cloud data warehouse) and ELT pipelines.This paradigm is…