Kovid

Photo by Jake Blucker on UnsplashDATA ENGINEERINGA peek into the different query optimizers and how they workSQL is a simple language with very few rules, which is what makes it so popular. It is also enriched with a great number of keywords and features that allow you to interact with your data in all kinds of ways. With this flexibility comes a lot of variance in query writing styles and choices.Once you issue a query to your database, it has to parse your query to understand its flow, but that’s not where the work of…

Photo by Maximalfocus on UnsplashDATA ENGINEERINGA brief introduction to various types of data workloadsDatabases are storage and processing products. They take data in from applications and store them. When you issue queries, database engines process them and give you back the output of your queries. The kinds of data captured by applications and how it is used by their users vary tremendously. Early on, the workloads for storage and processing were split into two main categories — transactional and analytical. Recently,…

Photo by Hanna Morris on UnsplashDATA ENGINEERINGPeeking into the database query execution engineOne of the most common problems that data analysts and data engineers face is non-performant queries, often referred to as slow queries. These queries are slow often not because there’s a shortage of resources to process the query, but because there’s an inefficient query you’ve written that uses far more resources than it should.Most data analysts and some data engineers don’t know much about database internals. How does one…

Photo by Shane Rounce on UnsplashHow Software Engineers & Data Engineers Think About DataAbout a year ago, I penned down my thoughts on the critical differences between how data engineers and data scientists approach data. I posited that the thought process of data engineers is deterministic because they work with deterministic systems. In contrast, the thought process of data scientists is probabilistic because they work with probabilistic systems. The difference in the thinking, in my opinion, is apparent and…