Dependencies

Supercharged Pandas: Tracing Dependencies with a Novel ApproachAn object-oriented approach to manage multiple files and dataframes, and tracing dependencies.Photo by Sibel Yıldırım on UnsplashHow will this benefit you:This article describes an object-oriented approach for data analysis. It outlines 2 novel approaches: (a) reduce repetitive file reads by assigning dataframes to the attributes of the Reports object, and (b) trace dependent methods recursively to construct attributes. These approaches have allowed me to be…

Do you really need it? Maybe your environment has already everything. Okay, we need it. Firstly, consider security. How safe is it to use? Then, maintenance. When was the last release? How often do releases occur? Issues. What is the ratio of open issues to closed ones? And other points: code quality, test coverage, library size, number of dependencies and license. Read All Do you really need it? Maybe your environment has already everything. Okay, we need it. Firstly, consider security. How safe is it to use? Then,…

and Data warehouse Gitflow pipelines to run it automaticallyimage by authorDo you unit test your data warehouse scripts?I am going to talk about unit tests for complex SQL queries which might consist of multiple operations (actions).I tried it before using BigQuery scripting:However, here is another (and probably better) way to do it.Dataform is a great free tool for SQL data transformation. It helps to keep your data warehouse clean and well-organized. It has nice dependency graphs explaining data lineage and it serves…

Dependency management and packaging with PoetryPhoto by Daniel K Cheung on UnsplashDependency management is among the most fundamental aspects of Python projects. It can soon turn into a nightmare especially when a package is maintained by many developers. It is therefore quite important to make use of the right tools that could eventually help the maintainers of a project to handle dependencies the right way.The concept of dependency management also involves package upgrades — just because you’ve specified a pin doesn’t…