Tabular

Graph methods are more general than you may thinkPhoto by Alina Grubnyak on UnsplashGraph data science methods are usually applied to data that has some inherent graphical nature, e.g., molecular structure data, transport network data, etc. However, graph methods can also be useful on data which does not display any obvious graphical structure, such as the tabular datasets used in machine learning tasks. In this article I will demonstrate simply and intuitively — without any mathematics or theory — that by representing…

Image created by author using Dall-E 2Can LLMs reduce the effort involved in anomaly detection, sidestepping the need for parameterization or dedicated model training?Follow along with this blog’s accompanying colab.This blog is a collaboration with Jason Lopatecki, CEO and Co-Founder of Arize AI, and Christopher Brown, CEO and Founder of Decision PatternsRecent advances in large language models (LLM) are proving to be a disruptive force in many fields (see: Sparks of Artificial General Intelligence: Early Experiments…

Enhance readability: tips for displaying datasets in the CLIImage by Dorothe on PixabayA couple of days ago I wanted to help my father solve a problem. His need was to aggregate, filter, and display some data as fast as possible. Well…the truth is that he printed the data (something like 10 pages each time!!) and search the data by hand! I saw his difficulties and decided to help him immediately.Nothing as difficult for someone who can analyze data as I am: the data was already in Excel format, so a Jupyter Notebook and…

Image by authorWhat happens when you unleash GPT-4 on a tabular Kaggle competition to predict home prices?Follow along with this blog’s accompanying Colab.This blog is a collaboration with Jason Lopatecki, CEO and Co-Founder of Arize AI, and Christopher Brown, CEO and Founder of Decision PatternsThere are two distinct groups in the ML ecosystem. One works with highly organized data collected in tables — the tabular-data-focused data scientist. The other works on deep learning applications including vision, audio, large…

Effortless data deduplication at scale.Photo by Sangga Rima Roman Selia on UnsplashIn today’s data-driven world, the importance of high-quality data to build quality systems cannot be overstated.The availability of reliable data is highly critical for teams to make informed decisions, develop effective strategies, and gain valuable insights.However, at times, the quality of this data gets compromised by various factors, one of which is the presence of fuzzy duplicates.A set of records are fuzzy duplicates when they look…

Learn now to produce and consume data in Kafka using a sample log of online retail transactionsPhoto by Tech Daily on UnsplashTime-series data comes in all shapes and sizes and it’s often produced in high frequencies in the form of sensor data and transaction logs. It’s also produced in huge volumes where the records are separated by milliseconds rather than hours or days.But what kind of system that can handle such a constant stream of data? An older approach would be to dump the raw data in Data Lake and process it in…

Tabular: Information that is displayed in a table with rows and columns is known as “tabular format”. A data table is an organized and practical approach to displaying a significant amount of information that contains repeated data items. The majority of office productivity software packages, including word processing software and spreadsheets, have capabilities for text and data entry in tabular format.Example:<table> <thead> <tr> <th>Column…

Go beyond finding clusters in tabular dataPhoto by Alex Shuper on UnsplashWe are also comfortable in finding clusters in rows and columns of data. But how about finding clusters within an image? Let me illustrate the subject using an example of an image from recent world cup football 2022 in Qatar.Shown here is a photo which I had taken during Brazil vs Serbia match. The photo has been taken just before the stunning goal from Richarlison.Photo of world cup football 2022 Brazil vs Serbia taken by myself (image by…

No more run-time and memory optimization, let’s get straight to workPhoto by freestocks on UnsplashPandas makes the tasks of analyzing tabular datasets an absolute breeze. The sleek API design offers a wide range of functionalities that covers almost every tabular data use case.However, it’s only when someone transitions towards scale that they experience the profound limitations of Pandas. I have talked about this before in the blog below:In a gist, almost all limitations of Pandas arise from its single-core…

Advanced numerical embeddings for better performancePhoto by Pawel Czerwinski on UnsplashThis is the third part in my exploration of Transformers for Tabular Data.In the Part 2 I’ve described linear numerical embeddings and how they are used in the FT-Transformer model. This post is going to explore more complex versions of the numerical embeddings, so if you haven’t read the previous part, I highly recommend starting there and returning to this post afterwards.FT-Transformer. Image by author.As a reminder, above you can…