Cavin

Why AI products are so difficult to designPhoto by DeepMind on UnsplashUser experience (UX) tools help designers create better products by making it easier to understand how users interact with them. However, there are few UX tools specifically designed for artificial intelligence (AI) products. This is a problem because AI products are becoming increasingly common and they present unique challenges for designers.Building AI applications is hard. Not only do you need to have a good understanding of the AI algorithms…

Idea generator, instruction, and TL;DR summaryPhoto by Roman Kraft on UnsplashGPT-3 is a computer system that is designed to generate natural language. It does this by taking in a piece of text and then predicting the next word or phrase that should come after it.In order to accomplish this, GPT-3 uses a deep learning algorithm that has been trained on a large corpus of text. This allows it to capture the statistical patterns of how words are used together in a sentence.GPT-3 can be used for a variety of tasks that…

Benchmark real-time anomaly detection methodsillustration of HBOS algorithm (image by author)In this blog post, we will compare the Histogram-based Outlier Score (HBOS) a proximity-based outlier detection algorithm that proved to be fast with linear time complexity, and an ensemble method called isolation Forest.We will discuss the working principle behind those two algorithms as well as their respective performance on the MacBook Pro M1 for 17 different datasets.The Outlier Detection DataSets (ODDS) group openly provides…

How I improved speed by 10xphoto by Saffu on UnsplashRecently, I’ve been working on an anomaly detection model for a project at work. I needed to improve the speed of my code by 10x.In this post, I’ll describe how I achieved this goal and some of the challenges I encountered along the way.Roughly speaking, anomaly detection techniques try to identify patterns in data that do not conform to typical behavior. This can be used to identify problems in a system, fraudulent behavior, or other unusual activities. In my case, we…

Machine learning for streaming data using PyOD And PySADsliding window for real-time anomaly detection (image by author)In this blog post, we are going to be talking about anomaly detection for streaming data and specifically two libraries for Python which are PyOD and PySAD.Simply put, anomaly detection is the identification of items, events, or observations that do not conform to an expected pattern. This could be something as simple as detecting fraudulent credit card transactions or something as complex as identifying…

Fight the curse of dimensionalityillustration to explain the curse of dimensionality (image by author)Feature engineering is the process of taking raw data and extracting features that are useful for modeling. With images, this usually means extracting things like color, texture, and shape. There are many ways to do feature engineering, and the approach you take will depend on the type of data you’re working with and the problem you’re trying to solve.But why would we need it with images?Images encapsulate a lot of…

Encoding categorical variables into numericPhoto by Pop & Zebra on UnsplashA machine learning algorithm needs to be able to understand the data it receives. For example, categories such as “small”, “medium”, and “large” need to be converted into numbers. To solve that, we can for example convert them into numeric labels with “1” for small, “2” for medium, and “3” for large.But is it really the best way?There are plenty of methods to encode categorical variables into numeric and each method comes with its own…

Combine feature extraction, similarity measure, and nearest neighbor graphExample of a k-nearest neighbor graph (image by author)When you start working on a dataset that consists of pictures, you’ll probably be asked such questions as: can you check if the pictures are good? Is there any anomaly? A quick-and-dirty solution would be to manually look at the data one by one and try to sort them out, but that might be tedious work depending on how many pictures you get.For example, in manufacturing, you could get a sample…

From a feature engineering perspectivefeature engineering with histogram for machine learning (image by author)Feature engineering is the process of using domain knowledge to create features that make machine learning algorithms work better. This is a crucial part of applied machine learning and is often the difference between successful and unsuccessful projects.On the other hand, histograms are known as one of the first steps to data preprocessing. It is an essential step for data exploration with simple basics: it…