Kafka

What drew you to translate this collection? David Yang (Courtesy Pushkin Press) Kafka has long been one of my favourite writers, so I jumped at the opportunity. I was immediately blown away by the art style, as I felt Nishiokya Kyodai did a fabulous job at capturing the uncanny and disorienting sense of impending doom that I had always gotten from reading Kafka’s prose. Moreover, since I had majored in German literature in my undergraduate days, I thought it was the perfect chance to combine my background in…

How to Stream and Apply Real-Time Prediction Models on High-Throughput Time-Series DataPhoto by JJ Ying on UnsplashMost of the stream processing libraries are not python friendly while the majority of machine learning and data mining libraries are python based. Although the Faust library aims to bring Kafka Streaming ideas into the Python ecosystem, it may pose challenges in terms of ease of use. This document serves as a tutorial and offers best practices for effectively utilizing Faust.In the first section, I present an…

This article is part of a project that’s split into two main phases. The first phase focuses on building a data pipeline. This involves getting data from an API and storing it in a PostgreSQL database. In the second phase, we’ll develop an application that uses a language model to interact with this database.Ideal for those new to data systems or language model applications, this project is structured into two segments:This initial article guides you through constructing a data pipeline utilizing Kafka for streaming,…

Generative AI (GenAI) enables advanced AI use cases and innovation but also changes how the enterprise architecture looks. Large Language Models (LLM), vector databases, and retrieval augmentation generation (RAG) require new data integration patterns and data engineering best practices. Data streaming with Apache Kafka and Apache Flink is key to ingesting and curating incoming data sets in real-time at scale, connecting various databases and analytics platforms, and decoupling independent business units and data…

Franz Kafka’s Metamorphoses finds a mention in Abhishek Anicca’s The Grammar of My Body. The author alludes to the novella while describing the time his condition left him with limited mobility. Since the bed, kitchen stove and the refrigerator were on the floor, he could crawl to them. “This was around the same time I was reading Metamorphoses,” he writes, “Not by choice though.” Abhishek Anicca in performance (Reframe Institute of Art and Expression) As the memoir progresses, it becomes clear that the…

My earliest reading memory We had very few books at home, growing up, so I always looked forward to the Japanese textbooks that were handed out in school every term. I would read the entire book in one day, and over and over again after that. Textbooks might seem boring, but they gave me a way into contemporary poetry, haiku and tanka poems.My favourite book growing upDemian by Hermann Hesse. Of all the protagonists, Sinclair appealed to me because of his introspective nature and lack of confidence.It surprised me that I…

Too Long; Didn't ReadKafka is a message broker in which some services generate messages and others receive them. In this article we will write a small application for consuming messages from kafka. And of course e2e tests. First we need to create a controller that will process messages. Then run the microservices in the main services in the NestJs framework. Too Long; Didn't ReadKafka is a message broker in which some services generate messages and others receive them. In this article we will write a small application…

Aviation and travel are notoriously vulnerable to social, economic, and political events, as well as the ever-changing expectations of consumers. The coronavirus was just a piece of the challenge. This post explores how Lufthansa leverages data streaming powered by Apache Kafka as cloud-native middleware for mission-critical data integration projects and as data fabric for AI/machine learning scenarios such as real-time predictions in fleet management. An interactive conversation with Lufthansa as an on-demand video is…

Honkai: Star Rail just dropped its 1.2 update, which means new content and characters. We get to see the next chapter of the Xianzhou Luofu plotline and pull for Stellaron Hunters Blade and Kafka, the latest limited-time five-star characters. They’re part of a mysterious organization that follows the will of someone named Elio, who supposedly can tell the future and even manipulate it. As the “five-star” title suggests, Blade and Kafka are some of the strongest teammates you can have. They play a significant part in…

Many mission-critical use cases require low latency data processing. Running these workloads close to the edge is mandatory if the applications cannot run in the cloud. This blog post explores architectures for low latency deployments leveraging a combination of cloud-native infrastructure at the edge, such as AWS Wavelength, 5G networks from Telco providers, and event streaming with Apache Kafka to integrate and process data in motion. The blog post is structured as follows: Definition of "low latency data processing"…