Luhui

GENERATIVE AI OVERVIEWGenerative AI for Text-to-Image, Text-to-3D, and Text-to-VideoThe Starry Night (by the author using Stable Diffusion)Generative AI is nascent but emerging exponentially. It has been stealing the limelight in the AI field since OpenAI debuted GPT-3 and DALL·E.2022 is the year of text-to-content generation (aka AIGC). In April 2022, OpenAI released DALL·E 2, described in the paper about CLIP and diffusion models. It was the first time to create realistic images and art from a text description in…

How to scale out training large models like GPT-3 & DALL-E 2 in PyTorchPhoto by Mark Harpur on UnsplashRecent years have witnessed exponential growth in the scale of distributed parallel training and the size of deep learning models. In particular, Transformer-based language models have been stealing the show. The notorious GPT-3 blew out with 175 billion parameters and 96 attention layers with a 3.2 M batch size and 499 billion words. Exactly half a year later, Google published Switch Transformer with 1.6 trillion…

Distributed model parallel training for large models in PyTorchPhoto by Daniela Cuevas on UnsplashRecent years have seen an exponential increase in the scale of deep learning models and the challenge of distributed parallel training. For example, the famous GPT-3 has 175 billion parameters and 96 attention layers with a 3.2 M batch size and 499 billion words. Amazon SageMaker training platform can achieve a throughput of 32 samples per second on 120 ml.p4d.24xlarge instances and 175 billion parameters. If we increase this…

Five steps to transform with data-centric AI engineeringThe past decade of AI/ML research and development has been historically significant. From cognitive learning to intelligent analytics, the evolution of deep learning (DL) has kicked off an unprecedented new era of AI. The latest artificial general intelligence (AGI) and self-supervised learning (SSL) are also compelling and promising. Gartner estimates the worldwide AI software market to reach $62 Billion in 2022, an increase of 21.3% from 2021. IDC predicts that…

Reinvent MLOps and accelerate AI democratizationPhoto by Sincerely Media on UnsplashThe past decade of AI/ML has been historic. With machine perception’s breakthrough milestones and AlphaGo’s fantastic victory, AGI (artificial general intelligence) and SSL (self-supervised learning) also hit the road. PwC predicts AI will contribute $15.7 trillion to the global economy by 2030. IDC forecasts that AI research and applications investments will hit $500 billion by 2024.AI can significantly improve performance and scalability…

Modern databases and the future of dataPhoto by Luca Bravo on UnsplashDatabases are essential technology like programming languages and operating systems. Business needs drive technology development. Over the past 30 years, hundreds of different databases have emerged from SQL to NoSQL and NewSQL. They have two primary workloads: OLTP (OnLine Transactional Processing) and OLAP (OnLine Analytical Processing), in various hardware architectures of shared-everything (e.g., Oracle RAC), shared-memory, shared-disk,…