Deploy Long-Running ETL Pipelines to ECS with Fargate
To keep things simple and costs to a minimumETL Pipeline | Image by authorETL stands for Extract, Transform, and Load. An ETL pipeline is essentially just a data transformation process — extracting data from one place, doing something with it, and then loading it back to the same or a different place.If you are working with natural language processing via APIs, which I’m guessing most will start doing, you can easily hit the timeout threshold of AWS Lambda when processing your data, especially if at least one function…