Import big data into AWS as fast as possible
Industry
Automotive
Product name
AWS – Data processing cloud scaling
Client
AWS | U+
Scroll down
A quick look
Our client needed to import big data as fast as possible but at the same time ensure enough cloud infrastructure capacity to run the client's e-commerce website and mobile app.
Our Mission
Build a scaling mechanism to spin-up or kill-off Fargate ECS services processing Messaging Queues.
Our Approach
Our solution was a Lambda function which scales the ECS services processing the Messaging Queues. The Lambda function is based on an algorithm that calculates the available IOPS of the database, in order to make sure that the ECS services process the Messaging Queues as fast as possible, without causing any interruption in the availability of the web page.
Outcomes
- Easy communication between the Lambda and ECS via the Python AWS SDK
- The cloudwatch event to trigger the Lambda in a desired time period
- Prioritization of the ECS tasks via a configuration file in the Lambda. This was impossible to achieve with a native ECS cron solution for example.
- Accurate calculation of the available RDS IOPS