44 Tehama St.
San Francisco, CA 94105

Our work

Import big data into AWS as fast as possible

Industry

Product name

Client

Scroll down

A quick look

Our client needed to import big data as fast as possible but at the same time ensure enough cloud infrastructure capacity to run the client's e-commerce website and mobile app.

media.media-12
Implementation schema

Our Mission

Build a scaling mechanism to spin-up or kill-off Fargate ECS services processing Messaging Queues.

Our Approach

Our solution was a Lambda function which scales the ECS services processing the Messaging Queues. The Lambda function is based on an algorithm that calculates the available IOPS of the database, in order to make sure that the ECS services process the Messaging Queues as fast as possible, without causing any interruption in the availability of the web page.

Outcomes

  • Easy communication between the Lambda and ECS via the Python AWS SDK
  • The cloudwatch event to trigger the Lambda in a desired time period
  • Prioritization of the ECS tasks via a configuration file in the Lambda. This was impossible to achieve with a native ECS cron solution for example.
  • Accurate calculation of the available RDS IOPS