AWS Lambda now supports concurrency limits on individual functions: https://aws.amazon.com/about-aws/whats-new/2017/11/set-concurrency-limits-on-individual-aws-lambda-functions/
amazon web services - Can I limit concurrent invocations of an AWS Lambda? - Stack Overflow
how to increase the concurrency of lambda?
What can I do about lambda concurrency being set to 10 in new accounts even though default is 1000?
AWS Lambda Concurrency limit
It’s per account and region, organization won’t make any difference in your case.
More on reddit.comVideos
I created a workflow using Lambdas triggered by EventSourceMapping+SQS and set 2000 ReservedConcurrentExecutions. To test I make a recursive copy from local to the bucket but when monitoring the queue the number of In Transit messages is always below 1k.
After digging more I tried to set the MaximumConcurrency in EventSourceMapping at max (1k), which I understood that will scale the triggers, but nothing changed, I always see a slow slope at the In Transit messages.
Am I understanding any concept wrong? Does this amount of concurrency affects bandwidth in each lambda?
AWS Lambda now supports concurrency limits on individual functions: https://aws.amazon.com/about-aws/whats-new/2017/11/set-concurrency-limits-on-individual-aws-lambda-functions/
I would suggest you to use Kinesis Streams (or alternatively DynamoDB + DynamoDB Streams, which essentially have the same behavior).
You can see Kinesis Streams as as queue. The good part is that you can use a Kinesis Stream as a Trigger to you Lambda function. So anything that gets inserted into this queue will automatically be passed over to your function, in order. So you will be able to process those S3 events one by one, one Lambda execution after the other (one instance at a time).
In order to do that, you'll need to create a Lambda function with the simple purpose of getting S3 Events and putting them into a Kinesis Stream. Then you'll configure that Kinesis Stream as your Lambda Trigger.
When you configure the Kinesis Stream as your Lambda Trigger I suggest you to use the following configuration:
- Batch size: 1
- This means that your Lambda will be called with only one event from Kinesis. You can select a higher number and you'll get a list of events of that size (for example, if you want to process the last 10 events in one Lambda execution instead of 10 consecutive Lambda executions).
- Starting position: Trim horizon
- This means it'll behave as a queue (FIFO)
A bit more info on AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AWS Lambda.
I hope this helps anyone with a similar problem.
P.S. Bear in mind that Kinesis Streams have their own pricing. Using DynamoDB + DynamoDB Streams might be cheaper (or even free due to the non-expiring Free Tier of DynamoDB).