[FIXED] Amazon Lambda re-invoked every minute with multiple request IDs despite having no errors

Issue

I have a Lambda function in Python 3.7 which gets invoked explicitly and synchronously via Boto3. It is configured with a timeout of 5 minutes. While the 1st invocation is in progress for a minute, a second invocation is started with another request ID. Another minute later, the request is again retried, and this time it finished quicky. as it detects the system state has changed. An example sequence of invocations

1. 11:20:29 to 11:21:53 - log stream A - Req. ID: 68cfbd89-9f4c-4e8c-9de0-b9e548983692
2. 11:21:30 to 11:22:56 - log stream B - Req. ID: 1b569ffb-67b8-4e82-8e99-46424f45e2e5
3. 11:22:30 to 11:22:33 - log stream A - Req. ID: d9d60271-4626-43ab-bb3a-f14be057af13

The request ID that's returned from my Lambda call is d9d60271-4626-43ab-bb3a-f14be057af13.

The stack that invokes the Lambda is as follows:

• Jenkins running a Pipeline script.
• The script runs a Makefile which executes 2 sequential tasks.
  • Build a docker container
  • Run a script inside the container.
• The script that runs inside the container is in Python 3.7. It uses Boto3 to call some AWS APIs, then invokes the lambda with InvocationType="RequestResponse".

None of the invocations exits with an unhandled exception or any obvious error. Is Lambda retrying my calls? Could any other element of this stack, notably Jenkins, be "transparently" retrying the task, and if so, how would I establish that?

Solution

The problem turned out to arise from Boto3 on the client-side, and not from Lambda! When a lambda returns nothing for more than a minute, Boto3 by default times out. To fix this, I had to override Boto's default config:

lambda_config = botocore.config.Config(region_name=region, read_timeout=300)
lambda_client = boto3.client('lambda', config=lambda_config)

The way I demonstrated this was a client-side problem was by setting the parameter reserved_concurrent_executions to 1, thus preventing concurrent executions of my Lambda, and noting that I get an exception from Boto.

Answered By - Isac Casapu
Answer Checked By - Marilyn (JavaFixing Volunteer)