[FIXED] AWS Lambda Cannot deserialize instance of `java.lang.String` out of START_OBJECT token

Issue

I have a very simple Java 11 Lambda:

public class GetArticleHandler implements RequestHandler<APIGatewayV2ProxyRequestEvent, APIGatewayV2ProxyResponseEvent> {

    @Inject
    private GetArticleService getArticleService;

    @Override
    public APIGatewayV2ProxyResponseEvent handleRequest(APIGatewayV2ProxyRequestEvent req, Context context) {

        String path = req.getPath();

        Article article = getArticleService.get(path);

        return generateResponse(req, article);
    }

    private APIGatewayV2ProxyResponseEvent generateResponse(APIGatewayV2ProxyRequestEvent req, Article article) {
        APIGatewayV2ProxyResponseEvent res = new APIGatewayV2ProxyResponseEvent();
        res.setHeaders(Collections.singletonMap("timeStamp", String.valueOf(System.currentTimeMillis())));
        res.setStatusCode(200);
        res.setBody(article.toString());
        return res;
    }

}

It is wired up to AWS APIGateway via a CloudFormation deployment, using the following template (note that this is an extract from that template):

Resources:
  UTableArticle:
    Type: AWS::DynamoDB::Table
    Properties:
      KeySchema:
        - AttributeName: id
          KeyType: HASH
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      ProvisionedThroughput:
        ReadCapacityUnits: 1
        WriteCapacityUnits: 1
      TableName: !Sub ${AWS::StackName}-Article
    UpdateReplacePolicy: Retain
    DeletionPolicy: Retain
    
  FunctionAssumeRoleRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
                
  DynamoActionsPolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyDocument:
        Statement:
          - Action:
              - dynamodb:BatchGetItem
              - dynamodb:GetRecords
              - dynamodb:GetShardIterator
              - dynamodb:Query
              - dynamodb:GetItem
              - dynamodb:Scan
              - dynamodb:BatchWriteItem
              - dynamodb:PutItem
              - dynamodb:UpdateItem
              - dynamodb:DeleteItem
            Effect: Allow
            Resource:
              - !GetAtt [ UTableArticle, Arn ]
              - !Ref AWS::NoValue
        Version: "2012-10-17"
      PolicyName: DynamoActionsPolicy
      Roles:
        - !Ref FunctionAssumeRoleRole
  BFunctionGetArticle:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: !Ref ArtefactRepositoryBucket
        S3Key: !Join [ '', [!Ref ArtefactRepositoryKeyPrefix, '.zip' ] ]
      Handler: !Ref 'GetArticleHandler'
      Role: !GetAtt [ FunctionAssumeRoleRole, Arn ]
      Runtime: java11
      Environment:
        Variables:
          TABLE_NAME: !Ref UTableArticle
          PRIMARY_KEY: id
    DependsOn:
      - DynamoActionsPolicy
      - FunctionAssumeRoleRole
  BFunctionGWPermissionGetIdArticle:
    Type: AWS::Lambda::Permission
    DependsOn:
      - BlogRestApi
      - BFunctionGetArticle
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !GetAtt [ BFunctionGetArticle, Arn ]
      Principal: apigateway.amazonaws.com
      SourceArn: !Join ['', ['arn:', !Ref 'AWS::Partition', ':execute-api:', !Ref 'AWS::Region', ':', !Ref 'AWS::AccountId', ':', !Ref BlogRestApi, '/*/GET/article/{id}'] ]
  BlogRestApi:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: Article
  AGWDeploymentArticle:
    Type: AWS::ApiGateway::Deployment
    Properties:
      RestApiId: !Ref BlogRestApi
      Description: Automatically created by the RestApi construct
    DependsOn:
      - MethodArticleIdGet
      - MethodArticleIdPatch
      - ResourceArticleId
      - MethodArticleGet
      - MethodArticlePost
      - ResourceArticle
  BAGDeploymentStageProdArticle:
    Type: AWS::ApiGateway::Stage
    Properties:
      RestApiId: !Ref BlogRestApi
      DeploymentId: !Ref AGWDeploymentArticle
      StageName: prod
  ResourceArticle:
    Type: AWS::ApiGateway::Resource
    Properties:
      ParentId: !GetAtt [ BlogRestApi, RootResourceId ]
      PathPart: article
      RestApiId: !Ref BlogRestApi
  MethodArticleGet:
    Type: AWS::ApiGateway::Method
    Properties:
      HttpMethod: GET
      ResourceId: !Ref ResourceArticle
      RestApiId: !Ref BlogRestApi
      AuthorizationType: NONE
      Integration:
        IntegrationHttpMethod: POST
        Type: AWS_PROXY
        Uri: !Join [ "", ['arn:', !Ref 'AWS::Partition', ':apigateway:', !Ref 'AWS::Region', ':lambda:path/2015-03-31/functions/', !GetAtt [ BFunctionListArticles, Arn ], '/invocations' ] ]
  ResourceArticleId:
    Type: AWS::ApiGateway::Resource
    Properties:
      ParentId: !Ref ResourceArticle
      PathPart: "{id}"
      RestApiId: !Ref BlogRestApi
  MethodArticleIdGet:
    Type: AWS::ApiGateway::Method
    Properties:
      HttpMethod: GET
      ResourceId: !Ref ResourceArticleId
      RestApiId: !Ref BlogRestApi
      AuthorizationType: NONE
      Integration:
        IntegrationHttpMethod: POST
        Type: AWS_PROXY
        Uri: !Join [ "", ['arn:', !Ref 'AWS::Partition', ':apigateway:', !Ref 'AWS::Region', ':lambda:path/2015-03-31/functions/', !GetAtt [ BFunctionGetArticle, Arn ], '/invocations' ] ]

CloudFromation deploys correctly and I can make calls through a cURL on the deployment as a whole or I can go to the API Gateway resource and conduct a test there. In either case, a call into the Lambda gets stuck at the Jackson deserialization on entry, and in the logs in CloudWatch, i get the error:

An error occurred during JSON parsing: java.lang.RuntimeException
java.lang.RuntimeException: An error occurred during JSON parsing
Caused by: java.io.UncheckedIOException: com.amazonaws.lambda.thirdparty.com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.lang.String` out of START_OBJECT token
 at [Source: (ByteArrayInputStream); line: 1, column: 1]
    at com.amazonaws.services.lambda.runtime.serialization.factories.JacksonFactory$InternalSerializer.fromJson(JacksonFactory.java:182)
Caused by: com.amazonaws.lambda.thirdparty.com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.lang.String` out of START_OBJECT token
 at [Source: (ByteArrayInputStream); line: 1, column: 1]
    at com.amazonaws.lambda.thirdparty.com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
    at com.amazonaws.lambda.thirdparty.com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1442)
    at com.amazonaws.lambda.thirdparty.com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1216)
    at com.amazonaws.lambda.thirdparty.com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1126)
    at com.amazonaws.lambda.thirdparty.com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:63)
    at com.amazonaws.lambda.thirdparty.com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10)
    at com.amazonaws.lambda.thirdparty.com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1719)
    at com.amazonaws.lambda.thirdparty.com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1228)
    at com.amazonaws.services.lambda.runtime.serialization.factories.JacksonFactory$InternalSerializer.fromJson(JacksonFactory.java:180)

This error seems to be telling me that Jackson is attempting to deserialize the API Gateway event as a string (which, of course, it is not). Given that I have specified the Lambda as: GetArticleHandler implements RequestHandler<APIGatewayV2ProxyRequestEvent, APIGatewayV2ProxyResponseEvent> I expected that Jackson would try to deserialize the API Gateway event into a APIGatewayV2ProxyRequestEvent. But no matter how I specify the RequestHandler (for example, I've tried specifying Map<String,Object> instead), it keeps trying to deserialise the event as if it were a string. Can anyone tell me what's going on here? Is there something that I'm missing?

Solution

It was hard to track this down, but it comes down to the need to supply a RequestTemplate to the AWS::ApiGateway::Method. The way I did this was in the CloudFormation template:

  MethodArticleIdGet:
    Type: AWS::ApiGateway::Method
    Properties:
      HttpMethod: GET
      ResourceId: !Ref ResourceArticleId
      RestApiId: !Ref BlogRestApi
      AuthorizationType: NONE
      Integration:
        IntegrationHttpMethod: POST
        Type: AWS_PROXY
        RequestTemplates:
          application/json: !Ref ParamRequestMappingTemplate
        Uri: !Join [ "", ['arn:', !Ref 'AWS::Partition', ':apigateway:', !Ref 'AWS::Region', ':lambda:path/2015-03-31/functions/', !GetAtt [ BFunctionGetArticle, Arn ], '/invocations' ] ]

and then adding the ParamRequestMappingTemplate:

Parameters:

  <snip>

  ParamRequestMappingTemplate:
    Type: String
    Description: 'Read from resources/templates'

  <snip>

so that I could feed in the parameter via the --parameter-overrides in the cloudformation deploy call with a file reference to a .vsl file containing:

#set($allParams = $input.params())
{
    #foreach($type in $allParams.keySet())
    #set($params = $allParams.get($type))
    "$type" : {
      #foreach($paramName in $params.keySet())
      "$paramName" : "$util.escapeJavaScript($params.get($paramName))"
      #if($foreach.hasNext),#end
      #end
    }
    #if($foreach.hasNext),#end
    #end
}

which is a modification on an AWS script that passes all headers, path args and query parameters as mapped elements.

I then modelled the request parameters in the following class:

public class RequestParams {
    String path;

    Map<String, String> header;

    Map<String, String> queryString;
}

and then remodelled the Lamabda and handler method:

public class GetArticleHandler implements RequestHandler<RequestParams, Response<Article>> {

    Injector injector = Guice.createInjector(new GetArticleHandlerModule());

    private GetArticleService getArticleService = injector.getInstance(GetArticleService.class);

    public void setGetArticleService(GetArticleService getArticleService) {
        this.getArticleService = getArticleService;
    }

    @Override
    public Response<Article> handleRequest(RequestParams params, Context context) {
        LOGGER.init(context, "GetArticle", null);

        LOGGER.info(this, params.getPath());

        Article article = getArticleService.get(params.getPath());

        return new Response<>(article);
    }
}

With the provision of this, the error went away.

Although, it should be noted that the API Gateway layer also requires the response be modelled as:

public class Response<B> {
    @JsonProperty("isBase64Encoded")
    boolean isBase64Encoded;
    int statusCode;
    Map<String, String> headers;
    B body;

    public Response(B body) {
        this.setBase64Encoded(false);
        this.setStatusCode(200);
        this.setHeaders(Map.of("Access-Control-Allow-Origin", "*"));
        this.setBody(body);
    }
}

I'm still having problems with this as the response after Jackson serialization stubbornly outputs:

Tue May 26 08:25:04 UTC 2020 : Endpoint response body before transformations: {"statusCode":200,"headers":{"Access-Control-Allow-Origin":"*"},"body":{"tags":[]},"base64Encoded":false}

In other words, it always serialises isBase64Encoded as base64Encoded no matter what I do.

and this results in the error:

Tue May 26 08:25:04 UTC 2020 : Execution failed due to configuration error: Malformed Lambda proxy response
Tue May 26 08:25:04 UTC 2020 : Method completed with status: 502

oh the humanity!

Answered By - Michael Coxon
Answer Checked By - Willingham (JavaFixing Volunteer)