Feature request: For idempotency feature, allow no return value / undefined from lambda handlers.

[kuber-]

Use case

To make functions with side effects and no return value idempotent, undefined /no return value needs to be handled. Currently, the library fails with error message: Failed to update success record to idempotency store. This error was caused by: Pass options.removeUndefinedValues=true to remove undefined values from map/array/set..

Solution/User Experience

Update DynamoDBPersistenceLayer to handle undefined with appropriate Marshall options while updating the records in the persistence store.

Alternative solutions

No response

Acknowledgment

• This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #typescript channel on our Powertools for AWS Lambda Discord: Invite link

[dreamorosi]

Hi @kuber-, thank you for taking the time to open this issue.

As we discussed on Discord, the short-term solution while we work on a more permanent fix would be to simply return something. This will allow the utility to work normally and especially if you don't care about the result,

I have just tested this with Powertools for AWS Lambda (Python) and they handle this by marshalling the None value correctly, for example:

from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer,
    idempotent,
)
import boto3
from aws_lambda_powertools.utilities.typing import LambdaContext

persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable")
dynamodb_local_client = boto3.client("dynamodb", endpoint_url="http://localhost:8000")
persistence_layer.client = dynamodb_local_client


@idempotent(persistence_store=persistence_layer)
def lambda_handler(event: dict, context: LambdaContext):
    print("expensive operation")
    return None

if __name__ == "__main__":
    print(lambda_handler({
        "key": "value"
    }, LambdaContext()) )

Results in this item being added to DynamoDB:

ConsumedCapacity: null
Count: 1
Items:
- data:
    S: 'null'
  expiration:
    N: '1715351592'
  id:
    S: test-func.__main__.lambda_handler#88bac95f31528d13a072c05f2a1cf371
  status:
    S: COMPLETED
ScannedCount: 1

Which when retried, results in the value being unmarshalled back to None.

---

In this version of Powertools we use @util-dynamodb from the AWS SDK v3 and we call the marshall/unmarshall helpers with default options.

Both utilities accept an options object (example) that allows you to customize the serialization operation.

I will need to look into it and run some tests as I don't know if we should use the convertEmptyValues or removeUndefinedValues option in this case, but I think we should work on this and so I am adding it to the backlog.

[kuber-]

Thanks for your help @dreamorosi 🙏 ! I look forward to its implementation. In the meantime, returning a value works perfectly fine for me.

[dreamorosi]

After some tests I think the removeUndefinedValues option is kind of useless and won't help in this case.

Setting it to true does remove the undefined value from the object being marshalled, however the SDK at that point complains because now one of the of the expression attributes used in the expression is not defined/not present:

await client.send(
  new UpdateItemCommand({
    TableName: 'IdempotencyTable',
    Key: marshall({ id: '123' }),
    UpdateExpression: 'SET #expiration = :expiration, #data = :data',
    ExpressionAttributeNames: { '#expiration': 'expiration', '#data': 'data' },
    ExpressionAttributeValues: marshall(
      {
        ':expiration': 123,
        ':data': undefined,
      },
      { removeUndefinedValues: true }
    ),
  })
);

// ValidationException: Invalid UpdateExpression: An expression attribute value used in expression is not defined; attribute value: :data

This was brought up to the AWS SDK team in aws/aws-sdk-js-v3#4280 but apparently it was then closed as working as intended (🫠). Likewise, the convertEmptyValues option I mentioned in my previous comment is also useless in this instance.

Even if the SDK was smart enough to remove the attribute from the UpdateExpression and ExpressionAttributeNames when the corresponding ExpressionAttributeValues entry is undefined, I'm not convinced removing the value is the right option in this case and I agree much more with the approach used by boto3 of converting this to null instead.

I will try to work on a PR to handle this aligning with boto3. If that's not possible, I will instead implement logic that handles this on our side - something along these lines:

const updateExpressionOps = ['#expiration = :expiration'];
const expressionAttributeNames: Record<string, string> = {
  '#expiration': 'expiration',
};
const expressionAttributeValues: Record<string, unknown> = {
  ':expiration': 123,
};

const data = undefined;

if (data !== undefined) {
  updateExpressionOps.push('#data = :data');
  expressionAttributeNames['#data'] = 'data';
  expressionAttributeValues[':data'] = data;
}

await client.send(
  new UpdateItemCommand({
    TableName: 'IdempotencyTable',
    Key: marshall({ id: '123' }),
    UpdateExpression: `SET ${updateExpressionOps.join(', ')}`,
    ExpressionAttributeNames: expressionAttributeNames,
    ExpressionAttributeValues: marshall(expressionAttributeValues),
  })
);