datadog:handler not initialized

[fabiokr]

Expected Behavior

Show additional info on datadog:handler not initialized error.

Actual Behavior

Only a log message is shown currently.

Steps to Reproduce the Problem

I have a Lambda function that runs with the datadog Lambda Layers on Node 14. I have checked this issues #298 #209 and my environment looks ok as far as I can tell (I use a plain Node setup, no webpack etc, and the packages are not part of package.json).

I think it might be related to this: #209 (comment)

But, it is hard to find out what exactly could be calling it out of place.

May we could have an additional debug log which prints the stack trace?

Specifications

• Datadog Lambda Layer version: Datadog-Extension:40
• Node version: Datadog-Node14-x:87

[astuyve]

Hi @fabiokr,

Thanks for reaching out. This error can occur when users call sendDistributionMetricWithDate before the Lambda Handler has been wrapped by the library. That can happen in a few ways:

1. Users call this method but do not finish the full installation steps for their function.
2. Users call sendDistributionMetricWithDate during the init process of a lambda function.

Please verify when this method is being called in your codebase and if you're still having issues, I suggest opening a support ticket with our team so we can review the code together.

Thanks!

[fabiokr]

Hi @fabiokr,

Thanks for reaching out. This error can occur when users call sendDistributionMetricWithDate before the Lambda Handler has been wrapped by the library. That can happen in a few ways:

1. Users call this method but do not finish the full installation steps for their function.
2. Users call sendDistributionMetricWithDate during the init process of a lambda function.

Please verify when this method is being called in your codebase and if you're still having issues, I suggest opening a support ticket with our team so we can review the code together.

Thanks!

Thanks for the input. I have followed the steps, and I think it might be the case that it is being called during init. But, I cannot identify where. That is why I was thinking that error message could include some additional info, like the stack trace, so I can dig into what is calling the sendDistributionMetric function.

[astuyve]

Ah I see, thanks for that context!

We can consider adding a stack trace to this error log, but we don't actually throw an error because we do not wish to crash user code. Since this error is recoverable, we simply move on.

I've created a simple reproduction case:

const { sendDistributionMetricWithDate } = require('datadog-lambda-js');

console.log('sending metric')
sendDistributionMetricWithDate('myMetric', 5)
console.log('sent metric')

module.exports.hello = async (event) => {
//... my handler code

And here you can see the logs:

2023-03-22T15:41:21.464-04:00 | 2023-03-22T19:41:21.464Z undefined INFO sending metric
2023-03-22T15:41:21.465-04:00 | 2023-03-22T19:41:21.464Z undefined ERROR {"status":"error","message":"datadog:handler not initialized"}
2023-03-22T15:41:21.465-04:00 | 2023-03-22T19:41:21.465Z undefined INFO sent metric

If you're having issues identifying where this is coming from, I'd suggest using your editor to search for sendDistributionMetric and trace those method calls to discover if they are called before the handler is initialized.

[fabiokr]

Ah I see, thanks for that context!

We can consider adding a stack trace to this error log, but we don't actually throw an error because we do not wish to crash user code. Since this error is recoverable, we simply move on.

I've created a simple reproduction case:

const { sendDistributionMetricWithDate } = require('datadog-lambda-js');

console.log('sending metric')
sendDistributionMetricWithDate('myMetric', 5)
console.log('sent metric')

module.exports.hello = async (event) => {
//... my handler code

And here you can see the logs:

2023-03-22T15:41:21.464-04:00 | 2023-03-22T19:41:21.464Z undefined INFO sending metric
2023-03-22T15:41:21.465-04:00 | 2023-03-22T19:41:21.464Z undefined ERROR {"status":"error","message":"datadog:handler not initialized"}
2023-03-22T15:41:21.465-04:00 | 2023-03-22T19:41:21.465Z undefined INFO sent metric

If you're having issues identifying where this is coming from, I'd suggest using your editor to search for sendDistributionMetric and trace those method calls to discover if they are called before the handler is initialized.

I tried completely disabling my calls to sendDistributionMetric, but I'm still seeing the issue. I also don't have the package on my deployed node_modules directory, so it uses the one from the lambda extension.

I could pinpoint the issue to where I am requiring the library, if I disable that, then the warning is gone. It looks something like this:

let sendDistributionMetric;

if (process.env.NODE_ENV !== 'test' && process.env.DD_ENV) {
  // use the datadog lambda layer lib if available
  sendDistributionMetric = require('datadog-lambda-js').sendDistributionMetric;
}

export function metric(name, value, group, tags = []) {
  const metricName = buildMetricName(name, group);

  if (process.env.NODE_ENV === 'test' || !sendDistributionMetric) {
    loggedMetric('count', name, value, group, tags);
  } else {
    sendDistributionMetric(metricName, value, ...tags);
  }
}

I have that as a way to make it also work on test environments without the lib. I don't have datadog-lambda-js on my package.json. If I comment sendDistributionMetric = require('datadog-lambda-js').sendDistributionMetric;, then the warning is gone. Any thoughts?

[astuyve]

You've pinpointed where it's loaded - but this error message is generated when the method is called. I'd suggest tracing your function calls to the metric function you have created here, and identify if there are cases where that is called yet the lambda handler has not been called first.

Please let me know if that helps!

[fabiokr]

@astuyve I have no idea why, but I just upgraded the lambda layers from Datadog-Node14-x:87 and Datadog-Extension:40 to Datadog-Node14-x:88 and Datadog-Extension:41 and the errors stopped. Definitely looks like it was not something on our end given I just upgraded the layers. Closing this for now.

[astuyve]

Hi @fabiokr - you can find the full changes here. The only code change in this release did not modify the metric method or handler methods at all, so if this problem still exists I expect it will resurface at some point.

Best of luck!