From 431b05634b708f42fb79d1022681b6db9252cd1b Mon Sep 17 00:00:00 2001 From: Abhinav Vedmala Date: Tue, 8 Oct 2024 19:06:17 -0400 Subject: [PATCH] SFN Troubleshooting Updates (#25634) * SFN Troubleshooting Updates - Add note to recommend updating Step Functions and Lambda Forwarder - Add warning for log pipelines * Update troubleshooting.md --------- Co-authored-by: Bryce Eadie --- .../en/serverless/step_functions/troubleshooting.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/content/en/serverless/step_functions/troubleshooting.md b/content/en/serverless/step_functions/troubleshooting.md index acc82be95b143..afea90fb22e30 100644 --- a/content/en/serverless/step_functions/troubleshooting.md +++ b/content/en/serverless/step_functions/troubleshooting.md @@ -5,8 +5,7 @@ title: Troubleshooting Serverless Monitoring for AWS Step Functions ## I cannot see any traces ### Verify that your Step Function is configured to send all logs - -- Ensure that the `DD_TRACE_ENABLED` environment variable is set to `true` on the Lambda function in your AWS console. +- Ensure that the `DD_TRACE_ENABLED` tag is set to `true` on the Step Function in your AWS console. - In your AWS console, open your Step Function's logging tab. Ensure that _Log level_ is set to `ALL`, and that _Include execution data_ is selected. - Ensure that the CloudWatch log group (also found on the logging tab) has a subscription filter to the Datadog Lambda Forwarder in the same region. @@ -30,6 +29,14 @@ If your organization has an existing all-encompassing index with a low limit, pl **Note**: Indexing logs is not a requirement for getting traces and may incur additional cost. If you are troubleshooting a specific issue, you may wish to temporarily send logs to an index, debug, and delete the index afterwards. See [Indexes][6] for more information. +### Verify that your Step Function is using the latest version +- AWS may release updates to the Step Function API or introduce newer versions of the Step Function definitions. Older versions may result in unexpected log formatting or behavior. +- It's also recommended that you are using the latest version of the Datadog Lambda Forwarder to avoid discrepancies in how logs are forwarded. + +### Caution when using custom log pipelines +- Custom log pipelines can offer flexibility in processing logs, but altering the log format too much can lead to issues downstream, such as logs not being parsed or recognized. +- Avoid making significant changes to the Step Function log structure that change the JSON format. + ## Lambda traces are not merging with Step Function traces - Verify that you can see both Lambda traces and Step Function traces in Datadog. - Verify that you are using Python layer v95+ or Node.js layer v112+.