diff --git a/packages/aws_elb_otel/changelog.yml b/packages/aws_elb_otel/changelog.yml index 736bdf231fe..1baaa93a2d2 100644 --- a/packages/aws_elb_otel/changelog.yml +++ b/packages/aws_elb_otel/changelog.yml @@ -1,4 +1,9 @@ # newer versions go on top +- version: "0.1.1" + changes: + - description: Add "Alternative setup using awss3receiver" section to README + type: enhancement + link: https://github.com/elastic/integrations/pull/15844 - version: "0.1.0" changes: - description: Initial draft of the AWS ELB Access Logs OpenTelemetry Assets package diff --git a/packages/aws_elb_otel/docs/README.md b/packages/aws_elb_otel/docs/README.md index 38dcb833f03..f4db9414dd1 100644 --- a/packages/aws_elb_otel/docs/README.md +++ b/packages/aws_elb_otel/docs/README.md @@ -23,6 +23,58 @@ The EDOT Cloud Forwarder for AWS supports collecting logs from: For the full documentation on how to set up the EDOT Cloud Forwarder, follow this link: [EDOT Cloud Forwarder for AWS](https://www.elastic.co/docs/reference/opentelemetry/edot-cloud-forwarder/aws). +## Alternative setup using [AWS S3 receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awss3receiver) + +The alternative setup allows you to bypass the "Serverless only" limitation. + +### Prerequisites + +- An S3 bucket for storing the logs +- A load balancer configured to export access logs to the S3 bucket. Check the [official AWS docs](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/enable-access-logging.html) on how to export access logs +- An SQS queue receiving notifications on object creation in the S3 bucket +- `awss3receiver` and `awslogsencodingextension` + +### Configuration example + +For details on configuration refer to the following documentation: [awss3receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awss3receiver#aws-s3-receiver), [awslogsencodingextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/encoding/awslogsencodingextension#aws-logs-encoding-extension), [elasticsearchexporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter#configuration-options) + +```yaml +extensions: + awslogs_encoding/elbaccess: + format: elbaccess + +receivers: + awss3: + sqs: + queue_url: "" + region: "" + s3downloader: + region: "" + s3_bucket: '' + s3_prefix: 'AWSLogs/' + encodings: + - extension: awslogs_encoding/elbaccess + +exporters: + debug: + verbosity: detailed + elasticsearch/otel: + endpoints: https://: + user: elastic + password: + mapping: + mode: otel + metrics_dynamic_index: + enabled: true + +service: + extensions: [awslogs_encoding/elbaccess] + pipelines: + logs: + exporters: [debug, elasticsearch/otel] + receivers: [awss3] +``` + ## Logs reference ### AWS ELB access logs diff --git a/packages/aws_elb_otel/manifest.yml b/packages/aws_elb_otel/manifest.yml index b4b15d6be24..08b09fc13af 100644 --- a/packages/aws_elb_otel/manifest.yml +++ b/packages/aws_elb_otel/manifest.yml @@ -1,7 +1,7 @@ format_version: 3.5.0 name: aws_elb_otel title: "AWS ELB OpenTelemetry Assets" -version: 0.1.0 +version: 0.1.1 source: license: "Elastic-2.0" description: "AWS ELB logs for OpenTelemetry Collector" diff --git a/packages/aws_vpcflow_otel/changelog.yml b/packages/aws_vpcflow_otel/changelog.yml index e23f954bb3f..d4896e56806 100644 --- a/packages/aws_vpcflow_otel/changelog.yml +++ b/packages/aws_vpcflow_otel/changelog.yml @@ -1,4 +1,9 @@ # newer versions go on top +- version: "0.1.1" + changes: + - description: Add "Alternative setup using awss3receiver" section to README + type: enhancement + link: https://github.com/elastic/integrations/pull/15844 - version: "0.1.0" changes: - description: Initial draft of the AWS VPC Flow Logs OpenTelemetry Assets Package diff --git a/packages/aws_vpcflow_otel/docs/README.md b/packages/aws_vpcflow_otel/docs/README.md index bc2d5a042cb..bd8af769ff2 100644 --- a/packages/aws_vpcflow_otel/docs/README.md +++ b/packages/aws_vpcflow_otel/docs/README.md @@ -21,6 +21,61 @@ From the AWS side, to collect VPC Flow logs, you need: For step-by-step instructions on how to set up an EDOT Cloud Forwarder for AWS, see the [EDOT Cloud Forwarder for AWS](https://www.elastic.co/docs/reference/opentelemetry/edot-cloud-forwarder/aws) guide. +## Alternative setup using [AWS S3 receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awss3receiver) + +The alternative setup allows you to bypass the "Serverless only" limitation. + +### Prerequisites + +- A Virtual Private Cloud (VPC) +- An S3 bucket for storing flow logs +- A flow log configured with the S3 bucket as the destination +- An SQS queue receiving notifications on object creation in the S3 bucket +- `awss3receiver` and `awslogsencodingextension` + +### Configuration example + +For details on configuration refer to the following documentation: [awss3receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awss3receiver#aws-s3-receiver), [awslogsencodingextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/encoding/awslogsencodingextension#aws-logs-encoding-extension), [elasticsearchexporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter#configuration-options) + +```yaml +extensions: + awslogs_encoding/vpcflow: + format: vpcflow + vpcflow: + file_format: plain-text + +receivers: + awss3: + sqs: + queue_url: "" + region: "" + s3downloader: + region: "" + s3_bucket: '' + s3_prefix: 'AWSLogs/' + encodings: + - extension: awslogs_encoding/vpcflow + +exporters: + debug: + verbosity: detailed + elasticsearch/otel: + endpoints: https://: + user: elastic + password: + mapping: + mode: otel + metrics_dynamic_index: + enabled: true + +service: + extensions: [awslogs_encoding/vpcflow] + pipelines: + logs: + exporters: [debug, elasticsearch/otel] + receivers: [awss3] +``` + ## Logs Reference For a complete list of all available logs and their detailed descriptions, refer to the [OpenTelemetry AWS Logs encoding extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/encoding/awslogsencodingextension#vpc-flow-log-record-fields) diff --git a/packages/aws_vpcflow_otel/manifest.yml b/packages/aws_vpcflow_otel/manifest.yml index c4fa0f6279c..cd6bbf52eb3 100644 --- a/packages/aws_vpcflow_otel/manifest.yml +++ b/packages/aws_vpcflow_otel/manifest.yml @@ -1,7 +1,7 @@ format_version: 3.5.0 name: aws_vpcflow_otel title: "AWS VPC Flow Logs OpenTelemetry Assets" -version: 0.1.0 +version: 0.1.1 source: license: "Elastic-2.0" description: "AWS VPC Flow Logs OpenTelemetry Assets"