This code sample demonstrates how to natively export Oracle Cloud Infrastructure (OCI) Observability data over to Splunk via an HTTP Event Collector.
See Splunk HTTP Event Collector REST API to get a sense of how Splunk HEC works. If you’re new to OCI Functions, I'd recommend running through the Quick Start guide on OCI Functions before proceeding.
Here is the basic flow-of-control:
- All OCI Services emit audit, metric and logging events which are ingested by the Monitoring and Logging services respectively.
- The Service Connector receives these events, invoking a Function to handle them as they arrive.
- The Function transforms the events into Splunk format and post to Splunk.
Let's survey the OCI Services involved.
The Logging Service is a highly scalable and fully managed single pane of glass for all the logs in your tenancy. Logging provides access to log data from Oracle Cloud Infrastructure resources. These logs include critical diagnostic information that describes how resources are performing and being accessed.
The Monitoring Service receives timestamp-value pairs (aka metric data points) which also carry contextual dimensions and metadata about the services or applications that emitted them.
The Auditing Service provides visibility into activities related to your OCI resources and tenancy. Audit log events can be used for security audits, to track usage of and changes to Oracle Cloud Infrastructure resources, and to help ensure compliance with standards or regulations.
The Service Connector Hub is designed to handle Audit, Logging and Monitoring event streams on-demand and at scale. See Service Connector Hub documentation for details.
We need to deploy a small amount of logic that transforms log and raw metric events into a format that Splunk can accept and then sends the transformed events in batch to the Splunk event collection endpoint.
OCI Functions is a natural fit for the task. OCI Functions is based on the Fn Project, an open source, container native, serverless platform that can be run anywhere - any cloud or on-premises. Fn Project is easy to use, extensible, and performant.
Let's compare the OCI and Splunk message payload formats, understand the mapping task, and see what the resulting transformed messages look like.
Here is an example OCI Metrics event:
{
"namespace": "oci_vcn",
"resourceGroup": null,
"compartmentId": "ocid1.compartment.oc1...",
"name": "VnicFromNetworkBytes",
"dimensions": {
"resourceId": "ocid1.vnic.oc1.phx..."
},
"metadata": {
"displayName": "Bytes from Network",
"unit": "bytes"
},
"datapoints": [
{
"timestamp": 1652196912000,
"value": 5780.0,
"count": 1
}
]
}
Here is an example OCI Logging event:
{
"datetime": 1689108090000,
"logContent": {
"data": {
"action": "ACCEPT",
"bytesOut": 4132,
"destinationAddress": "147.154.1.1",
"destinationPort": 443,
"endTime": 1689108091,
"flowid": "75d72a74",
"packets": 9,
"protocol": 6,
"protocolName": "TCP",
"sourceAddress": "10.0.0.1",
"sourcePort": 60830,
"startTime": 1689108090,
"targetSystem": "SPLUNK",
"status": "OK",
"version": "2"
},
"id": "159f15e5",
"oracle": {
"compartmentid": "ocid1.compartment.oc1...",
"ingestedtime": "2023-07-11T20:42:24.573Z",
"loggroupid": "ocid1.loggroup.oc1.phx...",
"logid": "ocid1.log.oc1.phx...",
"tenantid": "ocid1.tenancy.oc1...",
"vniccompartmentocid": "ocid1.compartment.oc1...",
"vnicocid": "ocid1.vnic.oc1.phx...",
"vnicsubnetocid": "ocid1.subnet.oc1.phx..."
},
"source": "-",
"specversion": "1.0",
"time": "2023-07-11T20:41:30.000Z",
"type": "com.oraclecloud.vcn.flowlogs.DataEvent"
}
},
By default, the Function will transform OCI events as needed
to instruct the Splunk platform to Extract JSON Fields from the events as described here.
There are several formats supported by Splunk, See Format events for HTTP Event Collector to get a sense of all the options.
Here is the Extract JSON Fields logic:
{
"sourcetype": "_json",
"event": {
<single OCI event record goes here in its entirety>
}
}
Here is an example of a transformed metric event using this logic:
{
"sourcetype": "_json",
"event": {
"namespace": "oci_vcn",
"resourceGroup": null,
"compartmentId": "ocid1.compartment.oc1...",
"name": "VnicFromNetworkBytes",
"dimensions": {
"resourceId": "ocid1.vnic.oc1.phx..."
},
"metadata": {
"displayName": "Bytes from Network",
"unit": "bytes"
},
"datapoints": [
{
"timestamp": 1652196912000,
"value": 5780.0,
"count": 1
}
]
}
}
There also is an optional feature called a Payload Map. Use it if you need greater control over the transformation.
In this mode, the Function uses a JSON map where the l-values are the keys that Splunk expects to see and the r-values in the map are used
as keys to look up values in the OCI event payload. If the Function finds a match anywhere in the
event (included nested within the JSON), the event value will be assigned to the Splunk-expected key and
included it in the transformed output.
Here is the default payload JSON:
{
"fields": {
"name": "name",
"namespace": "namespace",
"timestamp": "timestamp",
"value": "value",
"count": "count",
"type": "type",
"source": "source",
"displayName": "displayName",
"compartmentid": "compartmentid",
"ingestedtime": "ingestedtime",
"sourceAddress": "sourceAddress",
"destinationAddress": "destinationAddress",
"tenantid": "tenantid"
}
}
Here is the resulting output of a Log Event using the default payload JSON:
{
"fields": {
"type": "com.oraclecloud.vcn.flowlogs.DataEvent",
"source": "-",
"compartmentid": "ocid1.compartment.oc1...",
"ingestedtime": "2023-07-11T20:44:06.225Z",
"sourceAddress": "129.146.14.168",
"destinationAddress": "10.0.0.219",
"tenantid": "ocid1.tenancy.oc1..."
}
}
Here is the resulting output of a Raw Metric Event using the default payload JSON:
{
"fields": {
"name": "VnicFromNetworkMirrorBytes",
"namespace": "oci_vcn",
"timestamp": 1652196492000,
"value": 0.0,
"count": 1,
"displayName": "Mirrored Bytes from Network"
}
}
The default map is really just a placeholder. To customize the behavior, create your own JSON and pass it to your Function by setting it as a configuration parameter (see Function Configuration below).
Note that the l-values and r-values in the default payload JSON above are identical. That causes the OCI key=value
pairs to be passed as-is. If you need to pass a different key for Splunk, just change the l-value like so:
{
"fields": {
"splunk_key_1": "name",
"splunk_key_2": "namespace",
"splunk_key_3": "timestamp",
"splunk_key_4": "value",
"splunk_key_5": "count"
"splunk_key_6": "displayName"
}
}
Here is the resulting output of a Raw Metric Event using this JSON:
{
"fields": {
"splunk_key_1": "VnicFromNetworkMirrorBytes",
"splunk_key_2": "oci_vcn",
"splunk_key_3": 1652196492000,
"splunk_key_4": 0.0,
"splunk_key_5": 1,
"splunk_key_6": "Mirrored Bytes from Network"
}
}
Refer to the Splunk HEC examples page as a guide.
Here are the steps to set up OCI.
Create a compartment to contain the following:
- Virtual Cloud Network
- Application + Function
- Service Connector
Let's assume you create a compartment with name ABC.
Create a user group where we can assign developer related policies.
Let's assume you create a user group with name functions-developers.
See common policies.
Here is an example IAM policy structure:
Allow users in functions-developers to create, deploy and manage Functions and Applications at the tenancy level.
Allow group functions-developers to manage repos in tenancy
Allow group functions-developers to manage serviceconnectors in tenancy
Allow group functions-developers to manage metrics in tenancy
Allow group functions-developers to manage logging-family in tenancy
Allow group functions-developers to use cloud-shell in tenancy
Allow group functions-developers to use virtual-network-family in tenancy
Allow Functions Service to use OCIR Repositories at the tenancy level:
Allow service faas to use repos in tenancy
Allow Service Connector to use Functions, Metrics and Logs at the compartment level:
Allow service service-connector-hub to use functions-family in compartment ABC
Allow service service-connector-hub to use metrics in compartment ABC
Allow service service-connector-hub to use logging-family in compartment ABC
Follow the steps found in the Quick Start guide on OCI Functions. The Cloud Shell instructions are recommended.
Once you have your Function Application build, configure the Function by setting up the following Configuration:
| Environment Variable | Default | Purpose |
|---|---|---|
| SPLUNK_HEC_ENDPOINT | not-configured | REST API endpoint for reaching Splunk HTTP Event Collector. SPLUNK_HEC_ENDPOINT trial account example: https://{your-subdomain}.splunkcloud.com:8088/services/collector |
| SPLUNK_HEC_TOKEN | not-configured | API token token obtained from Splunk |
| SEND_TO_SPLUNK | True | Determines whether messages are forwarded to Splunk |
| VERIFY_SSL | True | This should be enabled in production |
| LOGGING_LEVEL | INFO | Controls function logging outputs. Choices: INFO, WARN, CRITICAL, ERROR, DEBUG |
| BATCH_SIZE | 100 | Number of records posted to Splunk per call |
| USE_PAYLOAD_MAP | False | If False, send entire record to Splunk as _json. If True, use the Payload Map. |
| CUSTOM_PAYLOAD_MAP | Stringified JSON that is your custom JSON payload map |
Now let’s set up a simple service connector instance that takes Monitoring sources and passes them to our Function.
Because your Function requires a VCN, you can use that VCN as the metric source to test against. Let's test
with the oci_vcn Monitoring namespace because it will quickly generate a lot of useful events.
Select Monitoring as the source and the Function as the target. Configure your source as the
compartment where the VCN resides and select the Monitoring namespace (oci_vcn) that you want to
pick up. Select your Application and the Function within it as the target.
This troubleshooting guide can also be very helpful.
Enable Function Invocation Service logs to see the logs generated by the Function.
It's a good idea to set LOGGING_LEVEL = DEBUG as a general rule while first deploying the Function.
- Increase
BATCH_SIZEto a larger value - Set
LOGGING_LEVEL = INFO - Increase Function timeout and / or memory allocation
When you have the Service Connector configured, metrics should appear in Splunk after a few minutes.
You now have a low-maintenance, serverless function that can send raw metrics over to Splunk in near-real time. I encourage you to experiment with the payload mappings to get the most value out of this integration.
For more information, see the following resources:
LiveLabs is the place to explore Oracle's products and services using workshops designed to enhance your experience building and deploying applications on the Cloud and On-Premises. ur library of workshops cover everything from how to provision the world's first autonomous database to setting up a webserver on our world class OCI Generation 2 infrastructure, machine learning and much more. Use your existing Oracle Cloud account, a Free Tier account or a LiveLabs Cloud Account to build, test, and deploy applications on Oracle's Cloud.
Visit LiveLabs now to get started. Workshops are added weekly, please visit frequently for new content.
Copyright (c) 2023, Oracle and/or its affiliates. All rights reserved. Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.