Skip to main content

Logs

We use Splunk Cloud for log management at Gjensidige. Each AKS cluster has Splunk Connect For Kubernetes installed which is built on top of the FluentD data collector. For your logs to be picked up by the collector they have to be written to stdout or stderr ("the console"). File based logs are not supported.

Gjensidige's Splunk Cloud instance is available at gjensidige.splunkcloud.com 🔎

Required request headers

To increase traceability across Gjensidige's application stack, the following headers are required when sending requests:

  • X-RequestId: Unique ID for one single request
  • X-SessionId: Unique ID for a user session. A session might consist of multiple requests
  • X-ClientId: Unique ID for the client/application initiating the request, typically a frontend application

The value of these headers should be searchable in Splunk by adding them to your application logs.

Logging in JSON :bulb:

As a part of their best practices, Splunk are recommending to log in JSON. Logging in JSON greatly improves searchability and readability in Splunk. "Write Logs for Machines, use JSON" is recommended for some interesting reading about logging in JSON.

Hottips :fire:

An example of how to configure Spring Boot applications to log with JSON format can be found in the guide Preparing a Spring Boot App for Kubernetes.

Splunk Index​

Each team gets their own Splunk Index when onboarded to Splunk Cloud. The index name is created on the format team-<team-name>-<environment>. For example would Team Platform's index in Test environment be team-platform-test. All resources created in your team Kubernetes Namespace will automatically be sent to your team Splunk Index.

It's possible to override which index logs are sent to on a Deployment level, but this is generally not recommended and should only be used in special scenarios. To override Splunk Index for a Deployment, add the following annotation:

deployment.yaml
annotations:
  splunk.com/index: "your-splunk-index"

Splunk Sourcetype​

Sourcetype are auto generated for every container in your Deployment on the format kube:container:<container-name>. It's recommended to use the generated Sourcetype, but it can be manually configured by adding the following annotation to your Deployment:

deployment.yaml
annotations:
  splunk.com/sourcetype: "your-sourcetype"

Searching logs in Splunk Cloud​

Navigate to gjensidige.splunkcloud.com and select Apps -> Search & Reporting. Using your index and sourcetype, you can filter logs for your container in Splunk Cloud with the following search:

index="your-team-index" sourcetype="kube:container:your-container-name"

Gjensidige Security Logging Standards​

You must adhere to Gjensidige Security Logging Standards

You are responsible and accountable for ensuring that your application logs follow Gjensidige Security Logging Standards. We send logs to Splunk Cloud and this standard ensures that we don't violate our agreement with them as well as EU laws.

Personally Identifiable Information (PII)

Read Datatilsynet's definition writeup on Hva er en personopplysning?