Rotation of Google Container Engine Application Logs

If you are running a Google Container Engine (GKE) cluster and delivering your applications logs using a logging agent – filebeat for instance – to your centralized logging solution you may want to control how these log files on the nodes are rotated.

Messages that are written to stdout/stderr by applications are picked up by the docker engine and saved under /var/lib/docker/containers in json format. You can run filebeat as a DaemonSet to deliver these messages. See this note for instructions.

Running a ps command on the nodes provides information on how the docker engine handles log rotation:

1
2
3
4
5
6
$ ps ax -o command | grep log-opt
/usr/bin/docker daemon -s overlay --registry-mirror=https://mirror.gcr.io \
  --host=fd:// -p /var/run/docker.pid --iptables=false --ip-masq=false \
  --log-level=warn --bip=169.254.123.1/24 \
  --insecure-registry 10.0.0.0/8 \
  --log-driver=json-file --log-opt=max-size=10m --log-opt=max-file=5

10MB per max 5 files seems reasonable. However, depending on the disk size of the nodes and the velocity of the application logs you may want to adjust these parameters to meet your logging needs.

If you already have a running cluster there are two options to choose from in order to modify the log rotation parameters:

You can modify the instance template of the instance group that each cluster node is a member of, and recreate the cluster nodes manually. Running nodes will be garbage collected and the nodes replacing them will be created using the new template. The second option is to create a new cluster with the new log rotation parameters and migrate your applications over.

In either case, however, if you upgrade the cluster, the new log rotation parameters will be lost and you will have to perform this operation again.

First, go to the instance template of the instance group that is used by the cluster:

MIG-template

In this example, my-cluster consists of instances of the group gke-my-cluster-default-pool-cc447591-grp and the instance template for this group is gke-my-cluster-default-pool-cc447591.

Click the COPY link to create a copy of this template. On the next page titled Create an instance template scroll down to the Metadata section and expand the value of key kube-env:

MIG-newtemplate

Add:

DOCKER_LOG_MAX_SIZE: Xmb
DOCKER_LOG_MAX_FILE: Y

to the kube-env.

Consider the disk size of the cluster nodes, the number of applications running on them and the velocity of log messages from these applications before changing these parameters. Default values for maximum file size and maximum number of files are 10mb and 5, respectively. Give this new template a unique name and save it.

Next, we will update the instance template of the instance group to the new template we’ve just created:

1
2
3
$ gcloud compute instance-groups managed set-instance-template \
   instance_group_name \
   --template new_template_name

The instance group is now assigned the new template. Running nodes will need to be recreated for the new template, and the new log rotation parameters to be in effect.

Finally, get a list of nodes in the cluster with kubectl get nodes and run recreate-instances gcloud command to recreate some or all of the nodes at a time:

1
2
$ gcloud compute instance-groups managed recreate-instances instance_group_name \
  --instances=node1,node2,...

Tags: 

Related Notes: