Can I run the Google Monitoring Agent inside the Kubernetes Pod?

Question

It seems that the Google Monitoring Agent (powered by Stackdriver) should be installed on each Node (i.e. each instance of the calculation, i.e. each machine) of the Kubernet cluster.

However, new plugins such as Nginx , Redis , ElasticSearch ..., these agents must know the IP addresses of these services. This means that kube-proxy is up and running, which should mean running the Google Monitoring agent on the Pod.

These two conflicts: on the one hand, the agent controls the entire machine, and on the other, it controls the services running on one or more machines.

Can these Stackdriver plugins work in the Google Container Engine (GKE) / Kubernetes cluster?

Answer 1

To control each machine (memory, processor, disk ...), you can install an agent on each node (i.e., on each instance of computing your GKE cluster). Please note that it will not work with automatic scaling in the sense that the newly created nodes will not be installed by the agent.

To track services (number of requests / s, connection to the client ...), you can install the agent plugin in another container, so that, for example, Nginx Pod starts two containers:

• Nginx
• Google Monitoring Agent with Nginx Plugin

Note. Not fully tested.

Answer 2

You can install the StackDriver agent in your Docker file.

I managed to get this working for the couchdb container as follows:

 FROM klaemo/couchdb RUN apt-get update RUN apt-get install curl lsb-release -y RUN curl -O https://repo.stackdriver.com/stack-install.sh RUN apt-get install libyajl2 -y COPY couchdb.conf /opt/stackdriver/collectd/etc/collectd.d/couchdb.conf CMD bash stack-install.sh --write-gcm && service stackdriver-agent restart && couchdb 

Answer 3

I tried using the Stackdriver container in the container to collect statistics about Nginx / Uwsgi in the same container. I had some conclusions that may not be as useful. Just for reference.

To create a stackdriver image, you can reference the docker file created by Keto. https://hub.docker.com/r/keto/stackdriver/~/dockerfile/

 FROM centos:centos7 MAINTAINER Mikael Keto # add stackdriver repository RUN curl -o /etc/yum.repos.d/stackdriver.repo https://repo.stackdriver.com/stackdriver-el7.repo # install stackdriver RUN yum -y install initscripts stackdriver-agent && yum clean all RUN mkdir -p /var/lock/subsys; exit 0 ADD run.sh /run.sh RUN chmod 755 /run.sh CMD ["/run.sh"] 

Run.sh looks like below

 #!/usr/bin/env bash /opt/stackdriver/stack-config --write-gcm --no-start /etc/init.d/stackdriver-agent start while true; do sleep 60 agent_pid=$(cat /var/run/stackdriver-agent.pid 2>/dev/null) ps -p $agent_pid > /dev/null 2>&1 if [ $? != 0 ]; then echo "Stackdriver agent pid not found!" break; fi done 

In the GKE / K8S deployment yaml file,

 apiVersion: extensions/v1beta1 kind: Deployment ... - name: stackdriver-agent image: gcr.io/<project_id>/stackdriver-agent:<your_version> command: ['/run.sh'] 

In my test, I found

• It will report statistics based on [hostname] instead of [content_name].
• It will collect a lot of system statistics that make sense for the node, but since it is in the container, this is completely pointless.

Well, I hope to find some way to collect both the statistics of the boxes and the nodes that I need, but I have not found an easy way to do this. I did this using the Google Python API library, but it takes too much time.

Answer 4

There is another way to use Dockerfile. When creating a docker image, first install the necessary libraries for installing the agent stack.

 FROM mongo RUN apt-get update && apt-get install -y curl lsb-release # COPY credential COPY gcloud-credential.json /etc/google/auth/application_default_credentials.json ENV GOOGLE_APPLICATION_CREDENTIALS "/etc/google/auth/application_default_credentials.json" # download Stackdriver Agent installer RUN curl -O https://repo.stackdriver.com/stack-install.sh RUN chmod +x /stack-install.sh # COPY stackdriver mongodb plugin COPY mongodb.conf /opt/stackdriver/collectd/etc/collectd.d/mongodb.conf 

Then install the agent using the POD life cycle.

 spec: containers: - image: your_mongo_image name: my-mongo ports: - containerPort: 27017 lifecycle: postStart: exec: command: ["/stack-install.sh", "--write-gcm"]