GCE - No stack memory metrics for nodes

Question

I installed my Kubernetes 1.3.4 cluster on GCE using

export KUBE_ENABLE_CLUSTER_MONITORING=google

This works pretty well, I get application logs (for some reason in the "Container Engine" section, but good), as well as pod and node metrics.

The only thing missing is node memory metrics, only the processor is displayed (see screenshot)

No memory metrics

In heap logs, I see tons of lines like this

 { metadata: { severity: "ERROR" projectId: "<project-id>" serviceName: "container.googleapis.com" zone: "europe-west1-d" labels: { container.googleapis.com/cluster_name: "production" compute.googleapis.com/resource_type: "instance" compute.googleapis.com/resource_name: "fluentd-cloud-logging-production-minion-group-p0w8" container.googleapis.com/instance_id: "6772154497331326454" container.googleapis.com/pod_name: "heapster-v1.1.0-2102007506-23b3e" compute.googleapis.com/resource_id: "6772154497331326454" container.googleapis.com/stream: "stderr" container.googleapis.com/namespace_name: "kube-system" container.googleapis.com/container_name: "heapster" } timestamp: "2016-09-13T14:40:08.000Z" projectNumber: "930564692351" } textPayload: "E0913 14:40:08.665035 1 gcm.go:179] Error while sending request to GCM googleapi: Error 400: Timeseries 76, point: start is not older than end, for a cumulative metric, invalidParameter " insertId: "pt5bo7g132r266" log: "heapster" } 

Not sure if this is related.

Any ideas?

Answer 1

If you use your cluster using GCE instead of GKE, you must install the stacker agent and check the credentials that the agent uses to communicate with the stackdriver link

If you are using linux, you can install the agent by doing:

 curl -sSO https://dl.google.com/cloudagents/install-monitoring-agent.sh sudo bash install-monitoring-agent.sh 

and you can verify your credentials by running the following command:

 sudo cat $GOOGLE_APPLICATION_CREDENTIALS sudo cat /etc/google/auth/application_default_credentials.json