Monitoring

CloudOps for Kubernetes can deploy Prometheus. Prometheus is a monitoring tool and a time-series database for various metrics.

Enable Prometheus

To enable Prometheus on your cluster:

1. Download the Docker Compose configuration file for your cluster.

   tip

   If you followed the recommended process for bootstrapping your cluster, the file name is docker-compose.override.yml.

2. Add the following settings to your Docker Compose configuration file.

   # enables the Prometheus Helm chart
   TF_VAR_enable_prometheus: "true"
   

3. Update your CloudOps for Kubernetes cluster.

Accessing Prometheus

After you deploy the Prometheus server, you can access it by completing the following steps:

1. Run the following command to get the pod name of the Prometheus server:

   export POD_NAME=$(kubectl get pods \
     --namespace prometheus \
     -l "app=prometheus,component=server" \
     -o jsonpath="{.items[0].metadata.name}")
   

2. Run the following command to port-forward the Prometheus server to port 9090 on your local machine:

   kubectl --namespace prometheus port-forward $POD_NAME 9090
   

3. Go to localhost:9090 in your browser.

Example Queries

The following section includes a list of example Prometheus queries that are available to help you.

Total CPU cores used by a type of container

This query allows you to see the total number of CPU cores used by all instances of a container with the same name. For example, you would use the following query for cortex:

sum (
  rate (
    container_cpu_usage_seconds_total{container_name="cortex"}[5m]
  )
)

Total memory used by a type of container

This query allows you to see the total amount of memory in bytes used by all instances of a container with the same name. For example, you would use the following query for cortex:

sum (
  container_memory_usage_bytes{container_name="cortex"}
)

Number of replicas in a deployment

This query allows you to see the number of replicas for a given Kubernetes Deployment. For example, you would use the following query for ep-cortex-deployment:

kube_deployment_status_replicas_available{deployment="ep-cortex-deployment"}

Percentage of requested CPU cores in use

This query allows you to see the percentage of requested CPU cores that are actively being used by all instances of a container with the same name. For example, you would use the following query for cortex:

sum (
  rate (
    container_cpu_usage_seconds_total{container_name="cortex"}[5m]
  )
)
/
sum (
  kube_pod_container_resource_requests_cpu_cores{container="cortex"}
)
*
100

For example, if in a moment of time:

• There is one pod with one container named cortex
• It requests 2 CPU cores
• It is actively using 1 core
• Prometheus would show 50% utilization at this moment

In another example, in another moment of time:

• There are 2 pods with one container each named cortex
• They each request 2 CPU cores
• Both are actively using 0.1 of a core
• Prometheus would show 5% utilization at this moment

Percentage of requested memory in use

This query allows you to see the percentage of requested memory that is actively being used by all instances of a container with the same name. For example, you would use the following query for cortex:

sum (
  container_memory_usage_bytes{container_name="cortex"}
)
/
sum  (
  kube_pod_container_resource_requests_memory_bytes{container="cortex"}
)
*
100