Grafana agent operator

The Grafana Agent Operator is a Kubernetes operator that makes it easier to deploy Grafana Agent and collect telemetry data from your Pods

Grafana agent overview:

Grafana Agent Operator works by watching for Kubernetes custom resources that specify how to collect telemetry data from your Kubernetes cluster and where to send it. Agent Operator manages corresponding Grafana Agent deployments in your cluster by watching for changes against the custom resources.
Grafana Agent Operator works in two phases — it discovers a hierarchy of custom resources and it reconciles that hierarchy into a Grafana Agent deployment.

The grafana agent operator architecture

The full hierarchy of custom resources is as follows:

GrafanaAgent:

MetricsInstance:

• PodMonitor
• Probe
• ServiceMonitor

LogsInstance:

• PodLogs

Grafana agent operator architecture

This table describes these custom resources:

How I manipulate grafana agent operator in the real life

And hereafter, I will talk about my experience in implementing Grafana agent operator in real world. This section will include:

• How to deploy Grafana operator
• How to use Grafana operator to create Grafana agent
• After implementing this model, what will we have

Grafana agent operator

I use Argocd to deploy helm chart. I will talk about the implementation of Argocd and how to organize the structure of folder to manage tools and eviroment clearer and easier. In my Argocd folder structure, you just create a helm chart like this:

After that, You can access to Argocd to check the result:

Grafana operator pod will query k8s api to get crd and filter by labels

Grafana agent custom resource

This is my Grafana agent crs structure:

Note:

• You should setup Externel Secret to sync Secret from AWS Secret Manager, then use it to intergrate with Grafana Loki, Mimir
• Log-pod incude PodLogs cr files, point to Pods to get logs
• Monitor-pod incude PodMonitor cr files, point to Pods to get metrics
• Monitor-service incude ServiceMonitor cr files, point to service/endpoint to get metrics

Example: monitor-service/cadvisor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cadvisor
  labels:
    instance: primary
spec:
  namespaceSelector:
    matchNames:
      - grafana-agent
  selector:
    matchLabels:
      app.kubernetes.io/name: kubelet
  endpoints:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      interval: 60s
      port: https-metrics
      path: /metrics/cadvisor
      scheme: https
      tlsConfig:
        insecureSkipVerify: false
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        serverName: kubernetes
      honorLabels: true
      relabelings:
        # Override job
        - action: replace
          targetLabel: job
          replacement: integrations/kubernetes/cadvisor
        # Override __address__
        - replacement: kubernetes.default.svc.cluster.local:443
          targetLabel: __address__
        # Override __metrics_path__
        - replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
          regex: (.+)
          sourceLabels:
            - __meta_kubernetes_endpoint_address_target_name
          targetLabel: __metrics_path__
        # Add this label to tolerate default kube-prometheus rules
        - replacement: "/metrics/cadvisor"
          targetLabel: metrics_path
      metricRelabelings:
        - action: keep
          regex: container_network_transmit_packets_dropped_total|container_memory_......
          sourceLabels:
            - __name__

Grafana agent match with service which has labels: instance: primary in grafana-agent namespace. Then it point to endpoints to action some relabelings and metricRelabelings before write metrics to time series DB.

How to integrate Grafana agent operator with another helm chart

Nowadays, Almost helm charts defined service monitor in helm repository. It’s easy for you to enable service monitor like this.
Ex: kube-state-helm-chart

After that, Grafana agent operator will deploy a ServiceMonitor resource to your k8s cluster.
It is simple, right!

Grafana agent operator under the hood: Actually what Grafana agent operator really does is reconcile all custom resource type: PodLogs, PodMonitor, ServiceMonitor and generates a final config file of Grafana agent.