Setting up Data source
This guide provides sample scrape job to be used for Prometheus deployment’s scrape-configmap and service monitors to be used with Prometheus operator for the different architectural topologies for integrating Prometheus (connecting a data source link) with Chaos center.
Topologies
Listed below are three among many topologies in which a data source can be setup for collecting chaos delegate cluster's metrics along with chaos metrics for chaos center.
For Control plane chaos delegate / Self chaos delegate with Prometheus - scraping chaos-exporter metrics and events along with other metrics
Scrape jobs:
- Chaos exporter
- job_name: 'chaos-exporter'
static_configs:
- targets: ['chaos-exporter.litmus.svc.cluster.local:8080']
relabel_configs:
- target_label: instance
replacement: 'chaos-exporter-service'
- Kube state metrics exporter
- job_name: 'kube-state-metrics'
static_configs:
- targets: ['kube-state-metrics.monitoring.svc.cluster.local:8080']
Detailed setup guide for Prometheus deployment with scrape configuration
Service monitors:
- Chaos exporter
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: chaos-exporter
labels:
k8s-app: chaos-exporter
namespace: litmus
spec:
jobLabel: app
selector:
matchLabels:
app: chaos-exporter
namespaceSelector:
matchNames:
- litmus
endpoints:
- port: tcp
interval: 1s
metricRelabelings:
- targetLabel: instance
replacement: 'chaos-exporter-service'
- Kube state metrics exporter
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/name: kube-state-metrics
app.kubernetes.io/version: latest
k8s-app: kube-state-metrics
name: kube-state-metrics
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
honorLabels: true
interval: 30s
port: http-metrics
relabelings:
- action: labeldrop
regex: (pod|service|endpoint|namespace)
scheme: http
scrapeTimeout: 30s
tlsConfig:
insecureSkipVerify: true
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 30s
port: telemetry
scheme: http
tlsConfig:
insecureSkipVerify: true
jobLabel: app.kubernetes.io/name
selector:
matchLabels:
app.kubernetes.io/name: kube-state-metrics
Detailed setup guide for Prometheus operator with service monitors
Note:
Scrape jobs as per (above links) can also be added under this additionalScrapeConfigs for the Prometheus community’s Kube prometheus stack
Pod labels to be used with Prometheus operator installed via helm as part of the Prometheus community’s Kube prometheus stack configured for podMonitors or with Prometheus’s inbuilt Kubernetes service discovery
Chaos exporter
labels
labels:
app: chaos-exporter
release: prometheus-stack
pod monitor
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: chaos-exporter-monitor
namespace: monitoring
labels:
release: prometheus-stack
spec:
selector:
matchLabels:
app: chaos-exporter
namespaceSelector:
matchNames:
- litmus
podMetricsEndpoints:
- port: tcp
- interval: 1s
metricRelabelings:
- targetLabel: instance
replacement: 'chaos-exporter-service'
Black box exporter
labels
labels:
app: prometheus-blackbox-exporter
release: prometheus-stack
pod monitor
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: black-box-exporter-monitor
namespace: monitoring
labels:
release: prometheus-stack
spec:
selector:
matchLabels:
app: prometheus-blackbox-exporter
namespaceSelector:
matchNames:
- monitoring
podMetricsEndpoints:
- port: http
- interval: 1s
Detailed setup guide for Prometheus operator with pod monitors
For Multiple chaos delegate s with multiple prometheus instances - scraping chaos-exporter metrics and events along with other metrics
- Scrape job and service monitor remain same as in case of Control plane chaos delegate / self chaos delegate , the individual Prometheus instances can be connected as separate data sources to the Chaos center. Separate dashboards can be created by selecting specific chaos delegate s and their corresponding data source which is essentially a Prometheus time series database, collecting metrics from the chaos delegate cluster for application / infra metrics, chaos events and chaos verdicts.
For Multiple chaos delegate s with single prometheus - scraping chaos-exporter metrics and events along with other metrics
- Separate dashboards can be created by selecting specific chaos delegate s and the data source which is essentially a Prometheus time series database, collecting metrics from the chaos delegate cluster for application / infra metrics, chaos events and chaos verdicts.
Scrape jobs:
- Chaos-exporter - chaos delegate -1
- job_name: 'chaos-exporter-agent-1'
static_configs:
- targets: ['<AGENT_1_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']
relabel_configs:
- target_label: instance
replacement: 'chaos-exporter-service'
- Chaos-exporter - chaos delegate -2
- job_name: 'chaos-exporter-agent-2'
static_configs:
- targets: ['<AGENT_2_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']
relabel_configs:
- target_label: instance
replacement: 'chaos-exporter-service'
- Kube state metrics exporter - chaos delegate -1
- job_name: 'kube-state-metrics-agent-1'
static_configs:
- targets: ['<AGENT_1_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']
- Kube state metrics exporter - chaos delegate -2
- job_name: 'kube-state-metrics-agent-2'
static_configs:
- targets: ['<AGENT_2_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']
Detailed setup guide for Prometheus deployment with scrape configuration
Service endpoint, spec and monitors:
- Chaos-exporter - chaos delegate -1
kind: Service
apiVersion: v1
metadata:
name: chaos-exporter-agent-1
namespace: monitoring
spec:
type: ClusterIP
ports:
- name: tcp
port: 8080
targetPort: 8080
---
kind: Endpoints
apiVersion: v1
metadata:
name: chaos-exporter-agent-1
namespace: monitoring
subsets:
- addresses:
- ip: <AGENT_1_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>
ports:
- name: tcp
port: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: chaos-exporter-agent-1
name: chaos-exporter-agent-1
namespace: monitoring
spec:
jobLabel: app
selector:
matchLabels:
app: chaos-exporter-agent-1
namespaceSelector:
matchNames:
- monitoring
endpoints:
- interval: 1s
port: tcp
metricRelabelings:
- targetLabel: instance
replacement: 'chaos-exporter-service'
- Chaos-exporter - chaos delegate -2
kind: Service
apiVersion: v1
metadata:
name: chaos-exporter-agent-2
namespace: monitoring
spec:
type: ClusterIP
ports:
- name: tcp
port: 8080
targetPort: 8080
---
kind: Endpoints
apiVersion: v1
metadata:
name: chaos-exporter-agent-2
namespace: monitoring
subsets:
- addresses:
- ip: <AGENT_2_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>
ports:
- name: tcp
port: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: chaos-exporter-agent-2
name: chaos-exporter-agent-2
namespace: monitoring
spec:
jobLabel: app
selector:
matchLabels:
app: chaos-exporter-agent-2
namespaceSelector:
matchNames:
- monitoring
endpoints:
- interval: 1s
port: tcp
metricRelabelings:
- targetLabel: instance
replacement: 'chaos-exporter-service'
- Kube state metrics exporter - chaos delegate -1
kind: Service
apiVersion: v1
metadata:
name: kube-state-metrics-agent-1
namespace: monitoring
spec:
type: ClusterIP
ports:
- name: tcp
port: 8080
targetPort: 8080
---
kind: Endpoints
apiVersion: v1
metadata:
name: kube-state-metrics-agent-1
namespace: monitoring
subsets:
- addresses:
- ip: <AGENT_1_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>
ports:
- name: tcp
port: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: kube-state-metrics-agent-1
name: kube-state-metrics-agent-1
namespace: monitoring
spec:
jobLabel: app
selector:
matchLabels:
app: kube-state-metrics-agent-1
namespaceSelector:
matchNames:
- monitoring
endpoints:
- interval: 30s
port: tcp
- Kube state metrics exporter - chaos delegate -2
kind: Service
apiVersion: v1
metadata:
name: kube-state-metrics-agent-2
namespace: monitoring
spec:
type: ClusterIP
ports:
- name: tcp
port: 8080
targetPort: 8080
---
kind: Endpoints
apiVersion: v1
metadata:
name: kube-state-metrics-agent-2
namespace: monitoring
subsets:
- addresses:
- ip: <AGENT_2_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>
ports:
- name: tcp
port: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: kube-state-metrics-agent-2
name: kube-state-metrics-agent-2
namespace: monitoring
spec:
jobLabel: app
selector:
matchLabels:
app: kube-state-metrics-agent-2
namespaceSelector:
matchNames:
- monitoring
endpoints:
- interval: 30s
port: tcp
Detailed setup guide for Prometheus operator with service monitors