Prometheus Federation

Case 1:

我的 Prometheus佈好後,如何將 Infra level 的 metric 收到我的 Prometheus (前提為Infra有自己一套Prometheus)?透過 Prometheus Federation 機制,在我的 Prometheus 設定 job 去抓 Infra Prometheus 所收集的 metric (類似 DB replication 的概念)。

以下設定範例,可同時抓 infra metric (container_ 開頭. 用來看 container CPU, memory 等資訊), kubernetes metrics (kube_ 開頭. 用來看 pod restart, container OOM, PVC utilization 等資訊),也可抓 istio metric (看 service 之間的 traffic, error, latency 等資訊)。

prometheus.yml 中新增此段落:

...
# Get infra metrics (cpu, memory, disk), kubernetes metrics, and istio metrics from infra prometheus.
- job_name: 'federate'
metrics_path: '/federate'
honor_labels: true
params:
'match[]':
# AP infra metrics, kubernetes metrics
- '{namespace=~"(my-ns-a|my-ns-b).*"}'
# istio metrics
- '{destination_workload_namespace=~"(my-ns-a|my-ns-b).*"}'
static_configs:
- targets: ['prometheus-operated.infra-ns.svc.cluster.local:9090']

Case 2 :水平收

起一套 Prometheus (B) 去掃另外一台 Prometheus (A)有收的 metric, 並進行過濾:

Prometheus A (被收的那套) config:

...
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['10.xxx.xxx.xxx:9090']
- job_name: 'node-exporter'
static_configs:
- targets: ['10.xxx.xxx.xxx:9100']
- job_name: 'cAdvisor'
static_configs:
- targets: ['10.xxx.xxx.xxx:8080']

Prometheus A 本身收了 3 個 job: prometheus, node-exporter, cAdvisor

Prometheus B (要去收Prometheus A的那套) config: (過濾job跟name)

...
scrape_configs:
- job_name: 'federate'
metrics_path: '/federate'
params:
'match[]':
- '{job="cAdvisor", name="kind-control-plane"}'
# - '{xxx}' # 可以再指定其他過濾條件,跟上面的條件是 or 關係
static_configs:
- targets: ['10.xxx.xxx.xxx:9090']

Prometheus B 用 federate 串接 A 已經有收的 metric, 透過 match 來過濾要收的 metric. 此例子指定收 “job=cAdvisor” 且 “容器名稱為 kind-control-plane” 的 metric。

視覺話: 建另外一個 grafana dashboard 設 Prometheus B 為 data source, 可以看到只收了過濾後的cAdvisor metrics —name只有 kind-control-plan。

測試 federate 結果 in http get:

https://my-prometheus.app.myurl.com/federate?match[]={namespace%3D%22titot%22}

Another example: (用namespace過濾)

scrape_configs:
- job_name: 'federate'
metrics_path: '/federate'
params:
'match[]':
- '{namespace=~"(my-ns1|my-ns2)"}'
# - '{xxx}' # 可以再指定其他過濾條件,跟上面的條件是 or 關係
static_configs:
- targets: ['prometheus.app.myurl.com']

參考:

https://yunlzheng.gitbook.io/prometheus-book/part-ii-prometheus-jin-jie/readmd/scale-prometheus-with-federation

--

--

Data Engineer from Taiwan, recently working on EFK and Kubernetes projects.

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store