According to Elasticsearch's official guide, there are no hard limits on shard size, but experience shows that shards between 10GB and 50GB typically work well for logs and time series data.

  1. Install elasticsearch-exporter

2. Add a Prometheus rule

- alert: ElasticsearchShardTooLarge
expr: sum by(index, cluster, instance) (elasticsearch_indices_store_size_bytes_primary)/count by (index, cluster, instance)(elasticsearch_indices_shards_docs)/1024/1024/1024 >50
for: 5m
labels:
severity: warning
service: EFK
frequency: daily
annotations:
summary: Elasticsearch Single Shard > 50G
action: Edit template setting - number_of_shards

3. Setup alertmanager.yml, and fire this alert once a day

route:
- match:
frequency: daily
service: EFK
group_by: [cluster, instance]
receiver: efk-receiver
active_time_intervals:
- morning
repeat_interval: 50m
time_intervals:
- name: morning
time_intervals:
- times:
- start_time: 00:00 # 8-9 AM in GMT+8 timezone
end_time: 01:00

--

--

[Type A] Manually move a shard to another data node

POST _cluster/reroute
{
“commands” : [
{
“move” : {
“index” : “logstash-default-gigacim-2022.01.17”, “shard” : 9,
“from_node” : “f12glog29_d3”, “to_node” : “f12glog21_d2”
}
}
]
}

[Type B] Move only today’s shards away from a data node

Check (sort by…

--

--

Error message in a Kubernetes pod:

[Faraday::ConnectionFailed] SSL_connect SYSCALL returned=5 errno=0 state=SSLv3/TLS write client hello (OpenSSL::SSL::SSLError)

Possible Solution:

1. Add AuthorizationPolicy to allow traffic

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: allow-elasticsearch
spec:
action: ALLOW
rules:
- from:
- source:
ipBlocks:
- 172.0.0.0/8
to:
- operation:
ports: [“9200”]

2. Check if…

--

--

Jasmine H

Jasmine H

Data Engineer from Taiwan, recently working on EFK and Kubernetes projects.