Situation: When indexing new shards, ramdomly one or more elasticsearch data node servers remain high CPU usage for hours.

Solution: Move an active shard from the server to another data node server.

Steps:

GET _cat/shards?v&s=node,store:desc

Choose one with a bigger size and reroute it using:

POST _cluster/reroute
{
"commands" : [
{
"move" : {
"index" : "logstash-default-myapp-2022.01.17", "shard" : 9,
"from_node" : "mydatanode_1", "to_node" : "mydatanode_5"
}
}
]
}

To bypass the limitation: “Password has been used already. Choose another”,

Use this script below to continuously change the password 25 times automatically

change_pw.ps

#!/bin/sh
passwd <<EOF
myOriginalPassword
efc039ad5cA@
efc039ad5cA@
EOF
echo "------- change 1 --------"
passwd <<EOF
efc039ad5cA@
66ac8373cdA@
66ac8373cdA@
EOF
echo "------- change 2 --------"
passwd <<EOF
66ac8373cdA@
3fc5caae1eA@
3fc5caae1eA@
EOF
echo "------- change 3 --------"
passwd <<EOF
3fc5caae1eA@
69a93cc3caA@
69a93cc3caA@
EOF
echo "------- change 4 --------"…

Situation:

Possible Reason:

There was a primary with no replicas allocated to the node when it got disconnected from the master. When rejoining the cluster, the locally allocated shard copy on the node was not able to free previously used resources in the time. After 5 unsuccessful allocation attempts, the master gives up and needs manual triggering to give it another allocation attempt.

https://github.com/elastic/elasticsearch/issues/23199)

Solution:

[WAY 1] Retry Failed

POST /_cluster/reroute?retry_failed

[WAY2] Reroute/move the shard

POST /_cluster/reroute
{
“commands” : [
{
“move” : {
“index” : “indexname–2021.11.02”, “shard” : 0,
“from_node” : “elastic_data_nodeA”, “to_node” :
“elastic_data_nodeB”
}
}
]
}

Add file IgnoreSSLFactory.java to the same path of MainApplicaiton.java (default under /android/app/src/main/java/com/[yourapp]/

//IgnoreSSLFactory.java
package com.yourapp; //Change this
import com.facebook.react.modules.network.OkHttpClientFactory;
import com.facebook.react.modules.network.OkHttpClientFactory;
import com.facebook.react.modules.network.OkHttpClientProvider;
import com.facebook.react.modules.network.ReactCookieJarContainer;
import java.security.cert.CertificateException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeUnit;
import android.util.Log;
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLSession;
import javax.net.ssl.SSLSocketFactory;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager…

Situation:

Seeing upstream connect error or disconnect/reset before headers. reset reason connection failure message after k8s kibana deployment

Environment: K8s, with istio injection

Solution :

2. Add server.host: "0.0.0.0" in kibana.yml

data:
kibana.yml: |
server.host: "0.0.0.0"
server.port: 5601
......

Other possible reasons: https://stackoverflow.com/questions/63408608/upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connect

Use Packed centralized ELK image : sebp/elk

docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -v elk-data:/var/lib/elasticsearch -it --name elkstack sebp/elk

http://localhost:9200 — Elasticsearch

http://localhost:5601 — Kibana web

http://localhost:5044 — Logstash

Enter container : docker exec -it <container-name> /bin/bash

Edit Logstash config : /opt/logstash/bin/logstash -e…

Jasmine H

Data Engineer from Taiwan

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store