Situation:

Possible Reason:

There was a primary with no replicas allocated to the node when it got disconnected from the master. When rejoining the cluster, the locally allocated shard copy on the node was not able to free previously used resources in the time. After 5 unsuccessful allocation attempts, the master gives up and needs manual triggering to give it another allocation attempt.

https://github.com/elastic/elasticsearch/issues/23199)

Solution:

[WAY 1] Retry Failed

POST /_cluster/reroute?retry_failed

[WAY2] Reroute/move the shard

POST /_cluster/reroute
{
“commands” : [
{
“move” : {
“index” : “indexname–2021.11.02”, “shard” : 0,
“from_node” : “elastic_data_nodeA”, “to_node” :
“elastic_data_nodeB”
}
}
]
}

Add file IgnoreSSLFactory.java to the same path of MainApplicaiton.java (default under /android/app/src/main/java/com/[yourapp]/

//IgnoreSSLFactory.java
package com.yourapp; //Change this
import com.facebook.react.modules.network.OkHttpClientFactory;
import com.facebook.react.modules.network.OkHttpClientFactory;
import com.facebook.react.modules.network.OkHttpClientProvider;
import com.facebook.react.modules.network.ReactCookieJarContainer;
import java.security.cert.CertificateException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeUnit;
import android.util.Log;
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLSession;
import javax.net.ssl.SSLSocketFactory;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager…

Kibana上設定Ingest pipeline

在不改動到 Fluentd config 的情境下,透過 Kibana介面設定 Ingest Pipeline來做做欄位處理。等於是在資料進到 elasticsearch才進行處理,而不是在 log shipper 端處理,缺點是這樣做會加重 elasticsearch cluster 的 load …

Situation:

Seeing upstream connect error or disconnect/reset before headers. reset reason connection failure message after k8s kibana deployment

Environment: K8s, with istio injection

Solution :

2. Add server.host: "0.0.0.0" in kibana.yml

data:
kibana.yml: |
server.host: "0.0.0.0"
server.port: 5601
......

Other possible reasons: https://stackoverflow.com/questions/63408608/upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connect

Free Link

Replica settings can be modified initially OR after index created, different from shard settings which cannot be modified after index created.

Check current index template

GET /_template

Add a new template (e.g. named logstash_index_replica) for replica indexes. Change “order” and “number_of_replicas” as needed.

PUT /_template/logstash_index_replica 
{…

Jasmine H

Data Engineer from Taiwan

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store