Posted over 4 years ago. Visible to the public.

Elasticsearch unassigned replica shards on single node instances

If you have a single node elasticsearch instance and indices with replicas enabled your cluster state will be yellow. If you have replica shards they should be moved to a different node for high availability purposes. With a single node this can't be accomplished. So you either build a ES cluster or you disable the replicas.

Building a cluster is beyond the scope of this card, but we can have a look how we can disable the replicas.

get the unassigned shards

curl -s -XGET 'localhost:9200/_cat/shards?pretty' | grep UNASSIGNED

If it's a replica (r in the column befor the UNASSIGNED) and you are on a single node elasticsearch you need to disable replicas

get templates with patterns and search which template will be used for the unassigned shard

curl -XGET 'localhost:9200/_template?pretty' \
    | jq 'to_entries[] | { key, index_pattern: .value.index_patterns}'

how to change template configuration

via puppet

You can manage templates with our puppet module. This template will be applied on all indices that start with foobar_. You only need to specify the api_port if your instance runs on a different port than 9200.

makandra::elasticsearch::templates:
  'instance_foo':
    api_port: 9200
    content:
      'index_patterns': ['foobar_*']
      'settings':
        'number_of_shards': 1
        'number_of_replicas': 0

Attention: This will replace all existing settings of the template. Make sure to include all default template settings by first checking:

curl -XGET 'localhost:9200/_template/instance_foo?pretty"

manual

Overwrite the template settings:

curl -XPUT 'localhost:9200/_template/footemplate?pretty' \
    -H 'Content-Type: application/json'                  \
    -d '{
        "index_patterns":
            [ "foobar.*(" ], "settings": {
                    "index":
                    { "number_of_shards": 1, "number_of_replicas": 0 }
            }
        }'

Cleanup

Now you have fixed the issue for future indices matching the index_pattern of the edited template. But you need to remove the unassigend replica shards already there. Use the index names returned by the first HTTP request and disable replicas for the indices:

curl -XPUT 'localhost:9200/$INDEXNAME/_settings?pretty' \
    -H 'Content-Type: application/json'                 \
    -d '{ "index": { "number_of_replicas": 0 } }'

xpack monitoring index template

If the troubeling template is the xpack monitoring template .monitoring-es or any other monitoring template automaticall created (for e.g. .monitoring-kibana) you can't change the settings because xpack will overwrite them. You need to create a separate template which will be applied after the first one: Configuring Indices for Monitoring Archive

Avatar
Claus-Theodor Riegg
Last edit
About 1 year ago
Moritz Kraus
About this deck
We are makandra and do test-driven, agile Ruby on Rails software development.