So I my index growth too fast and now has 60 million docs in 3 shards (single node).
I want to buy more machines and split content into more shards. How can I do this?
It's just connect new nodes to the cluster and update shards number in master?
Add a comment
|
2 Answers
Afaik elasticsearch cannot yet redistribute indexed documents automatically (see here). You would have to reindex all content. The problem behind it is, that documents are distributed to shards according to a hash value modulo number of shards. Just adding shards and keeping indexing would keep adding documents to the old shards too.
Elasticsearch allows to distribute documents according to a custom function (routing parameter). You could distribute all new content to the new shards, but this makes deletions difficult, because now you have to know if a document is "old" or "new". Further it ruins your uniform index statistics which may bias ranking in nonobvious ways.
Bottom line: adding shards to an existing index requires reindexing all contents or some heavy hacking.
Comments
You already have 3 shards, so if you add 2 nodes Elasticsearch will automatically reallocate 2 shards to the other 2 nodes, giving all shard 3 times more power.
If you want to add more shards, you need to reindex your data. This can be done by creating a new index with the desired number of shards and copying your data to that index (see https://www.elastic.co/guide/en/elasticsearch/guide/current/reindex.html)
