Delete Data from Elasticsearch Maxed out index

Question

I have an single Elasticsearch index which has maximum number of documents. i want to delete some data from this index. i have tried running a delete_by_query but it fails. any sugestions for this, which i would able to keep the index and its data and also delete some old data from the same index ?

This is the request i was running.

curl -X GET "1localhost:9200/produdtion-index/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
                    "bool": {
                        "must": [
                            { "range": { "submitDate": { "lte": "20200421" } } }   
                        ]
                    }
                }
}
'

What error do you get? Something like "maximum documents per shard reached"? — Val
– Val, Commented Jul 24, 2020 at 13:11
If you're missing storage space, it's very hard to get around those issues. You should plan your storage space according to your needs before indexing your data. Now, you should increase your storage space if you want to go forward — Val
– Val, Commented Jul 24, 2020 at 13:44
Deleting documents doesn't reclaim the space immediately, that just marks the document deleted in the underlying segment files, so that wouldn't solve your issue in the short term. What you can try to do is to forcemerge the index, but if you're missing space, it might not work out as the process will require additional space to rewrite the segments without the deleted documents. — Val
– Val, Commented Jul 24, 2020 at 13:50
it just means that the forcemerge call ran successfully, not much more. Do you see some reclaimed space? — Val
– Val, Commented Jul 27, 2020 at 15:47
As I said, calling forcemerge is bandaid, you need to provision enough disk space prior to indexing data. Now your best option is to increase disk space (storage is cheap) — Val
– Val, Commented Jul 27, 2020 at 16:08

Nu-ONE · Accepted Answer · 2020-07-28 18:09:34Z

1

Try index life cycle managementhttps://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html You can split the index based on the number of documents and then delete them

answered Jul 28, 2020 at 18:09

Nu-ONE

6991 gold badge5 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Rajitha Fernando Over a year ago

Thank You mate. not the solution for my problem but it helped me to find the re-index api.

Collectives™ on Stack Overflow

Delete Data from Elasticsearch Maxed out index

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related