How to configure "search.max_open_scroll_context" in elasticsearch?

Question

I an using scroll to get data in elasticsearch (version: 7.0.0). However, when I used it, the exception was thrown.

Request:

GET /index-name/_search?scroll=1m
{
    "size": 100,
    "query": {
        "match_all" : {}
    }
}

Response

{
  "error": {
    "root_cause": [
      {
        "type": "exception",
        "reason": "Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "nr-v1",
        "node": "jVqXndodRtqsZ4Srh9eHSg",
        "reason": {
          "type": "exception",
          "reason": "Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."
        }
      }
    ]
  },
  "status": 500
}

And I check the node status using GET /_nodes/stats/indices/search, the response:

{
  "_nodes": {
    "total": 3,
    "successful": 3,
    "failed": 0
  },
  "cluster_name": "bprc",
  "nodes": {
    "AdT9tX_jRqiuRyNnYunRdg": {
      "timestamp": 1557036722528,
      "name": "es2",
      "transport_address": "192.168.99.74:9300",
      "host": "192.168.99.74",
      "ip": "192.168.99.74:9300",
      "roles": [
        "master",
        "data",
        "ingest"
      ],
      "attributes": {
        "ml.machine_memory": "16819453952",
        "ml.max_open_jobs": "20",
        "xpack.installed": "true"
      },
      "indices": {
        "search": {
          "open_contexts": 502,
          "query_total": 3171,
          "query_time_in_millis": 2490,
          "query_current": 0,
          "fetch_total": 538,
          "fetch_time_in_millis": 951,
          "fetch_current": 0,
          "scroll_total": 2020,
          "scroll_time_in_millis": 1948250008,
          "scroll_current": 502,
          "suggest_total": 0,
          "suggest_time_in_millis": 0,
          "suggest_current": 0
        }
      }
    },
......

My question: 1. How to fix the problem about the exception that "Trying to create too many scroll contexts. Must be less than or equal to: [500]."

          "scroll_total": 2020,
          "scroll_time_in_millis": 1948250008,
          "scroll_current": 502

what is the function of the three attributes?

@NishantSaini Yes, I use multi search to execute several term query or match_phrase query. Is there any question? my code can work well on elasticsearch6。 — huangjs
– huangjs, Commented May 5, 2019 at 8:18
You need to make sure to clear each scroll when it's done. and if you're using multi search, do not send more than 500 queries at a time — Val
– Val, Commented May 5, 2019 at 8:23
I have cleared scroll id after each query. However, the problem still exists. — huangjs
– huangjs, Commented May 5, 2019 at 12:07

Swoogan · Accepted Answer · 2019-08-21 19:06:37Z

21

I was able to fix this issue by increasing the maximum number of scroll contexts:

curl -x "" -X PUT localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d'{
    "persistent" : {
        "search.max_open_scroll_context": 1024
    },
    "transient": {
        "search.max_open_scroll_context": 1024
    }
}'

answered Aug 21, 2019 at 19:06

Swoogan

5,6686 gold badges39 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

zhuguowei · Accepted Answer · 2019-09-14 08:31:14Z

6

I have met the same problem, my solution is clear the scroll explicitly after every use

    ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
    clearScrollRequest.addScrollId(scrollId);
    restHighLevelClient.clearScroll(clearScrollRequest, RequestOptions.DEFAULT);

Please see https://www.elastic.co/guide/en/elasticsearch/client/java-rest/master/java-rest-high-clear-scroll.html

and do not need to change the max_open_scroll_context

answered Sep 14, 2019 at 8:31

zhuguowei

8,54520 gold badges83 silver badges115 bronze badges

1 Comment

Vicky Over a year ago

I dint have permission to change the cluster settings, clearing the scroll context solved my problem, thank you for the answer.

Collectives™ on Stack Overflow

How to configure "search.max_open_scroll_context" in elasticsearch?

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related