how to match an array value by it's key in a key value pair elasticsearch array?

Question

I have an array of key value pairs. Is it possible to exact match value of key & then do a check on it's value's range value?

Example: In below doc oracle_props is an array with name, value pairs. I need to check if it has "oracle_cursors" key and then check if it's value is less than 1000.

GET /eg/message/_percolate
{
   "doc": {
      "client": {
         "name": "Athena",
         "version": 1,
         "db": {
            "@type": "Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 64bit",
            "oracle_props": [
               {
                  "@name": "open_cursors",
                  "@value": 4000
               },
               {
                  "@name": "USER_ROLE_PRIVS_COUNT",
                  "@value": 1
               },
               {
                  "@name": "CREATE_PERMISSION",
                  "@value": "Y"
               }
            ]
         }
      }
   }
}

Below is my percolator.

I also need to check the following so that it gives back 3 as my result

"client.name" must be "Athena"
"client.db.@type" must be "Oracle" then only go ahead and check below points
"client.db.oracle_props.@name" field is not found
check if it has "oracle_cursors" key and then check if it's value is < 1000

1 & 2 are and operations and any of 3 or 4 satisfies it should result 3. I need help with point 4, below is my query. Also please suggest if there is a better way.

PUT /eg/.percolator/3
{
   "query": {
      "filtered": {
         "filter": {
            "or": [
               {
                  "missing": {
                     "field": "client.db.oracle_props.@name"
                  }
               }
            ]
         },
         "query": {
            "bool": {
               "must": [
                  {
                     "match": {
                        "client.name": "Athena"
                     }
                  },
                  {
                     "match": {
                        "client.db.@type": "Oracle"
                     }
                  }
               ]
            }
         }
      }
   }
}

Update

Can I have something like below

{
     "match": {
                    "client.db.oracle_props[name='open_cursors'].value": 4000
                 }
              }

More tries

I followed elasticsearch nested query and changed the mapping to nestedtype by re-indexing. Can anyone find problem why am i getting nested: NullPointerException;?

PUT /eg/.percolator/3
{
   "nested" : {
        "path" : "client.db.oracle_props",
        "score_mode" : "avg",
        "query" : {
            "bool" : {
                "must" : [
                    {
                        "match" : {"client.db.oracle_props.@name" : "open_cursors"}
                    },
                    {
                        "range" : {"client.db.oracle_props.@value" : {"lt" : 4000}}
                    }
                ]
            }
        }
    }
}

mapping change

...
"properties": {
               "@type": {
                  "type": "string"
               },
               "oracle_props": {
                   "type" : "nested",
                  "properties": {
                     "@name": {
                        "type": "string"
                     },
                     "@value": {
                        "type": "long"
                     }
                  }
               }
            }
...

Thorsten · Accepted Answer · 2014-07-15 16:02:17Z

5

+50

Let's get into it:

You seem to map your nested path wrong, oracle_props is a child item of db in your example document, but not in your mapping, where it appears directly as child of your root.
You are mapping oracle_props.@value as long, but assign a text Y to it at the CREATE_PERMISSION nested doc
You query for range lt 4000, which excludes 4000, lte would fit for you

I didn't get your requirement for the missing value, hence I skipped that.

To get you to the right path, I has to simplify it a bit (since I couldn't follow all the mess in your question, sorry)

I'm not going into percolation either, and renamed everything to twitter/tweet, since this was easier for me to copy from my examples.

1) Create empty index "twitter"

curl -XDELETE 'http://localhost:9200/twitter/'
curl -XPUT 'http://localhost:9200/twitter/'

2) create geo_point mapping for the actual "tweet"

curl -XPUT 'http://localhost:9200/twitter/tweet/_mapping' -d '
{
    "tweet": {
        "properties": {
            "db": {
                "type": "object",
                "properties": {
                    "@type": {
                        "type": "string"
                    },
                    "oracle_props": {
                        "type": "nested",
                        "properties": {
                            "@name": {
                                "type": "string"
                            },
                            "@value": {
                                "type": "string"
                            }
                        }
                    }
                }
            }
        }
    }
}'

3) Let's check if the mapping was set

curl -XGET 'http://localhost:9200/twitter/tweet/_mapping?pretty=true'

4) Post some tweets, with nested data

curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{
    "name": "Athena",
    "version": 1,
    "db": {
        "@type": "Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 64bit",
        "oracle_props": [
            {
                "@name": "open_cursors",
                "@value": 4000
            },
            {
                "@name": "USER_ROLE_PRIVS_COUNT",
                "@value": 1
            },
            {
                "@name": "CREATE_PERMISSION",
                "@value": "Y"
            }
        ]
    }
}'

5) Query nested only

curl -XGET localhost:9200/twitter/tweet/_search -d '{
    "query": {
        "nested" : {
            "path" : "db.oracle_props",
            "score_mode" : "avg",
            "query" : {
                "bool" : {
                    "must" : [
                        {
                            "term": {
                                "db.oracle_props.@name": "open_cursors"
                            }
                        },
                        {
                            "range": {
                                "db.oracle_props.@value": {
                                    "lte": 4000
                                }
                            }
                        }
                    ]
                }
            }
        }
    }
}';

6) Query "Athena" and "Oracle"

curl -XGET localhost:9200/twitter/tweet/_search -d '{
    "query" : {
        "bool" : {
            "must" : [
                {
                    "match" : {"tweet.name" : "Athena"}
                },
                {
                    "match" : {"tweet.db.@type" : "Oracle"}
                }
            ]
        }
    }
}'

7) Combine the former two queries

curl -XGET localhost:9200/twitter/tweet/_search -d '{
    "query" : {
        "bool" : {
            "must" : [
                {
                    "match" : {"tweet.name" : "Athena"}
                },
                {
                    "match" : {"tweet.db.@type" : "Oracle"}
                },
                {
                    "nested" : {
                        "path" : "db.oracle_props",
                        "score_mode" : "avg",
                        "query" : {
                            "bool" : {
                                "must" : [
                                    {
                                        "term": {
                                            "db.oracle_props.@name": "open_cursors"
                                        }
                                    },
                                    {
                                        "range": {
                                            "db.oracle_props.@value": {
                                                "lte": 4000
                                            }
                                        }
                                    }
                                ]
                            }
                        }
                    }
                }
            ]
        }
    }
}'

Results as

{
    "took": 2,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "failed": 0
    },
    "hits": {
        "total": 1,
        "max_score": 2.462332,
        "hits": [
            {
                "_index": "twitter",
                "_type": "tweet",
                "_id": "1",
                "_score": 2.462332,
                "_source": {
                    "name": "Athena",
                    "version": 1,
                    "db": {
                        "@type": "Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 64bit",
                        "oracle_props": [
                            {
                                "@name": "open_cursors",
                                "@value": 4000
                            },
                            {
                                "@name": "USER_ROLE_PRIVS_COUNT",
                                "@value": 1
                            },
                            {
                                "@name": "CREATE_PERMISSION",
                                "@value": "Y"
                            }
                        ]
                    }
                }
            }
        ]
    }
}

answered Jul 15, 2014 at 16:02

Thorsten

5,6446 gold badges37 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

AabinGunz Over a year ago

Thanks it worked. Also if I assign oracle_props.value mapping as string range will not work on long but string and I need to do a range match and value also contains string. Is there any way to ignore string in such case?

Thorsten Over a year ago

I'm not sure about that, sorry.

Andy Over a year ago

@AbhishekSimon - do a multi-type field. Make one of them a long, and one a String. This will make it index the proper tokens for string contains, and the long value compare. See, elasticsearch.org/guide/en/elasticsearch/reference/0.90/…

AabinGunz Over a year ago

@AndyPryor & Thorsten: Thanks for all the help.

AabinGunz Over a year ago

I'll put this in a new question and will explain the problem with your twitter example. Please check here

|

Andy · Accepted Answer · 2014-07-15 14:39:25Z

0

You need to use a Nested Document. See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-nested-type.html

answered Jul 15, 2014 at 14:39

Andy

8,9498 gold badges48 silver badges69 bronze badges

3 Comments

AabinGunz Over a year ago

Thanks I am able to solve my problem with Thorsten's solution

Andy Over a year ago

@AbhishekSimon Cool, yeah sorry didn't have time to do a full solution, but Thorsten's looks good. The gist of it is, without nested types it's all flat in the index.

AabinGunz Over a year ago

Can you please check a similar question here

Collectives™ on Stack Overflow

how to match an array value by it's key in a key value pair elasticsearch array?

2 Answers 2

7 Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

7 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related