Elasticsearch - provided multi-field value

Question

I have documents in my index with a duration field, represented in a period/duration format, e.g. P1DT3H4S.

Ideally I'd have Elasticsearch parse that into milliseconds in a multi-field. Apprently Elasticsearch cannot do that.

The next best thing is to convert the period into milliseconds myself and store it in a multi-field. Is this possible?

What I would like to end up with:

duration with a value like PT4M20S.

duration.millis with a corresponding value of 260000

it's possible to have ES compute it for you, but the result would not be stored in a multi-field, just in two separate fields. Would that be ok for you? — Val
– Val, Commented Jun 22, 2021 at 12:34
I could do that in code and store in a separate field. But how do you make Elasticsearch compute it? — siltalau
– siltalau, Commented Jun 22, 2021 at 12:41

Val · Accepted Answer · 2021-06-22 12:45:41Z

1

You can save yourself from that logic and have ES do it for you using an ingest pipeline with a script processor.

First you need to create an ingest pipeline that will take the duration string and parse it into milliseconds:

PUT _ingest/pipeline/duration2ms
{
  "processors": [
    {
      "script": {
        "source": """
            ctx.durationInMillis = Duration.parse(ctx.duration).getSeconds() * 1000;
          """
      }
    }
  ]
}

Then when you index your documents you can reference that pipeline, like this:

PUT index/_doc/1234?pipeline=duration2ms
{
  "duration": "P1DT3H4S"
}

And the document that will be indexed will look like this:

{
  "duration" : "P1DT3H4S",
  "durationInMillis" : 97204000
}

answered Jun 22, 2021 at 12:45

Val

218k14 gold badges377 silver badges384 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

siltalau Over a year ago

Could I in this case have an alias named 'duration.millis' that refers to durationInMillis?

Val Over a year ago

What difference does it make? You can name that field however you like

siltalau Over a year ago

Since duration is already a keyword, it cannot be an object. So I cannot simply store a document that has both a duration (keyword) field and a duration.millis (long) field

Val Over a year ago

Yes, and that's why you need two different fields, multi-fields won't work here. You could have an object field {"duration": {"keyword": "P1DT3H4S", "millis": 97204000}}

siltalau Over a year ago

Technically this works just fine. The problem I have is that the duration field is public and has to have the value as the ISO duration. Also, the users can query the data using lucene syntax. I would really have liked to have the users be able to refer to the millisecond value as duration.millis.

|

Collectives™ on Stack Overflow

Elasticsearch - provided multi-field value

1 Answer 1

8 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Related