0

In Spring Boot Elasticsearch Repository, I am using the PaginationAndSorting Spring capability (Pagination), which works great for fields that are 'date', but causes an error when I try to sort by a 'text' field. Is this behavior expected or is there something I can do to fix it so that I can sort by all field types.

LogstashLogRepository.java:

public interface LogstashLogRepository extends ElasticsearchRepository<LogstashLog, String> {
   
    /**
     * Repository's findByLevel does a query for all LogstashLogs with given level.
     * @param logstashLevel level to match, choices are SECURITY and ACTIVITY.
     * @return LogstashLogs with matching level.
     */
    @Query("{\"bool\": {\"must\": [{\"match\": {\"level\": \"?0\"}}]}}")
    Page<LogstashLog> findByLevel(LogstashLog.LogstashLevel logstashLevel, Pageable pageable);
}

LogstashLogController.java:

@RestController
@RequestMapping(value = "/")
public class LogstashLogController {

    /**
     * Object mapper.
     */
    @Autowired
    private ObjectMapper objectMapper;

    /**
     * Spring Boot ES Repository.
     */
    @Autowired
    private LogstashLogRepository logstashLogRepository;

   

    /**
     * Retrieves all Logstash logs by level.
     *
     * @param levelRequest level requested.
     * @return ResponseEntity.
     */
    @Operation(summary = "Retrieve all Logstash logs specified by a level.")
    @GetMapping("/level")
    public ResponseEntity<List<LogstashLog>> findLogstashLogsByLevel(@RequestBody final LogstashLog.LogstashLevel levelRequest,
                                                               @PageableDefault(size = 20) final Pageable pageable) {
        Page<LogstashLog> pagedLogstashLogs = logstashLogRepository.findByLevel(levelRequest, pageable);
        List<LogstashLog> logstashLogs = pagedLogstashLogs.stream().toList();
        return new ResponseEntity<>(logstashLogs, HttpStatus.OK);
    }
}

LogstashLog.java (model/dto):

@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
@Document(indexName = "logstash-log")
public class LogstashLog {

    @Id
    private String id;

    @JsonAlias("@timestamp")
    @Field(name = "@timestamp")
    private Instant timestamp;

    @JsonAlias("Class")
    @JsonProperty("class")
    @Field(name = "class")
    private String loggingClass;

    @JsonAlias("Level")
    private LogstashLevel level;

    @JsonAlias("Date")
    private Instant date;

    @JsonAlias("Data")
    private LogstashLogData data;

    public enum LogstashLevel {
        SECURITY,
        ACTIVITY
    }
}

mapping for the metadata in elastic search (I use an elk stack, so this data is put in via logstash)


"logstash-log": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "class": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "data": {
          "properties": {
            "message": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "user": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "date": {
          "type": "date"
        },
        "event": {
          "properties": {
            "original": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "level": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }

If I use timestamp (or date), I can sort just fine, and get back something like this:

[
    {
        "id": "aafC544C-o-vp9msel90",
        "timestamp": "2024-04-16T16:34:25.916453227Z",
        "level": "SECURITY",
        "date": "2024-04-16T16:34:25.914062Z",
        "data": {
            "user": "John Smith",
            "message": "Title Security Chicken"
        },
        "class": "com.example.ExampleController"
    },
    {
        "id": "ZqfC544C-o-vp9msel9c",
        "timestamp": "2024-04-16T16:34:25.908138479Z",
...

But if I try to sort by class (or id, or level, etc.), I get: Request processing failed: org.springframework.data.elasticsearch.UncategorizedElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed] with root cause","exception":"co.elastic.clients.elasticsearch._types.ElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed ....

I have modified LogstashRepository to not return anything with Pagination and instead tried to do it on the results. But since I use a @Query annotation instead of a NativeSearchQuery, I'm afraid that there's not much I can do to the results since they're not recognized as a type of Query. Do I have to switch to using Query itself instead of the annotation so I can use ES's sort instead of Pagination's sort?

1 Answer 1

0

The answer for this was stupidly simple. Elasticsearch can't sort on text unless you tell it to via fielddata=true (though they don't recommend this method) or change it to a type keyword instead of text -- both would require a fix to how logstash enters the data. Alternatively, the search itself could just say "*.keyword", so in my case "class.keyword" will correctly do the Sort.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.