In Spring Boot Elasticsearch Repository, I am using the PaginationAndSorting Spring capability (Pagination), which works great for fields that are 'date', but causes an error when I try to sort by a 'text' field. Is this behavior expected or is there something I can do to fix it so that I can sort by all field types.
LogstashLogRepository.java:
public interface LogstashLogRepository extends ElasticsearchRepository<LogstashLog, String> {
/**
* Repository's findByLevel does a query for all LogstashLogs with given level.
* @param logstashLevel level to match, choices are SECURITY and ACTIVITY.
* @return LogstashLogs with matching level.
*/
@Query("{\"bool\": {\"must\": [{\"match\": {\"level\": \"?0\"}}]}}")
Page<LogstashLog> findByLevel(LogstashLog.LogstashLevel logstashLevel, Pageable pageable);
}
LogstashLogController.java:
@RestController
@RequestMapping(value = "/")
public class LogstashLogController {
/**
* Object mapper.
*/
@Autowired
private ObjectMapper objectMapper;
/**
* Spring Boot ES Repository.
*/
@Autowired
private LogstashLogRepository logstashLogRepository;
/**
* Retrieves all Logstash logs by level.
*
* @param levelRequest level requested.
* @return ResponseEntity.
*/
@Operation(summary = "Retrieve all Logstash logs specified by a level.")
@GetMapping("/level")
public ResponseEntity<List<LogstashLog>> findLogstashLogsByLevel(@RequestBody final LogstashLog.LogstashLevel levelRequest,
@PageableDefault(size = 20) final Pageable pageable) {
Page<LogstashLog> pagedLogstashLogs = logstashLogRepository.findByLevel(levelRequest, pageable);
List<LogstashLog> logstashLogs = pagedLogstashLogs.stream().toList();
return new ResponseEntity<>(logstashLogs, HttpStatus.OK);
}
}
LogstashLog.java (model/dto):
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
@Document(indexName = "logstash-log")
public class LogstashLog {
@Id
private String id;
@JsonAlias("@timestamp")
@Field(name = "@timestamp")
private Instant timestamp;
@JsonAlias("Class")
@JsonProperty("class")
@Field(name = "class")
private String loggingClass;
@JsonAlias("Level")
private LogstashLevel level;
@JsonAlias("Date")
private Instant date;
@JsonAlias("Data")
private LogstashLogData data;
public enum LogstashLevel {
SECURITY,
ACTIVITY
}
}
mapping for the metadata in elastic search (I use an elk stack, so this data is put in via logstash)
"logstash-log": {
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"class": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"data": {
"properties": {
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"user": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"date": {
"type": "date"
},
"event": {
"properties": {
"original": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"level": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
If I use timestamp (or date), I can sort just fine, and get back something like this:
[
{
"id": "aafC544C-o-vp9msel90",
"timestamp": "2024-04-16T16:34:25.916453227Z",
"level": "SECURITY",
"date": "2024-04-16T16:34:25.914062Z",
"data": {
"user": "John Smith",
"message": "Title Security Chicken"
},
"class": "com.example.ExampleController"
},
{
"id": "ZqfC544C-o-vp9msel9c",
"timestamp": "2024-04-16T16:34:25.908138479Z",
...
But if I try to sort by class (or id, or level, etc.), I get: Request processing failed: org.springframework.data.elasticsearch.UncategorizedElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed] with root cause","exception":"co.elastic.clients.elasticsearch._types.ElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed ....
I have modified LogstashRepository to not return anything with Pagination and instead tried to do it on the results. But since I use a @Query annotation instead of a NativeSearchQuery, I'm afraid that there's not much I can do to the results since they're not recognized as a type of Query. Do I have to switch to using Query itself instead of the annotation so I can use ES's sort instead of Pagination's sort?