If you have well defined criteria, you could try something like precomputing an index over the keys you are filtering over. Initially it will take the same time as doing a single filter operation but after will be instant.
const MONTHS = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"];
const bench = (tag, fn) => {
const now = performance.now();
fn();
console.log(`${tag} executed in ${performance.now() - now}ms`);
};
// this is just to create a ton of elements.
const data = Array.from({ length: 1000000 }, (_, i) => {
return { id: i, month: MONTHS[Math.random() * 12 | 0]};
});
const index = new Map();
for (datum of data) {
const bucket = index.get(datum.month);
if (bucket == null) {
index.set(datum.month, [datum]);
} else {
bucket.push(datum);
}
}
bench('filter [Dec, Nov]', () => {
['Dec', 'Nov'].reduce((r, f) => r.concat(index.get(f)), []);
});
bench('filter [Nov, Feb, May]', () => {
['Nov', 'Feb', 'May'].reduce((r, f) => r.concat(index.get(f)), []);
});
bench('filter [Apr, Dec, Feb, Jan, Jul]', () => {
['Apr', 'Dec', 'Feb', 'Jan', 'Jul'].reduce((r, f) => r.concat(index.get(f)), []);
});
On my PC/Browser im filtering 1M records in roughly 0.3ms - 2ms depending on the complexity of the filters. That's within a frame(60fps).
If you want, you could create a function that generates an index for you that you can use to filter on.
const createIndexOn = (date, selector) => {
const index = new Map();
for (datum of data) {
const bucket = index.get(datum.month);
if (bucket == null) {
index.set(datum.month, [datum]);
} else {
bucket.push(datum);
}
}
return index;
};
const monthIndex = createIndexOn(data, (d) => d.month);
If you need to do something like compound filters where you have say a month and a name, you could create an index on month, filter the results by month, create a index based on the results over name and filter by name. You could also create 2 discrete indexes and just take an intersection. Either way its pretty fast.
This all is predicated that you profiled the code and know that looping over 6000*|Filters| is slow(its probably fine honestly until you hit say 100k). The problem could be else where. For example if you are using react and you are filtering every time a component changes, you should memo the results or use other performance tuning tools.
