How to distinguish elements from offset time in sliding window?

Question

I am wondering whether it is possible to distinguish elements from offset time in Apache Flink sliding windows. Let me explain it in more details.

I want to take elements from eg.: 13:00 to 13:59:59. However, I also added an offset/slide so I have also elements from 12:30 so I can calculate things based on the values from previous hour, but I do not want to count them again (it was already counted during previous window). Is it possible out of the box to skip calculation of the elements from previous hour?

My stream definition:

var myStream= env.fromSource(source, watermarkStrategy, "data")
    .keyBy(MyType::getKey)
    .connect(otherStream)
    .process(new EnrichWithMyTypeProcessFunction())
    .assignTimestampsAndWatermarks(watermarkStrategyWithMyType)
    .keyBy(elem -> elem.getKey())
    .window(SlidingEventTimeWindows.of(Time.minutes(90), Time.minutes(60), Time.minutes(-30)))
    .trigger(EventTimeTrigger.create())
    .sideOutputLateData(lateDataTag)
    .process(new CalculateFromHourFunction());

Thanks for any hint!

kkrugler · Accepted Answer · 2024-12-02 14:22:01Z

1

I think you could do this using a 30 minute slide, and no .trigger(). In your CalculateFromHourFunction you'll see all of the elements from the 90 minute window (12:30 to 13:59:59.999, in your example), so you can do whatever calculation you need that informs the 1 hour window with values from the preceding 30 minutes.

This isn't the most efficient approach, since each window element would be collected and processed twice, once for the 30 minute pre-window, and a second time for the 1 hour actual window. You could do better with a custom KeyedProcessFunction, where you do your own windowing and use state to store the 30 minute pre-window result, but that would be a lot more complex.

answered Dec 2, 2024 at 14:22

kkrugler

9,5606 gold badges26 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to distinguish elements from offset time in sliding window?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related