Python Streaming pipeline on Google Dataflow missing some Element metrics

Question

I am running a Python dataflow (beam 2.5) and was wondering why the step metrics are not showing after a beam.Map(lambda x: x.decode('utf-8')).

Everything is working fine, but the number of elements vanish.

As you can see, the numbers from this point on are zero.

Is there something that I am missing?

Note an answer, but is there a reason to use beam 2.5 instead of 2.9? — John Hanley
– John Hanley, Commented Feb 8, 2019 at 22:25

Pablo · Accepted Answer · 2019-03-06 22:42:59Z

1

Some features are not fully supported, and that's the case for some monitoring metrics. These will gradually be added as support is improved.

answered Mar 6, 2019 at 22:42

Pablo

11.1k1 gold badge49 silver badges72 bronze badges

Sign up to request clarification or add additional context in comments.

1 Answer 1