Dataflow Display Data missing from composite transform

Question

I'm trying the new display data functionality in Dataflow to make additional details show up in the Google Cloud Dataflow UI. However, the display data for custom PTransform's doesn't show up. In my Dataflow pipeline, I have a transform like:

Pipeline p = // ..
p.apply(new PTransform<PCollection<Integer>, PCollection<Integer>>() {
  @Override
  public PCollection<Integer> apply(PCollection<Integer> input) {
    return input
      .apply(/* .. */)
      .apply(/* .. */)
      .apply(/* .. */);
  }

  @Override
  public void populateDisplayData(DisplayData.Builder builder) {
    builder.add(DisplayData.item("foo", "bar"));
  }
});

When I run the Dataflow job, the UI doesn't seem to show the foo=bar display data.

Scott Wegner · Accepted Answer · 2016-07-26 18:26:43Z

Display data is not supported on composite transforms. Instead, you can attach display data to any user-defined function which your transform executes. For example, if one of the inner transforms in the above composite uses a ParDo, you could re-write the display data methods as:

Pipeline p = // ..
p.apply(new PTransform<PCollection<Integer>, PCollection<Integer>>() {
  @Override
  public PCollection<Integer> apply(PCollection<Integer> input) {
    return input
      .apply(/* .. */)
      .apply(/* .. */)
      .apply(ParDo.of(new DoFn<Integer, Integer>() {
        @Override
        public void processElement(ProcessContext c) { /* .. */ } 

        @Override
        public void populateDisplayData(DisplayData.Builder builder) {
          builder.add(DisplayData.item("foo", "bar"));
        }            
      }));
  }
});

There is a JIRA issue to extend display data functionality to composite transforms in Beam and Dataflow SDKs.

Collectives™ on Stack Overflow

Dataflow Display Data missing from composite transform

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related