How to convert wrappedArray column in spark dataset to java array?

Question

In my project I got a column after applying collect_list aggregate function, which is of format wrappedArray , I wanted to know is there anyway to convert that wrappedArray to normal array or list So that I can iterate over that column and extract values row by row in that column?

Language:- JAVA Using Apache spark library

By this I am getting that column which is of wrappedArray type

dataset.groupBy( "department","salary","tag")
.agg(collect_list(col("id")));

dataset.foreach(row ->
      row.getAs("id)")

);

I am confused how should I proceed further to iterate in that column.

vinsce · Accepted Answer · 2022-11-04 20:43:50Z

1

If you have a WrappedArray there are several ways to convert it to a Java collection. You can leverage scala.collection.JavaConverters helper methods:

WrappedArray<Long> wrappedArray = row.getAs("ids");
List<Long> list = scala.collection.JavaConverters.seqAsJavaList(wrappedArray.seq());

Alternatively, you can use the Row.getList method to obtain directly a Java List:

List<Long> list = row.getList(row.fieldIndex("ids"));

answered Nov 4, 2022 at 20:43

vinsce

1,3581 gold badge10 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Procrastinator · Accepted Answer · 2023-06-14 06:45:19Z

0

To convert scala wrappedArray column to Java list in java spark. here line is of type Row.

List<Long> list = scala.collection.JavaConverters.seqAsJavaList(line.getAs("fieldName")));

edited Jun 14, 2023 at 6:45

Procrastinator

2,67445 gold badges31 silver badges37 bronze badges

answered Jun 10, 2023 at 1:59

Rakesh

514 bronze badges

Collectives™ on Stack Overflow

How to convert wrappedArray column in spark dataset to java array?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related