DataFrame to Json Array in Spark

Question

I am writing Spark Application in Java which reads the HiveTable and store the output in HDFS as Json Format.

I read the hive table using HiveContext and it returns the DataFrame. Below is the code snippet.

 SparkConf conf = new SparkConf().setAppName("App");
 JavaSparkContext sc = new JavaSparkContext(conf);
 HiveContext hiveContext = new org.apache.spark.sql.hive.HiveContext(sc);

DataFrame data1= hiveContext.sql("select * from tableName")

Now I want to convert DataFrame to JsonArray. For Example, data1 data looks like below

|  A  |     B     |
-------------------
|  1  | test      |
|  2  | mytest    |

I need an output like below

[{1:"test"},{2:"mytest"}]

I tried using data1.schema.json() and it gives me the output like below, not an Array.

{1:"test"}
{2:"mytest"}

What is the right approach or function to convert the DataFrame to jsonArray without using any third Party libraries.

philantrovert · Accepted Answer · 2017-07-17 11:24:21Z

9

data1.schema.json will give you a JSON string containing the schema of the dataframe and not the actual data itself. You will get :

String = {"type":"struct",
          "fields":
                  [{"name":"A","type":"integer","nullable":false,"metadata":{}},
                  {"name":"B","type":"string","nullable":true,"metadata":{}}]}

To convert your dataframe to array of JSON, you need to use toJSON method of DataFrame:

val df = sc.parallelize(Array( (1, "test"), (2, "mytest") )).toDF("A", "B")
df.show()

+---+------+
|  A|     B|
+---+------+
|  1|  test|
|  2|mytest|
+---+------+

df.toJSON.collect.mkString("[", "," , "]" )
String = [{"A":1,"B":"test"},{"A":2,"B":"mytest"}]

answered Jul 17, 2017 at 11:24

philantrovert

10.1k3 gold badges43 silver badges65 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user2731629 Over a year ago

In Java, mkString not working. toString() wont get any arguments.

philantrovert Over a year ago

If you're using Java 8, you can use StringJoiner

Akiner Alkan · Accepted Answer · 2019-01-16 15:04:02Z

1

in JAVA you can do it with the following way:

String jsonToReturn = df.toJSON().collectAsList().toString();
reutrn jsonToReturn;

and return it as a response if it's a server-side.

edited Jan 16, 2019 at 15:04

Akiner Alkan

7,0684 gold badges42 silver badges80 bronze badges

answered Jan 16, 2019 at 13:40

user3652695

111 silver badge4 bronze badges

Collectives™ on Stack Overflow

DataFrame to Json Array in Spark

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related