I'm using Spark 2.3.1.
I have a SparkSQL dataframe like this
|bigrams |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|[a control, control circuit, circuit utilizes, utilizes patient, patient information, information and, and treatment-platform, treatment-platform information, information to, to optimize, optimize a, a radiation-treatment, radiation-treatment plan, plan by, by permitting, permitting isocenters, isocenters of, of various, various radiation-treatment, radiation-treatment fields, fields as, as comprise, comprise parts, parts of, of a, a same, same treatment, treatment plan, plan to, to not, not be, be coincidental, coincidental with, with one, one another, another to, to thereby, thereby yield, yield an, an optimized, optimized treatment, treatment plan., plan. the, the patient, patient information, information can, can pertain, pertain to, to one, one or, or more, more physical, physical aspects, aspects of, of the, the patient, patient as, as desired., desired. by, by one, one approach,, approach, the, the foregoing, foregoing can, can comprise, comprise scattering, scattering the, the isocenters, isocenters of, of the, the various, various radiation-treatment, radiation-treatment fields, fields around, around a, a predetermined, predetermined point, point (such, (such as,, as, for, for example,, example, the, the center, center of, of the, the treatment, treatment volume, volume and/or, and/or some, some or, or all, all of, of the, the beams)., beams). this, this approach, approach can, can comprise, comprise causing, causing an, an area, area of, of highest, highest energy, energy flux, flux for, for a, a given, given field, field to, to be, be non-coincident, non-coincident for, for at, at least, least some, some of, of the, the radiation-treatment, radiation-treatment fields, fields as, as are, are specified, specified by, by the, the radiation-treatment, radiation-treatment plan.]|
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I want to split this dataframe row's record by (,) and save each record in new row like this
|bigrams |
+--------------------+
|a control |
|control circuit |
|circuit utilizes |
....
+--------------------+
