Quantcast
Channel: How can I change column types in Spark SQL's DataFrame? - Stack Overflow
Viewing all articles
Browse latest Browse all 24

Answer by Ravi for How can I change column types in Spark SQL's DataFrame?

$
0
0

In case if you want to change multiple columns of a specific type to another without specifying individual column names

/* Get names of all columns that you want to change type. In this example I want to change all columns of type Array to String*/    val arrColsNames = originalDataFrame.schema.fields.filter(f => f.dataType.isInstanceOf[ArrayType]).map(_.name)//iterate columns you want to change type and cast to the required typeval updatedDataFrame = arrColsNames.foldLeft(originalDataFrame){(tempDF, colName) => tempDF.withColumn(colName, tempDF.col(colName).cast(DataTypes.StringType))}//displayupdatedDataFrame.show(truncate = false)

Viewing all articles
Browse latest Browse all 24

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>