Quantcast
Channel: How can I change column types in Spark SQL's DataFrame? - Stack Overflow
Viewing all articles
Browse latest Browse all 24

Answer by Vibha for How can I change column types in Spark SQL's DataFrame?

$
0
0

Another solution is as follows:

1) Keep "inferSchema" as False

2) While running 'Map' functions on the row, you can read 'asString' (row.getString...)

//Read CSV and create datasetDataset<Row> enginesDataSet = sparkSession            .read()            .format("com.databricks.spark.csv")            .option("header", "true")            .option("inferSchema","false")            .load(args[0]);JavaRDD<Box> vertices = enginesDataSet            .select("BOX","BOX_CD")            .toJavaRDD()            .map(new Function<Row, Box>() {                @Override                public Box call(Row row) throws Exception {                    return new Box((String)row.getString(0),(String)row.get(1));                }            });

Viewing all articles
Browse latest Browse all 24

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>