spark-sql产生小文件合并

数据结构算法 water 5年前 (2019-04-24) 6156℃ 0评论

hdfs dfs -getmerge /data/search/gome/suggestdata/part-* /tmp/part-00000

hdfs dfs -moveFromLocal /tmp/part-00000 /data/search/gome/suggestdata/part-00000

spark.sql("SET hive.merge.sparkfiles = true")
 spark.sql("SET hive.merge.mapredfiles = true")
 spark.sql("SET hive.merge.mapfiles = true")
 spark.sql("set hive.merge.smallfiles.avgsize = 128000000")
 spark.sql("set hive.merge.size.per.task = 128000000")

转载请注明：学时网 » spark-sql产生小文件合并