use coalesce in place of repartition to avoid shuffle

Debasish Das · Debasish Das · commit 3c49d98d8016 · 2015-07-24T00:46:30.000-07:00
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/IndexedRowMatrix.scala b/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/IndexedRowMatrix.scala
@@ -276,7 +276,7 @@ class IndexedRowMatrix(
       value.map { sim =>
         MatrixEntry(i, sim._1, sim._2)
       }
-    }.repartition(defaultParallelism)
+    }.coalesce(defaultParallelism)
 
     blockedBig.unpersist()
 

Original file line number	Diff line number	Diff line change
`@@ -276,7 +276,7 @@ class IndexedRowMatrix(`
`276`	`276`	`value.map { sim =>`
`277`	`277`	`MatrixEntry(i, sim._1, sim._2)`
`278`	`278`	`}`
`279`		`- }.repartition(defaultParallelism)`
	`279`	`+ }.coalesce(defaultParallelism)`
`280`	`280`
`281`	`281`	`blockedBig.unpersist()`
`282`	`282`