-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SQL][minor] use catalyst type converter in ScalaUdf #6182
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Merged build triggered. |
|
Merged build started. |
|
Use the same benchmark: import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.types._
case class Floor(child: Expression) extends UnaryExpression with Predicate {
override def foldable = child.foldable
def nullable = child.nullable
override def toString = s"Floor $child"
override def eval(input: Row): Any = {
child.eval(input) match {
case null => null
case ts: Int => ts - ts % 300
}
}
}
object T {
def benchmark(count: Int, expr: Expression): Unit = {
var i = 0
val row = new GenericRow(Array[Any](123, 21, 42))
val s = System.currentTimeMillis()
while (i < count) {
expr.eval(row)
i += 1
}
val e = System.currentTimeMillis()
println (s"${expr.getClass.getSimpleName} -- ${e - s} ms")
}
def main(args: Array[String]) {
def func(ts: Int) = ts - ts % 300
val udf0 = ScalaUdf(func _, IntegerType, BoundReference(0, IntegerType, true) :: Nil)
val udf1 = Floor(BoundReference(0, IntegerType, true))
benchmark(1000000, udf0)
benchmark(1000000, udf0)
benchmark(1000000, udf0)
benchmark(1000000, udf1)
benchmark(1000000, udf1)
benchmark(1000000, udf1)
}
}before: after: |
|
Test build #32813 has started for PR 6182 at commit |
|
Test build #32813 has finished for PR 6182 at commit
|
|
Merged build finished. Test PASSed. |
|
Test PASSed. |
|
LGTM; seems like a pretty straightforward optimization. I'm a bit new to Spark SQL, though, so I'll wait for committer with SQL experience to do the final sign-off on this. |
|
LGTM |
|
I am merging it to master and branch 1.4. |
It's a follow-up of #5154, we can speed up scala udf evaluation by create type converter in advance. Author: Wenchen Fan <[email protected]> Closes #6182 from cloud-fan/tmp and squashes the following commits: 241cfe9 [Wenchen Fan] use converter in ScalaUdf (cherry picked from commit 2f22424) Signed-off-by: Yin Huai <[email protected]>
It's a follow-up of apache#5154, we can speed up scala udf evaluation by create type converter in advance. Author: Wenchen Fan <[email protected]> Closes apache#6182 from cloud-fan/tmp and squashes the following commits: 241cfe9 [Wenchen Fan] use converter in ScalaUdf
It's a follow-up of apache#5154, we can speed up scala udf evaluation by create type converter in advance. Author: Wenchen Fan <[email protected]> Closes apache#6182 from cloud-fan/tmp and squashes the following commits: 241cfe9 [Wenchen Fan] use converter in ScalaUdf
It's a follow-up of apache#5154, we can speed up scala udf evaluation by create type converter in advance. Author: Wenchen Fan <[email protected]> Closes apache#6182 from cloud-fan/tmp and squashes the following commits: 241cfe9 [Wenchen Fan] use converter in ScalaUdf
It's a follow-up of #5154, we can speed up scala udf evaluation by create type converter in advance.