Skip to content

Commit d60ab5f

Browse files
kiszkhvanhovell
authored andcommitted
[SPARK-18745][SQL] Fix signed integer overflow due to toInt cast
## What changes were proposed in this pull request? This PR avoids that a result of a cast `toInt` is negative due to signed integer overflow (e.g. 0x0000_0000_1???????L.toInt < 0 ). This PR performs casts after we can ensure the value is within range of signed integer (the result of `max(array.length, ???)` is always integer). ## How was this patch tested? Manually executed query68 of TPC-DS with 100TB Author: Kazuaki Ishizaki <[email protected]> Closes #16235 from kiszk/SPARK-18745.
1 parent b08b500 commit d60ab5f

File tree

1 file changed

+4
-4
lines changed

1 file changed

+4
-4
lines changed

sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -670,9 +670,9 @@ private[execution] final class LongToUnsafeRowMap(val mm: TaskMemoryManager, cap
670670
var offset: Long = Platform.LONG_ARRAY_OFFSET
671671
val end = len * 8L + Platform.LONG_ARRAY_OFFSET
672672
while (offset < end) {
673-
val size = Math.min(buffer.length, (end - offset).toInt)
673+
val size = Math.min(buffer.length, end - offset)
674674
Platform.copyMemory(arr, offset, buffer, Platform.BYTE_ARRAY_OFFSET, size)
675-
writeBuffer(buffer, 0, size)
675+
writeBuffer(buffer, 0, size.toInt)
676676
offset += size
677677
}
678678
}
@@ -710,8 +710,8 @@ private[execution] final class LongToUnsafeRowMap(val mm: TaskMemoryManager, cap
710710
var offset: Long = Platform.LONG_ARRAY_OFFSET
711711
val end = length * 8L + Platform.LONG_ARRAY_OFFSET
712712
while (offset < end) {
713-
val size = Math.min(buffer.length, (end - offset).toInt)
714-
readBuffer(buffer, 0, size)
713+
val size = Math.min(buffer.length, end - offset)
714+
readBuffer(buffer, 0, size.toInt)
715715
Platform.copyMemory(buffer, Platform.BYTE_ARRAY_OFFSET, array, offset, size)
716716
offset += size
717717
}

0 commit comments

Comments
 (0)