[SPARK-25704][CORE] Allocate a bit less than Int.MaxValue #22705
Conversation
JVMs don't you allocate arrays of length exactly Int.MaxValue, so leave a little extra room. This is necessary when reading blocks >2GB off the network (for remote reads or for cache replication).
|
Test build #97290 has finished for PR 22705 at commit
|
| // Though in theory you should be able to index into an array of size Int.MaxValue, in practice | ||
| // jvms don't let you go up to limit. It seems you may only need - 2, but we leave a little | ||
| // extra room. | ||
| val maxArraySize = Int.MaxValue - 512 |
There was a problem hiding this comment.
Looks like the max is safely more like Int.MaxValue - 5 from things like https://www.quora.com/What-is-the-maximum-size-of-the-array-in-Java You could probably push it further, but whatever.
There was a problem hiding this comment.
We already had seen the similar problem in other places. Can we use this value here, too?
There was a problem hiding this comment.
great suggestion, thanks, just updated
|
Test build #97399 has finished for PR 22705 at commit
|
squito
changed the title
|
Jenkins, retest this please |
|
Test build #97423 has finished for PR 22705 at commit
|
|
retest this please |
|
Test build #97465 has finished for PR 22705 at commit
|
|
LGTM |
JVMs don't you allocate arrays of length exactly Int.MaxValue, so leave a little extra room. This is necessary when reading blocks >2GB off the network (for remote reads or for cache replication). Unit tests via jenkins, ran a test with blocks over 2gb on a cluster Closes #22705 from squito/SPARK-25704. Authored-by: Imran Rashid <[email protected]> Signed-off-by: Imran Rashid <[email protected]>
|
merged to master / 2.4. Something went wrong w/ the merge script when putting it on 2.4 (I think I accidentally had a dirty working dir at the time), but it was actually a clean cherry pick, so I pushed it manually here: |
JVMs don't you allocate arrays of length exactly Int.MaxValue, so leave a little extra room. This is necessary when reading blocks >2GB off the network (for remote reads or for cache replication). Unit tests via jenkins, ran a test with blocks over 2gb on a cluster Closes apache#22705 from squito/SPARK-25704. Authored-by: Imran Rashid <[email protected]> Signed-off-by: Imran Rashid <[email protected]>
4 participants
JVMs don't you allocate arrays of length exactly Int.MaxValue, so leave
a little extra room. This is necessary when reading blocks >2GB off
the network (for remote reads or for cache replication).
Unit tests via jenkins, ran a test with blocks over 2gb on a cluster