Skip to content

Improve build perf with test splitting and some sensible caching #129

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 39 commits into from
Mar 16, 2017
Merged

Improve build perf with test splitting and some sensible caching #129

merged 39 commits into from
Mar 16, 2017

Conversation

@dansanduleac
Copy link

What changes were proposed in this pull request?

  • split scala tests in SBT based on the number of containers at the test class level
    • currently each test class has a weight of 1
    • enables easy future improvement of caching test metrics (how long it takes) and using that as the weight
  • add some more sensible caching to make the build start faster
  • create two conda envs for the python versions we want to support (+ numpy) rather than installing numpy using pip, which is slow (needs to be compiled upon install)

How was this patch tested?

Passes tests

Robert Kruszewski and others added 30 commits March 7, 2017 23:33
only one container
0d232ad
@dansanduleac
Only clean in the case of container 2, because the flags have changed
effd212
@dansanduleac
Cache all built classes after the install, restore cache before calli…
a2c080d 
…ng it next time
@dansanduleac
Cache the target/analysis dirs as well
baf7d82
@dansanduleac
Fix invocation of container 2
62922a2
@dansanduleac
Just do rsync -v
f32081f
@dansanduleac
Make sbt fetch all the external deps to ~/.ivy2 so it gets cached
c23dba4
@dansanduleac
Actually do this instead
306f60b
@dansanduleac
More rsync stats as to what was copied
b45e683
@dansanduleac
Include target/maven-status in the cache
f031d39
@dansanduleac
Improve target/ caching I think
61bb45f
@dansanduleac
Merge remote-tracking branch 'origin/rk/only-one-container' into ds/b…
1b25ab3 
…uild-perf
@dansanduleac
Use circle parallelism to split SBT test goals by goals
601c7ac
@dansanduleac
Fix run-tests.py prematurely exiting before copying the reports to ci…
1557a78 
…rcle
@dansanduleac
Download miniconda through pyenv, and create envs for the tests rathe…
0f7cd19 
…r than using pip
@dansanduleac
Revert "Use circle parallelism to split SBT test goals by goals"
e7d716f 
This reverts commit 601c7ac.
@dansanduleac
Don't do the publish until the tests have actually run
f45fd21
@dansanduleac
Move publishing to deployment which just calls dev/publish
1430462
@dansanduleac
Implement test-level parallelism across the entire set of tests run f…
3294221 
…rom SBT when in circle
@dansanduleac
Use SBT cached resolution for faster update, hopefully
1fbc191
@dansanduleac
Cache compilation results too in the dependencies stage
2e4c6c5
@dansanduleac
fix style, add apache copyright
cb2cee7
@dansanduleac
Set sbt's target to something else, so it doesn't clash with maven's …
ad05d01 
…and destroy all the caching benefit...
@dansanduleac
Stop grep -v-ing for info
da42682
@dansanduleac
Exclude target_sbt/ from stuff too
8167a57
@dansanduleac
exclude target_sbt from rat tests
1bf9ec1
@dansanduleac
Revert "Stop grep -v-ing for info"
4b7ba2b 
This reverts commit da42682.
@dansanduleac
oops that has to be a list!
ec936ff
@dansanduleac
Fix cached resolution, it must be applied to the root project too whi…
df47910 
…ch means we should set it in global where it will be picked up from
@dansanduleac
Revert "Set sbt's target to something else, so it doesn't clash with …
015cbe4 
…maven's and destroy all the caching benefit..." + rest

This reverts commit 1bf9ec1.
This reverts commit 8167a57.
This reverts commit ad05d01.
@dansanduleac dansanduleac mentioned this pull request Mar 16, 2017
Merged
@robert3005 robert3005 merged commit 55fb50d into master Mar 16, 2017
@robert3005 robert3005 deleted the ds/build-perf branch March 16, 2017 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dansanduleac @robert3005