Added atomic checkpoint creation

[fgerzer]

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
  • Checkpoint saving isn't atomic #688
• Did you read the contributor guideline?
• Did you make sure to update the docs?
  • No visible changes
• Did you write any new necessary tests?
  • Old behaviour continues; new behaviour relies on timing outside the code.

What does this PR do?

Fixes #688 .

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

[williamFalcon]

@fdiehl can you rebase master?

[fgerzer]

@williamFalcon Done.

(I had originally used an older commit because the tests previously failed on master due to tensorboard issues. They are passing with the current master.)