integrations: CML

[daavoo]

From a high level user perspective, I think it would be interesting to have some kind of integration between dvclive and CML.

Just like there is already a custom dvclive behavior when used along with DVC (i.e. generating checkpoints) maybe a similar approach could be used when dvclive is also used along with CML.

I was thinking on a use case like dvclive taking care of using cml-send-comment inside next_step if CML is detected.

Conceptually i think that it would be something similar to the existing make_checkpoint:

https://github.com/iterative/dvclive/blob/master/dvclive/metrics.py#L105

This might be just an specific use case of #90

[dberenbaum]

This seems like a general need for integration between cml and dvc checkpoints. Do you see any functionality as being specific to dvclive?

[daavoo]

This seems like a general need for integration between cml and dvc checkpoints. Do you see any functionality as being specific to dvclive?

Given the discussion so far in iterative/dvc#6182, it looks like the exp push integration won't require any changes from the dvclive side.

I probably didn't express it right but I was also taking about an additional integration, which will be something like using cml-send-comment in the context of a long running experiment (i.e. deep learning) as a way to periodically report the status of the training.

From an user perspective what I would want is, by using dvclive in an environment where CML is installed, getting a P.R. comment reporting the dvclive metrics summary each time an epoch is finished (which would translate to each time dvclive.next_step is called).

This is a feature (sending the metrics to somewhere) that I have been repeatedly implementing through workarounds when using other ML Loggers and I though that ot could be nice to have it working out of the box with dvclive<>CML.

The feature probably doesn't make too much sense in the context of training a lot of short experiments and it's probably not a deal breaker for attracting new users.

[dberenbaum]

The feature probably doesn't make too much sense in the context of training a lot of short experiments and it's probably not a deal breaker for attracting new users.

Yes, this could be really useful, and absolutely dvclive should address issues for long-running experiments! iterative/dvc#6182 will at least make it possible to pull the latest checkpoint and see the associated metrics and plots, but we could make the experience smoother and better integrate cml.

Seems like this has 2 requirements:

1. Generate some default report to send.
2. Trigger cml-send-comment at checkpoints or some other point in code (should it be coupled with checkpoints/next_step?).

[shcheklein]

@iterative/cml I know that you have been discussing (and even prototyping) something like this? Could you share your thoughts, outcomes?

[pared]

@dberenbaum I think we could even start with pretty printing metrics to some env var that CML could pick up and show in the build log.

[casperdcl]

I'd say generic callback support (#90) would be best. The user can use that to run cml-send-comment --update per-checkpoint.

Vaguely related (though this is automatic rather than manual): auto-push per checkpoint iterative/dvc#6182

[DavidGOrtega]

From a high level user perspective, I think it would be interesting to have some kind of integration between dvclive and CML.

That would be awesome. I think that the best integration would be if dvclive would work as tensorboard. Something that you can launch and give you a url in return. We wrapped that under cml-tensorboard-dev.

@iterative/cml I know that you have been discussing (and even prototyping) something like this? Could you share your thoughts, outcomes?

I implemented something like that in cml-publish. The idea was to initiate a daemon when you used cml-publish to watch for file changes of the file that you published with cml-publish. If the daemon see a new change republishes the file again in the same url. It works like badges, so the images in the report change. The daemon was a very simple piece of software using chokidar.

[dberenbaum]

That sounds cool, @DavidGOrtega! Definitely cleaner than a separate report for each checkpoint. In this case, can dvclive republish to the same url at each step without needing a daemon?

[DavidGOrtega]

Yes, it could. Something that actually @pared also liked when I shared the idea in general and gave the same feedback.
Its not implemented in CML (yet), its a local experiment like the extension that we did

[daavoo]

I'm currently working on what I consider the minimum viable integration between DVCLive and CML

• DVCLive P.R. live: Add report option. #215
• Integration example https://github.com/iterative/test-dvclive-cml
• Current result https://github.com/iterative/test-dvclive-cml/commit/8460322df5a586d89882367310c1a4d534202c24

---

The current prototype doesn't depend on DVC (or DVC checkpoints) but still works with both.

It is focused on using cml send-comment and cml publish to share the progress of a training stage while the python process it's being run.

• If CML is installed, on each step update (i.e. next_step()), DVCLive generates a markdown report using the metrics already logged.
• The report includes a table (uses tabulate) and rendered images for the plots (uses matplotlib and cml publish).
• Calls cml send-comment once the report is ready.

It currently uses subprocess.run to call cml commands.

It only works for posting multiple comments (1 per step) because of iterative/cml#888
I still have to try to set up a workflow supporting "P.R. comments" instead of "commit comments".

---

@iterative/cml I would like to hear some thoughts about the downsides of raw subprocess.run v.s. potential python bindings to see how "blocker" those bindings are.

[DavidGOrtega]

@daavoo I can't follow up the links.

I did a wrapper for python called CMLpy. Basically it exposes all the CML commands.
I just only need to do the installer. We should sync. I'm going to send you an invite