[RLlib] Cleanup examples folder #13. Fix main examples docs page for RLlib. #45382

sven1977 · 2024-05-16T09:53:55Z

Cleanup examples folder #13. Fix main examples docs page for RLlib.

Why are these changes needed?

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: sven1977 <[email protected]>

…rithm_config_dissolve_resources_method

Signed-off-by: sven1977 <[email protected]>

…nup_examples_folder_11_fractional_gpus

…p_examples_folder_11_fractional_gpus

Signed-off-by: sven1977 <[email protected]>

…p_examples_folder_11_fractional_gpus # Conflicts: # rllib/utils/test_utils.py

Signed-off-by: sven1977 <[email protected]>

angelinalg

Just some style nits and a few typos.

doc/source/rllib/rllib-advanced-api.rst

angelinalg · 2024-05-20T20:29:48Z

rllib/utils/error.py

-    `num_gpus_per_worker` to 0 (they may be set to 1 by default for your
-    particular RL algorithm)."""
+    machine does not have any GPUs, you should set the config keys
+    `num_gpus_per_learner` and `num_gpus_per_env_runner` to 0 (they may be set to


Suggested change

`num_gpus_per_learner` and `num_gpus_per_env_runner` to 0 (they may be set to

`num_gpus_per_learner` and `num_gpus_per_env_runner` to 0. They may be set to

angelinalg · 2024-05-20T20:29:58Z

rllib/utils/error.py

-    particular RL algorithm)."""
+    machine does not have any GPUs, you should set the config keys
+    `num_gpus_per_learner` and `num_gpus_per_env_runner` to 0 (they may be set to
+    1 by default for your particular RL algorithm)."""


Suggested change

1 by default for your particular RL algorithm)."""

1 by default for your particular RL algorithm."""

angelinalg · 2024-05-20T20:30:14Z

rllib/utils/error.py

+    1 by default for your particular RL algorithm)."""

 ERR_MSG_INVALID_ENV_DESCRIPTOR = """The env string you provided ('{}') is:
 a) Not a supported/installed environment.


Suggested change

a) Not a supported/installed environment.

a) Not a supported or installed environment.

angelinalg · 2024-05-20T20:30:34Z

rllib/utils/test_utils.py

        trainable: The Trainable sub-class to run in the tune.Tuner. If None (default),
            use the registered RLlib Algorithm class specified by args.algo.
        tune_callbacks: A list of Tune callbacks to configure with the tune.Tuner.
            In case `args.wandb_key` is provided, will append a WandB logger to this


Suggested change

In case `args.wandb_key` is provided, will append a WandB logger to this

In case `args.wandb_key` is provided, appends a WandB logger to this

angelinalg · 2024-05-20T20:30:51Z

rllib/utils/test_utils.py

+        keep_config: Set this to True, if you don't want this utility to change the
+            given `base_config` in any way and leave it as-is. This is helpful
+            for example script that want to demonstrate how to set those settings
+            that are usually taken care of automatically in this function (e.g.


Suggested change

that are usually taken care of automatically in this function (e.g.

that are usually taken care of automatically in this function (e.g.,

…nup_examples_folder_13_folder_readme Signed-off-by: sven1977 <[email protected]> # Conflicts: # doc/source/rllib/rllib-advanced-api.rst # doc/source/rllib/rllib-learner.rst # rllib/BUILD # rllib/algorithms/algorithm.py # rllib/algorithms/algorithm_config.py # rllib/algorithms/dreamerv3/tests/test_dreamerv3.py # rllib/core/learner/learner.py # rllib/core/learner/scaling_config.py # rllib/examples/checkpoints/restore_1_of_n_agents_from_checkpoint.py # rllib/examples/gpus/fractional_gpus_per_learner.py # rllib/tuned_examples/dreamerv3/atari_100k.py # rllib/tuned_examples/dreamerv3/atari_200M.py # rllib/tuned_examples/dreamerv3/dm_control_suite_vision.py # rllib/utils/test_utils.py

Signed-off-by: sven1977 <[email protected]>

…nup_examples_folder_13_folder_readme

Signed-off-by: sven1977 <[email protected]>

…nup_examples_folder_13_folder_readme

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Sven Mika <[email protected]>

Signed-off-by: sven1977 <[email protected]>

…r_readme' into cleanup_examples_folder_13_folder_readme

…nup_examples_folder_13_folder_readme

Signed-off-by: sven1977 <[email protected]>

…nup_examples_folder_13_folder_readme Signed-off-by: sven1977 <[email protected]> # Conflicts: # rllib/examples/inference/policy_inference_after_training.py

Signed-off-by: sven1977 <[email protected]>

simonsays1980

LGTM.

simonsays1980 · 2024-06-11T16:43:43Z

doc/source/rllib/key-concepts.rst

 RLlib uses `Ray actors <actors.html>`__ to scale training from a single core to many thousands of cores in a cluster.
 You can `configure the parallelism <rllib-training.html#specifying-resources>`__ used for training by changing the ``num_env_runners`` parameter.
-Check out our `scaling guide <rllib-training.html#scaling-guide>`__ for more details here.
+See this `scaling guide <rllib-training.html#scaling-guide>`__ for more details here.


The scaling guide also needs to be overhauled.

simonsays1980 · 2024-06-11T16:56:30Z

rllib/algorithms/algorithm_config.py

+                an index into the available
                CUDA devices. For example if `os.environ["CUDA_VISIBLE_DEVICES"] = "1"`
-                then a `local_gpu_idx` of 0 will use the GPU with ID=1 on the node.
+                and `local_gpu_idx=0`, RLlib uses the GPU with ID=1 on the node.


This feels counterintuitive. The GPU index 0 does not equal the environment variable 1 and we have two or more GPUs for a single learner. A user would expect a single GPU for a single learner when multiple GPUs are available on a node to be indicated with an ID or index. Do I misunderstand sth here?

simonsays1980 · 2024-06-11T16:57:23Z

rllib/algorithms/algorithm_config.py

-                `num_learners` x `train_batch_size_per_learner` and can
-                be accessed via the property `AlgorithmConfig.total_train_batch_size`.
+                `num_learners` x `train_batch_size_per_learner` and you can
+                access it with the property `AlgorithmConfig.total_train_batch_size`.


We should refer hereto in the scaling guide ~ if not done yet.

…nup_examples_folder_13_folder_readme

Signed-off-by: sven1977 <[email protected]>

…s page for RLlib. (ray-project#45382) Signed-off-by: Richard Liu <[email protected]>

sven1977 added 13 commits May 16, 2024 04:33

wip

cb937c6

Signed-off-by: sven1977 <[email protected]>

wip

c931bed

Signed-off-by: sven1977 <[email protected]>

wip

02d3d04

Signed-off-by: sven1977 <[email protected]>

wip

3ada50a

Signed-off-by: sven1977 <[email protected]>

Merge branch 'master' of https://github.com/ray-project/ray into algo…

bccdde6

…rithm_config_dissolve_resources_method

wip

496c5ee

Signed-off-by: sven1977 <[email protected]>

Merge branch 'master' of https://github.com/ray-project/ray into clea…

edd0a91

…nup_examples_folder_11_fractional_gpus

Merge branch 'algorithm_config_dissolve_resources_method' into cleanu…

4cdebb3

…p_examples_folder_11_fractional_gpus

wip

68aa7ba

Signed-off-by: sven1977 <[email protected]>

wip

794b960

Signed-off-by: sven1977 <[email protected]>

Merge branch 'algorithm_config_dissolve_resources_method' into cleanu…

3a6d05e

…p_examples_folder_11_fractional_gpus # Conflicts: # rllib/utils/test_utils.py

wip

03be7f5

Signed-off-by: sven1977 <[email protected]>

wip

e6a8a2e

Signed-off-by: sven1977 <[email protected]>

sven1977 requested review from a team, ArturNiederfahrenhorst, avnishn, kouroshHakha, maxpumperla and simonsays1980 as code owners May 16, 2024 09:53

sven1977 assigned simonsays1980 and angelinalg May 16, 2024

sven1977 added rllib RLlib related issues rllib-docs-or-examples Issues related to RLlib documentation or rllib/examples rllib-newstack labels May 16, 2024

wip

d65082f

Signed-off-by: sven1977 <[email protected]>

angelinalg approved these changes May 20, 2024

View reviewed changes

sven1977 added 3 commits June 3, 2024 21:23

wip

5d1fa21

Signed-off-by: sven1977 <[email protected]>

Merge branch 'master' of https://github.com/ray-project/ray into clea…

5322241

…nup_examples_folder_13_folder_readme

sven1977 and others added 12 commits June 10, 2024 12:21

wip

d0995ae

Signed-off-by: sven1977 <[email protected]>

wip

0916ce7

Signed-off-by: sven1977 <[email protected]>

Merge branch 'master' of https://github.com/ray-project/ray into clea…

f36f7dc

…nup_examples_folder_13_folder_readme

Apply suggestions from code review

23fcf8d

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Sven Mika <[email protected]>

Apply suggestions from code review

8e2afcc

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Sven Mika <[email protected]>

Apply suggestions from code review

8966d52

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Sven Mika <[email protected]>

fix

cbc7f5b

Signed-off-by: sven1977 <[email protected]>

Merge remote-tracking branch 'origin/cleanup_examples_folder_13_folde…

9d509c3

…r_readme' into cleanup_examples_folder_13_folder_readme

Merge branch 'master' of https://github.com/ray-project/ray into clea…

11b207e

…nup_examples_folder_13_folder_readme

fix

ed21e41

Signed-off-by: sven1977 <[email protected]>

fix

3ea64bf

Signed-off-by: sven1977 <[email protected]>

fix

f192bc3

Signed-off-by: sven1977 <[email protected]>

sven1977 changed the title ~~[RLlib] Cleanup examples folder 13: Add READMEs to folder and all sub-folders.~~ [RLlib] Cleanup examples folder #13. Fix main examples docs page for RLlib. Jun 11, 2024

sven1977 added 3 commits June 11, 2024 14:15

fix

2dbe142

Signed-off-by: sven1977 <[email protected]>

Merge branch 'master' of https://github.com/ray-project/ray into clea…

73820dc

…nup_examples_folder_13_folder_readme Signed-off-by: sven1977 <[email protected]> # Conflicts: # rllib/examples/inference/policy_inference_after_training.py

fix

5172242

Signed-off-by: sven1977 <[email protected]>

sven1977 enabled auto-merge (squash) June 11, 2024 13:52

github-actions bot added the go add ONLY when ready to merge, run all tests label Jun 11, 2024

simonsays1980 approved these changes Jun 11, 2024

View reviewed changes

Merge branch 'master' of https://github.com/ray-project/ray into clea…

80566dd

…nup_examples_folder_13_folder_readme

github-actions bot disabled auto-merge June 12, 2024 10:03

fix

41102e5

Signed-off-by: sven1977 <[email protected]>

sven1977 enabled auto-merge (squash) June 12, 2024 11:21

sven1977 merged commit c84bf37 into ray-project:master Jun 12, 2024

sven1977 deleted the cleanup_examples_folder_13_folder_readme branch June 12, 2024 12:59

richardsliu pushed a commit to richardsliu/ray that referenced this pull request Jun 12, 2024

[RLlib] Cleanup examples folder ray-project#13. Fix main examples doc…

57a3dc2

…s page for RLlib. (ray-project#45382) Signed-off-by: Richard Liu <[email protected]>

	`num_gpus_per_learner` and `num_gpus_per_env_runner` to 0 (they may be set to
	`num_gpus_per_learner` and `num_gpus_per_env_runner` to 0. They may be set to

	1 by default for your particular RL algorithm)."""
	1 by default for your particular RL algorithm."""

	a) Not a supported/installed environment.
	a) Not a supported or installed environment.

	In case `args.wandb_key` is provided, will append a WandB logger to this
	In case `args.wandb_key` is provided, appends a WandB logger to this

	that are usually taken care of automatically in this function (e.g.
	that are usually taken care of automatically in this function (e.g.,

[RLlib] Cleanup examples folder #13. Fix main examples docs page for RLlib. #45382

[RLlib] Cleanup examples folder #13. Fix main examples docs page for RLlib. #45382

Uh oh!

Conversation

sven1977 commented May 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

angelinalg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

simonsays1980 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sven1977 commented May 16, 2024 •

edited

Loading