Skip a failing Bagua test for manual optimization

[awaelchli]

What does this PR do?

Unblocks master.
The test started failing over night with:

strategies/test_bagua_strategy.py:64: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../src/pytorch_lightning/trainer/trainer.py:602: in fit
    call._call_and_handle_interrupt(
../../src/pytorch_lightning/trainer/call.py:36: in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
../../src/pytorch_lightning/strategies/launchers/subprocess_script.py:88: in launch
    return function(*args, **kwargs)
../../src/pytorch_lightning/trainer/trainer.py:644: in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
../../src/pytorch_lightning/trainer/trainer.py:1078: in _run
    self.strategy.setup(self)
../../src/pytorch_lightning/strategies/bagua.py:209: in setup
    self._configure_bagua_model(trainer)
../../src/pytorch_lightning/strategies/bagua.py:221: in _configure_bagua_model
    self.model = self._setup_model(model)
../../src/pytorch_lightning/strategies/bagua.py:234: in _setup_model
    return BaguaDistributedDataParallel(
/usr/local/lib/python3.9/dist-packages/bagua/torch_api/data_parallel/distributed.py:148: in __init__
    self.inner = BaguaDistributedDataParallel(
/usr/local/lib/python3.9/dist-packages/bagua/torch_api/data_parallel/bagua_distributed.py:153: in __init__
    self._bagua_init_algorithm()
/usr/local/lib/python3.9/dist-packages/bagua/torch_api/data_parallel/bagua_distributed.py:400: in _bagua_init_algorithm
    self._bagua_autotune_register_tensors()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <bagua.torch_api.data_parallel.bagua_distributed.BaguaDistributedDataParallel object at 0x7f7551791490>

     def _bagua_autotune_register_tensors(self):
        """
        Register tensors on autotune server, and return first bucketing suggestions
        """
        autotune_tensor_list = [
            TensorDeclaration(
                {
                    "name": tensor.bagua_tensor_name,
                    "num_elements": tensor.numel(),
                    "dtype": to_bagua_datatype(tensor.dtype),
                }
            )
            for tensor in self._bagua_tensors
        ]
    
        rsp = self._bagua_autotune_client.register_tensors(
            model_name=self.bagua_module_name,
            tensor_list=autotune_tensor_list,
        )
>       assert rsp.status_code == 200, "Unexpected rsp={}".format(rsp)
E       AssertionError: Unexpected rsp=<Response [500]>

/usr/local/lib/python3.9/dist-packages/bagua/torch_api/data_parallel/bagua_distributed.py:372: AssertionError

Bagua version installed: bagua-cuda116==0.9.1

Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

cc @Borda @awaelchli @wangraying

[github-actions]

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, pytorch, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, pytorch, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, pytorch, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.10, oldest)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.10)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.9, 1.11)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.12)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.7, 1.10, oldest)	success	✅
pl-cpu (windows-2022, pytorch, 3.9, 1.11)	success	✅
pl-cpu (windows-2022, pytorch, 3.10, 1.12)	success	✅
pl-cpu (windows-2022, pytorch, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.7, 1.10, oldest)	success	✅
pl-cpu (slow, macOS-11, pytorch, 3.7, 1.11)	success	✅
pl-cpu (slow, ubuntu-20.04, pytorch, 3.7, 1.11)	success	✅
pl-cpu (slow, windows-2022, pytorch, 3.7, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.13)	success	✅

These checks are required after the changes to tests/tests_pytorch/strategies/test_bagua_strategy.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to tests/tests_pytorch/strategies/test_bagua_strategy.py.

🟢 pytorch_lightning: Azure HPU

Check ID	Status
pytorch-lightning (HPUs)	success	✅

These checks are required after the changes to tests/tests_pytorch/strategies/test_bagua_strategy.py.

🟢 pytorch_lightning: Azure IPU

Check ID	Status
pytorch-lightning (IPUs)	success	✅

These checks are required after the changes to tests/tests_pytorch/strategies/test_bagua_strategy.py.

---

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

[woqidaideshi]

@awaelchli
After an investigation of this problem, I found that this failure is caused by scikit-optimize library used by Bagua. In detail, scikit-optimize library currently uses an old builtin type of numpy (i.e., numpy.int), which is deprecated by numpy>=1.24.0.

To avoid this problem, users can choose to use numpy<1.24.0.

In the meantime, scikit-optimize is trying to fix this problem (PR: scikit-optimize/scikit-optimize#1123). After that, users can use the updated scikit-optimize with numpy>=1.24.0 in Bagua.