[BUG]: Several bugs in examples/language/gpt/titans

[gouchangjiang]

🐛 Describe the bug

Several small bug in the examples provided in directory language/gpt/titans:

1. the default option of argument '--use_dummy_dataset' in function main of file train_gpt.py should be set to False, and 'store_false', otherwise even if I set the environment variable DATA, and run the train_gpt.py without the option '--use_dummy_dataset', the training will use dummy data.
2. Please add the py file webtext, otherwise the train_gpt.py will complain that 'WebtextDataset cannot be imported from dataset.webtext'.
3. The throughput is 0, Loss is nan from epoch 1 to epoch 8, then it stalls. The total epoch is 10. I am using the config 'gpt2_small_zero3_pp1d.py'. Nothing modified. 4 Nodes equipped with 8 A100 each are provided. ColossalAI is launched with Slurm. PyTorch is 1.12.1 and CUDA is of version 11.3. Two screeshot of out are provided:

The full output logs are [here](https://drive.google.com/file/d/1dSAeIqXzZv-9FIMdFUVeNDz9vJapU1ip/view?usp=sharing)

Environment

PyTorch is 1.12.1 and CUDA is of version 11.3. Colossal-AI is built from source without CUDA pre-compiled kernel.

[feifeibear]

Thanks, I have fixed some of the bugs. However, using dummy dataset seems still not ready. I will see it in detail.

[feifeibear]

I have fixed the bug. @gouchangjiang can you pull the latest code and try again?

[gouchangjiang]

Thanks. It get stalls when using configuration gpt3_zero3_pp1d and dummy dataset on 4 nodes 36 GPUS in totoal.

[feifeibear]

Running GPT3 is not easy. Can you start with a smaller model first? For example, a 10B model? Let's check you can run a model on multi-node.

[feifeibear]

Can we close this issue (if you run the default config successfully?) and start another one titled How to Run GPT3 using Titans?

[gouchangjiang]

Oh, sorry, the configuration is gpt2_small_zero3_pp1d provided in the directory config.

[feifeibear]

Never mind. Can you run it successfully on 1 node?

[gouchangjiang]

Unfortunately. It report a problem 'AssertionError: Expected the world size 2 to be equal to data parallel size (0) * pipeline parallel size (4) * tensor parallel size (2)' if I run it simply on a node by bash run.sh

[gouchangjiang]

Let me use 8 GPUs.

[feifeibear]

I guess you only use 1 GPU.

[gouchangjiang]

I forget that I was on my develop branch. I siwtched back to main. And run it with 8 GPUs. The same phenomenon happens: load layernorm kernel many many times infinitely