Skip to content

Commit a3e5496

Browse files
authored
[example] improved the clarity yof the example readme (#2427)
* [example] improved the clarity yof the example readme * polish workflow * polish workflow * polish workflow * polish workflow * polish workflow * polish workflow
1 parent 2125667 commit a3e5496

File tree

2 files changed

+33
-20
lines changed

2 files changed

+33
-20
lines changed

.github/workflows/auto_example_check.yml

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ jobs:
3939
res=`python .github/workflows/scripts/example_checks/detect_changed_example.py --fileNameList $changedFileName`
4040
echo "All changed examples are $res"
4141
42-
if [ "$x" = "[]" ]; then
42+
if [ "$res" = "[]" ]; then
4343
echo "anyChanged=false" >> $GITHUB_OUTPUT
4444
echo "matrix=null" >> $GITHUB_OUTPUT
4545
else
@@ -54,7 +54,8 @@ jobs:
5454
if: |
5555
github.event.pull_request.draft == false &&
5656
github.base_ref == 'main' &&
57-
github.event.pull_request.base.repo.full_name == 'hpcaitech/ColossalAI' && github.event_name == 'pull_request'
57+
github.event.pull_request.base.repo.full_name == 'hpcaitech/ColossalAI' && github.event_name == 'pull_request' &&
58+
needs.detect-changed-example.outputs.anyChanged == 'true'
5859
name: Test the changed example
5960
needs: detect-changed-example
6061
runs-on: [self-hosted, gpu]

examples/README.md

Lines changed: 30 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,40 @@
1-
## Examples folder document
1+
# Colossal-AI Examples
22

33
## Table of Contents
4-
<ul>
5-
<li><a href="#Example-folder-description">Example folder description</a> </li>
6-
<li><a href="#Integrate-Your-Example-With-System-Testing">Integrate Your Example With System Testing</a> </li>
7-
</ul>
84

9-
## Example folder description
5+
- [Colossal-AI Examples](#colossal-ai-examples)
6+
- [Table of Contents](#table-of-contents)
7+
- [Overview](#overview)
8+
- [Folder Structure](#folder-structure)
9+
- [Integrate Your Example With Testing](#integrate-your-example-with-testing)
1010

11-
This folder provides several examples using colossalai. The images folder includes model like diffusion, dreambooth and vit. The language folder includes gpt, opt, palm and roberta. The tutorial folder is for concept illustration, such as auto-parallel, hybrid-parallel and so on.
11+
## Overview
1212

13+
This folder provides several examples accelerated by Colossal-AI. The `tutorial` folder is for everyone to quickly try out the different features in Colossal-AI. Other folders such as `images` and `language` include a wide range of deep learning tasks and applications.
1314

14-
## Integrate Your Example With System Testing
15+
## Folder Structure
1516

16-
For example code contributor, to meet the expectation and test your code automatically using github workflow function, here are several steps:
17+
```text
18+
└─ examples
19+
└─ images
20+
└─ vit
21+
└─ test_ci.sh
22+
└─ train.py
23+
└─ README.md
24+
└─ ...
25+
└─ ...
26+
```
1727

28+
## Integrate Your Example With Testing
1829

19-
- (must) Have a test_ci.sh file in the folder like shown below in 'File Structure Chart'
20-
- The dataset should be located in the company's machine and can be announced using environment variable and thus no need for a separate terminal command.
21-
- The model parameters should be small to allow fast testing.
22-
- File Structure Chart
30+
Regular checks are important to ensure that all examples run without apparent bugs and stay compatible with the latest API.
31+
Colossal-AI runs workflows to check for examples on a on-pull-request and weekly basis.
32+
When a new example is added or changed, the workflow will run the example to test whether it can run.
33+
Moreover, Colossal-AI will run testing for examples every week.
2334

24-
└─examples
25-
└─images
26-
└─vit
27-
└─requirements.txt
28-
└─test_ci.sh
35+
Therefore, it is essential for the example contributors to know how to integrate your example with the testing workflow. Simply, you can follow the steps below.
36+
37+
1. Create a script called `test_ci.sh` in your example folder
38+
2. Configure your testing parameters such as number steps, batch size in `test_ci.sh`, e.t.c. Keep these parameters small such that each example only takes several minutes.
39+
3. Export your dataset path with the prefix `/data` and make sure you have a copy of the dataset in the `/data/scratch/examples-data` directory on the CI machine. Community contributors can contact us via slack to request for downloading the dataset on the CI machine.
40+
4. Implement the logic such as dependency setup and example execution

0 commit comments

Comments
 (0)