|
1 | 1 | # Test Quality Assurance Rules
|
2 | 2 |
|
3 |
| -## Simple Enhancement Workflow |
4 |
| - |
5 |
| -When user requests test case enhancement or **test reproducibility verification**: |
| 3 | +## CRITICAL: Follow These Steps EXACTLY - No Deviations |
6 | 4 |
|
7 | 5 | ### 1. Problem Resolution
|
8 | 6 |
|
9 | 7 | - Use active file context or user-provided problem name
|
10 | 8 | - If unclear, run: `poetry run python -m leetcode_py.tools.check_test_cases --threshold=10 --max=1`
|
11 | 9 |
|
12 |
| -### 2. Enhancement Process |
| 10 | +### 2. Test Reproducibility Verification Process |
| 11 | + |
| 12 | +**MANDATORY 6-Step Process - Execute in Order:** |
13 | 13 |
|
14 | 14 | ```bash
|
15 |
| -# Simple 4-step process: |
16 |
| -# 1. Update JSON template with more test cases (12-15 total) |
17 |
| -# 2. Backup original |
18 |
| -mv leetcode/{problem_name} .cache/leetcode/{problem_name} |
19 |
| -# 3. Regenerate with enhanced tests |
20 |
| -make p-gen PROBLEM={problem_name} FORCE=1 && make p-lint PROBLEM={problem_name} |
21 |
| -# 4. Restore original solution, keep enhanced tests |
22 |
| -cp .cache/leetcode/{problem_name}/solution.py leetcode/{problem_name}/solution.py |
23 |
| -``` |
| 15 | +# Step 1: Backup original files |
| 16 | +cp -r leetcode/{problem_name} leetcode/{problem_name}_backup |
24 | 17 |
|
25 |
| -### 3. Verification |
| 18 | +# Step 2: Regenerate from JSON template (use Makefile, NOT poetry run) |
| 19 | +make p-gen PROBLEM={problem_name} FORCE=1 |
26 | 20 |
|
27 |
| -- Run `make p-test PROBLEM={problem_name}` |
28 |
| -- Fix any incorrect expected values in test cases |
29 |
| -- Update JSON template with corrections |
| 21 | +# Step 3: Restore original solution ONLY |
| 22 | +cp leetcode/{problem_name}_backup/solution.py leetcode/{problem_name}/solution.py |
30 | 23 |
|
31 |
| -### 4. Restore Backup |
| 24 | +# Step 4: Verify mypy passes (CRITICAL for CI) |
| 25 | +poetry run mypy leetcode/{problem_name}/test_solution.py --explicit-package-bases --install-types --non-interactive --check-untyped-defs |
32 | 26 |
|
33 |
| -```bash |
34 |
| -# Copy enhanced test_solution.py to backup |
35 |
| -cp leetcode/{problem_name}/test_solution.py .cache/leetcode/{problem_name}/ |
36 |
| -# Restore all original files (preserves user edits) |
37 |
| -rm -rf leetcode/{problem_name} |
38 |
| -mv .cache/leetcode/{problem_name} leetcode/{problem_name} |
| 27 | +# Step 5: Verify tests pass (expected to fail if solution is incomplete) |
| 28 | +make p-test PROBLEM={problem_name} |
| 29 | + |
| 30 | +# Step 6: Cleanup |
| 31 | +rm -rf leetcode/{problem_name}_backup |
39 | 32 | ```
|
40 | 33 |
|
| 34 | +### 3. What NOT to Do |
| 35 | + |
| 36 | +- ❌ **NEVER edit cookiecutter templates** (`{{cookiecutter.problem_name}}/` files) |
| 37 | +- ❌ **NEVER use `poetry run python -m leetcode_py.cli.main gen`** - use `make p-gen` instead |
| 38 | +- ❌ **NEVER modify helpers.py manually** - let regeneration handle it |
| 39 | +- ❌ **NEVER skip mypy verification** - this is the main CI issue |
| 40 | +- ❌ **NEVER assume tests will pass** - they may fail if solution is incomplete |
| 41 | + |
| 42 | +### 4. What to Do |
| 43 | + |
| 44 | +- ✅ **ALWAYS use `make p-gen PROBLEM={problem_name} FORCE=1`** for regeneration |
| 45 | +- ✅ **ALWAYS verify mypy passes** before considering task complete |
| 46 | +- ✅ **ALWAYS restore original solution** after regeneration |
| 47 | +- ✅ **ALWAYS check JSON template** if mypy fails (look for `assert_assert_` bugs) |
| 48 | + |
41 | 49 | ## Test Case Standards
|
42 | 50 |
|
43 | 51 | ### Coverage Requirements
|
@@ -84,24 +92,34 @@ poetry run python -m leetcode_py.tools.check_test_cases --threshold=12
|
84 | 92 | make p-gen PROBLEM={problem_name} FORCE=1
|
85 | 93 | ```
|
86 | 94 |
|
87 |
| -## Test Reproducibility Verification |
| 95 | +## Common Issues & Solutions |
88 | 96 |
|
89 |
| -Use this same workflow when CI tests fail due to reproducibility issues: |
| 97 | +### Issue: `assert_assert_missing_number` Error |
90 | 98 |
|
91 |
| -**Process Name**: Test Reproducibility Verification |
| 99 | +**Cause**: JSON template has `helpers_assert_name: "assert_missing_number"` but template adds `assert_` prefix |
| 100 | +**Solution**: Change JSON to `helpers_assert_name: "missing_number"` so template generates `assert_missing_number` |
92 | 101 |
|
93 |
| -**When to Use**: |
| 102 | +### Issue: mypy Import Errors |
94 | 103 |
|
95 |
| -- CI test failures in reproducibility checks |
96 |
| -- Inconsistent test results between environments |
97 |
| -- Missing edge cases causing coverage gaps |
98 |
| -- Need to ensure 100% code coverage |
| 104 | +**Cause**: Regenerated helpers.py doesn't match test imports |
| 105 | +**Solution**: Use `make p-gen` (not poetry run) and verify JSON template is correct |
| 106 | + |
| 107 | +### Issue: Tests Fail After Regeneration |
| 108 | + |
| 109 | +**Expected**: Tests may fail if solution is incomplete (returns 0 or placeholder) |
| 110 | +**Action**: This is normal - focus on mypy passing, not test results |
99 | 111 |
|
100 | 112 | ## Success Criteria
|
101 | 113 |
|
102 |
| -- All tests pass with enhanced test cases |
103 |
| -- Minimum 12 comprehensive test cases per problem |
104 |
| -- Original solution code preserved |
105 |
| -- **Enhanced test cases in final test_solution.py** |
106 |
| -- JSON template updated for future regeneration |
107 |
| -- **100% code coverage including edge cases** |
| 114 | +- ✅ **mypy passes** with no errors (CRITICAL for CI) |
| 115 | +- ✅ **Test structure matches JSON template** exactly |
| 116 | +- ✅ **Original solution preserved** (user's code intact) |
| 117 | +- ✅ **helpers.py generated correctly** (no `assert_assert_` bugs) |
| 118 | +- ✅ **Reproducibility verified** (can regenerate consistently) |
| 119 | + |
| 120 | +## When to Use This Workflow |
| 121 | + |
| 122 | +- GitHub Actions CI failures due to mypy errors |
| 123 | +- Test reproducibility verification requests |
| 124 | +- Need to ensure test structure matches JSON template |
| 125 | +- CI test failures in reproducibility checks |
0 commit comments