|
1 | 1 | # Test Case Enhancement Rules
|
2 | 2 |
|
3 |
| -## Assistant Workflow for Adding Comprehensive Test Cases |
| 3 | +## Simple Enhancement Workflow |
4 | 4 |
|
5 |
| -### Automated Workflow (Recommended) |
| 5 | +When user requests test case enhancement: |
6 | 6 |
|
7 |
| -For single problem enhancement: |
| 7 | +### 1. Problem Resolution |
8 | 8 |
|
9 |
| -1. **Use automation script**: `./.amazonq/rules/test-case-enhancement/scripts/enhance_single_problem.sh <problem_name>` |
10 |
| -2. **Verify results**: `./.amazonq/rules/test-case-enhancement/scripts/verify_enhancement.sh <problem_name>` |
| 9 | +- Use active file context or user-provided problem name |
| 10 | +- If unclear, run: `poetry run python .templates/check_test_cases.py --threshold=10 --max=1` |
11 | 11 |
|
12 |
| -For batch enhancement: |
13 |
| - |
14 |
| -1. **Find problems**: `./.amazonq/rules/test-case-enhancement/scripts/find_problems_needing_enhancement.sh` |
15 |
| -2. **Batch enhance**: `./.amazonq/rules/test-case-enhancement/scripts/enhance_batch_problems.sh` |
16 |
| - |
17 |
| -### Manual Workflow (Fallback) |
18 |
| - |
19 |
| -When user requests to enhance test cases for a problem, the assistant will: |
20 |
| - |
21 |
| -### 1. Problem Resolution (Priority Order) |
22 |
| - |
23 |
| -- **FIRST**: Try to resolve from context - check active file path or user-provided problem name |
24 |
| -- **SECOND**: If context resolution fails, THEN run `poetry run python .templates/check_test_cases.py --threshold=10 --max=1` to auto-detect 1 problem with <10 test cases |
25 |
| -- **LAST**: If both above fail, ask user to explicitly specify problem name |
26 |
| - |
27 |
| -### 2. Test Case Generation |
28 |
| - |
29 |
| -- Read `leetcode/{problem_name}/README.md` for problem understanding |
30 |
| -- Analyze existing test cases in `leetcode/{problem_name}/tests.py` |
31 |
| -- Generate comprehensive test cases covering: |
32 |
| - - **Edge cases**: Empty inputs, single elements, boundary values |
33 |
| - - **Corner cases**: Maximum/minimum constraints, special patterns |
34 |
| - - **Normal cases**: Typical scenarios with varied complexity |
35 |
| - - **Error cases**: Invalid inputs (if applicable) |
36 |
| - |
37 |
| -### 3. Initial Validation |
38 |
| - |
39 |
| -- Run `make p-test PROBLEM={problem_name}` to verify current implementation |
40 |
| -- **If errors found**: |
41 |
| - - DO NOT update implementation automatically |
42 |
| - - Only update test cases if they're incorrect |
43 |
| - - If implementation seems wrong, ASK USER first before modifying |
44 |
| - |
45 |
| -### 4. JSON Template Update |
46 |
| - |
47 |
| -- Update corresponding `.templates/leetcode/json/{problem_name}.json` |
48 |
| -- Add new test cases to `test_cases` field in proper format |
49 |
| -- Maintain existing test structure and naming conventions |
50 |
| - |
51 |
| -### 5. Backup and Regeneration Process |
52 |
| - |
53 |
| -- **Backup**: Move `leetcode/{problem_name}/` to `.cache/leetcode/{problem_name}/` |
54 |
| -- **Regenerate**: Run `make p-gen PROBLEM={problem_name} FORCE=1` |
55 |
| -- **Lint check**: Run `make p-lint PROBLEM={problem_name}` |
56 |
| -- **Iterate**: If lint fails, update JSON and regenerate until passes |
57 |
| - |
58 |
| -### 6. Solution Preservation |
59 |
| - |
60 |
| -- Copy `solution.py` from backup to newly generated structure |
61 |
| -- Run `make p-test PROBLEM={problem_name}` to verify tests pass |
62 |
| -- **If tests fail**: Go back to step 4, update JSON, and iterate until passes |
63 |
| - |
64 |
| -### 7. Cleanup and Restore |
65 |
| - |
66 |
| -- **CRITICAL**: Remove entire newly generated `leetcode/{problem_name}/` directory |
67 |
| -- **CRITICAL**: Restore original structure from `.cache/leetcode/{problem_name}/` backup |
68 |
| -- **CRITICAL**: Only THEN copy enhanced `test_solution.py` from generated files to restored structure |
69 |
| -- **CRITICAL**: Preserve existing solution class parametrization - if original test had multiple solution classes, restore them |
70 |
| -- Verify final state with `make p-test PROBLEM={problem_name}` |
71 |
| -- Clean up backup directory after successful verification |
72 |
| - |
73 |
| -## Test Case Quality Standards |
74 |
| - |
75 |
| -### Coverage Requirements |
76 |
| - |
77 |
| -- **Minimum 10 test cases** per problem |
78 |
| -- **Total test cases after enhancement must exceed the threshold** used for detection (e.g., if threshold=10, final count must be >10) |
79 |
| -- **Edge cases**: 20-30% of total test cases |
80 |
| -- **Normal cases**: 50-60% of total test cases |
81 |
| -- **Corner cases**: 20-30% of total test cases |
82 |
| - |
83 |
| -### Test Case Categories |
84 |
| - |
85 |
| -#### Edge Cases |
86 |
| - |
87 |
| -- Empty inputs: `[]`, `""`, `None` |
88 |
| -- Single element: `[1]`, `"a"` |
89 |
| -- Boundary values: `[0]`, `[1]`, `[-1]` |
90 |
| -- Maximum/minimum constraints from problem description |
91 |
| - |
92 |
| -#### Corner Cases |
93 |
| - |
94 |
| -- Duplicate elements: `[1,1,1]` |
95 |
| -- Sorted/reverse sorted arrays: `[1,2,3]`, `[3,2,1]` |
96 |
| -- All same elements: `[5,5,5,5]` |
97 |
| -- Alternating patterns: `[1,0,1,0]` |
98 |
| - |
99 |
| -#### Normal Cases |
100 |
| - |
101 |
| -- Mixed positive/negative numbers |
102 |
| -- Various array sizes within constraints |
103 |
| -- Different data patterns and structures |
104 |
| -- Representative problem scenarios |
105 |
| - |
106 |
| -### JSON Format Requirements |
107 |
| - |
108 |
| -- Use single quotes for Python strings in test cases |
109 |
| -- Follow existing parametrize format |
110 |
| -- Maintain type hints in parametrize_typed |
111 |
| -- Ensure test_cases string is valid Python list syntax |
112 |
| -- **NEVER include custom solution classes** in test_imports - only import the main solution class specified in solution_class_name |
113 |
| -- **PRESERVE existing solution class parametrization** - if original test had multiple solution classes, restore them after JSON regeneration |
114 |
| - |
115 |
| -## Automation Scripts 🤖 |
116 |
| - |
117 |
| -**RECOMMENDED APPROACH**: Use prebuilt scripts in `.amazonq/rules/test-case-enhancement/scripts/` to streamline the enhancement process: |
118 |
| - |
119 |
| -### Find Problems Needing Enhancement |
| 12 | +### 2. Enhancement Process |
120 | 13 |
|
121 | 14 | ```bash
|
122 |
| -# Find problems with ≤10 test cases (default) |
123 |
| -./.amazonq/rules/test-case-enhancement/scripts/find_problems_needing_enhancement.sh |
124 |
| - |
125 |
| -# Custom threshold and max results |
126 |
| -./.amazonq/rules/test-case-enhancement/scripts/find_problems_needing_enhancement.sh 8 15 |
| 15 | +# Simple 4-step process: |
| 16 | +# 1. Update JSON template with more test cases (12-15 total) |
| 17 | +# 2. Backup original |
| 18 | +mv leetcode/{problem_name} .cache/leetcode/{problem_name} |
| 19 | +# 3. Regenerate with enhanced tests |
| 20 | +make p-gen PROBLEM={problem_name} FORCE=1 && make p-lint PROBLEM={problem_name} |
| 21 | +# 4. Restore original solution, keep enhanced tests |
| 22 | +cp .cache/leetcode/{problem_name}/solution.py leetcode/{problem_name}/solution.py |
127 | 23 | ```
|
128 | 24 |
|
129 |
| -### Enhance Single Problem |
| 25 | +### 3. Verification |
| 26 | + |
| 27 | +- Run `make p-test PROBLEM={problem_name}` |
| 28 | +- Fix any incorrect expected values in test cases |
| 29 | +- Update JSON template with corrections |
| 30 | + |
| 31 | +### 4. Restore Backup |
130 | 32 |
|
131 | 33 | ```bash
|
132 |
| -# Enhance specific problem (handles full workflow automatically) |
133 |
| -./.amazonq/rules/test-case-enhancement/scripts/enhance_single_problem.sh <problem_name> |
| 34 | +# Copy enhanced test_solution.py to backup |
| 35 | +cp leetcode/{problem_name}/test_solution.py .cache/leetcode/{problem_name}/ |
| 36 | +# Restore all original files (preserves user edits) |
| 37 | +rm -rf leetcode/{problem_name} |
| 38 | +mv .cache/leetcode/{problem_name} leetcode/{problem_name} |
134 | 39 | ```
|
135 | 40 |
|
136 |
| -### Batch Enhancement |
| 41 | +## Test Case Standards |
137 | 42 |
|
138 |
| -```bash |
139 |
| -# Enhance up to 10 problems with ≤10 test cases (default) |
140 |
| -./.amazonq/rules/test-case-enhancement/scripts/enhance_batch_problems.sh |
| 43 | +### Coverage Requirements |
141 | 44 |
|
142 |
| -# Custom batch size and threshold |
143 |
| -./.amazonq/rules/test-case-enhancement/scripts/enhance_batch_problems.sh 5 8 |
144 |
| -``` |
| 45 | +- **Minimum 12 test cases** per problem |
| 46 | +- **Edge cases**: Empty inputs, single elements, boundary values |
| 47 | +- **Corner cases**: Maximum/minimum constraints, duplicates, sorted arrays |
| 48 | +- **Normal cases**: Mixed scenarios with varied complexity |
145 | 49 |
|
146 |
| -### Verify Enhancement Results |
| 50 | +### JSON Format |
147 | 51 |
|
148 |
| -```bash |
149 |
| -# Verify enhancement was successful |
150 |
| -./.amazonq/rules/test-case-enhancement/scripts/verify_enhancement.sh <problem_name> |
151 |
| -``` |
| 52 | +- Use single quotes for Python strings: `'hello'` not `"hello"` |
| 53 | +- Follow existing parametrize format |
| 54 | +- Ensure valid Python list syntax in test_cases field |
152 | 55 |
|
153 |
| -## Manual Commands Reference |
| 56 | +## Quick Commands |
154 | 57 |
|
155 | 58 | ```bash
|
156 |
| -# Find problems needing more test cases |
157 |
| -poetry run python .templates/check_test_cases.py --threshold=10 --max=1 |
| 59 | +# Find problems needing enhancement |
| 60 | +poetry run python .templates/check_test_cases.py --threshold=10 |
158 | 61 |
|
159 | 62 | # Test specific problem
|
160 | 63 | make p-test PROBLEM={problem_name}
|
161 | 64 |
|
162 | 65 | # Generate from JSON template
|
163 | 66 | make p-gen PROBLEM={problem_name} FORCE=1
|
164 | 67 |
|
165 |
| -# Lint specific problem |
| 68 | +# Lint check |
166 | 69 | make p-lint PROBLEM={problem_name}
|
167 | 70 | ```
|
168 | 71 |
|
169 |
| -## Error Handling |
170 |
| - |
171 |
| -- **Implementation errors**: Ask user before modifying solution code |
172 |
| -- **Test failures**: Update JSON template and regenerate |
173 |
| -- **Lint failures**: Fix JSON format and iterate |
174 |
| -- **Backup failures**: Ensure `.cache/leetcode/` directory exists |
175 |
| - |
176 | 72 | ## Success Criteria
|
177 | 73 |
|
178 | 74 | - All tests pass with enhanced test cases
|
179 |
| -- Total test cases exceed the threshold used for detection |
180 |
| -- Minimum 10 comprehensive test cases per problem |
181 |
| -- Original solution code preserved and working |
| 75 | +- Minimum 12 comprehensive test cases per problem |
| 76 | +- Original solution code preserved |
| 77 | +- **Enhanced test cases in final test_solution.py** |
182 | 78 | - JSON template updated for future regeneration
|
183 |
| -- Clean final state with no temporary files |
184 |
| - |
185 |
| -## Automation Benefits |
186 |
| - |
187 |
| -Using the prebuilt scripts provides several advantages: |
188 |
| - |
189 |
| -### ⚡ **Speed & Efficiency** |
190 |
| - |
191 |
| -- **Single command execution** - No need to remember complex multi-step workflows |
192 |
| -- **Batch processing** - Enhance multiple problems simultaneously |
193 |
| -- **Automatic cleanup** - No manual file management required |
194 |
| - |
195 |
| -### 🛡️ **Safety & Reliability** |
196 |
| - |
197 |
| -- **Automatic backups** - Original code always preserved |
198 |
| -- **Error handling** - Graceful failure recovery |
199 |
| -- **Validation checks** - Syntax and structure verification |
200 |
| - |
201 |
| -### 📊 **Visibility & Tracking** |
202 |
| - |
203 |
| -- **Progress reporting** - Real-time status updates |
204 |
| -- **Result verification** - Automatic test case counting |
205 |
| -- **Success metrics** - Clear pass/fail indicators |
206 |
| - |
207 |
| -### 🔄 **Consistency** |
208 |
| - |
209 |
| -- **Standardized process** - Same workflow every time |
210 |
| -- **Quality assurance** - Built-in lint and validation checks |
211 |
| -- **Reproducible results** - Eliminates human error |
212 |
| - |
213 |
| -**Recommendation**: Always use automation scripts for test case enhancement unless debugging specific issues requires manual intervention. |
0 commit comments