Skip to content

Conversation

@marioevz
Copy link
Member

@marioevz marioevz commented Jun 7, 2025

🗒️ Description

This PR introduces the EIPChecklist class which contains a list of all EIP checklist items for easier marking of tests that cover an item in the EIP checklist.

Calling any value in this class directly returns the marker which can be used in the tests or marks parameter of the parametrize object:

import pytest
from ethereum_test_tools import StateTestFiller
from ethereum_test_checklists import EIPChecklist

@EIPChecklist.TransactionType.Test.IntrinsicValidity.GasLimit.Exact()
def test_exact_intrinsic_gas(state_test: StateTestFiller):
    """Test transaction with exact intrinsic gas limit."""
    # Test implementation
    pass

It also fixes a breaking bug in the checklist generation code that marked all EIPs for a given test id when a marker was used anywhere in any test in any folder (whoops).

Suggestions for improvement of the usage of this new class are welcome.

🔗 Related Issues

✅ Checklist

  • All: Set appropriate labels for the changes.
  • All: Considered squashing commits to improve commit history.
  • All: Added an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

@marioevz marioevz requested a review from danceratopz June 7, 2025 13:41
@marioevz marioevz force-pushed the eip-checklist-enum branch from 36ad5a4 to 3ad8a32 Compare June 9, 2025 15:02
Copy link
Member

@danceratopz danceratopz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The nested classes are insane 😄 but LGTM, thanks!

@danceratopz danceratopz added type:feat type: Feature scope:fill Scope: fill command labels Jun 10, 2025
@danceratopz danceratopz changed the title feat(checklists): Add EIPChecklist enum to access EIP Checklist Items feat(checklists): add EIPChecklist enum to specify EIP checklist items Jun 10, 2025
@danceratopz danceratopz added the scope:checklists Scope: ethereum_test_checklists package label Jun 10, 2025
Copy link
Member

@danceratopz danceratopz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something's not quite right when running fill.

@danceratopz
Copy link
Member

Something's not quite right when running fill.
@marioevz I tried out the markers within the tests and found two issues:

Issue 1: Marker Without Parentheses Not Working

Problem: Using @EIPChecklist.Opcode.Test.StackOverflow (without parentheses) didn't work - pytest wouldn't execute the test because the marker wasn't being applied correctly.

Root Cause: The original implementation used __new__ in the ChecklistItem class, which only handled the parentheses case (). When used without parentheses, Python would call the
class directly but the marker wouldn't be applied to the test function.

Solution: Moved the marker logic from ChecklistItem.__new__ to ChecklistItemMeta.__call__ in the metaclass. This allows the metaclass to handle both cases:

  • @EIPChecklist.Item() - called with parentheses, returns a marker
  • @EIPChecklist.Item - called without parentheses as a decorator, applies the marker directly to the function

Issue 2: MyPy Type Checking Errors

Problem: MyPy showed "StackComplexOperations" not callable [operator] error when using @EIPChecklist.Opcode.Test.StackComplexOperations() because mypy couldn't understand that the
metaclass makes classes callable.

Root Cause: MyPy doesn't understand metaclass magic that makes classes callable at runtime. The ChecklistItemMeta metaclass provides __call__ but mypy can't infer this.

Solution: Created a comprehensive .pyi stub file that explicitly tells mypy all EIPChecklist classes are callable by inheriting from _CallableChecklistItem with proper @overload
signatures. Also built a CLI tool to auto-generate the stub file when the class structure changes.

Copy link
Member

@danceratopz danceratopz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yo! LGTM! Thanks, the IDE auto-completion really improves UX!

Copy link
Member

@danceratopz danceratopz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @marioevz, I just fixed-up the CI fails due to incorrect eip checklist ids (hopefully 🤞).

I didn't change the string ID format to the new enum format to avoid conflicts; they're probably best solved in the branches working on these new tests.

@danceratopz
Copy link
Member

I didn't change the string ID format to the new enum format to avoid conflicts; they're probably best solved in the branches working on these new tests.

Well at least to reduce them, if all the test PRs are merged we can of course address them here!

@marioevz marioevz merged commit 97b54c6 into main Jun 16, 2025
27 checks passed
@marioevz marioevz deleted the eip-checklist-enum branch June 16, 2025 15:39
danceratopz added a commit that referenced this pull request Aug 11, 2025
Also removes a file that was unintentionally added in #1718.
danceratopz added a commit that referenced this pull request Sep 1, 2025
Also removes a file that was unintentionally added in #1718.
kclowes pushed a commit to kclowes/execution-spec-tests that referenced this pull request Oct 20, 2025
…ems (ethereum#1718)

* fix: move unit test

* Create Enums - Incomplete/broken attempt

* fix(filler/checklist): Template copying bug

* fix: EIPChecklist usage

* docs: Update usage

* refactor(eip_checklist): Remove new_* from all IDs

* fix: tox (minus spellcheck)

* fix: typing

* fix(docs,checklists): Rename section to be more descriptive

* docs: remove references to string EIP checklist notation

* docs: backtick `EIPChecklist` instances

* docs: update changelog

* docs: fix title

* fix(checklists): allow ommission of parentheses in checklist markers

* style(checklists): add stubs to solve mypy issues using checklist markers

* test(checkmarks): add fw tests that apply EIPChecklists in pytest.param

* fix(docs): fix bad links to exception tests

* fix(checklists): update test markers to use correct checklist IDs

Update test files to use the correct checklist IDs that match the template:
- Change 'new_precompile' prefix to 'precompile'
- This fixes mkdocs build warnings about missing checklist items

The checklist template uses 'precompile/test/*' IDs but tests were using
'new_precompile/test/*' IDs, causing template lookup failures during
documentation generation.

Also add missing docstrings to fix linting errors.

---------

Co-authored-by: danceratopz <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

scope:checklists Scope: ethereum_test_checklists package scope:fill Scope: fill command type:feat type: Feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants