Running eslint-remote-tester-runner for each new version published

[Belco90]

@AriPerkkio I forgot to mention: another possibility would be running eslint-remote-tester-runner within this project without blocking the CI at all, that's up to us 😄 . So every time we release a new version, we trigger the remote tester runner in parallel and leave it doing its thing to get the report when gets finished. Is that possible? I guess so right?

Originally posted by @Belco90 in #272 (comment)

[Belco90]

@MichaelDeBoey @AriPerkkio moving conversation here 😅

As discussed I'd go for creating a new action file in this plugin which runs the remote tester. It would be executed on each new package published (not really sure how to hook the CI to this? New tag pushed?), and just by getting a failed pipeline would be enough for us to know something went wrong linting some repo.

Additionally, put the flag in our README.

[AriPerkkio]

and just by getting a failed pipeline would be enough for us to know something went wrong

The eslint-remote-tester-run-action will create a new issue in the repository it is configured in when errors are found. There's no need to check pipeline or its logs for errors. Example from eslint-plugin-unicorn: Results of weekly scheduled smoke test #1122

It would be executed on each new package published (not really sure how to hook the CI to this?

There's the release workflow event which can be used. So in the smoke-test.yml you would have:

For example, you can run a workflow when a release has been published.

on:
 release:
   types: [published]

[Belco90]

@AriPerkkio oooh I see! Even better then. I'll work on this after I get pending issues solved (unless @MichaelDeBoey wants to take care of it now)

[AriPerkkio]

Here is example of integration for reference:
jsx-eslint/eslint-plugin-react#2963

Following the setup from first commit should be enough.

When deciding which rules to include and which rule settings, try to cover as many branches of the code as possible. The configuration doesnt have to "make sense", e.g. you can mix react and angular rules in same setup. Goal is to test all rules as widely as possible.

[MichaelDeBoey]

@AriPerkkio is it also possible to somehow export a baseConfig that we can just extend, like you do in your own eslint-plugin-testing-library.config.js file please?

That way we don't have to bother about all the config stuff (like repositories, pathIgnorePattern, extensions, rulesUnderTesting, concurrentTasks, cache & default eslintrc stuff) and can just extend the eslintrc config with plugin & rules?

This can also be something that eslint-remote-tester-run-action handles by doing a deep merge of our config with the CI config, so you don't have export that baseConfig. (this is what I think is the best approach for it)

[AriPerkkio]

I'm not really planning on providing a base configuration - the default values of configuration are for that. Ideas for improving these are definitely welcome. 👍

For repositories and pathIgnorePattern there is eslint-remote-tester-repositories available. All that you'll need to configure is cache, extensions and eslintrc:

const { getRepositories, getPathIgnorePattern } = require('eslint-remote-tester-repositories');

module.exports = {
    repositories: getRepositories({ randomize: true }),
    pathIgnorePattern: getPathIgnorePattern(),
    cache: false, // This should probably be set by eslint-remote-tester-run-action automatically 🤔 
    extensions: ['js', 'jsx', 'ts', 'tsx'], // This should probably be default value in eslint-remote-tester 🤔 
    eslintrc: {
        // This is really plugin specific and should not be provided by eslint-remote-tester
    }
};

[MichaelDeBoey]

@AriPerkkio I would not suggest doing that in eslint-remote-tester, but rather in eslint-remote-tester-run-action.

I'll create an issue on the repo to discuss it further there.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.