Automate testing; add code coverage

[iaincollins]

There isn't any formal testing and at this point it's a defect in the project and should be a priority.

This issue is something of a blocker for #226 (automating publication of new releases).

The focus should probably be on having end-to-end acceptances tests at first, as those have more value than unit tests. We don't need to test everything, but we do need some sort of smoketest. Puppeteer seems like the best candidate for this.

Options

• Mocha + Chai
• Jest + Enzyme (great for testing React and Node.js projects in the same repo)
• Puppeteer

Scope

There is already a docker compose suite that spins up MySQL, Postgres and MongoDB instances.

• I think it would be sensible to test all those databases with two OAuth providers plus the email sign in flow (even if the email flow has to be partly mocked).
• We should also test the same flows with no database configured.
• Ideally we would also test with a database and JWT enabled (but if the above two flows work, this is likely to work also).
• We should also test the database schema for each of those databases; a snapshot test for those is probably sufficient, just so schema changes are flagged.

Feedback

As this is a topic that attracts a lot of bike-shedding I'm going to come out and say that we are really only looking for feedback from either existing contributors or from people interested in helping with authoring tests.

[JeffersonBledsoe]

Just wanted to chime in, I'd be up for helping with writing some E2E tests! I've yet to use next-auth, but plan to use it on a couple of personal projects. Would a fresh look on things be beneficial for test writing, or would you rather somebody with deeper knowledge of next-auth?

Could I also recommend taking a look at Cypress as another potential option for the E2E tests which may involve the browser. It's API is just as nice as Puppeteer, but I've found it to be more stable and has easier to write best-practise tests (mostly due to great documentation and a good community around it)

[iaincollins]

Just wanted to chime in, I'd be up for helping with writing some E2E tests! I've yet to use next-auth, but plan to use it on a couple of personal projects. Would a fresh look on things be beneficial for test writing, or would you rather somebody with deeper knowledge of next-auth?

That would be fantastic - a fresh pair of eyes would be really helpful!

I'm very happy to collaborate on this, to help where existing knowledge of next-auth is useful.

I would have started with tests but from a practical perspective I really needed to have a working solution for a couple of projects (as v1 isn't viable for serverless) and wouldn't have been able to do that as quickly; but I'd really like that to be the focus now, to give us confidence when working on changes post-release.

Could I also recommend taking a look at Cypress as another potential option for the E2E tests. It's API is just as nice as Puppeteer, but I've found it to be more stable and has easier to write best-practise tests (mostly due to great documentation and a good community around it)

Absolutely! That seems like a promising choice. I got a lot of value out of PhantomJS over the years and have been looking for a similar replacement since it's no longer supported.

Puppeteer is great but not specifically for testing; Cypress seems like it's what I've been missing!

[iaincollins]

PS: If you want to read about the horrible database "tests" (and I use that term in inverted commas) I've just been writing about them here #274

Because of the database dependancy and the behaviour between them varies, a thing I'd like to do is be able to run the entire test suite against multiple supported databases (e.g. MySQL, Postgres, MongoDB); the simplest way to do that locally is probably to run server that is tested against within a Docker image, so that the database ports don't have to be run on the host, and don't collide with anything already running locally.

I figure we could stub the OAuth provider response for a couple of easy-to-integrate-with providers (Google, Twitter…) or even just use the IdentityServer4 endpoint to get started?

https://next-auth.js.org/providers/identity-server4

[JeffersonBledsoe]

@iaincollins Any input on how to get started with this? Are you mostly looking for a full E2E setup (i.e. with Next.JS, API routes, etc.) initially, or more of an acceptance test all done in Node? If the latter, I'll make a start on something during the week based on comment on #274 and get a draft PR on the go :)

[iaincollins]

I was thinking E2E would be ideal, but I appreciate that is a huge undertaking.

Given where we are, anything would be very gratefully received!

[JeffersonBledsoe]

E2E works well for me! Any recommendations on how to setup next.js only for the tests? Any way to mock it, or should next.js be used as a dev dependency and some sort of test app created within this repo?

[iaincollins]

I have an idea about how I'd like to approach that actually!

If you are able to get a test working in the interim for https://next-auth-example.now.sh/ that would be wonderful! Doesn't matter if not full sign in flow - appreciate might be a bunch of work in scaffolding, but checking something like /api/auth/signin returns sign-in buttons would be helpful!

If you are able to do that, I am happy to work on a one-line command that can spin up Next.js + NextAuth.js + Database in a Docker, and expose it on a port, in such a way we should just be able to connect to it from the tests (e.g. http://localhost:8080). I think we probably should add it the code for that in repo (as a very simple project), but it would better to actually run it from inside a Docker container.

I say this not just because that way we don't also have to export database ports to the host - though that is one reason but also because NPM module behaviour gets tricky with repos inside other repos.

Because NPM automatically checks up the directory tree if there is a dependancy missing that should be declared then things seem to work locally but don't work when deployed (also true of things declared as devDependancies or peerDependancies that should actually be dependancies); linking also gets a bit annoying when doing this because of React.

I ran into into the above issues with v1 a couple of times - and it's why I ended up moving the example site into a separate repo this time.

Anyway, I'm thinking if we could do this, it would be an easy step to then add a GitHub action to have the tests run on every PR.

After that, am happy to look at how we can do mocking! I have some ideas about that, that would work with a setup like this. Specifically I'm thinking a dummy oAuth service - e.g. we emulate one by running a tiny web service on localhost:8080 in the Docker container, rather than have to mock function calls in NextAuth.js - that way we are able to test more of the app.

Does that sound like a decent plan? I hope this makes sense!

I appreciate there are some limitations of this approach, like getting code coverage might be next to impossible, so am open to push back if you have other ideas!

[JeffersonBledsoe]

I've started a draft PR on next-auth-example. Feel free to comment on the tiny bit that's there now/ adjust the scope of tests I've noted down for the PR!

[iaincollins]

Thank you! That is awesome! I'll leave comments there!