`nim doc` recursive should allow specifying a directory and flattened structure

[mratsim]

Currently using the nim doc for the global project via

nim doc --project --index:on --git.url: --git.commit: <main_filename>.nim
nim buildIndex -o:htmldocs/theindex.html htmldocs

will recursively generate doc for each Nim file of the project and put them next to the Nim files.

Providing the options to:

• Put all the docs in a target directory (keeping the original hierarchy)
• Put all the docs in a target directory (using a flattened hierarchy)

would help automating doc generation and their publication.

[timotheecour]

did you see #13223 (comment) ?
this works:

nim doc --project --index:on --outdir:htmldocs compiler/nim.nim
nim buildIndex -o:htmldocs/theindex.html htmldocs

#13223 made --project more reliable, as well as introducing --docroot but there are more things that need to be fixed:

• nim buildIndex -o:htmldocs/theindex.html htmldocs extra step shouldn't be needed and be implied by --index:on (or at least have option for that since it's what you'll often want to have and it's more DRY)
• fix a recent regression introduced sometime after fix #13150 nim doc --project now works reliably #13223 where --docroot would cause nimdoc.out.css to be misplaced
• dochack.js should be installed by default when you use --project so that search works by default
• I can't think of a valid use case for recursively generate doc for each Nim file of the project and put them next to the Nim files so IMO we should make --outdir:htmldocs be implied as a default (unless overridden by some flag, eg --outdir:xxx); in fact it's downright harmful to do so as it clobbers your sources (or worse, could potentially overwrite some files in your sources) and may be hard to gitignore if you have other html files etc; if really needed we can add an option (off by default) to output each generated file next to its source
• likewise doc2 outputs in current work dir #6583 regression is related (outputs to $pwd, which has similar drawbacks) ; I'm fixing that in a PR, where it'll probably default to htmldocs (like it did at some point in nim history IIRC)
• (EDIT) update https://nim-lang.github.io/Nim/docgen.html to reflect reality

nim doc is special because it installs other things (eg nimdoc.out.css, idx files, html files, and subdirs) so it really should be encapsulated inside htmldocs (by default) to avoid clobbering either pwd or source files

Put all the docs in a target directory (using a flattened hierarchy)

I had originally suggested that to deal with design issues with the old --project semantics (relative imports + module name clashes) but see discussion here #13150 (comment) which led to the new semantics in #13223

[mratsim]

Ah the docgen documentation is lagging behind

[timotheecour]

@mratsim #14324 should fix this (in particular all points mentioned in #14376 (comment)); see that comment for why we don't have a flattened structure option.
feel free to chime in