Migrate documentation to mkdocs

[vemonet]

Hi, I am opening this issue to start a discussion around migrating the documentation to a more modern solution, and a more user friendly state after discussing it with @nicholascar

Why material for mkdocs?

• Mature, actively maintained, used by many high profile python projects (fastapi, pydantic, ansible...)
• User friendly: modern and familiar look and feel with a user friendly navigation
• Contributor friendly: markdown based, good developer experience to run locally and deploy remotely
• Supported out-of-the-box, and even recommended, by readthedocs: https://docs.readthedocs.com/platform/stable/intro/mkdocs.html

Migration

I have started exploring how to migrate the documentation to mkdocs, without losing existing features:

• Reference to API docs for specific classes and functions can be done even more easily than with sphinx
  • :class:~rdflib.graph.Graph becomes [Graph][rdflib.graph.Graph]
  • Or [add()][rdflib.graph.Graph.add]
• Deploying the reference API docs from docstring for the whole project is slightly more verbose than previously, but enable better control over how it is displayed
  • ⚠️ by default mkdocstring will not automatically create new pages for submodules of a module. Instead it will put everything in the same page
  • ✅ the solution is a small python script that recursively go through all submodules and create new pages for each submodule with mkdocs-gen-files. These pages can be referenced in the mkdocs.yml nav but are never materialized in the repository, so we have control over the side navigation, while still getting all of the code automatically documented
    • Note that when running mkdocs it will print the list of pages that are not referenced in the nav (in our case we are fine with it because we expect the user to access top modules through the navigation, and then navigate to submodules through the top modules page).
    • We could also put everything in the nav, but that might make it a bit harder to read. There are options to automatically generate the navigation through the script, but in my opinion it is too cluttered, I prefer that this is manually defined for a better display and focus. Let me know what you think
• Importing a definition in a documentation page (previously done with .. autoclass:: rdflib.term.URIRef) can be done with ::: rdflib.term.URIRef.

  • But with mkdocs it adds a lot more things (e.g. properties), which makes the documentation less readable. We could improve it by changing the global mkdocstring options, or applying specific options to this specific reference:

    ::: rdflib.term.URIRef
        options:
          show_source: false
          show_bases: false
          show_category_heading: false
          show_if_no_docstring: false
          show_root_heading: false
          show_root_toc_entry: false
          inherited_members: false

  • I propose we just replace them by a link to the reference in the API doc, so if the user wants to see more about this function they just need to click. This will make the documentation more lean and readable

• Automated test of all code blocks in the documentation done with pytest-markdown-docs
  • just run poetry run pytest --markdown-docs
  • this is the only way to make sure your documentation examples are not broken (really simple and powerful concept, rust have it built-in the standard lib, it should be a default of pytest to be honest). And you get more tests for free
  • For now I have not yet made sure those tests passes, it will be a future work (it is disabled by default for now)
• Update docstring to Google style:
  • readable, and does not require to duplicate the type annotations (rdflib already has proper type annotations in the functions definitions)
  • Compliant with PEP 257
  • Well supported by modern IDEs
• Examples doc: in its current state it is quite hard to access. https://rdflib.readthedocs.io/en/stable/apidocs/examples.html only has the docstring, so users need to manually go to the various files in the examples folder in the source code (there are not even links to the source code from the documentation, so the journey to get there is more complex than it should be).
  • Maybe that would be a good moment to make it better accessible, e.g. directly include it as code blocks in markdown? In a Examples category in the top nav
  • Or completely review them because they kind of overlap with "regular" documentation for the main features of the lib

Questions

1. Would a migration to mkdocs be interesting for the project?
2. Any remarks on the proposed plan?
3. Any outstanding requests? (modifications to the docs you would like to see while we do this migration)

I just converted a handful of files to markdown, the next step will be to convert all docstring and documentation pages from rst to markdown

I have pushed the changes to this branch, I will create a PR when it will be a bit clearer which direction to take

• Setup mkdocs configuration: vemonet@ded6528
• Add automated docs tests: vemonet@8cecbaa

You can deploy it locally with:

poetry run mkdocs serve

Some screenshots to give an idea of what it would look like:

[nicholascar]

1. Would a migration to mkdocs be interesting for the project?

Yes! I think this is a great idea. mkdocs is clearly the in vogue Python documentation system and I, and several other maintainers, use it elsewhere.

2. Any remarks on the proposed plan?

Go for it!

3. Any outstanding requests? (modifications to the docs you would like to see while we do this migration)

A couple:

• We need to ensure the examples and the tools are documented properly, both appearing in the docco menues and with reasonable descriptions for each example & tool. People so often repeat things done her like convert RDF to CSV. Docco will help stop this. Also, I'd like people to be able to easily add more examples, even tiny scripts, so if its easy to add a new .py file with basic documentation in it and see it appear in all the correct places, that would be great.

• what will we do with https://rdflib.readthedocs.io/?

  • If this is this forever locked to Sphinx, I guess we will just have to fix this at RDFLib 7.x - whatever the last v7 release is
  • we can perhaps put new docco that can't use readthedocs at https://rdflib.dev, which we already use (a little bit)

! looks like readthedocs supports mkdocs: https://about.readthedocs.com/tools/mkdocs/

[vemonet]

what will we do with https://rdflib.readthedocs.io/?
! looks like readthedocs supports mkdocs: https://about.readthedocs.com/tools/mkdocs/

As you noticed @nicholascar we can just keep readthedocs as it supports mkdocs, and https://rdflib.readthedocs.io/ is the number 1 result in google for rdflib search (for me in private mode at least)

To be fair I like more https://rdflib.dev because it's neater and shorter, but it does not appears at all in google results 1st page for now. That said, if you put the readthedocs page down, and update all links to use rdflib.dev, the search engines should be updated properly relatively fast (afaik mkdocs is well optimized for SEO).

From our experience migrating nanopub.org to nanopub.net took a few days (or weeks, not sure how much time, but not too long) in search engines, but now it is the first answer when searching for nanopublications

We need to ensure the examples and the tools are documented properly, both appearing in the docco menues and with reasonable descriptions for each example & tool. People so often repeat things done her like convert RDF to CSV. Docco will help stop this. Also, I'd like people to be able to easily add more examples, even tiny scripts, so if its easy to add a new .py file with basic documentation in it and see it appear in all the correct places, that would be great.

In my opinion it's 2 different use-cases:

• Tools are modules of rdflib, so they should be docstring documented just like the rest of rdflib (e.g. plugins) and we can add a "Tools" sections in the left navbar of the "API Reference" (under Plugins)
• Examples are about showing the code to the users on how to do specific things, so the whole example code should be clearly available on the documentation website (users should not have to go in the github interface to check the code or clone the repo).
  • So using .py files with docstring might not be the nicest solution here
  • I would propose to just convert all examples to .md files with big python codeblocks, and have an examples/ folder in the docs/ folder. This way they will be directly available and readable from the documentation website + users will be able to copy them in one click if they want to paste it in a file to run them (no dev wants to have to clone a huge repo just to access an example file) + we can add as much explanation as we want easily
  • pytest-markdown-docs should run these blocks as part of the tests, which will make sure the examples are working and not deprecated

• Importing a definition in a documentation page (previously done with .. autoclass:: rdflib.term.URIRef) can be done with ::: rdflib.term.URIRef.
  • But with mkdocs it adds a lot more things (e.g. properties), which makes the documentation less readable. We could improve it by changing the global mkdocstring options, or applying specific options to each specific reference
  • I propose we just replace them by a link to the reference in the API doc, so if the user wants to see more about this function they just need to click. This will make the documentation more lean and readable

What is your opinion on this issue? Should we just use links to the API reference in these cases, and stop trying to integrate docstrings directly in the main documentation?

Imo it makes the documentation a bit confusing over time and harder to maintain, because sometimes docstring will change without considering how it impacts the main documentation. Which might make the documentation harder to read and less consistent overall.

---

I plan to act in this order:

1. Migrate all docstrings to markdown with google style (without changing their content much, apart when obvious issues)
2. Migrate all .rst documentation pages to .md and replace .. autoclass:: references with direct links to the API Reference page
3. Convert existing examples .py scripts to .md without changing the essence of their content (might start to fix obvious issues though)
4. At this point we might consider to start sending a PR?
5. Run tests over the whole documentation and make sure every single documented example runs as expected

Not sure about the exact timeline though

[ajnelson-nist]

Question from someone unfamiliar with mkdocs, on your plan to convert example .py scripts to .md: How would the Python in the Markdown be tested for code drift? As Python files, they can be type-reviewed, auto-formatted, and smoke-tested. Is there testing that can still be done when the Python's split into blocks in a Markdown file?

[vemonet]

Good point @ajnelson-nist I did not think about this

• Testing: this package enables to run every python codeblocks in all .md files using pytest: https://github.com/modal-labs/pytest-markdown-docs
• Type checking: did not find any readily available tools. There might be some solutions where you pipe a tool that extract python codeblocks with mypy... But that might be a pain to maintain and add a lot of complexity
• Code formatting: there is some solution with mdformat + plugin mdformat-black but the plugin latest release is in 2020, so not sure it is still really working and sustainable

Considering this, it would be better to keep them in their .py files if we want to keep automated formatting and type checking, and there are 2 solutions to include them in the documentation:

1. just add a documentation page for Examples with direct links to the files on GitHub (simple and clean, code in GitHub UI is quite readable, sometimes we just want the code)
2. I could also try to include them directly in the docs page through the usual API reference, mkdocs has an option to include all of the module code in the reference. But I don't know how readable it will be (to be tested I guess?)

[ajnelson-nist]

Having an index of the examples with direct links to the files on GitHub sounds reasonable to me. Then, drift-monitoring between documentation and example files would be a matter of reviewing the Markdown for broken links. Thank you for the consideration, @vemonet .