Setting up Python Projects: Part IV | by Johannes Schmidt
Whether you’re a seasoned developer or just getting started with 🐍 Python, it’s important to know how to build robust and maintainable projects. This tutorial will guide you through the process of setting up a Python project using some of the most popular and effective tools in the industry. You will learn how to use GitHub and GitHub Actions for version control and continuous integration, as well as other tools for testing, documentation, packaging and distribution. The tutorial is inspired by resources such as Hypermodern Python and Best Practices for a new Python project. However, this is not the only way to do things and you might have different preferences or opinions. The tutorial is intended to be beginner-friendly but also cover some advanced topics. In each section, you will automate some tasks and add badges to your project to show your progress and achievements.
The repository for this series can be found at github.com/johschmidt42/python-project-johannes
- OS: Linux, Unix, macOS, Windows (WSL2 with e.g. Ubuntu 20.04 LTS)
- Tools: python3.10, bash, git, tree
- Version Control System (VCS) Host: GitHub
- Continuous Integration (CI) Tool: GitHub Actions
It is expected that you are familiar with the versioning control system (VCS) git. If not, here’s a refresher for you: Introduction to Git
Commits will be based on best practices for git commits & Conventional commits. There is the conventional commit plugin for PyCharm or a VSCode Extension that help you to write commits in this format.
Overview
Structure
- Documentation framework (mkdocs, diataxis)
- Configuration (mkdocs.yml)
- Build documentation locally (index.html)
- GitHub Pages (gh-pages)
- CI (pages.yml)
- Docstrings (mkdocstrings)
- Badge (Documentation)
- Bonus (Plugins: Swagger)
As developers, we love writing code. But code alone can be difficult to comprehend sometimes. That’s why we need to make our code readable, usable and understandable for others who might encounter it. Whether we have customers or colleagues who require documentation, or whether we just want to help our future selves in a few months, we should document our code! It will make our lives easier and our code better, trust me!
There are tools that allow us to generate really nice-looking and modern documentation from Markdown files and docstrings automatically. These tools reduce the effort as we link the already existing information in the code and the pages that we manually create. In this series, we introduced fastAPI, a REST API framework that uses the mkdocs library as its documentation framework. Their documentation pages/static website looks like this:
If you think that this documentation looks really good and are interested in setting up you own documentation with mkdocs and the material theme, follow along! You don’t need any frontend development skills to build a stunning documentation. You can see the final result here.
sphinx is another popular documentation library, but it uses reStructuredText instead of markdown as the default plaintext markup language. I personally prefer mkdocs for that reason.
So let’s get started by creating a new branch: feat/docs
Create a new dependency group called docs and add the mkdocs library and the material theme to it. We use a separate group because we only want to use the libraries that are needed to create the documentation in our CI pipeline.
> poetry add --group docs mkdocs mkdocs-material
For our landing page, we must create a Markdown file index.md that gives a short description of the project and allows us to navigate to other pages (markdown files). I will follow the best practices for project documentation as described by Daniele Procida in the Diataxis documentation framework Therefore, besides the index.md I will create four additional markdown files in the docs directory:
To create our landing page, we need a Markdown file called index.md that gives a brief overview of the project and links to other pages (markdown files). I will use the best practices for project documentation from Daniele Procida’s Diataxis documentation framework. So, besides the index.md, I will make four more markdown files in the docs directory:
> mkdir docs
> cd docs/ && tree
.
├── explanation.md
├── how-to-guides.md
├── index.md
├── reference.md
└── tutorials.md
Each file will be filled with some text in markdown. So the content of the index.md could look like this:
The markdown pages are referenced in this file.
To build documentation based on these files, we need to add yet another configuration file where we set a few options: mkdocs.yml
This file lets us set the navigation tab, the site name, the theme, the option to use directory urls and more. We will also add plugins (mkdocstrings etc.) to this yml file later to get more cool features in our documentation page. Don’t forget to check out the Bonus part at the bottom!
Instead of building the navigation ourselves, we could simply point to the folder where our documentation is stored and let it be generated automatically:
Building the site locally is as simple as running:
> mkdocs build
INFO - Cleaning site directory
INFO - Building documentation to directory: /Users/johannes/workspace/python-project-johannes/site
INFO - Documentation built in 0.40 seconds
This makes a directory called site that has an index.html file. We can open it in our browser and see our static documentation site:
We can also navigate through the pages that we created:
Now we have a basic but nice-looking documentation that we can see locally. Let’s share it with everyone by deploying the site content with GitHub Pages (to the gh-pages branch). mkdocs makes this very easy for us, so we just need to run
> mkdocs gh-deploy -m "docs: update documentation" -v --force
which returns information about the steps being performed:
...
INFO - Documentation built in 0.55 seconds
WARNING - Version check skipped: No version specified in previous deployment.
INFO - Copying '/Users/johannes/workspace/python-project-johannes/site' to 'gh-pages' branch and pushing to GitHub.
Enumerating objects: 55, done.
Counting objects: 100% (55/55), done.
Delta compression using up to 8 threads
Compressing objects: 100% (51/51), done.
Writing objects: 100% (55/55), 473.92 KiB | 3.18 MiB/s, done.
Total 55 (delta 8), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (8/8), done.
remote:
remote: Create a pull request for 'gh-pages' on GitHub by visiting:
remote: https://github.com/johschmidt42/python-project-johannes/pull/new/gh-pages
remote:
remote: GitHub found 1 vulnerability on johschmidt42/python-project-johannes's default branch (1 moderate). To find out more, visit:
remote: https://github.com/johschmidt42/python-project-johannes/security/dependabot/1
remote:
To github.com:johschmidt42/python-project-johannes.git
* [new branch] gh-pages -> gh-pages
INFO - Your documentation should shortly be available at: https://johschmidt42.github.io/python-project-johannes/
It tells us that the build artifacts (html, css files etc.) were generated and pushed to the remote branch gh-pages. That is the default branch, but we can name it whatever we want, e.g. my-super-cool-branch-for-docs . Behind the scenes, mkdocs will use the ghp-import tool to commit them to the gh-pages branch and push the gh-pages branch to GitHub. After a short time, our site should be available at
https://johschmidt42.github.io/python-project-johannes/
If the site doesn’t show up when you open the URL in a browser, you need to do these steps in your Settings section of the Github repository:
This is all very well explained in GitHub Pages & MkDocs deployment.
We can check that the branch is created and the content has some website files (html files etc.).
Let’s put these new commands in our Makefile before we move on to CI/CD:
# Makefile...
##@ Documentation
docs-build: ## build documentation locally
@mkdocs build
docs-deploy: ## build & deploy documentation to "gh-pages" branch
@mkdocs gh-deploy -m "docs: update documentation" -v --force
...
clean-docs: ## remove output files from mkdocs
@rm -rf site
To keep our documentation updated, we need a GitHub actions workflow that runs these commands every time we commit to our default branch. We can create a workflow by creating a file called .github/workflow/pages.yml and adding some content similar to what we did in lint.yml or test.yml.
The documentation is updated by a build & deploy job when we merge a PR to the main branch. Awesome!
We can also use docstrings in our code to generate documentation with the mkdocs plugin mkdocstrings. Let’s make a new branch: feat/docs-docstrings and add the library to our docs group:
> poetry add --group docs "mkdocstrings[python]"
To use docstrings in the source code to appear in the documentation, we need to first create docstrings! We will follow the Google python docstrings style, as it is my favourite docstrings style and also supported with the plugin. Please note that it does not make much sense to add docstrings to fastAPI endpoints, because the documentation for these endpoints should be provided in the decorator function parameters. However, we will create docstrings for app.py anyway and additionally create another file service.py for demonstration purposes.
Our app.py looks like this now:
and we created on the same level as the app.py the src/example_app/service.py with this very generic content, that has some docstring tests:
In our pyproject.toml we now have the mkdocstrings in our dependency group docs. Please note that we have added docstring-tests in the src code (service.py ) but pytest would only look for tests in the tests directory. That’s because the attribute testpaths only points to the tests directory. So we have to update this part by adding the src directory as input for pytest to look for tests as well. Our pyproject.toml is updated like this:
# pyproject.toml[tool.poetry.group.docs.dependencies]
mkdocs = "^1.3.1"
mkdocs-material = "^8.4.3"
mkdocstrings = {extras = ["python"], version = "^0.19.0"}
...
[tool.pytest.ini_options]
testpaths = ["src", "tests"]
addopts = "-p no:cacheprovider" # deactivating pytest caching.
But that’s not the only change, we need to do. By default, pytest does not look for docstring tests, we also need to add the --docstest-modules flag when running pytest.
So our Makefile will be updated to this:
# Makefileunit-tests:
@pytest --doctest-modules
unit-tests-cov:
@pytest --doctest-modules --cache-clear --cov=src --junitxml=pytest.xml --cov-report=html --cov-report term-missing | tee pytest-coverage.txt
...
Our test.yml does not need to change as we’re using these commands that we just updated.
Ok, so we have added docstrings & included them in our testing pipeline. Now we can semi-automatically generate documentation based on the source code. I say semi-automatic because we have to do the following step. We need to add some blocks with a special ::: notation in one of the markdown files for the documentation. This tells mkdocstrings which files to use for autodocs. I will use the references.md file, which is for the technical documentation, and add these blocks for example_app.app.py and example_app.service.py:
# docs/reference.md...
::: example_app.app
options:
show_root_heading: true
::: example_app.service
options:
show_root_heading: true
......
Because our handler (python) needs to find the module example_app, we can conveniently add this information in the mkdocs.yml:
# mkdocs.yml...
plugins:
- mkdocstrings:
handlers:
python:
paths: [src]
...
The path points to the src directory that contains the package example_app. There is a better explanation on how to find the modules in the “finding modules” documentation.
We are ready to create documentation from our source code. We just need to build the documentation and deploy it to the gh-pages branch with the CI (commit to main)
Before we jump to the last section, we shouldn’t forget to add a search bar to our site and add the Github URL next to it:
# mkdocs.yml...
plugins:
- search:
- mkdocstrings:
handlers:
python:
paths: [src]
repo_url: https://github.com/johschmidt42/python-project
And now we see both features on our documentation site:
To explore the full potential of the material theme, let me demonstrate some of the features that we can leverage:
The features section of the material theme in the mkdocs.yml file is used to enable or disable specific features of the theme.
In this case, the following features are enabled:
navigation.tabs: This feature enables tabs in the navigation bar.navigation.indexes: This feature enables indexes in the navigation bar.navigation.instant: This feature enables instant search in the navigation bar.
The result looks like this now:
There is only the obligatory badge adding process for us to be left for the core of this section.
To get the badge, we can click on a workflow run
and select the main branch. The badge markdown can be copied and added to the README.md:
Our landing page of the GitHub now looks like this ❤:
If you are curious about how this badge reflects the latest status of the pipeline run in the main branch, you can check out the statuses API on GitHub.
Mastering the Art of Python Project Setup: A Step-by-Step Guide
12 min read
Oct 21, 2022
Whether you’re a seasoned developer or just getting started with 🐍 Python, it’s important to know how to build robust and maintainable projects. This tutorial will guide you through the process of setting up a Python project using some of the most popular and effective tools in the industry. You will learn how to use GitHub and GitHub Actions for version control and continuous integration, as well as other tools for testing, documentation, packaging and distribution. The tutorial is inspired by resources such as Hypermodern Python and Best Practices for a new Python project. However, this is not the only way to do things and you might have different preferences or opinions. The tutorial is intended to be beginner-friendly but also cover some advanced topics. In each section, you will automate some tasks and add badges to your project to show your progress and achievements.
The repository for this series can be found at github.com/johschmidt42/python-project-johannes
- OS: Linux, Unix, macOS, Windows (WSL2 with e.g. Ubuntu 20.04 LTS)
- Tools: python3.10, bash, git, tree
- Version Control System (VCS) Host: GitHub
- Continuous Integration (CI) Tool: GitHub Actions
It is expected that you are familiar with the versioning control system (VCS) git. If not, here’s a refresher for you: Introduction to Git
Commits will be based on best practices for git commits & Conventional commits. There is the conventional commit plugin for PyCharm or a VSCode Extension that help you to write commits in this format.
Overview
Structure
- Documentation framework (mkdocs, diataxis)
- Configuration (mkdocs.yml)
- Build documentation locally (index.html)
- GitHub Pages (gh-pages)
- CI (pages.yml)
- Docstrings (mkdocstrings)
- Badge (Documentation)
- Bonus (Plugins: Swagger)
As developers, we love writing code. But code alone can be difficult to comprehend sometimes. That’s why we need to make our code readable, usable and understandable for others who might encounter it. Whether we have customers or colleagues who require documentation, or whether we just want to help our future selves in a few months, we should document our code! It will make our lives easier and our code better, trust me!
There are tools that allow us to generate really nice-looking and modern documentation from Markdown files and docstrings automatically. These tools reduce the effort as we link the already existing information in the code and the pages that we manually create. In this series, we introduced fastAPI, a REST API framework that uses the mkdocs library as its documentation framework. Their documentation pages/static website looks like this:
If you think that this documentation looks really good and are interested in setting up you own documentation with mkdocs and the material theme, follow along! You don’t need any frontend development skills to build a stunning documentation. You can see the final result here.
sphinx is another popular documentation library, but it uses reStructuredText instead of markdown as the default plaintext markup language. I personally prefer mkdocs for that reason.
So let’s get started by creating a new branch: feat/docs
Create a new dependency group called docs and add the mkdocs library and the material theme to it. We use a separate group because we only want to use the libraries that are needed to create the documentation in our CI pipeline.
> poetry add --group docs mkdocs mkdocs-material
For our landing page, we must create a Markdown file index.md that gives a short description of the project and allows us to navigate to other pages (markdown files). I will follow the best practices for project documentation as described by Daniele Procida in the Diataxis documentation framework Therefore, besides the index.md I will create four additional markdown files in the docs directory:
To create our landing page, we need a Markdown file called index.md that gives a brief overview of the project and links to other pages (markdown files). I will use the best practices for project documentation from Daniele Procida’s Diataxis documentation framework. So, besides the index.md, I will make four more markdown files in the docs directory:
> mkdir docs
> cd docs/ && tree
.
├── explanation.md
├── how-to-guides.md
├── index.md
├── reference.md
└── tutorials.md
Each file will be filled with some text in markdown. So the content of the index.md could look like this:
The markdown pages are referenced in this file.
To build documentation based on these files, we need to add yet another configuration file where we set a few options: mkdocs.yml
This file lets us set the navigation tab, the site name, the theme, the option to use directory urls and more. We will also add plugins (mkdocstrings etc.) to this yml file later to get more cool features in our documentation page. Don’t forget to check out the Bonus part at the bottom!
Instead of building the navigation ourselves, we could simply point to the folder where our documentation is stored and let it be generated automatically:
Building the site locally is as simple as running:
> mkdocs build
INFO - Cleaning site directory
INFO - Building documentation to directory: /Users/johannes/workspace/python-project-johannes/site
INFO - Documentation built in 0.40 seconds
This makes a directory called site that has an index.html file. We can open it in our browser and see our static documentation site:
We can also navigate through the pages that we created:
Now we have a basic but nice-looking documentation that we can see locally. Let’s share it with everyone by deploying the site content with GitHub Pages (to the gh-pages branch). mkdocs makes this very easy for us, so we just need to run
> mkdocs gh-deploy -m "docs: update documentation" -v --force
which returns information about the steps being performed:
...
INFO - Documentation built in 0.55 seconds
WARNING - Version check skipped: No version specified in previous deployment.
INFO - Copying '/Users/johannes/workspace/python-project-johannes/site' to 'gh-pages' branch and pushing to GitHub.
Enumerating objects: 55, done.
Counting objects: 100% (55/55), done.
Delta compression using up to 8 threads
Compressing objects: 100% (51/51), done.
Writing objects: 100% (55/55), 473.92 KiB | 3.18 MiB/s, done.
Total 55 (delta 8), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (8/8), done.
remote:
remote: Create a pull request for 'gh-pages' on GitHub by visiting:
remote: https://github.com/johschmidt42/python-project-johannes/pull/new/gh-pages
remote:
remote: GitHub found 1 vulnerability on johschmidt42/python-project-johannes's default branch (1 moderate). To find out more, visit:
remote: https://github.com/johschmidt42/python-project-johannes/security/dependabot/1
remote:
To github.com:johschmidt42/python-project-johannes.git
* [new branch] gh-pages -> gh-pages
INFO - Your documentation should shortly be available at: https://johschmidt42.github.io/python-project-johannes/
It tells us that the build artifacts (html, css files etc.) were generated and pushed to the remote branch gh-pages. That is the default branch, but we can name it whatever we want, e.g. my-super-cool-branch-for-docs . Behind the scenes, mkdocs will use the ghp-import tool to commit them to the gh-pages branch and push the gh-pages branch to GitHub. After a short time, our site should be available at
https://johschmidt42.github.io/python-project-johannes/
If the site doesn’t show up when you open the URL in a browser, you need to do these steps in your Settings section of the Github repository:
This is all very well explained in GitHub Pages & MkDocs deployment.
We can check that the branch is created and the content has some website files (html files etc.).
Let’s put these new commands in our Makefile before we move on to CI/CD:
# Makefile...
##@ Documentation
docs-build: ## build documentation locally
@mkdocs build
docs-deploy: ## build & deploy documentation to "gh-pages" branch
@mkdocs gh-deploy -m "docs: update documentation" -v --force
...
clean-docs: ## remove output files from mkdocs
@rm -rf site
To keep our documentation updated, we need a GitHub actions workflow that runs these commands every time we commit to our default branch. We can create a workflow by creating a file called .github/workflow/pages.yml and adding some content similar to what we did in lint.yml or test.yml.
The documentation is updated by a build & deploy job when we merge a PR to the main branch. Awesome!
We can also use docstrings in our code to generate documentation with the mkdocs plugin mkdocstrings. Let’s make a new branch: feat/docs-docstrings and add the library to our docs group:
> poetry add --group docs "mkdocstrings[python]"
To use docstrings in the source code to appear in the documentation, we need to first create docstrings! We will follow the Google python docstrings style, as it is my favourite docstrings style and also supported with the plugin. Please note that it does not make much sense to add docstrings to fastAPI endpoints, because the documentation for these endpoints should be provided in the decorator function parameters. However, we will create docstrings for app.py anyway and additionally create another file service.py for demonstration purposes.
Our app.py looks like this now:
and we created on the same level as the app.py the src/example_app/service.py with this very generic content, that has some docstring tests:
In our pyproject.toml we now have the mkdocstrings in our dependency group docs. Please note that we have added docstring-tests in the src code (service.py ) but pytest would only look for tests in the tests directory. That’s because the attribute testpaths only points to the tests directory. So we have to update this part by adding the src directory as input for pytest to look for tests as well. Our pyproject.toml is updated like this:
# pyproject.toml[tool.poetry.group.docs.dependencies]
mkdocs = "^1.3.1"
mkdocs-material = "^8.4.3"
mkdocstrings = {extras = ["python"], version = "^0.19.0"}
...
[tool.pytest.ini_options]
testpaths = ["src", "tests"]
addopts = "-p no:cacheprovider" # deactivating pytest caching.
But that’s not the only change, we need to do. By default, pytest does not look for docstring tests, we also need to add the --docstest-modules flag when running pytest.
So our Makefile will be updated to this:
# Makefileunit-tests:
@pytest --doctest-modules
unit-tests-cov:
@pytest --doctest-modules --cache-clear --cov=src --junitxml=pytest.xml --cov-report=html --cov-report term-missing | tee pytest-coverage.txt
...
Our test.yml does not need to change as we’re using these commands that we just updated.
Ok, so we have added docstrings & included them in our testing pipeline. Now we can semi-automatically generate documentation based on the source code. I say semi-automatic because we have to do the following step. We need to add some blocks with a special ::: notation in one of the markdown files for the documentation. This tells mkdocstrings which files to use for autodocs. I will use the references.md file, which is for the technical documentation, and add these blocks for example_app.app.py and example_app.service.py:
# docs/reference.md...
::: example_app.app
options:
show_root_heading: true
::: example_app.service
options:
show_root_heading: true
......
Because our handler (python) needs to find the module example_app, we can conveniently add this information in the mkdocs.yml:
# mkdocs.yml...
plugins:
- mkdocstrings:
handlers:
python:
paths: [src]
...
The path points to the src directory that contains the package example_app. There is a better explanation on how to find the modules in the “finding modules” documentation.
We are ready to create documentation from our source code. We just need to build the documentation and deploy it to the gh-pages branch with the CI (commit to main)
Before we jump to the last section, we shouldn’t forget to add a search bar to our site and add the Github URL next to it:
# mkdocs.yml...
plugins:
- search:
- mkdocstrings:
handlers:
python:
paths: [src]
repo_url: https://github.com/johschmidt42/python-project
And now we see both features on our documentation site:
To explore the full potential of the material theme, let me demonstrate some of the features that we can leverage:
The features section of the material theme in the mkdocs.yml file is used to enable or disable specific features of the theme.
In this case, the following features are enabled:
navigation.tabs: This feature enables tabs in the navigation bar.navigation.indexes: This feature enables indexes in the navigation bar.navigation.instant: This feature enables instant search in the navigation bar.
The result looks like this now:
There is only the obligatory badge adding process for us to be left for the core of this section.
To get the badge, we can click on a workflow run
and select the main branch. The badge markdown can be copied and added to the README.md:
Our landing page of the GitHub now looks like this ❤:
If you are curious about how this badge reflects the latest status of the pipeline run in the main branch, you can check out the statuses API on GitHub.
Denial of responsibility! Techno Blender is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – [email protected]. The content will be deleted within 24 hours.