Pre-commit hooks and linting configurations for Python projects

In this article, we'll configure pre-commit hooks to automatically lint our code before committing changes to Github.

Since we're using Python, we can add autopep8 to make our code align with PEP8 guidelines or if the project includes YAML files we can use related packages to check the validation of these files.

We'll go through a new project but you can also apply it to an existing codebase by following the same steps. So, create a new folder and also activate a virtual environment to isolate dependencies:

python3 -m venv venv
source venv/bin/activate

Initialize the project with git init

Then, start by installing pre-commit package for Python:

pip install pre-commit

We should create a YAML file at the root level of the project named .pre-commit-config.yaml which will include specific configurations or rules applied to the source code.

Next, run the following command to set up the git hook scripts which will automatically run once changes are committed by git commit command:

pre-commit install

As a simple configuration you can use below:

.pre-commit-config.yaml

repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v3.2.0
    hooks:
      - id: end-of-file-fixer
      - id: check-added-large-files
      - id: requirements-txt-fixer
      - id: check-json
      - id: check-merge-conflict
-   repo: https://github.com/pycqa/flake8
    rev: '4.0.1'
    hooks:
    -   id: flake8

Now, run pre-commit run --all-files to see it in action:

Fix End of Files.....................................(no files to check)Skipped
Check for added large files..........................(no files to check)Skipped
Fix requirements.txt.................................(no files to check)Skipped
Check JSON...........................................(no files to check)Skipped
Check for merge conflicts............................(no files to check)Skipped
flake8...............................................(no files to check)Skipped

Currently, we don't have any files to check so create a new Python file named main.py as below:

main.py

import os

def test():
    pass

In order to apply pre-commit hooks, we need to stage our changes with git add main.py

Run pre-commit run --all-files one more time:

Fix End of Files.........................................................Passed
Check for added large files..............................................Passed
Fix requirements.txt.................................(no files to check)Skipped
Check JSON...........................................(no files to check)Skipped
Check for merge conflicts................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

main.py:1:1: F401 'os' imported but unused
main.py:3:1: E302 expected 2 blank lines, found 1

So, it seems flake8 validated our code since we have repo mapping in our pre-commit configuration file:

-   repo: https://github.com/pycqa/flake8
    rev: '4.0.1'
    hooks:
    -   id: flake8

Some of the hooks are auto-fixing the invalid changes. If you leave more than one blank line at the end of the file and run pre-commit once again, the extra lines will be removed.

Let's add a few more linting tools for our code:

repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v3.2.0
    hooks:
      - id: end-of-file-fixer
      - id: check-added-large-files
      - id: requirements-txt-fixer
      - id: check-json
      - id: check-merge-conflict
-   repo: https://github.com/pycqa/flake8
    rev: '4.0.1'
    hooks:
    -   id: flake8

-   repo: https://github.com/pre-commit/mirrors-isort
    rev: v4.3.21
    hooks:
    -   id: isort
-   repo: https://github.com/asottile/pyupgrade
    rev: v1.17.1
    hooks:
    -   id: pyupgrade
        args: ['--py3-plus', '--py36-plus']
-   repo: https://github.com/pre-commit/mirrors-autopep8
    rev: 'v1.5.7'
    hooks:
    -   id: autopep8

Now, the imports will be also sorted in the correct order using isort package and the code style automatically format to conform to the PEP 8 style guidelines.

You can customize the functionalities of these tools by creating the required files and adding your configurations. For instance, create a new file named .flake8 and the following lines:

.flake8

[flake8]
ignore = E402, E127, W503, W504, N807, E501
exclude = .git, __pycache__, __init__.py, chart/, config/
max-complexity = 10
count = true
max-line-length = 120
statistics = true

We're ignoring some of the rules and overriding the default configurations for flake8.

Let's add another file to customize isort as well:

isort.cfg

[settings]
default_section = THIRDPARTY
indent = '    '
known_first_party = app,config
known_third_party = pypika,motor,asyncio_pool,uvloop
multi_line_output = 0
sections = FUTURE,STDLIB,THIRDPARTY,FIRSTPARTY,LOCALFOLDER

You can take a look at the documentation of isort to understand the configurations in more detail.

Great! Now, these pre-commit hooks will be applied to your files each time you commit your changes. Also, pre-commit hook failures must be fixed before pushing the Github.

Feel free to increase the number of repos in the pre-commit configuration file such as adding linting for YAML files. More information is available in the official documentation.

Support 🌏

If you feel like you unlocked new skills, please share with your friends and stay connected :)