git pre-commit Hooks for Python

2018-12-08 Kenneth H. East [view page source]

As part of an effort to move to a fully automated CI/CD process, I wanted to begin using git pre-commit hooks for my python code. These are my notes on setting that up.

Introduction

My Goals:

Enforce consistent code formatting via black.
Automatic sorting of python import statements via isort.
Ensure comliance with PEP 8 via flake8.
Add additional tests to flake8 using flake8-bugbear.

Installation and Setup

Assume the following:

$PROJ contains a path to the project directory.
A virtualenv has been created and activated.
Application code and tests exist in $PROJ/app.

Install `flake8`

You probably already have flake8 installed. I also use flake8-bugbear, which looks for problems not found by flake8. Install them via:

$ pip install flake8
$ pip install flake-bugbear

To ensure that all is well, run the command flake8 --version and confirm that flake8-bugbear appears in the output. Mine looked like this:

$ flake8 --version
3.6.0 (flake8-bugbear: 18.8.0, mccabe: 0.6.1, pycodestyle: 2.4.0, pyflakes: 2.0.0) CPython 3.7.1 on Darwin

Both flake8 and flake8-bugbear will take their configuration from a .flake8 file in the root of the project. Here is mine:

# file: $PROJ/.flake8
#
# This config expects that the flake8-bugbear extension to be installed.
# bugbear looks at the line length and allows a slight variance as opposed
# to a hard limit.  When it detects excessive line lengths, it returns B950.
# This config looks for B950 and ignores the default flake8 E501 line length error.

[flake8]
max-complexity = 10
max-line-length = 88
select = C,E,F,W,B,B950
ignore =
       # Use bugbear line length detection instead of default
       E501,
       # PEP8 allows hanging indent, but E126 dosn't seem to.
       E126,

# Local Variables:
# mode: conf
# End:

Install `pre-commit`

The pre-commit python package is the easiest way to get git's pre-commit hooks setup. Install it:

$ pip install pre-commit
$ pre-commit install

The pre-commit install updates $PROJ/.git/hooks/pre-commit to use the pre-commit python package. This always needs to be done, even if you are cloning a project that is already using pre-commit.

pre-commit is configured by .pre-commit-config.yaml file which should be located at the top of the project. I started out by creating the following:

# file: $PROJ/.pre-commit-config.yaml
#
repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v2.0.0
    hooks:
    -  id: flake8

This config tells pre-commit where the repository is that contains the implementations are for the hooks to be run, as well as the version of the repository to use. The first time that a hook is run, pre-commit will download the hook's implementation from the repository. In this config, I tell it to run a single hook: flake8.

Initialize a git repo in $PROJ and stage a python file from $PROJ/app.

One can now test that pre-commit is running flake8 on staged files via pre-commit run -v. You should see output as to whether or not the staged file passed PEP 8. This is not the normal way to use pre-commit, but it's a easy way to ensure that things are properly configured.

The normal way to 'use' pre-commit is to simply run git commit. If the specified hooks pass, the commit procedes normally. If a hook fails, an error message is printed and the commit does not take place. In this case, correct the problems, stage the corrections, and re-commit.

Install `black`

While I try to do a good job of formatting as I code, I wanted to ensure that my code is consistently formatted. To that end, I decided to use black. It follow an opinionated coding style; there is little that can be tweaked. Thus the allusion to Henry Ford's, "any color you want, as long as it's black." When black is run on a python file, if the file precisely meets the standard, the file is not modified. If a file does not meet the standard, black modifies the file's contents so that it does meet the standard. This is an important point -- black can modify your files.

Install black via pip install black. black is configured via the pyproject.toml file. I created a $PROJ/pyproject.toml containing:

[tool.black]
line-length = 88
py36 = true
skip-string-normalization = true
include = '\.pyi?$'
exclude = '''
/(
    \.git
  | \.hg
  | \.mypy_cache
  | \.tox
  | \.venv
  | _build
  | buck-out
  | build
  | dist
)/
'''

Because I use single quotes for strings unless forced to do otherwise, I added skip-string-normalization = true to the standard black configuration.

With black installed, I updated $PROJ/.pre-commit-config.yaml so that black's pre-commit hook would be run:

# file: $PROJ/.pre-commit-config.yaml
#
repos:
-   repo: https://github.com/ambv/black
    rev: stable
    hooks:
    - id: black
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v2.0.0
    hooks:
    - id: flake8

When black is run on a file, it modifies that file to comply with black's formatting standard. If a file already meets that standard, the file is not modified. Thus, when git commit fails due black's pre-commit hook, the modifications made to the file by black must be staged. For example:

$ # Stage a file that was modified
$ git add app/app.py

$ # This will fail due to pre-commit hook - black will reformat $PROJ/app/app.py
$ git commit -m 'example commit 1'
black....................................................................Failed
hookid: black

Files were modified by this hook. Additional output:

reformatted app/app.py
All done! ✨ 🍰 ✨
1 file reformatted.

Flake8...................................................................Passed

$ # We can see that black modified $PROJ/app/app.py
$ git status
On branch master
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

    modified:   app/app.py

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   app/app.py

$ # Stage the changes made by black.
$ git add app/app.py

$ # This will succeed as all pre-commit hooks will pass
$ git commit -m 'example commit 2'
black................................................(no files to check)Skipped
Flake8...............................................(no files to check)Skipped
[master 15b53b9] example commit 2
 1 file changed, 6 insertions(+)

Install `isort`

black reformats import statements, as does isort. However, black does not sort import statements; it reformats them without changing order. Therefore, isort is needed to ensure that import statements are in thr proper order.

To enusre that isort reformats import statements according to the same rules as black, I created $PROJ/.isort.cfg containing:

# file: $PROJ/.isort.cfg
#
[settings]
multi_line_output=3
include_trailing_comma=True
force_grid_wrap=0
combine_as_imports=True
line_length=88

I then added an isort pre-commit hook to $PROJ/.pre-commit-config.yaml:

# file: $PROJ/.pre-commit-config.yaml
#
repos:
-   repo: https://github.com/ambv/black
    rev: stable
    hooks:
    - id: black
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v2.0.0
    hooks:
    - id: flake8
-   repo: https://github.com/pre-commit/mirrors-isort
    rev: v4.3.4
    hooks:
    - id: isort

Within emacs, I use py-isort.el which provides the commands M-x py-isort-buffer and M-x py-isort-region. I generally run these manually whenever I modify import statements.

py-isort.el has a before-save-hook which causes emacs to isort a buffer prior to saving it, but I prefer to have this done via isort's pre-commit hook. My emacs configuration for python is here.

Other `pre-commit` Hooks

pre-commit has a nice set of pre-commit hooks available for use. After looking them over, my $PROJ/.pre-commit-config.yaml ended up looking like this:

# file: $PROJ/.pre-commit-config.yaml
#
repos:
-   repo: https://github.com/ambv/black
    rev: stable
    hooks:
    - id: black
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v2.0.0
    hooks:
    - id: flake8
    - id: detect-aws-credentials
    - id: detect-private-key
    - id: double-quote-string-fixer
    - id: requirements-txt-fixer
    - id: trailing-whitespace

If I were not using black, I would add the add-trailing-comma hook as well. Additional interesting hooks:

blacken-docs - runsblack on python code blocks in documentation files
shellcheck, shfmt - for shell scripts
bandit - check for python code vulnerabilities
encryption-check - ensure ansible vault files are encrypted
sign-commit - adds signature verification
dockerfile and teraform linters

KHE

Introduction

Installation and Setup

Install flake8

Install pre-commit

Install black

Install isort

Other pre-commit Hooks

Install `flake8`

Install `pre-commit`

Install `black`

Install `isort`

Other `pre-commit` Hooks