git pre-commit Hooks for Python
As part of an effort to move to a fully automated CI/CD process, I wanted to begin using
git
pre-commit hooks for my python code. These are my notes on setting that up.
Introduction
My Goals:
- Enforce consistent code formatting via
black
. - Automatic sorting of python
import
statements viaisort
. - Ensure comliance with
PEP 8
viaflake8
. - Add additional tests to
flake8
usingflake8-bugbear
.
Installation and Setup
Assume the following:
-
$PROJ
contains a path to the project directory. - A virtualenv has been created and activated.
- Application code and tests exist in
$PROJ/app
.
Install flake8
You probably already have flake8
installed. I also use flake8-bugbear
, which looks
for problems not found by flake8
. Install them via:
$ pip install flake8 $ pip install flake-bugbear
To ensure that all is well, run the command flake8 --version
and confirm that
flake8-bugbear
appears in the output. Mine looked like this:
$ flake8 --version 3.6.0 (flake8-bugbear: 18.8.0, mccabe: 0.6.1, pycodestyle: 2.4.0, pyflakes: 2.0.0) CPython 3.7.1 on Darwin
Both flake8
and flake8-bugbear
will take their configuration from a .flake8
file
in the root of the project. Here is mine:
# file: $PROJ/.flake8 # # This config expects that the flake8-bugbear extension to be installed. # bugbear looks at the line length and allows a slight variance as opposed # to a hard limit. When it detects excessive line lengths, it returns B950. # This config looks for B950 and ignores the default flake8 E501 line length error. [flake8] max-complexity = 10 max-line-length = 88 select = C,E,F,W,B,B950 ignore = # Use bugbear line length detection instead of default E501, # PEP8 allows hanging indent, but E126 dosn't seem to. E126, # Local Variables: # mode: conf # End:
Install pre-commit
The pre-commit
python package is the easiest way to get git's pre-commit hooks setup.
Install it:
$ pip install pre-commit $ pre-commit install
The pre-commit
install updates $PROJ/.git/hooks/pre-commit
to use the pre-commit
python package. This always needs to be done, even if you are cloning a project that is
already using pre-commit
.
pre-commit
is configured by .pre-commit-config.yaml
file which should be located at
the top of the project. I started out by creating the following:
# file: $PROJ/.pre-commit-config.yaml # repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v2.0.0 hooks: - id: flake8
This config tells pre-commit
where the repository is that contains the implementations
are for the hooks to be run, as well as the version of the repository to use. The first
time that a hook is run, pre-commit
will download the hook's implementation from the
repository. In this config, I tell it to run a single hook: flake8
.
Initialize a git
repo in $PROJ
and stage a python file from $PROJ/app
.
One can now test that pre-commit
is running flake8
on staged files via pre-commit
run -v
. You should see output as to whether or not the staged file passed PEP 8. This
is not the normal way to use pre-commit
, but it's a easy way to ensure that things are
properly configured.
The normal way to 'use' pre-commit
is to simply run git commit
. If the specified
hooks pass, the commit procedes normally. If a hook fails, an error message is printed
and the commit does not take place. In this case, correct the problems, stage the
corrections, and re-commit.
Install black
While I try to do a good job of formatting as I code, I wanted to ensure that my code is
consistently formatted. To that end, I decided to use black
. It follow an
opinionated coding style; there is little that can be tweaked. Thus the allusion to
Henry Ford's, "any color you want, as long as it's black." When black
is run on a
python file, if the file precisely meets the standard, the file is not modified. If a
file does not meet the standard, black
modifies the file's contents so that it does
meet the standard. This is an important point -- black
can modify your files.
Install black
via pip install black
. black
is configured via the pyproject.toml
file. I created a $PROJ/pyproject.toml
containing:
[tool.black] line-length = 88 py36 = true skip-string-normalization = true include = '\.pyi?$' exclude = ''' /( \.git | \.hg | \.mypy_cache | \.tox | \.venv | _build | buck-out | build | dist )/ '''
Because I use single quotes for strings unless forced to do otherwise, I
added skip-string-normalization = true
to the standard black
configuration.
With black
installed, I updated $PROJ/.pre-commit-config.yaml
so that black
's
pre-commit hook would be run:
# file: $PROJ/.pre-commit-config.yaml # repos: - repo: https://github.com/ambv/black rev: stable hooks: - id: black - repo: https://github.com/pre-commit/pre-commit-hooks rev: v2.0.0 hooks: - id: flake8
When black
is run on a file, it modifies that file to comply with black
's
formatting standard. If a file already meets that standard, the file is not modified.
Thus, when git commit
fails due black
's pre-commit hook, the modifications made to
the file by black
must be staged. For example:
$ # Stage a file that was modified $ git add app/app.py $ # This will fail due to pre-commit hook - black will reformat $PROJ/app/app.py $ git commit -m 'example commit 1' black....................................................................Failed hookid: black Files were modified by this hook. Additional output: reformatted app/app.py All done! ✨ 🍰 ✨ 1 file reformatted. Flake8...................................................................Passed $ # We can see that black modified $PROJ/app/app.py $ git status On branch master Changes to be committed: (use "git reset HEAD <file>..." to unstage) modified: app/app.py Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git checkout -- <file>..." to discard changes in working directory) modified: app/app.py $ # Stage the changes made by black. $ git add app/app.py $ # This will succeed as all pre-commit hooks will pass $ git commit -m 'example commit 2' black................................................(no files to check)Skipped Flake8...............................................(no files to check)Skipped [master 15b53b9] example commit 2 1 file changed, 6 insertions(+)
Install isort
black
reformats import
statements, as does isort
. However, black
does not sort
import
statements; it reformats them without changing order. Therefore, isort
is
needed to ensure that import
statements are in thr proper order.
To enusre that isort
reformats import statements according to the same rules as
black
, I created $PROJ/.isort.cfg
containing:
# file: $PROJ/.isort.cfg # [settings] multi_line_output=3 include_trailing_comma=True force_grid_wrap=0 combine_as_imports=True line_length=88
I then added an isort
pre-commit hook to $PROJ/.pre-commit-config.yaml
:
# file: $PROJ/.pre-commit-config.yaml # repos: - repo: https://github.com/ambv/black rev: stable hooks: - id: black - repo: https://github.com/pre-commit/pre-commit-hooks rev: v2.0.0 hooks: - id: flake8 - repo: https://github.com/pre-commit/mirrors-isort rev: v4.3.4 hooks: - id: isort
Within emacs
, I use py-isort.el
which provides
the commands M-x py-isort-buffer
and M-x py-isort-region
. I generally run these
manually whenever I modify import statements.
py-isort.el
has a before-save-hook
which causes emacs
to isort
a buffer prior to
saving it, but I prefer to have this done via isort
's pre-commit hook. My emacs
configuration for python is
here.
Other pre-commit
Hooks
pre-commit
has a nice set of pre-commit hooks
available for use. After looking them over, my $PROJ/.pre-commit-config.yaml
ended up looking like this:
# file: $PROJ/.pre-commit-config.yaml # repos: - repo: https://github.com/ambv/black rev: stable hooks: - id: black - repo: https://github.com/pre-commit/pre-commit-hooks rev: v2.0.0 hooks: - id: flake8 - id: detect-aws-credentials - id: detect-private-key - id: double-quote-string-fixer - id: requirements-txt-fixer - id: trailing-whitespace
If I were not using black
, I would add the add-trailing-comma
hook as
well. Additional interesting hooks:
-
blacken-docs
- runsblack
on python code blocks in documentation files -
shellcheck
,shfmt
- for shell scripts -
bandit
- check for python code vulnerabilities -
encryption-check
- ensure ansible vault files are encrypted -
sign-commit
- adds signature verification -
dockerfile
andteraform
linters