Chapter01 Getting Started Click

If you find this content useful, consider buying this book:

If you enjoyed this book considering buying a copy

Chapter 1: Getting Started with Click #

Alfredo Deza

Building command-line tools is fun. For me, though, and for the longest time, it meant struggling with the different types of libraries Python had to offer. It all began with optparse, which is deprecated since Python 2.7, and it is unfortunately still in the standard library after all these years. It was replaced with argparse which, as you will see soon, it is still complicated to deal with. It is crucial to understand what Python has to offer to grasp why the alternative (the Click framework) is better.

Click is a project created by Armin Ronacher. If you haven’t heard of him before, you probably have heard about projects he created or co-authored:

I’ve used most of these before, and have had nothing but great experiences with them. What brings us to this chapter is Click and how to get started creating powerful tooling with it.

What is wrong with the alternatives #

The standard library offerings (via optparse and argparse) work fine. From this point forward, I’ll briefly touch on argparse only since optparse is deprecated (I strongly suggest you avoid it). My problem with argparse is that it is tough to reason about, and it wasn’t created to support some of the use cases the Click supports. One of these examples is sub-commands. Let’s build a command-line

import argparse

def main():
    parser = argparse.ArgumentParser(
        description="Tool with sub-commands"
    )

    subparsers = parser.add_subparsers(help='Sub-commands')

    sub_parser_1 = subparsers.add_parser('sub1')
    sub_parser_2 = subparsers.add_parser('sub2')

    parser.parse_args()

if __name__ == '__main__':
    main()

Crafting the tool requires dealing with a class called ArgumentParser and then adding to it. I don’t have anything against classes, but this reads convoluted, to say the least. Save it as sub-commands.py and run it to see how it behaves:

$ python sub-commands.py -h
usage: sub-commands.py [-h] {sub1,sub2} ...

Tool with sub-commands

positional arguments:
  {sub1,sub2}  Sub-commands

optional arguments:
  -h, --help   show this help message and exit

What are “positional arguments” ? I explicitly followed the documentation to add sub-commands (called sub_parsers in the code), why are these referred to as positional arguments? They aren’t! These are sub-commands. If you use them as a sub-command in the terminal you can verify this:

$ python sub-commands.py sub1 -h
usage: sub-commands.py sub1 [-h]

optional arguments:
  -h, --help  show this help message and exit

This is not good. Aside from how complex it is to deal with an instantiated class that grows with sub-commands, the implementation doesn’t match what it advertises. In essence, the sub-command is calling add_subparsers(), and they display in the help output as positional arguments which are sub-commands.

It is entirely possible to change the help output to force it to say something else, but this misses the point. I found the sub-command handling of argparse so rough that I ended up creating a small library that attempts to make it a bit easier. Even though that little library is in a few production tools, I would recommend you take a close look to Click, because it does everything that argparse does, and makes it simpler - including sub-commands.

Even if you aren’t going to use sub-commands at all, the interface to the class and objects that configure the command-line application creates much friction. It has taken me a while to find a suitable alternative, and although Click has been around for quite some time, it hasn’t been until recently that I’ve acknowledged how good it is.

A helpful Hello World #

The simplest use case for Click has a caveat: it doesn’t handle the help menu in a way that I prefer when building command-line tools. Have you ever tried a tool that doesn’t allow -h? How about it does allow -h but doesn’t like --help? What about one that doesn’t like -h, --help, or help?

$ find -h
find: illegal option -- h
usage: find [-H | -L | -P] [-EXdsx] [-f path] path ... [expression]
       find [-H | -L | -P] [-EXdsx] -f path [path ...] [expression]
$ find --help
find: illegal option -- -
usage: find [-H | -L | -P] [-EXdsx] [-f path] path ... [expression]
       find [-H | -L | -P] [-EXdsx] -f path [path ...] [expression]
$ find help
find: help: No such file or directory

Perhaps I’m raising the bar too high and find is too old of a tool? Let’s try with some modern ones:

$ docker -h
Flag shorthand -h has been deprecated, please use --help
$ python3 help
/usr/local/bin/python3: can't open file 'help': [Errno 2] No such file or directory

Do you think this level of helpfulness is over the top? Some tools do work with all three of them, like Skopeo and Coverage.py. Not only is it possible to be that helpful, but it should also be a priority if you are creating a new tool. As I mentioned, Click doesn’t do all three by default, but it doesn’t take much effort to get them all working. There are lots of basic examples for Click, and in this section, I’ll concentrate on creating one that allows all these help menus to show.

Starting with the most basic example so that we can check what is missing and then go fix it up, save the example as cli.py:

import click


@click.command()
def main():
    return

if __name__ == '__main__':
    main()

This is the first example of a command-line application using Click, and it looks very straightforward. It uses a decorator to go over the main() function, and has the piece of magic at the bottom to tell Python it should call the main() function if it executes directly. The first time I interacted with a command-line application built with Click, I was surprised to find it so easy to read, compared to argparse and other libraries I’ve interacted with in the past. Nothing should happen when you run this directly, there is no action or information to be displayed, but the help menus are all there ready to be displayed. Run it in the terminal with the three variations of help flags we are looking (-h, --help, and help):

$ python cli.py -h
Usage: cli.py [OPTIONS]
Try 'cli.py --help' for help.

Error: no such option: -h

No good! It doesn’t like -h. Try again with --help:

$ python cli.py --help
Usage: cli.py [OPTIONS]

Options:
  --help  Show this message and exit.

Much better. As you can see, the help menu is produced with no effort at all, just by having decorated the main() function. Finally, try with help and see what happens:

$ python cli.py help
Usage: cli.py [OPTIONS]
Try 'cli.py --help' for help.

Error: Got unexpected extra argument (help)

It doesn’t work, but there is an important distinction: it reported -h as an option that is not available and help as an unexpected extra argument. Argument vs. options doesn’t sound like much, but internally, Click already has an understanding of both. There is no need to configure anything. Now let’s make this work the way we envisioned in the beginning.

First, address the problem with -h not functioning. To do this, we need to (at last) get into some configuration of the framework. The changes mean that some unique interactions need to happen. In this case, that is working with the context. This context is a particular object that keeps options, flags, values, and internal configuration available for any command and sub-command. The framework pokes inside this context to check for any special directives. In this case, we want to change how it deals with the help option names, so define a dictionary that has these expanded:

import click

CONTEXT_SETTINGS = dict(help_option_names=['-h', '--help'])

@click.command(context_settings=CONTEXT_SETTINGS)
def main():
    return

if __name__ == '__main__':
    main()

The updated example doesn’t look that different, except for declaring CONTEXT_SETTINGS at the top with a dictionary. That dictionary sets a list onto help_option_names. Rerun it with -h:

$ python cli.py -h
Usage: cli.py [OPTIONS]

Options:
  -h, --help  Show this message and exit.

The help menu works, and it displays both flags as available options. This is great and solves the problem with -h not being recognized. But what about help? Things get tricky here because, without the dashes, it may very well be a sub-command. Declaring help as a sub-command is, in fact, part of the solution:

import click

CONTEXT_SETTINGS = dict(help_option_names=['-h', '--help'])

@click.group(context_settings=CONTEXT_SETTINGS)
def main():
    return

@main.command()
@click.pass_context
def help(ctx):
    print(ctx.parent.get_help())

if __name__ == '__main__':
    main()

It may be tricky to realize at first that there are other differences aside from creating the help() function acting as a sub-command. The @click.command() decorator gets removed from the main() function in favor of @click.group. That is how the framework can understand other sub-commands belong to the main() function. Another side-effect of this, is that a new decorator is available. The new decorator starts with the name of the function of the group (main() in this case is @main.command()). Finally, the context is requested with @click.pass_context and declared as an argument (ctx). This context is what allows the function to print the help menu from the parent command (in main()), instead of generating some other help menu. Run this once again to see how it behaves with the new sub-command:

$ python cli.py help
Usage: cli.py [OPTIONS] COMMAND [ARGS]...

Options:
  -h, --help  Show this message and exit.

Commands:
  help

Good! It now displays the help with any of the three combinations. It is a bit unfortunate that help shows in the Commands: section. If you are bothered by this, you can hide it from the help by adding an argument to the help() function:

@main.argument(hidden=True)
@click.pass_context
def help(ctx): #, topic, **kw):
    print(ctx.parent.get_help())

Rerun the cli.py script to check it out:

$ python cli.py help
Usage: cli.py [OPTIONS] COMMAND [ARGS]...

Options:
  -h, --help  Show this message and exit.

The downside? help is no longer part of the help menu itself, which is a good compromise for being helpful and allowing multiple flags to show the help menu.

Map a function to a command #

The previous examples have hinted at how you can start expanding a command-line tool with the Click framework. Let’s explore that further by filling all the gaps I skipped over to demonstrate some flags along with the help menu. This section creates a small tool to fix a problem I often encounter when working with SSH: when creating the configuration files and keys, I can’t ever get the permissions correctly. The SSH tool requires permissions to be of a certain type for different files; otherwise, it complains that permissions are too open. The problem with the error is that it doesn’t tell you how to fix it:

Permissions 0777 for '/Users/alfredo/.ssh/id_rsa' are too open.
It is recommended that your private key files are NOT accessible by others.
This private key will be ignored.

There can be multiple different files in the .ssh/ directory, these are just a few of the conditions that the program needs to ensure:

  • The .ssh/ directory needs to have 700 permission.
  • authorized_keys, known_hosts, config, and any public key (ending in .pub) needs to have 644 permissions.
  • All private keys need to have 600 permission bits.

Create a new file called ssh-medic.py, and start with the entry point to define what this tool is, including a good description in the docstring of the function:

import click

CONTEXT_SETTINGS = dict(help_option_names=['-h', '--help', 'help'])

@click.group(context_settings=CONTEXT_SETTINGS)
def main():
    """
    Ensure SSH directories and files have correct permissions:

    \b
    $HOME/.ssh      -> 700
    authorized_keys -> 644
    known_hosts     -> 644
    config          -> 644
    *.pub keys      -> 644
    All private key -> 600
    """
    return


if __name__ == '__main__':
    main()

The main() function’s docstring explains what the permission bits should be. It includes a special directive (\b), which tells Click not to wrap the lines around my formatting. The wrapping removes the lines breaks and makes my formatting look the same as in the script. Run the script with the help flag to verify the information:

$ python ssh-medic.py -h
Usage: ssh-medic.py [OPTIONS] COMMAND [ARGS]...

  Ensure SSH directories and files have correct permissions:

  $HOME/.ssh       -> 700
  authorized_keys  -> 644
  known_hosts      -> 644
  config           -> 644
  *.pub keys       -> 644
  All private keys -> 600

Options:
  -h, --help  Show this message and exit.

Next, I want to introduce a separate sub-command that checks the permissions and reports them back in the terminal. This feature is nice because it is a read-only command, nothing is going to happen except for some reporting. The main() function already is good to go because it is using the @click.group decorator, so all is needed is a new function that gets decorated with @main.command() to start expanding:

@main.command()
def check():
    click.echo('This sub-command innspects files in ~/.ssh and reports back')

I’m introducing a new helper from the framework: click.echo(), which is replacing print(). This helper utility is useful because it doesn’t mind Unicode or binary, and understands how to handle different use-cases transparently (unlike the print() function). Rerun the tool, using the new sub-command:

$ python ssh-medic.py check
This sub-command innspects files in ~/.ssh and reports back

The output verifies that the newly added function works as intended, so it is time to do something with it. The os and stat modules can help us with listing files and checking permissions respectively, so that gets imported and used to expand the check() function:

import os
import stat

@main.command()
def check():
    ssh_dir = os.path.expanduser('~/.ssh')
    files = [ssh_dir] + os.listdir(ssh_dir)
    for _file in files:
        file_stat = os.stat(os.path.join(ssh_dir, _file))
        permissions = oct(file_stat.st_mode)
        click.echo(f'{permissions[-3:]} -> {_file}')

There is quite a bit of newly added code in the function. First, the home directory (as indicated with the tilde) is expanded to a full path, stored to verify permissions later. The script can run from anywhere now. And it reports correctly on the full path rather than just the names of the directories. Then, it creates a files list with all the interesting files that are needed, and a loop goes over each one, calling stat to get all file metadata. The st_mode is the method that provides the information we need, and the function passes the ssh_dir joined with the file to produce an absolute path. Finally, the result is converted to an octal form, and reported with click.echo. Run it to see how it behaves in your system:

$ python ssh-medic.py check
700 -> /Users/alfredo/.ssh
644 -> config
600 -> id_rsa
644 -> authorized_keys
644 -> id_rsa.pub
644 -> known_hosts

The permissions on all my files are correct, but I have to correlate what I see with what the permissions should be. Not that useful yet. What is needed here is to perform that check for me instead. Create a mapping of the files and what they should be so that it can report back, and take into account that private keys can be named anything, and public keys may end up with the .pub suffix:

@main.command()
def check():
    ssh_dir = os.path.expanduser('~/.ssh')
    absolute_path = os.path.abspath(ssh_dir)
    files = [ssh_dir] + os.listdir(ssh_dir)

    # Expected permissions
    public_permissions = '644'
    private_permissions = '600'
    expected = {
        ssh_dir: '700',
        'authorized_keys': '644',
        'known_hosts': '644',
        'config': '644',
        '.ssh': '700',
    }

    for _file in files:
        # Public keys can use the .pub suffix
        if _file.endswith('.pub'):
            expected[_file] = public_permissions

        # Stat the file and get the octal permissions
        file_stat = os.stat(os.path.join(ssh_dir, _file))
        permissions = oct(file_stat.st_mode)[-3:]

        try:
            expected_permissions = expected[_file]
        except KeyError:
            # If the file doesn't exist, consider it as a private key
            expected_permissions = private_permissions
        # Only report if there are unexpected permissions
        if expected_permissions != permissions:
            click.echo(
                f'{_file} has {permissions}, should be {expected_permissions}'
            )

The function is now much longer and starting to get into the need for some refactoring. It’s OK for now to demonstrate how useful it is. The mapping is now in place and has some fallback values for public and private keys, which can be named almost anything. The loop is somewhat similar to before and checks if the permissions are adhering to the expected values. I’ve added a few keys with incorrect permissions in my system to test the new functionality:

$ python ssh-medic.py check
jenkins_rsa has 664, should be 600
jenkins_rsa.pub has 664, should be 644

Nothing reports back if all the files are OK, which is not very helpful. The script has lots of room to improve, like adding a fix sub-command that would change the permissions on the fly, and perhaps a dry-run flag that would not change anything but could show what would end up happening. Command-line tools are exciting, and they can grow in features, just make sure that you are keeping functions readable and small enough to test them. Anything that can be extracted into a separate utility for easier testing and maintainability is a positive change.