Chapter06 Use Advanced Click Features

If you find this content useful, consider buying this book:

If you enjoyed this book considering buying a copy

Chapter 6: Use Advanced Click Features #

Alfredo Deza

Once you are past simple command-line tools that have a particular objective, it is time to move on and explore solutions that perhaps you weren’t aware you needed. Recently at work, I was in charge of rethinking the reporting output of a command-line tool that was providing a table with columns and headers as the default format. The problem with a header-and-column format is that users will end up wanting more columns and get more information. This is what ended happening to this tool at work. It was reporting so many columns that the output would not fit in my large monitor. When the information reported has any chance of including more fields, it will. Users aren’t all the same; consumers are all different. Someone will want an extra field that doesn’t make sense for others. My proposal to get rid of the columns was to report vertically and nesting anything that requires grouping:

Item 1
  total: 30
  size: 23MB
  path: /var/lib
  severity: High
  issues:
    - issue-17
    - issue-29

The output now looks a bit like structured YAML. It doesn’t matter, because after taking a second look, it seemed I could do better. What if I could give visual cues that informed without using any more space? This is what coloring can do for you: red can indicate an error, while blue can be informational or for low-priority items. Using a red color for Item 1 would implicitly describe this item as high severity, allowing the removal of severity: High as it would be redundant.

Next, command-line applications tend to like configuration files to implement human-readable defaults, or to avoid too many flags and values in the execution, simplifying working with a tool. Although there is no specific handler for configuration files, in this chapter, we also cover how to integrate config files into Click command-line tools. Finally, sub-commands and other friendly terminal-based interfaces complete the circle of powerful extras to enhance a tool that needs to grow in complexity. In chapter 1, I briefly covered sub-commands to explain how terrible sub-commands are implemented in Python’s standard library (with argparse primarily), which is an excellent introduction. Let’s go into more detail in this chapter.

Subcommands #

We already covered a brief introduction of sub-commands in Chapter 1 while explaining the differences between what the Python standard library offers and how Click makes things way more interesting. Click, as with most everything it offers, is comprehensive yet still very simple most of the time.

In this section, I go into using Click to build a nested sub-command tool that talks to a real HTTP API. This is a typical case, where if an HTTP API exists, a companion command-line tool exists. A few years ago, I built a package repository that would store system packages (mostly RPM, and DEB) and then created repositories for different Linux Distributions automatically. The service also offered an API, and I had many users asking for tasks that I only had access to like:

  • Forcefully remove a repository
  • Recreate the repository (triggering a rebuild)
  • Upload a package or set of packages
  • Remove a package

Designing the command-line tool was fun, and it allowed me to think about how to compose these sub-commands to make sense. I encourage you to think deeply about how the commands read when executing them. One example of oddness when crafting sub-commands is in init systems (tools in charge of starting system services in a server). Which one of these would be nicer to work with?

  • service start nginx
  • service nginx start

There is no straightforward answer to which one is better. As with most things that are somewhat complex, the answer is that it depends. Using start nginx is great if you don’t plan to add anything else to the command after it because nginx is the argument. It is also easier to implement since nginx is an argument (think of it as a value) and not something that is pre-configured (Nginx is only available if installed). The counterpart to this is valid as well. If you need to nest other sub-commands, flags, or options, then nginx start can be easier since nginx would be an actual sub-command. The implementation would be trickier because it gets dynamically generated.

For this section, I demonstrate how to create a tool with sub-commands that communicates with an external HTTP API. I’ve chosen the Jenkins CI/CD Platform, which is easy to install and startup, and many workplaces tend to have at least one running somewhere. I’m not covering how to install and configure Jenkins here, but you need to know the full HTTP address and ensure you can reach it from your current workstation. A fully qualified domain name (FQDN) of the running Jenkins instance, along with the port and the protocol (HTTPS for SSL, for example) is required. An easy way to run Jenkins is with a container; you can find setup instructions in their Github repository. Using the container is what I chose for these examples, and after installing, I create a user (named admin) and generate a token. The token is crucial for talking to the HTTP API; without it, the examples don’t work.

Since I have Jenkins running locally with a user named admin, that means my URL is: http://localhost:8080/. And I’ve generated a new token using http://localhost:8080/user/admin/configure. That token is not to be shared, and it should only be available once to copy it from the web interface. For me, that token looks similar to this: 1102a43986753ffb17ee76db2164bba0c8. Now create a new virtual environment for this new tool, and install the Jenkins library, which is needed to talk to the HTTP API:

$ python3 -m venv venv
$ source venv/bin/activate
$ pip install python-jenkins
Collecting python-jenkins
  ...
Installing collected packages:
  idna, certifi, urllib3, chardet, requests, multi-key-dict,
  six, pbr, python-jenkins
  Running setup.py install for multi-key-dict ... done
Successfully installed
  certifi-2020.4.5.1 chardet-3.0.4 idna-2.9 multi-key-dict-2.0.3
  pbr-5.4.5 python-jenkins-1.7.0 requests-2.23.0 six-1.14.0 urllib3-1.25.9

Make sure you are installing python-jenkins, not jenkins which is a different library that doesn’t work to talk to the Jenkins API. After installing, and ensuring you have Jenkins running and a token available, start a Python shell to interact with the API:

>>> import jenkins
>>> conn = jenkins.Jenkins(
  "http://localhost:8080/",
  username="admin",
  password="secret-token")
>>> conn.get_jobs()
[]
>>> conn.get_whoami()
{'_class': 'hudson.model.User',
 'absoluteUrl': 'http://localhost:8080/user/admin',
 'description': None,
 'fullName': 'admin',
 'id': 'admin',
 'property': [{'_class': 'jenkins.security.ApiTokenProperty'},
  {'_class': 'hudson.model.MyViewsProperty'},
  {'_class': 'hudson.model.PaneStatusProperties'},
  {'_class': 'jenkins.security.seed.UserSeedProperty'},
  {'_class': 'hudson.search.UserSearchProperty', 'insensitiveSearch': True},
  {'_class': 'hudson.model.TimeZoneProperty'},
  {'_class': 'hudson.security.HudsonPrivateSecurityRealm$Details'},
  {'_class': 'hudson.tasks.Mailer$UserProperty',
   'address': 'admin@example.com'}]}

After importing the library, a connection object returns from jenkins.Jenkins. One thing to note here is that the library wants a password argument, which in reality, is the secret token that got generated in the Jenkins web interface. Then, using the connection object, you can interact with Jenkins with almost anything, there is almost a complete parity between the web interface and what the library offers. In the example, I ask to get all the jobs (there aren’t any, so an empty list returns), and then I use the get_whoami() method to get some metadata about my account.

Start by creating a new file with a single sub-command called jobs. This sub-command helps when interacting with Jenkins jobs as they come from the API. Remember to define the token that generated earlier. I save the example as staffing.py in reference to something that handles jobs:

import jenkins
import click

token = "secret"


@click.group()
def api():
    pass


@api.command()
def jobs(_list):
    """Interact with Jobs in a remote Jenkins server"""
    pass


if __name__ == '__main__':
    api()

Many things are missing from this first iteration, but now, the tool can be executed in the terminal and show some help menus, verifying that it is all wired up correctly:

$ python staffing.py jobs --help
Usage: staffing.py jobs [OPTIONS]

  Interact with Jobs in a remote Jenkins server

Options:
  --help  Show this message and exit.

Now add a flag to the jobs() function, that shows a list of jobs available in the Jenkins server:

url = 'http://localhost:8080'


@api.command()
@click.option('--list', '_list', is_flag=True, default=False)
def jobs(_list):
    """Interact with Jobs in a remote Jenkins server"""
    if _list:
        conn = jenkins.Jenkins(
            username='admin', password=token, url=url
        )
        jobs = conn.get_jobs()
        if jobs:
            for job in jobs:
                click.echo(job)
            return
        return click.echo(
            f'No jobs available in remote Jenkins server at: {url}'
        )

First, I include a new option that is aliased to _list so that it doesn’t override the built-in list function in Python. I define it as a flag and that it defaults to False. Next, if _list is passed in, it creates a connection which calls get_jobs() and loops over each one. If a job is available, it shows in the output; otherwise, it returns a helpful message saying nothing is available yet for the given URL.

$ python staffing.py jobs --list
No jobs available in remote Jenkins server at: http://localhost:8080

After creating a quick example job in Jenkins the output is very different:

$ python staffing.py jobs --list
{
    '_class': 'hudson.model.FreeStyleProject',
    'name': 'example',
    'url': 'http://localhost:8080/job/example/',
    'color': 'notbuilt',
    'fullname':
    'example'
}

The JSON output looks odd in the terminal, make it a little bit better to extract a subset of useful fields:

@api.command()
@click.option('--list', '_list', is_flag=True, default=False)
def jobs(_list):
    """Interact with Jobs in a remote Jenkins server"""
    if _list:
        conn = jenkins.Jenkins(
            username='admin', password=token, url=url
        )
        jobs = conn.get_jobs()
        if jobs:
            for job in jobs:
                click.echo(job['fullname'])
                click.echo(' ' + job['url'])
                click.echo(' Status: {0}'.format(job['color']))
                click.echo('')
            return
        return click.echo(
            f'No jobs available in remote Jenkins server at: {url}'
        )

Terminal output is now much better:

python staffing.py jobs --list
example
 http://localhost:8080/job/example/
 Status: notbuilt

The remote connection for Jenkins is needed for the next commands, so extract it into a separate function to make it easier to reuse:

def connection():
    return jenkins.Jenkins(
        username='admin', password=token, url=url
    )

To dynamically load jobs that are present in the remote Jenkins server as sub-commands, a custom class needs to get in place and handle the loading. Click allows this with the click.MultiCommand class and this is how the implementation looks for this case:

class DynamicJobs(click.MultiCommand):

    def list_commands(self, ctx):
        conn = connection()
        jobs = conn.get_jobs()
        return [i['name'] for i in jobs]

    def get_command(self, ctx, name):
        conn = connection()
        jobs = conn.get_jobs()

        if name in [i['name'] for i in jobs]:
            return Job(
                name=name,
                params=[
                    click.Option(
                        ['-d', '--delete'], is_flag=True)
                    ]
            )

This class can list jobs as sub-commands and map a remote job to an actual command (called Job()), which needs to be defined as well:

class Job(click.Command):

    def invoke(self, ctx):
        conn = connection()
        if ctx.params['delete']:
            click.echo(f'Deleting job: {self.name}')
            conn.delete_job(self.name)
        click.echo('Completed deletion of job')

Now create an empty sub-command called job(), which is the placeholder for all the dynamically loaded jobs. Whatever jobs exist in the remote Jenkins server, these jobs get listed here:

@api.group(cls=DynamicJobs)
def job():
    pass

With the two classes and the new job() sub-command, the command-line tool can verify that it is reading correctly from the remote server and mapping to sub-commands:

python staffing.py job --help
Usage: staffing.py job [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  example

In this output, example is the remote Jenkins job. If I create another job named deploy appears here as well. This is how it looks now:

python staffing.py job --help

Usage: staffing.py job [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  deploy
  example

The two classes added, DynamicJobs() which loads and matches, and Job() which does the actual work, are crucial to add behavior to these lazily loaded commands. An option exists for deleting the job, which you verify by using the name of the job as the command:

$ python staffing.py job deploy --help

Usage: staffing.py job deploy [OPTIONS]

Options:
  -d, --delete
  --help        Show this message and exit.

Go ahead and delete that job now and verify the output once again:

$ python staffing.py job deploy --delete

Deleting job: deploy
Completed deletion of job

Running the staffing.py tool once more shows that the job no longer exists, and thus, it no longer appears in the help menu:

$ python staffing.py job --help

Usage: staffing.py job [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  example

There are many ways to extend tools and sub-commands, and being able to interact with a remote API (Jenkins, in this case) is an excellent way to make it easier to handle complex interactions.

Utilities #

There are a few utilities that I like to mention about Click because they make it so much easier to work with. Handling colors, launching a pager or an editor, or even handling files seamlessly between stdin and regular file paths is straightforward. You will probably not need all of these, but knowing they exist is useful to address the different needs of command-line tools. I’m guilty of implementing a lot of these utilities before, with implementations that aren’t as nice as those included with Click.

Colored Output #

I’m guilty of implementing colored output (from scratch) many times. Proper colored output is difficult to get right. If you haven’t tried this before, it involves sending ASCII escape codes to the terminal, which, in turn, is interpreted as a specific color. Not only one needs to send these escape codes to the terminal (as part of the string), but a reset code is needed when the coloring needs to end. This is because, otherwise, the coloring wouldn’t know where to stop. Neither newlines or whitespace is a delimiter for colors to stop.

If you are curious, this is a handy mapping I’ve written many times for some basic colors:

colors = dict(
    blue='\033[34m',
    green='\033[92m',
    yellow='\033[33m',
    red='\033[91m',
    bold='\033[1m',
    ends='\033[0m'
)

The ends key, indicates that the coloring has ended. The implementation on strings looks something like this:

>>> yellow = colors['yellow']
>>> ends = colors['ends']
>>> message = '{0}This is a warning{0}'.format(yellow, ends)
>>> message
'\x1b[33mThis is a warning\x1b[33m'

If you print the string or write it directly to the terminal with some other method, it makes the string yellow while keeping any subsequent string or lines the default color. All this bookkeeping is tedious and requires more granular controls to determine if a terminal is capable of sending colors or disable them altogether. This level of control is useful for programs that run on continuous integration systems that run commands, capture the output, and implement it correctly. Luckily, Click addresses this problem head-on, and allows controlling colored output with a separate library.

There are two ways (unfortunately) to accomplish coloring of output through Click:

import click

# Output combining click.echo and click.style
click.echo(click.style('Command Line Tools Book!', fg='green'))

# Output with click.secho
click.secho('Command Line Tools Book!', fg='green', blink=True)

The API for combining both click.echo and click.style is what I prefer as it reads much better.

There are many combinations to use for coloring text output, and my suggestion is to try the simplest first while making a conscious effort to avoid adding more color combinations. Click allows changing the background to a different color from the foreground, use bold characters, and even blink. Don’t abuse these, and try to make it as minimal as possible.

File handling #

File handling is not easy. It is easy in the sense that opening a file and reading it in Python is straightforward, but inside command-line applications it isn’t. Many different constraints need to get preemptively addressed to have a solid command-line tool. Here are just a few things that can happen that one needs to guard against:

  • A file path can be misspelled by the user
  • The path might be correct but permissions aren’t sufficient to open it
  • After opening the file, the encoding is not handled properly, causing encoding errors
  • Relative paths or shortened paths (like ~/) need to get expanded
  • stdin is handled differently, even though it behaves like a file descriptor

You still need to handle all of the above, but the interface that Click offers is seamless and makes it less painful. Here are a few examples on what happens in error conditions:

>>> click.open_file('/etc/aliases.db').read()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lib/python/site-packages/click/utils.py", line 336, in open_file
    f, should_close = open_stream(filename, mode, encoding, errors, atomic)
  File "lib/python/site-packages/click/_compat.py", line 533, in open_stream
    return _wrap_io_open(filename, mode, encoding, errors), True
  File "lib/python/site-packages/click/_compat.py", line 511, in _wrap_io_open
    return io.open(file, mode, **kwargs)
PermissionError: [Errno 13] Permission denied: '/etc/aliases.db'

When the file can’t be found:

>>> click.open_file('/etc/bash')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lib/python/site-packages/click/utils.py", line 336, in open_file
    f, should_close = open_stream(filename, mode, encoding, errors, atomic)
  File "lib/python/site-packages/click/_compat.py", line 533, in open_stream
    return _wrap_io_open(filename, mode, encoding, errors), True
  File "lib/python/site-packages/click/_compat.py", line 511, in _wrap_io_open
    return io.open(file, mode, **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: '/etc/bash'

Finally, when there are valid files that can be read, the encoding (defaulting to utf-8) is automatically handled:

>>> >>> click.open_file('/etc/bashrc')
<_io.TextIOWrapper name='/etc/bashrc' mode='r' encoding='UTF-8'>
>>> click.open_file('/etc/passwd')