Chapter06 Use Advanced Click Features

If you find this content useful, consider buying this book:

  • Amazon
  • Purchase all books bundle
  • Purchase from pragmatic ai labs
  • Subscribe all content monthly
  • If you enjoyed this book considering buying a copy

    Chapter 6: Use Advanced Click Features #

    Alfredo Deza

    Once you are past simple command-line tools that have a particular objective, it is time to move on and explore solutions that perhaps you weren’t aware you needed. Recently at work, I was in charge of rethinking the reporting output of a command-line tool that was providing a table with columns and headers as the default format. The problem with a header-and-column format is that users will end up wanting more columns and get more information. This is what ended happening to this tool at work. It was reporting so many columns that the output would not fit in my large monitor. When the information reported has any chance of including more fields, it will. Users aren’t all the same; consumers are all different. Someone will want an extra field that doesn’t make sense for others. My proposal to get rid of the columns was to report vertically and nesting anything that requires grouping:

    Item 1
      total: 30
      size: 23MB
      path: /var/lib
      severity: High
      issues:
        - issue-17
        - issue-29
    

    The output now looks a bit like structured YAML. It doesn’t matter, because after taking a second look, it seemed I could do better. What if I could give visual cues that informed without using any more space? This is what coloring can do for you: red can indicate an error, while blue can be informational or for low-priority items. Using a red color for Item 1 would implicitly describe this item as high severity, allowing the removal of severity: High as it would be redundant.

    Next, command-line applications tend to like configuration files to implement human-readable defaults, or to avoid too many flags and values in the execution, simplifying working with a tool. Although there is no specific handler for configuration files, in this chapter, we also cover how to integrate config files into Click command-line tools. Finally, sub-commands and other friendly terminal-based interfaces complete the circle of powerful extras to enhance a tool that needs to grow in complexity. In chapter 1, I briefly covered sub-commands to explain how terrible sub-commands are implemented in Python’s standard library (with argparse primarily), which is an excellent introduction. Let’s go into more detail in this chapter.

    Subcommands #

    We already covered a brief introduction of sub-commands in Chapter 1 while explaining the differences between what the Python standard library offers and how Click makes things way more interesting. Click, as with most everything it offers, is comprehensive yet still very simple most of the time.

    In this section, I go into using Click to build a nested sub-command tool that talks to a real HTTP API. This is a typical case, where if an HTTP API exists, a companion command-line tool exists. A few years ago, I built a package repository that would store system packages (mostly RPM, and DEB) and then created repositories for different Linux Distributions automatically. The service also offered an API, and I had many users asking for tasks that I only had access to like:

    • Forcefully remove a repository
    • Recreate the repository (triggering a rebuild)
    • Upload a package or set of packages
    • Remove a package

    Designing the command-line tool was fun, and it allowed me to think about how to compose these sub-commands to make sense. I encourage you to think deeply about how the commands read when executing them. One example of oddness when crafting sub-commands is in init systems (tools in charge of starting system services in a server). Which one of these would be nicer to work with?

    • service start nginx
    • service nginx start

    There is no straightforward answer to which one is better. As with most things that are somewhat complex, the answer is that it depends. Using start nginx is great if you don’t plan to add anything else to the command after it because nginx is the argument. It is also easier to implement since nginx is an argument (think of it as a value) and not something that is pre-configured (Nginx is only available if installed). The counterpart to this is valid as well. If you need to nest other sub-commands, flags, or options, then nginx start can be easier since nginx would be an actual sub-command. The implementation would be trickier because it gets dynamically generated.

    For this section, I demonstrate how to create a tool with sub-commands that communicates with an external HTTP API. I’ve chosen the Jenkins CI/CD Platform, which is easy to install and startup, and many workplaces tend to have at least one running somewhere. I’m not covering how to install and configure Jenkins here, but you need to know the full HTTP address and ensure you can reach it from your current workstation. A fully qualified domain name (FQDN) of the running Jenkins instance, along with the port and the protocol (HTTPS for SSL, for example) is required. An easy way to run Jenkins is with a container; you can find setup instructions in their Github repository. Using the container is what I chose for these examples, and after installing, I create a user (named admin) and generate a token. The token is crucial for talking to the HTTP API; without it, the examples don’t work.

    Since I have Jenkins running locally with a user named admin, that means my URL is: http://localhost:8080/. And I’ve generated a new token using http://localhost:8080/user/admin/configure. That token is not to be shared, and it should only be available once to copy it from the web interface. For me, that token looks similar to this: 1102a43986753ffb17ee76db2164bba0c8. Now create a new virtual environment for this new tool, and install the Jenkins library, which is needed to talk to the HTTP API:

    $ python3 -m venv venv
    $ source venv/bin/activate
    $ pip install python-jenkins
    Collecting python-jenkins
      ...
    Installing collected packages:
      idna, certifi, urllib3, chardet, requests, multi-key-dict,
      six, pbr, python-jenkins
      Running setup.py install for multi-key-dict ... done
    Successfully installed
      certifi-2020.4.5.1 chardet-3.0.4 idna-2.9 multi-key-dict-2.0.3
      pbr-5.4.5 python-jenkins-1.7.0 requests-2.23.0 six-1.14.0 urllib3-1.25.9
    

    Make sure you are installing python-jenkins, not jenkins which is a different library that doesn’t work to talk to the Jenkins API. After installing, and ensuring you have Jenkins running and a token available, start a Python shell to interact with the API:

    >>> import jenkins
    >>> conn = jenkins.Jenkins(
      "http://localhost:8080/",
      username="admin",
      password="secret-token")
    >>> conn.get_jobs()
    []
    >>> conn.get_whoami()
    {'_class': 'hudson.model.User',
     'absoluteUrl': 'http://localhost:8080/user/admin',
     'description': None,
     'fullName': 'admin',
     'id': 'admin',
     'property': [{'_class': 'jenkins.security.ApiTokenProperty'},
      {'_class': 'hudson.model.MyViewsProperty'},
      {'_class': 'hudson.model.PaneStatusProperties'},
      {'_class': 'jenkins.security.seed.UserSeedProperty'},
      {'_class': 'hudson.search.UserSearchProperty', 'insensitiveSearch': True},
      {'_class': 'hudson.model.TimeZoneProperty'},
      {'_class': 'hudson.security.HudsonPrivateSecurityRealm$Details'},
      {'_class': 'hudson.tasks.Mailer$UserProperty',
       'address': 'admin@example.com'}]}
    

    After importing the library, a connection object returns from jenkins.Jenkins. One thing to note here is that the library wants a password argument, which in reality, is the secret token that got generated in the Jenkins web interface. Then, using the connection object, you can interact with Jenkins with almost anything, there is almost a complete parity between the web interface and what the library offers. In the example, I ask to get all the jobs (there aren’t any, so an empty list returns), and then I use the get_whoami() method to get some metadata about my account.

    Start by creating a new file with a single sub-command called jobs. This sub-command helps when interacting with Jenkins jobs as they come from the API. Remember to define the token that generated earlier. I save the example as staffing.py in reference to something that handles jobs:

    import jenkins
    import click
    
    token = "secret"
    
    
    @click.group()
    def api():
        pass
    
    
    @api.command()
    def jobs(_list):
        """Interact with Jobs in a remote Jenkins server"""
        pass
    
    
    if __name__ == '__main__':
        api()
    

    Many things are missing from this first iteration, but now, the tool can be executed in the terminal and show some help menus, verifying that it is all wired up correctly:

    $ python staffing.py jobs --help
    Usage: staffing.py jobs [OPTIONS]
    
      Interact with Jobs in a remote Jenkins server
    
    Options:
      --help  Show this message and exit.
    

    Now add a flag to the jobs() function, that shows a list of jobs available in the Jenkins server:

    url = 'http://localhost:8080'
    
    
    @api.command()
    @click.option('--list', '_list', is_flag=True, default=False)
    def jobs(_list):
        """Interact with Jobs in a remote Jenkins server"""
        if _list:
            conn = jenkins.Jenkins(
                username='admin', password=token, url=url
            )
            jobs = conn.get_jobs()
            if jobs:
                for job in jobs:
                    click.echo(job)
                return
            return click.echo(
                f'No jobs available in remote Jenkins server at: {url}'
            )
    

    First, I include a new option that is aliased to _list so that it doesn’t override the built-in list function in Python. I define it as a flag and that it defaults to False. Next, if _list is passed in, it creates a connection which calls get_jobs() and loops over each one. If a job is available, it shows in the output; otherwise, it returns a helpful message saying nothing is available yet for the given URL.

    $ python staffing.py jobs --list
    No jobs available in remote Jenkins server at: http://localhost:8080
    

    After creating a quick example job in Jenkins the output is very different:

    $ python staffing.py jobs --list
    {
        '_class': 'hudson.model.FreeStyleProject',
        'name': 'example',
        'url': 'http://localhost:8080/job/example/',
        'color': 'notbuilt',
        'fullname':
        'example'
    }
    

    The JSON output looks odd in the terminal, make it a little bit better to extract a subset of useful fields:

    @api.command()
    @click.option('--list', '_list', is_flag=True, default=False)
    def jobs(_list):
        """Interact with Jobs in a remote Jenkins server"""
        if _list:
            conn = jenkins.Jenkins(
                username='admin', password=token, url=url
            )
            jobs = conn.get_jobs()
            if jobs:
                for job in jobs:
                    click.echo(job['fullname'])
                    click.echo(' ' + job['url'])
                    click.echo(' Status: {0}'.format(job['color']))
                    click.echo('')
                return
            return click.echo(
                f'No jobs available in remote Jenkins server at: {url}'
            )
    
    

    Terminal output is now much better:

    python staffing.py jobs --list
    example
     http://localhost:8080/job/example/
     Status: notbuilt
    

    The remote connection for Jenkins is needed for the next commands, so extract it into a separate function to make it easier to reuse:

    def connection():
        return jenkins.Jenkins(
            username='admin', password=token, url=url
        )
    

    To dynamically load jobs that are present in the remote Jenkins server as sub-commands, a custom class needs to get in place and handle the loading. Click allows this with the click.MultiCommand class and this is how the implementation looks for this case:

    class DynamicJobs(click.MultiCommand):
    
        def list_commands(self, ctx):
            conn = connection()
            jobs = conn.get_jobs()
            return [i['name'] for i in jobs]
    
        def get_command(self, ctx, name):
            conn = connection()
            jobs = conn.get_jobs()
    
            if name in [i['name'] for i in jobs]:
                return Job(
                    name=name,
                    params=[
                        click.Option(
                            ['-d', '--delete'], is_flag=True)
                        ]
                )
    
    

    This class can list jobs as sub-commands and map a remote job to an actual command (called Job()), which needs to be defined as well:

    class Job(click.Command):
    
        def invoke(self, ctx):
            conn = connection()
            if ctx.params['delete']:
                click.echo(f'Deleting job: {self.name}')
                conn.delete_job(self.name)
            click.echo('Completed deletion of job')
    
    

    Now create an empty sub-command called job(), which is the placeholder for all the dynamically loaded jobs. Whatever jobs exist in the remote Jenkins server, these jobs get listed here:

    @api.group(cls=DynamicJobs)
    def job():
        pass
    

    With the two classes and the new job() sub-command, the command-line tool can verify that it is reading correctly from the remote server and mapping to sub-commands:

    python staffing.py job --help
    Usage: staffing.py job [OPTIONS] COMMAND [ARGS]...
    
    Options:
      --help  Show this message and exit.
    
    Commands:
      example
    

    In this output, example is the remote Jenkins job. If I create another job named deploy appears here as well. This is how it looks now:

    python staffing.py job --help
    
    Usage: staffing.py job [OPTIONS] COMMAND [ARGS]...
    
    Options:
      --help  Show this message and exit.
    
    Commands:
      deploy
      example
    

    The two classes added, DynamicJobs() which loads and matches, and Job() which does the actual work, are crucial to add behavior to these lazily loaded commands. An option exists for deleting the job, which you verify by using the name of the job as the command:

    $ python staffing.py job deploy --help
    
    Usage: staffing.py job deploy [OPTIONS]
    
    Options:
      -d, --delete
      --help        Show this message and exit.
    

    Go ahead and delete that job now and verify the output once again:

    $ python staffing.py job deploy --delete
    
    Deleting job: deploy
    Completed deletion of job
    

    Running the staffing.py tool once more shows that the job no longer exists, and thus, it no longer appears in the help menu:

    $ python staffing.py job --help
    
    Usage: staffing.py job [OPTIONS] COMMAND [ARGS]...
    
    Options:
      --help  Show this message and exit.
    
    Commands:
      example
    

    There are many ways to extend tools and sub-commands, and being able to interact with a remote API (Jenkins, in this case) is an excellent way to make it easier to handle complex interactions.

    Utilities #

    There are a few utilities that I like to mention about Click because they make it so much easier to work with. Handling colors, launching a pager or an editor, or even handling files seamlessly between stdin and regular file paths is straightforward. You will probably not need all of these, but knowing they exist is useful to address the different needs of command-line tools. I’m guilty of implementing a lot of these utilities before, with implementations that aren’t as nice as those included with Click.

    Colored Output #

    I’m guilty of implementing colored output (from scratch) many times. Proper colored output is difficult to get right. If you haven’t tried this before, it involves sending ASCII escape codes to the terminal, which, in turn, is interpreted as a specific color. Not only one needs to send these escape codes to the terminal (as part of the string), but a reset code is needed when the coloring needs to end. This is because, otherwise, the coloring wouldn’t know where to stop. Neither newlines or whitespace is a delimiter for colors to stop.

    If you are curious, this is a handy mapping I’ve written many times for some basic colors:

    colors = dict(
        blue='\033[34m',
        green='\033[92m',
        yellow='\033[33m',
        red='\033[91m',
        bold='\033[1m',
        ends='\033[0m'
    )
    

    The ends key, indicates that the coloring has ended. The implementation on strings looks something like this:

    >>> yellow = colors['yellow']
    >>> ends = colors['ends']
    >>> message = '{0}This is a warning{0}'.format(yellow, ends)
    >>> message
    '\x1b[33mThis is a warning\x1b[33m'
    

    If you print the string or write it directly to the terminal with some other method, it makes the string yellow while keeping any subsequent string or lines the default color. All this bookkeeping is tedious and requires more granular controls to determine if a terminal is capable of sending colors or disable them altogether. This level of control is useful for programs that run on continuous integration systems that run commands, capture the output, and implement it correctly. Luckily, Click addresses this problem head-on, and allows controlling colored output with a separate library.

    There are two ways (unfortunately) to accomplish coloring of output through Click:

    import click
    
    # Output combining click.echo and click.style
    click.echo(click.style('Command Line Tools Book!', fg='green'))
    
    # Output with click.secho
    click.secho('Command Line Tools Book!', fg='green', blink=True)
    

    The API for combining both click.echo and click.style is what I prefer as it reads much better.

    There are many combinations to use for coloring text output, and my suggestion is to try the simplest first while making a conscious effort to avoid adding more color combinations. Click allows changing the background to a different color from the foreground, use bold characters, and even blink. Don’t abuse these, and try to make it as minimal as possible.

    File handling #

    File handling is not easy. It is easy in the sense that opening a file and reading it in Python is straightforward, but inside command-line applications it isn’t. Many different constraints need to get preemptively addressed to have a solid command-line tool. Here are just a few things that can happen that one needs to guard against:

    • A file path can be misspelled by the user
    • The path might be correct but permissions aren’t sufficient to open it
    • After opening the file, the encoding is not handled properly, causing encoding errors
    • Relative paths or shortened paths (like ~/) need to get expanded
    • stdin is handled differently, even though it behaves like a file descriptor

    You still need to handle all of the above, but the interface that Click offers is seamless and makes it less painful. Here are a few examples on what happens in error conditions:

    >>> click.open_file('/etc/aliases.db').read()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "lib/python/site-packages/click/utils.py", line 336, in open_file
        f, should_close = open_stream(filename, mode, encoding, errors, atomic)
      File "lib/python/site-packages/click/_compat.py", line 533, in open_stream
        return _wrap_io_open(filename, mode, encoding, errors), True
      File "lib/python/site-packages/click/_compat.py", line 511, in _wrap_io_open
        return io.open(file, mode, **kwargs)
    PermissionError: [Errno 13] Permission denied: '/etc/aliases.db'
    

    When the file can’t be found:

    >>> click.open_file('/etc/bash')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "lib/python/site-packages/click/utils.py", line 336, in open_file
        f, should_close = open_stream(filename, mode, encoding, errors, atomic)
      File "lib/python/site-packages/click/_compat.py", line 533, in open_stream
        return _wrap_io_open(filename, mode, encoding, errors), True
      File "lib/python/site-packages/click/_compat.py", line 511, in _wrap_io_open
        return io.open(file, mode, **kwargs)
    FileNotFoundError: [Errno 2] No such file or directory: '/etc/bash'
    

    Finally, when there are valid files that can be read, the encoding (defaulting to utf-8) is automatically handled:

    >>> >>> click.open_file('/etc/bashrc')
    <_io.TextIOWrapper name='/etc/bashrc' mode='r' encoding='UTF-8'>
    >>> click.open_file('/etc/passwd')