Mastering Click: Writing Advanced Python Command-Line Apps

Welcome to the second Click tutorial on how to improve your command-line tools and Python scripts. I’ll show you some more advanced features that help you when things are getting a bit more complex and feature rich in you scripts.

You might wonder why I suggest using Click over argparse or optparse. I don’t think they are bad tools, they both have their place and being part of the standard library gives them a great advantage. However, I do think that Click is much more intuitive and requires less boilerplate code to write clean and easy-to-use command-line clients.

I go into more details about that in the first tutorial and give you a comprehensive introduction to Click as well. I also recommend you to take a look at that if this is the first time you hear the name “Click” so you know the basics. I’ll wait here for you.

Now that we are all starting from a similar knowledge level, let’s grab a cup of tea, glass of water or whatever it is that makes you a happy coder and learner ✨. And then we’ll dive into discovering:

• how you can read parameter values from environment variables,
• we’ll then separate functionality into multiple sub-commands
• and get the user to provide some input data on the command-line.
• We’ll learn what parameter types are and how you can use them
• and we’ll look at contexts in Click to share data between commands.

Sounds great? Let’s get right to it then.

Building on our existing Python command-line app

We’ll continue building on top of the example that I introduced in the previous tutorial. Together, we built a simple command-line tool that interacted with the OpenWeatherMap API.

It would print the current weather for a location provided as an argument. Here’s an example:

$ python cli.py --api-key <your-api-key> London
The weather in London right now: light intensity drizzle.

You can see the full source code on Github. As a little reminder, here’s what our final command-line tool looked like:

@click.command()
@click.argument('location')
@click.option(
    '--api-key', '-a',
    help='your API key for the OpenWeatherMap API',
)
def main(location, api_key):
    """
    A little weather tool that shows you the current weather in a LOCATION of
    your choice. Provide the city name and optionally a two-digit country code.
    Here are two examples:
    1. London,UK
    2. Canmore
    You need a valid API key from OpenWeatherMap for the tool to work. You can
    sign up for a free account at https://openweathermap.org/appid.
    """
    weather = current_weather(location, api_key)
    print(f"The weather in {location} right now: {weather}.")


if __name__ == "__main__":
    main()

In this tutorial, we’ll extend the existing tool by adding functionality to store data in a configuration file. You’ll also learn multiple ways to validate user input in your Python command-line apps.

Storing the API key in an environment variable

In the example, we have to specify the API key every time we are calling the command-line tool to access the underlying Web API. That can be pretty annoying. Let’s consider a few options that we have to improve how our tool handles this.

One of the first things that comes to mind is storing the API key in an environment variable in a 12-factor style.

$ export API_KEY="your-api-key"

We can then extract the API key from that variable in Python using os.getenv. Try it out yourself:

>>> import os
>>> api_key = os.getenv("API_KEY")
>>> print(api_key)
your-api-key

This works totally fine but it means that we have to manually integrate it with the Click parameter that we already have. Luckily, Click already allows us to provide parameter values as environment variables. We can use envvar in our parameter declaration:

@click.option(
    '--api-key', '-a',
    envvar="API_KEY",
)

That’s all! Click will now use the API key stored in an environment variable called API_KEY and fall back to the --api-key option if the variable is not defined. And since examples speak louder than words, here’s how you’d use the command with an environment variable:

$ export API_KEY="<your-api-key>"
$ python cli.py London
The weather in London right now: light intensity drizzle.

But you can still use the --api-key option with an API key as well:

$ python cli.py --api-key <your-api-key> London
The weather in London right now: light intensity drizzle.

You’re probably wondering about what happens when you have the environment variable defined and also add the option when running the weather tool. The answer is simple: the option beats environment variable.

We have now simplified running our weather command with just adding a single line of code.

Separating functionality into sub-commands

I am sure you agree that we can do better. If you’ve worked with a command-line tool like docker or heroku, you are familiar with how they manage a large set of functionality and handle user authentication.

Let’s take a look at the Heroku Toolbelt. It provides a --help option for more details:

$ heroku --help
Usage: heroku COMMAND

Help topics, type heroku help TOPIC for more details:

 access          manage user access to apps
 addons          tools and services for developing, extending, and operating your app
 apps            manage apps
 auth            heroku authentication
 authorizations  OAuth authorizations
 ... # there's more but we don't care for now

They use a mandatory argument as a new command (also called sub-command) that provides a specific functionality. For example heroku login will authenticate you and store a token in a configuration file if the login is successful.

Wouldn’t it be nice if we could do the same for our weather command? Well, we can! And you’ll see how easy it is as well.

We can use Click’s Commands and Groups to implement our own version of this. And trust me, it sounds more complicated than it actually is.

Let’s start with looking at our weather command and defining the command that we’d like to have. We’ll move the existing functionality into a command and name it current (for the current weather). We’d now run it like this:

$ python cli.py current London
The weather in London right now: light intensity drizzle.

So how can we do this? We start by creating a new entry point for our weather command and registering it as a group:

@click.group()
def main():
   pass

We have now turned our main function into a command group object that we can use to register new commands “below” it. What that means is, that we change our @click.command decorator to @main.command when wrapping our weather function. We’ll also have to rename the function from main to the name we want to give our command. What we end up with is this:

@main.command()
@click.argument('location')
@click.option(
    '--api-key', '-a',
    help='your API key for the OpenWeatherMap API',
)
def current(location, api_key):
    ...

And I’m sure you’ve already guessed it, this means we know run our command like this:

$ python cli.py current London
The weather in London right now: light intensity drizzle.

Storing the API key in a configuration file using another sub-command

The change we made above obviously doesn’t make sense on its own. What we wanted to add is a way to store an API key in a configuration file, using a separate command. I suggest we call it config and make it ask the user to enter their API key:

$ python cli.py config
Please enter your API key []: your-api-key

We’ll then store the key in a config file that we’ll put into the user’s home directory: e.g. $HOME/.weather.cfg for UNIX-based systems.

$ cat ~/.weather.cfg
your-api-key

We start with adding a new function to our Python module with the same name as our command and register it with our main command group:

@main.command()
def config():
    """
    Store configuration values in a file.
    """
    print("I handle the configuration.")

You can now run that new command and it will print the statement above.

$ python cli.py config
I handle the configuration.

Boom, we’ve now extended our weather tool with two separate commands:

$ python cli.py --help
<NEED CORRECT OUTPUT>

Asking the user for command-line input

We created a new command but it doesn’t to anything, yet. What we need is the API key from the user, so we can store it in our config file. Let’s start using the --api-key option on our config command and write it to the configuration file.

@main.command()
@click.option(
    '--api-key', '-a',
    help='your API key for the OpenWeatherMap API',
)
def config(api_key):
    """
    Store configuration values in a file.
    """
    config_file = os.path.expanduser('~/.weather.cfg')

    with open(config_file, 'w') as cfg:
        cfg.write(api_key)

We are now storing the API key provided by the user in our config file. But how can we ask the user for their API key like I showed you above? By using the aptly named click.prompt.

@click.option(
    '--api-key', '-a',
    help='your API key for the OpenWeatherMap API',
)
def config(api_key):
    """
    Store configuration values in a file.
    """
    config_file = os.path.expanduser('~/.weather.cfg')

    api_key = click.prompt(
        "Please enter your API key",
        default=api_key
    )

    with open(config_file, 'w') as cfg:
        cfg.write(api_key)

Isn’t it amazing how simple that was? This is all we need to have our config command print out the question asking the user for their API key and receiving it as the value of api_key when the user hits [Enter].

We also continue to allow the --api-key option and use it as the default value for the prompt which means the user can simply hit [Enter] to confirm it:

$ python cli.py config --api-key your-api-key
Please enter your API key [your-api-key]:

That’s a lot of new functionality but the code required is minimal. I’m sure you agree that this is awesome!

Introducing Click’s parameter types

Until now, we’ve basically ignored what kind of input we receive from a user. By default, Click assumes a string and doesn’t really care about anything beyond that. That makes it simple but also means we can get a lot of 🚮.

You probably guessed it, Click also has a solution for that. Actually there are multiple ways of handling input but we’ll be looking at Parameter Types for now.

The name gives a pretty good clue at what it does, it allows us to define a the type of our parameters. The most obvious ones are the builtin Python types such as str, int, float but Click also provides additional types: Path, File and more. The complete list is available in the section on Parameter Types.

Ensuring that an input value is of a specific type is as easy as you can make it. You simply pass the parameter type you’re expecting to the decorator as type argument when defining your parameter. Something like this:

@click.option('--api-key', '-a', type=str)
@click.option('--config-file', '-c', type=click.Path())

Looking at our API key, we expect a string of 32 hexadecimal characters. Take a moment to look at this Wikipedia article if that doesn’t mean anything to you or believe me when I say it means each character is a number between 0 and 9 or a letter between a and f.

There’s a parameter type for that, you ask? No, there is not. We’ll have to build our own. And like everything else, it’ll be super easy (I feel like a broken record by now 😇).

Building a custom parameter type to validate user input

What do we need implement our own parameter type? We have to do two things: (1) we define a new Python class derived from click.ParamType and (2) implement it’s convert method. Classes and inheritance might be a new thing for you, so make sure you understand the benefits of using classes and are familiar with Object-Oriented Programming.

Back to implementing our own parameter type. Let’s call it ApiKey and start with the basic boilerplate:

class ApiKey(click.ParamType):

    def convert(self, value, param, ctx):
        return value

The only thing that should need some more explanation is the list of arguments expected by the convert method. Why are there three of them (in addition to self) and where do they come from?

When we use our ApiKey as the type for our parameter, Click will call the convert method on it and pass the user’s input as the value argument. param will contain the parameter that we declared using the click.option or click.argument decorators. And finally, ctx refers to the context of the command which is something that we’ll be talking about later in this tutorial.

The last thing to note is the return value. Click expects us to either return the cleaned and validated value for the parameter or raise an exception if the value is not valid. If we raise an exception, Click will automatically abort and tell the user that their value is not of the correct type. Sweet, right?

That’s been a lot of talk and no code, so let’s stop here, take a deep breath and look at the implementation.

import re

class ApiKey(click.ParamType):
    name = 'api-key'

    def convert(self, value, param, ctx):
        found = re.match(r'[0-9a-f]{32}', value)

        if not found:
            self.fail(
                f'{value} is not a 32-character hexadecimal string',
                param,
                ctx,
            )

        return value

You can see that we’re only interested in the value of our parameter. We use a regular expression to check for a string of 32 hexadecimal characters. I won’t go into details on regular expressions here but Al Sweigart does in this PyCon video.

Applying a re.match will return a match object for a perfect match or None otherwise. We check if they match and return the unchanged value or call the fail() method provided by Click to explain why the value is incorrect.

Almost done. All we have to do now is plug this new parameter type into our existing config command.

@main.command()
@click.option(
    '--api-key', '-a',
    type=ApiKey(),
    help='your API key for the OpenWeatherMap API',
)
def config(api_key):
    ...

And we are done! A user will now get an error if their API key is in the wrong format and we can put an end to those sleepless nights 🤣.

$ python cli.py config --api-key invalid
Usage: cli.py [OPTIONS] COMMAND [ARGS]...

Error: Invalid value for "--api-key" / "-a": your-api-key is not a 32-character hexadecimal string

I’ve thrown a lot of information at you. I have one more thing that I’d like to show you before we end this tutorial. But if you need a quick break, go get yourself a delicious beverage, hot or cold, and continue reading when you feel refreshed. I’ll go get myself a ☕️ and be right back…

Using the Click context to pass parameters between commands

Alright, welcome back 😉. You probably thought about the command we created, our new API key option and wondered if this means we actually have to define the option on both of our commands, config and current. And your assumption would be correct. Before your eyes pop out and you shout at me “Hell no! I like my code DRY!”, there’s a better way to do this. And if DRY doesn’t mean anything to you, check out this Wikipedia arcticle on the “Don’t Repeat Yourself” principle.

How can we avoid defining the same option on both commands? We use a feature called the “Context”. Click executes every command within a context that carries the definition of the command as well as the input provided by the user. And it comes with a placeholder object called obj, that we can use to pass arbitrary data around between commands.

First let’s look at our group and how we can get access to the context of our main entrypoint:

@click.group()
@click.pass_context
def main(ctx):
   ctx.obj = {}

What we are doing here is telling Click that we want access to the context of the command (or group) and Click will pass it to our function as the first argument, I called it ctx. In the function itself, we can now set the obj attribute on the context to an empty dictionary that we can then fill with data. obj can also be an instance of a custom class that we implement but let’s keep it simple. You can imagine how flexible this is. The only thing you can’t do, is assign your data to anything but ctx.obj.

Now that we have access to the context, we can move our option --api-key to the main function and then save then store the API key in the context:

@click.group()
@click.option(
    '--api-key', '-a',
    type=ApiKey(),
    help='your API key for the OpenWeatherMap API',
)
@click.pass_context
def main(ctx, api_key):
    ctx.obj = {
        'api_key': api_key,
    }

I should mention that it doesn’t matter where you put the click.pass_context decorator, the context will always be the first argument. And with the API key stored in the context, we can now get access to it in both of our commands by adding the pass_context decorator as well:

@main.command()
@click.pass_context
def config(ctx):
    api_key = ctx.obj['api_key']
    ...

The only thing this changes for the user, is that the --api-key option has to come before the config or current commands. Why? Because the option is no associated with the main entry point and not with the sub-commands:

$ python cli.py --api-key your-api-key current Canmore
The weather in Canmore right now: overcast clouds.

I think that’s a small price to pay for keeping our code DRY. And even if you disagree with me, you still learned how the Click context can be used for sharing data between commands; that’s all I wanted anyways 😇.

Advanced Python CLIs with Click — Summary

Wow, we work though a lot of topics. You should have an even better knowledge of Click and it features now. Specifically we looked at:

• How to read parameter values from environment variables.
• How you can separate functionality into separate commands.
• How to ask the user for input on the command-line.
• What parameter types are in Click and how you can use them for input validation.
• How Click contexts can help you share data between commands.

I am tempted to call you a Master of Click 🏆 with all of the knowledge you have now. At this point, there should be little that you don’t know how to do. So start playing around with what you learned and improve you own command-line tools. Then come back for another tutorial on testing and packaging of Click commands.

Full code example

import re
import os
import click
import requests

SAMPLE_API_KEY = 'b1b15e88fa797225412429c1c50c122a1'


class ApiKey(click.ParamType):
    name = 'api-key'

    def convert(self, value, param, ctx):
        found = re.match(r'[0-9a-f]{32}', value)

        if not found:
            self.fail(
                f'{value} is not a 32-character hexadecimal string',
                param,
                ctx,
            )

        return value


def current_weather(location, api_key=SAMPLE_API_KEY):
    url = 'https://api.openweathermap.org/data/2.5/weather'

    query_params = {
        'q': location,
        'appid': api_key,
    }

    response = requests.get(url, params=query_params)

    return response.json()['weather'][0]['description']


@click.group()
@click.option(
    '--api-key', '-a',
    type=ApiKey(),
    help='your API key for the OpenWeatherMap API',
)
@click.option(
    '--config-file', '-c',
    type=click.Path(),
    default='~/.weather.cfg',
)
@click.pass_context
def main(ctx, api_key, config_file):
    """
    A little weather tool that shows you the current weather in a LOCATION of
    your choice. Provide the city name and optionally a two-digit country code.
    Here are two examples:
    1. London,UK
    2. Canmore
    You need a valid API key from OpenWeatherMap for the tool to work. You can
    sign up for a free account at https://openweathermap.org/appid.
    """
    filename = os.path.expanduser(config_file)

    if not api_key and os.path.exists(filename):
        with open(filename) as cfg:
            api_key = cfg.read()

    ctx.obj = {
        'api_key': api_key,
        'config_file': filename,
    }


@main.command()
@click.pass_context
def config(ctx):
    """
    Store configuration values in a file, e.g. the API key for OpenWeatherMap.
    """
    config_file = ctx.obj['config_file']

    api_key = click.prompt(
        "Please enter your API key",
        default=ctx.obj.get('api_key', '')
    )

    with open(config_file, 'w') as cfg:
        cfg.write(api_key)


@main.command()
@click.argument('location')
@click.pass_context
def current(ctx, location):
    """
    Show the current weather for a location using OpenWeatherMap data.
    """
    api_key = ctx.obj['api_key']

    weather = current_weather(location, api_key)
    print(f"The weather in {location} right now: {weather}.")


if __name__ == "__main__":
    main()