Python is approaching its third decade of good old age, and over the years many people have contributed to the creation of Python packages that perform specific functions and operations.
As of this writing, there are ~112K packages listed on the PyPI website. PyPI is short for “Python Package Index”, a central repository for free third-party Python modules.
This large and convenient module ecosystem is what makes Python so great to work with:
You see, most Python programmers are really assemblers of Python packages, which take care of a big chunk of the programming load required by modern applications.
Chances are that there is more than one Python package ready to be unleashed and help you with your specific programming needs.
For instance, while reading dbader.org, you may notice that the pages on the site render emoji quite nicely. You may wonder…
I’d like to use emoji on my Python app!
Is there a Python package for that?
Let’s find out!
Here’s what we’ll cover in this tutorial:
- Finding Python Packages
- What to Look for in a Python Package
- Installing Python Packages With Pip
- Capturing Installed Python Packages with Requirements Files
- Visualizing Installed Packages
- Installing Python Packages From a
- Uninstalling Python Packages With Pip
- Summary & Conclusion
Finding Python Packages
Let’s use the emoji use case as an example. We find emoji related Python packages by visiting the PyPI website and searching for
emoji via the search box on the top right corner of the page.
As of this writing, PyPI lists 94 packages, of which a partial list is shown below.
Notice the “
Weight*” header of the middle column. That’s a key piece of information. The weight value is basically a search scoring number, which the site calculates for each package to rank them and list them accordingly.
If we read the footnote it tells us that the number is calculated by “the occurrence of search term weighted by field (name, summary, keywords, description, author, maintainer).”
Does that mean that the top one is the best package?
Not necessarily. Although uncommon, a package maintainer may stuff
emoji into every field to try to top rank the package, which could well happen.
Conversely, many developers don’t do their homework and don’t bother filling out all the fields for their packages, which results in those packages being ranked lower.
You still need to research the packages listed, including a consideration for what your specific end use may be. For instance, a key question could be:
Which environment do you want to implement emoji on? A terminal-based app, or perhaps a Django web app?
If you are trying to display emoji on a django web app, you may be better off with the 10th package down the list shown above (package
For our use case, let’s assume that we are interested in emoji for a terminal based Python app.
Let’s check out the first one on our list (package
emoji 0.4.5) by clicking on it.
What to Look for in a Python Package
The following are characteristics of a good Python package:
- Decent documentation: By reading it we can get a clue as to whether the package could meet our need or not;
- Maturity and stability: It’s been around for some time, proven by both its age and its successive versions;
- Number of contributors: Healthy packages (especially complex ones) tend to have a healthy number of maintainers;
- Maintenance: It undergoes maintenance on a regular basis (we live in an ever-evolving world).
Although I would check it out, I wouldn’t rely too much on the development status listed for each package, that is, whether it’s a 4 - Beta or 5 - Production/Stable package. That classification is in the eye of the package creator and not necessarily reliable.
On our emoji example, the documentation seems decent. At the top of the page, we get a graphical indication of the package at work (see snippet below), which shows emoji on a Python interpreter. Yay!
The documentation for our emoji package also tells us about installing it, how to contribute to its development, etc., and points us to a GitHub page for the package, which is a great source of useful information about it.
By visiting its GitHub page, we can glean from it that the package has been around for at least two years, was last maintained in the past couple of months, has been starred 300+ times, has been forked 58 times, and has 10 contributors.
It’s looking good! We have identified a good candidate to incorporate emoji-ing into our Python terminal app.
How do we go about installing it?
Installing Python Packages With Pip
At this time, I am assuming that you already have Python installed on your system. There is plenty of info out there as to how to accomplish that.
Once you install Python, you can check whether
pip is installed by running
pip --version on a terminal.
I get the following output:
$ pip --version pip 9.0.1 from /Library/Frameworks/Python.framework/↵ Versions/3.5/lib/python3.5/site-packages (python 3.5)
Since Python 3.4,
pip is bundled with the Python installation package. If for some reason it is not installed, go ahead and get it installed.
I highly recommend also that you use a virtual environment (and more specifically,
virtualenvwrapper), a set of extensions that…
…include wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies.
For this tutorial, I have created a virtual environment called
pip-tutorial, which you will see going forward. My other tutorial walks you through setting up Python and
virtualenvwrapper on Windows.
Below you’ll see how package dependencies can bring complexity into our already complex development environments, which is the reason why using virtual environments is a must for Python development.
A great place to start learning about a terminal program is by running it without any options on the terminal. So, on your terminal, run
pip. You would get a list of Commands and General Options.
Below is a partial list of the results on my terminal:
From there on you could run
pip install --help to read on what the
install command does and what you need to specify to run it, for example. Of course, reading the pip documentation is another great place to start.
$ pip install --help Usage: pip install [options] <requirement specifier> [package-index-options] ... pip install [options] -r <requirements file> [package-index-options] ... pip install [options] [-e] <vcs project url> ... pip install [options] [-e] <local project path> ... pip install [options] <archive url/path> ... Description: Install packages from: - PyPI (and other indexes) using requirement specifiers. - VCS project urls. - Local project directories. - Local or remote source archives. pip also supports installing from "requirements files", which provide an easy way to specify a whole environment to be installed. Install Options: ...
Let’s take a quick detour and focus on the
freeze command next, which will be a key one in dealing with dependencies. Running
pip freeze displays a list of all installed Python packages. If I run it with my freshly installed virtual environment active, I should get an empty list, which is the case:
$ pip freeze
Now we can get the Python interpreter going by typing
python on our terminal. Once that’s done, let’s try to import the
emoji module, upon which
python will complain that there isn’t such a module installed, and rightfully so for we haven’t installed it, yet:
$ python Python 3.5.0 (default) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import emoji Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'emoji'
To finally install the package, we can go ahead and run
pip install emoji on our terminal. I get the following output:
$ pip install emoji==0.4.5 Collecting emoji==0.4.5 Installing collected packages: emoji Successfully installed emoji-0.4.5
🚫 Getting a pip install “invalid syntax” error?
Please note that the
pip install command needs to be run from the command-line inside your terminal program, and not inside the Python interpreter.
If you’re getting a “SyntaxError: invalid syntax” message from running
pip install, then try leaving the interpreter with Ctrl+C and run the pip command again from the terminal prompt.
Pip is a program that installs modules, so you can use them from Python. Once you have installed the module, then you can open the Python shell and import the module.
When installing packages with
pip, we can constrain
pip to install a version of our preference, by using the following operators:
A specific version of the package (
$ pip install emoji==0.4.1
A version other than the specified one (
$ pip install emoji!=0.4.1
A version equal to or greater than a specific version (
$ pip install emoji>=0.4.0
A version of the package in the specified range (
$ pip install emoji>=0.4.0, <=0.4.9
For a full specification of the version specifiers, refer to this page. Generally the most useful specifier here is
== to pip install a specific version of a package. If we don’t constrain
pip, it will grab the latest version of the package.
You may be wondering why you would want to install an older version of a Python package in the first place:
One good example of that is if you are following a tutorial which might have used the latest version of a package when it was written but which could be an older version by the time you are reading it. If you want to follow along, you would do well by installing the same version that the author used.
Another example is that if you start writing code for an app today, chances are that the packages that you use today will evolve, and new versions of it will be released in the future (while your app is “stuck” with the versions that you use today).
Programmers freeze requirements to keep track of the versions of the different packages that are installed on development and production environments. One of the objectives is to be able to replicate the environments as needed. Dan’s course on Python dependency management goes into more detail on that topic.
Let’s continue on and run
pip freeze again after installing the emoji package. You should now see it included in the list of all installed Python modules:
$ pip freeze emoji==0.4.5
pip freeze now lists the emoji package as an added dependency with a specific version number.
I now go back to my Python interpreter session, and run
import emoji, and this time around Python doesn’t complain, which is a good sign. I test it, and get the following output:
Success, at last! We just installed and then imported a third-party Python module. Great job 🙂
It’s typical for an application to have several interdependent packages. For instance, running
pip freeze on the virtual environment that I use to develop tumblingprogrammer.com, will output the following list of modules:
appdirs==1.4.3 beautifulsoup4==4.6.0 Django==1.11.1 django-bootstrap3==8.2.3 django-crispy-forms==1.6.1 django-debug-toolbar==1.8 (...) pyparsing==2.2.0 pytz==2017.2 PyYAML==3.12 selenium==3.4.1 six==1.10.0 sqlparse==0.2.3 tornado==4.5.1
That’s a total of 25 Python packages. And it’s a fairly simple application. Later on I’ll describe a way to visualize the interdependency between packages.
Capturing Installed Python Packages with Requirements Files
Developers get in the habit of freezing requirements every time that a package or a dependency gets installed on their projects. We do that by running the following pip command:
$ pip freeze > requirements.txt
This dumps the output of
pip freeze into a
requirements.txt file on the working directory.
Let’s assume now that for some reason we need to install
MarkupSafe version 0.11. Let’s assume also that we have gotten ahead, installed it, tested it, and that our app behaves as we expect it to.
pip freeze, which would only output our two packages, as shown below:
$ pip freeze emoji==0.4.5 MarkupSafe==0.11
To continue with our learning, let’s go ahead and install
Flask, a popular web microframework. We’ll grab the latest version of it by running
pip install flask.
I get the following output (if you are following along, yours might differ a little bit, for my computer had cached the files from a previous install):
$ pip install flask Collecting flask Using cached Flask-0.12.2-py2.py3-none-any.whl Collecting itsdangerous>=0.21 (from flask) Collecting Jinja2>=2.4 (from flask) Using cached Jinja2-2.9.6-py2.py3-none-any.whl Collecting click>=2.0 (from flask) Using cached click-6.7-py2.py3-none-any.whl Collecting Werkzeug>=0.7 (from flask) Using cached Werkzeug-0.12.2-py2.py3-none-any.whl Collecting MarkupSafe>=0.23 (from Jinja2>=2.4->flask) Installing collected packages: itsdangerous, MarkupSafe, Jinja2, click, Werkzeug, flask Found existing installation: MarkupSafe 0.11 Uninstalling MarkupSafe-0.11: Successfully uninstalled MarkupSafe-0.11 Successfully installed Jinja2-2.9.6 MarkupSafe-1.0 Werkzeug-0.12.2 click-6.7 flask-0.12.2 itsdangerous-0.24
Flask, being a more complex package, has some dependencies (
itsdangerous, etc.) which are installed with it automatically through the
pip install command.
I want to call your attention to the following lines, extracted from the above listing:
... Found existing installation: MarkupSafe 0.11 Uninstalling MarkupSafe-0.11: Successfully uninstalled MarkupSafe-0.11 ...
Take a close look…
You’ll see that
pip doesn’t have a way of reconciling conflicting dependencies. Without even warning us, it went ahead and replaced version 0.11 with version 1.0 of our
MarkupSafe package. And that could be trouble for our application.
At that point in time, we run our app tests (assuming that have them), and dig into our application to make sure that the changes between 0.11 and 1.0 of the
MarkupSafe package don’t break it.
If I were to face this situation in real life, I would roll back the changes first by uninstalling
Flask and its dependencies and restore the packages that I had before. Then I would upgrade
MarkupSafe to 1.0, test to make sure that the application works as expected. And then—and only then—would I re-install
Assuming that we have gone through rolling back, upgrading, testing, and re-installing
Flask, if we run
pip freeze now, we get 7 packages in total:
$ pip freeze click==6.7 emoji==0.4.5 Flask==0.12.2 itsdangerous==0.24 Jinja2==2.9.6 MarkupSafe==1.0 Werkzeug==0.12.2
Let’s go ahead and freeze our requirements into a
requirements.txt file by running
pip freeze > requirements.txt.
Now we’re going to add another package with dependencies to increase the complexity of our setup. We’ll install version 6.0 of a package called
alembic by running:
$ pip install alembic==0.6 Collecting alembic==0.6 Collecting Mako (from alembic==0.6) Collecting SQLAlchemy>=0.7.3 (from alembic==0.6) Requirement already satisfied: MarkupSafe>=0.9.2 in /Users/puma/.virtualenvs/pip-tutorial/lib/python3.5/site-packages (from Mako->alembic==0.6) Installing collected packages: Mako, SQLAlchemy, alembic Successfully installed Mako-1.0.7 SQLAlchemy-1.1.11 alembic-0.6.0
I now call your attention to the following line from the above listing:
... Requirement already satisfied: MarkupSafe>=0.9.2 in /Users/puma/.virtualenvs/pip-tutorial/lib/python3.5/site-packages (from Mako->alembic==0.6) ...
Which means that
alembic also depends on
MarkupSafe. More complexity, huh? Let’s run
$ pip freeze alembic==0.6.0 click==6.7 emoji==0.4.5 Flask==0.12.2 itsdangerous==0.24 Jinja2==2.9.6 Mako==1.0.7 MarkupSafe==1.0 SQLAlchemy==1.1.11 Werkzeug==0.12.2
The listing above showing all the packages on our emoji application is not very helpful at the moment, for it doesn’t give us info on dependencies (it only lists packages in alphabetical order). Let’s fix that.
Visualizing Installed Packages
One good package to have installed on our environment is
pipdeptree, which shows the dependency tree of packages. Let’s go ahead and install the latest version of it by running the following command:
$ pip install pipdeptree
Once it’s done, let’s run
pip freeze to see what we get:
$ pip freeze alembic==0.6.0 click==6.7 emoji==0.4.5 Flask==0.12.2 itsdangerous==0.24 Jinja2==2.9.6 Mako==1.0.7 MarkupSafe==1.0 pipdeptree==0.10.1 SQLAlchemy==1.1.11 Werkzeug==0.12.2
We now get 11 packages, as we have added
pipdeptree, which had no dependencies. Let’s run
pipdeptree on the terminal to see what it does. Below is the output that I get on my machine:
$ pipdeptree alembic==0.6.0 - Mako [required: Any, installed: 1.0.7] - MarkupSafe [required: >=0.9.2, installed: 1.0] - SQLAlchemy [required: >=0.7.3, installed: 1.1.11] emoji==0.4.5 Flask==0.12.2 - click [required: >=2.0, installed: 6.7] - itsdangerous [required: >=0.21, installed: 0.24] - Jinja2 [required: >=2.4, installed: 2.9.6] - MarkupSafe [required: >=0.23, installed: 1.0] - Werkzeug [required: >=0.7, installed: 0.12.2] pipdeptree==0.10.1 - pip [required: >=6.0.0, installed: 9.0.1] setuptools==36.2.0 wheel==0.29.0
We notice much more useful information here, including dependencies, and the minimum versions required for dependent packages to work properly.
Notice, once again, how
MarkupSafe is listed twice, as both
alembic) depend on it. That’s very useful information to troubleshoot things gone ugly.
We also notice other packages here that
pip freeze doesn’t list, including
wheel. The reason is that by default
pip freeze doesn’t list packages that
pip itself depends on.
We can use the
--all flag to show also those packages. Let’s test this by running
pip freeze --all, in which case we get:
$ pip freeze --all alembic==0.6.0 click==6.7 emoji==0.4.5 Flask==0.12.2 itsdangerous==0.24 Jinja2==2.9.6 Mako==1.0.7 MarkupSafe==1.0 pip==9.0.1 pipdeptree==0.10.1 setuptools==36.2.0 SQLAlchemy==1.1.11 Werkzeug==0.12.2 wheel==0.29.0
Another benefit of using
pipdeptree is that it warns us about conflicting dependencies, including circular ones (where packages depend on one another), but I have yet to see that in action. So far I couldn’t replicate the functionality on my system. You can find more info about the tool on its PyPI page.
Installing Python Packages From a
If you have a
requirements.txt file, you can install all the packages listed there by running the following command:
$ pip install -r /path/to/the/file/requirements.txt
This is very handy when we want to replicate environments and have access to a
requirements.txt that reflects the makeup of it.
Uninstalling Python Packages With Pip
In this section you’ll see how to uninstall individual Python packages from your system or active virtual environment, how you can remove multiple packages at once with a single command, and how you can remove all installed Python packages.
Uninstalling individual packages:
You can do so by running, as an example,
pip uninstall alembic. Let’s do that on our setup to see what happens. Here is the output on my end:
$ pip uninstall alembic Uninstalling alembic-0.6.0: /Users/puma/.virtualenvs/pip-tutorial/bin/alembic ... a bunch on other files ... /Users/puma/.virtualenvs/pip-tutorial/lib/python3.5/site-packages/alembic/util.py Proceed (y/n)? y Successfully uninstalled alembic-0.6.0
pipdeptree to see what our setup looks like:
$ pipdeptree emoji==0.4.5 Flask==0.12.2 - click [required: >=2.0, installed: 6.7] - itsdangerous [required: >=0.21, installed: 0.24] - Jinja2 [required: >=2.4, installed: 2.9.6] - MarkupSafe [required: >=0.23, installed: 1.0] - Werkzeug [required: >=0.7, installed: 0.12.2] Mako==1.0.7 - MarkupSafe [required: >=0.9.2, installed: 1.0] pipdeptree==0.10.1 - pip [required: >=6.0.0, installed: 9.0.1] setuptools==36.2.0 SQLAlchemy==1.1.11 wheel==0.29.0
If you look carefully, you may notice that the
alembic dependencies are still present, because
pip uninstall does not get rid of them, by design.
We have to manually do that (there are other options, which we will cover below). Therefore, it is extremely important that we freeze our requirements and commit changes to our
requirements.txt file every time that we install or uninstall packages so we know what our setup should look like if we need to roll back changes.
Uninstalling multiple Python packages at once:
You can also uninstall several packages at once, by using the following command-line syntax:
$ pip uninstall package1 package2 ...
Another option is reading the list of packages to uninstall from a requirements file. Similar to its
install counterpart, if you have a
requirements.txt file, you can uninstall all the packages listed there like so:
$ pip uninstall -r /path/to/the/file/requirements.txt
Note that we could wipe out all the packages on our setup, which could actually be quite useful. Let’s take a look at an example.
The output below is a list of my git commits log (
gl is an alias on my
bash profile for a prettified
$ gl * 40f4f37 - (HEAD -> master) all packages in (37 minutes ago) <Jose Pumar> * 2d00cf5 - emoji + markupsafe + flask + alembic (56 minutes ago) <Jose Pumar> * e52002b - emoji + MarkupSafe + Flask (84 minutes ago) <Jose Pumar> * 9c48895 - emoji + MarkupSafe (86 minutes ago) <Jose Pumar> * 3a797b3 - emoji + MarkSafe (2 hours ago) <Jose Pumar> * ... other commits...
If I change my mind and decide that I don’t need
alembic any more, I could delete all the packages by running
pip uninstall -r requirements.txt while on commit
40f4f37 (the HEAD).
If I do it, it gives me a bunch of warnings and asks me if I want to proceed several times (once for each package), to which I say yes. I could have avoided that by using the flag
-y, as in:
$ pip uninstall -y -r requirements.txt
-y flag tells
pip not to ask for confirmation of uninstall deletions. If we run
pip freeze after this operation, we’ll get an empty packages list, which is what we want.
We then checkout commit
e52002b (the last safe commit before we installed
alembic), and run
pip install -r requirements.txt to reinstate the packages that we had at that point in time.
Removing all installed Python packages:
Sometimes it can be useful to remove all installed packages in a virtual environment or on your system Python install. It can help you get back to a clean slate.
Running the following command will uninstall all Python packages in the currently active environment:
$ pip freeze | xargs pip uninstall -y
This command works by first listing all installed packages using the
freeze command, and then feeding the list of packages into the
pip uninstall command to remove them.
-y flag automatically confirms the uninstallation so you don’t have to stick around hammering the “y” key on your keyboard.
Installing and Uninstalling Python Packages with the “pip” Package Manager – Conclusion
Although we covered a lot of ground and shed light on key commands and major challenges that you may face when dealing with installing and uninstalling Python packages and their dependencies.
In summary, the workflow for the installation of a Python package with
pip is as follows:
- Make sure that you are using a virtual environment.
- Identify the need for a new package.
- Research potential candidate packages: Look for the maturity of the package, its documentation, etc. See what you can find regarding its dependencies. For example, other packages that have to be installed so the package works properly. Sometimes the documentation refers to them.
- Install the package and its dependent packages:
pipwill do this for you. Look for version upgrades in the
- Test your application to make sure that the package meets your needs and that the package and or its dependent packages don’t break it.
- Freeze your requirements: Run
pip freeze > requirements.txtif tests show your application is still okay and works as intended.
- Commit the changes to Git or the version control tool of your choice.
There is a lot more to cover, especially when it comes to dependency management, which has long-term implications on how we setup and configure our Python projects.
Such a complexity is one of the factors that make necessary to implement different settings and configurations to account for the distinct needs of our development, staging, and production environments.