Category: Programming

Python and pyenv: In Practice

Programming / July 3, 2018 / Shaun Hamman / 0

This post has been migrated from my older blog The Castle.

If you’re looking for an explanation of what all the moving parts are, make sure to read the previous post on this topic. If you’re ready to fire up your terminal and get your Python environment set up once and for all, read on!

A few details before we start:

• We’re going to assume you’re working in a Linux environment. While pyenv may still be helpful on Windows, Windows is much better at handling multiple Python installs.
• As to which particular flavor of Linux, we’ll be using CentOS 7. By and large this shouldn’t really matter, except that you’ll need to use your distribution’s package manager of choice. For us, it’s yum but you may use apt, zypper, pacman, emerge, dnf, or something else entirely. The process should be roughly the same, but some of the package names may differ.
• The pyenv setup and configuration utilizes bash scripts, so we will be using the bash shell for this guide. I’m not familiar with other shells, so I can’t say how much work it would be to modify those scripts to run in other shells.

Update 2018-07-10: My understanding of pyenv was slightly mistaken. pyenv is a bash extension, thus it cannot be run on Windows or in a shell other than bash. The majority of the functionality pyenv provides is implemented into Python itself as of version 3.3, so the virtual environment handling is available cross-platform/shell, but the ability to work across Python versions (including versions prior to 3.3) and to automatically enable and disable environments on a per directory basis is pyenv exclusive as far as I know.

With all that settled, let’s open up a terminal and get started!

CentOS, like most Linux distributions, comes with Python pre-installed. However, it has become best practice to basically never use this installation. It’s being used by the operating system, and updating or installing modules could potentially interfere with its operation. So we’re going to take some minimally invasive steps to separate ourselves from it.

sudo yum -y install gcc zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel openssl-devel xz xz-devel tk-devel libffi-devel python-pip

Update 2018-07-21: Added a few potentially missing dependencies. See https://github.com/pyenv/pyenv/wiki for the list of dependencies you should install for your particular linux distribution.

This will install a number of libraries that Python and/or pyenv need in order to run properly. Of particular note here is python-pip. Pip is the Python-specific package manager. It’s actually invoked by calling Python and passing a particular parameter string, but many tutorials will invoke it directly. This package provides a simple alias to allow you to do that. If you want to skip this step and invoke Pip using the Python executable, you can do so with the command:

python -m pip install

Next we’ll upgrade pip to the latest version.

sudo pip install --upgrade pip

Note: In general, you shouldn’t have to sudo every command. If you do, either your permissions are not configured appropriately, or you’re doing something you probably shouldn’t be doing. Here, we only use sudo when dealing with the system installation of Python. When dealing with pyenv or any installation of Python installed by pyenv, you should explicitly NOT use sudo. Using sudo when executing a non-system pip, for example, may install packages to the wrong installation of Python.

At this point, there is one last step to take before we are done messing with the built-in version of Python: installing pyenv. pyenv works by replacing the Python executables with proxy shims that will redirect your commands to the correct Python installation. It does this in such a way that the operating system won’t know it’s happening, and thus it shouldn’t have any effect on the system’s operation.

curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash

This downloads and executes the pyenv-installer script which installs pyenv and those shims. The final step to getting pyenv up and running is to open the ~/.bashrc file and add the following lines to the bottom of it:

export PATH=$HOME/.pyenv/bin:$PATH

eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

This adds pyenv to the beginning of your PATH and runs some initialization scripts any time you open a terminal. Now close the terminal and open a new one so that these initialization scripts run, and that’s it! pyenv is installed! There is one final thing we want to do though. I mentioned before that best practice is to not touch the system installation of Python. With pyenv, we can take steps to make sure we don’t accidently do that. On CentOS, the system version of Python is 2.7.5. What we can do is install a separate copy of Python 2.7.5 and tell pyenv that we want to use that if no other environment is specified. That way if we accidently install or upgrade something outside of a virtual environment, it will use this second installation and not our system one.

pyenv install 2.7.5
pyenv global 2.7.5

Since we haven’t used Python 2.7.5 from within pyenv before, the first line tells Python to go ahead and create its own installation of 2.7.5. The second line says that when we’re not using any other virtual environment, we should use this one. Now we are completely decoupled from the system installation. Now, let’s say we want to start a new project using Python 3.6.5. Since we haven’t installed Python 3.6.5 before, we tell pyenv to do so:

pyenv install 3.6.5

And then we create a new virtual environment for our particular project. This allows us to use the same Python version for multiple projects, but also ensures our installed dependencies and versions won’t interfere across projects. We tell pyenv to name our virtual environment “myproject” and that we want to use Python version 3.6.5.

pyenv virtualenv 3.6.5 myproject

Now you can switch to this environment to work on your project

pyenv activate myproject

and you can be sure that anything you do won’t have any impact on any other project or installation on your machine! Bonus: when you go to use python or pip, you can just run that command. No more python2, python3, python36, etc.! Just make sure that when you’re done working on your project that you deactivate your environment

pyenv deactivate

so that you don’t accidently mess it up in the meantime.

pyenv has one other really cool trick that you’ll want to use. Let’s say you have a handful of different projects using various Python versions and dependencies. This all works fine because each project has its own virtual environment. But it’s a hassle having to manually enable and disable the environments since you’re working on all the projects at the same time, and you have to remember to do actually do that. Well, pyenv can help you out here. You can mark a particular directory as using a particular environment, then any python commands you execute from within that directory (or any of its children) will automatically use the correct environment! You can set this up by navigating to your project’s root directory and running:

pyenv local myproject

This will create a .python-version file in this directory, which is what pyenv uses to know it needs to automatically change the virtual environment! Make sure to add this to your .gitignore or otherwise exclude this file from your source control.

And there you have it! You’re all set up and ready to write some Python! It’s actually not that complicated of a process, and the possibilities and workflows it opens up are pretty powerful, but most of the documentation around this process is scattered, broken, out of date, or just otherwise woefully lacking. If you have any questions or run into any trouble, please leave a comment below. Otherwise, enjoy writing some Python!

~ S

Python, Virtual Environments, and pyenv: Explained

Programming / June 13, 2018 / Shaun Hamman / 0

This post has been migrated from my older blog The Castle.

On a recent client project I was required to set up a Python development environment on a CentOS 7 virtual machine. I haven’t had to work with Python since version 2.3, so I was basically starting from scratch as far as prior knowledge went. I’m generally pretty good at figuring things out and following instructions, but my gosh what a mess!

Between the system insisting that there are no newer versions of Python past 2.7.5 (there’s a reason for this that I’ll detail in a moment, but the actual latest is currently 3.6.5) and every other blog post and documentation page I could find telling me to do different things to set it up, I don’t know how anyone can call Python “a language for beginners”.

I ended up spending three days trying to get the system to work without it complaining pip wasn’t installed or that I didn’t have permission to do something. Many blog posts, Stack Overflow questions, and re-created virtual machines later I finally got it working and (I think) I understand how everything fits together. My stance on it now? The way Python handles environments is pretty clever and powerful once you get it set up, but getting it to that point is a nightmare!

For those of you who are as lost as I was, let me explain all the pieces to you in simple, understandable terms. Then I’ll make a second post that shows you step by step how to get this all up and running slick as a whistle.

Python

Let’s talk about Python. You may have seen in various guides that the Python executable is referred to differently depending on which tutorial you’re reading. It can be either python or python# where # is seemingly any arbitrary number. Basically, because Python comes pre-installed in many Linux distributions, a number of system and third-party scripts started relying on it. The executable at that time was python. This persisted up until Python 2.x. When Python 3.0 was released, the parser changed fairly significantly. Rather than trying to handle versioning in a scalable, future-proof way, the Python developers decided that the Python 3.x executable would be named python3.

Here is where things get tricky. Because some distributions wanted to update to Python 3, but wanted their existing scripts to continue working, they aliased python to resolve into python3. This, apparently, broke things. So Python released an official statement saying that python should always refer to Python 2.x and that distributions should use python# to refer to a specific version installation. But, each distribution interprets the # differently, so Python 3.6.5 can be any of python, python3, python36, python365, python3.6 (yes, the executable has a dot in the filename), or python3.6.5. As a commonly installed module, pip follows this same convention and has the same problem. Interestingly, a growing contingent of users are beginning to frown on the use of pip install , instead perferring the more direct python -m pip install . I suspect this is because depending on your system setup, python and pip may reference different versions of Python, which won’t do what you think it does.

Virtual Environments

To make matters worse, there is also a potential conflict with dependency versions. Assume, for example, that your Linux distribution has a dependency on pyzip 0.1. You want to use pyzip 0.2 in your scripts, so you pip install –upgrade pyzip. One of two things happens here:

1. It throws an error while updating and you are effectively stuck with pyzip 0.1.
2. It upgrades pyzip to 0.2 and your system becomes unstable.

Obviously, these are both bad outcomes. In order to handle this, Python came up with the concept of Virtual Environments. Basically, a virtual environment is a sandbox. You select which environment you want to use, then you can install, upgrade, or uninstall any dependencies you want without regard for anything else on the system. It is all kept separate. You can create a new virtual environment just for your current project and the entire Python installation will be completely unaware of any dependencies (or versions thereof) of any other project as well as the system itself. This is where virtualenv comes in. virtualenv is an extension included with later versions of Python that allows you to create, delete, and manage these virtual environments. There is a catch though, it can only do so within a single Python version installation. So you still need separate python and python3 executables, and separate sets of virtual environments for each.

pyenv

Enter pyenv. pyenv is a virtualenv wrapper. When installed properly, pyenv adds a shim which basically takes over your system’s python and pip commands. pyenv then allows you to manage your virtual environments as well as all your Python versions all from a single executable. You can install new versions of Python, create new virtual environments for different projects, and never need to worry about what # to stick on the end of your python call or where Python is installed to ever again. You always call python or pip, then pyenv will automatically route your request to the correct version, installation, and set of dependencies for what you are doing.

Note the words “when installed properly”. Apparently, pyenv is particularly difficult to install by hand. Thus a sister project, pyenv-installer was born. It seems, though, that the installation instructions provided on the pyenv-installer don’t actually work. I will walk you through the correct way of installing it in the next post.

So I hope that helps clear all that up. Python, like most scripting languages, is a hot mess of libraries with obtuse names and wrappers upon wrappers upon wrappers, and most documentation assumes you have intimate knowledge of not just Python, but half the libraries involved. It can be very difficult to find your footing and much of the community documentation that is out there is long out of date. The information is there if you’re willing to dig, and my goal is to do some of that excavation so you don’t have to.

~ S