Introduction

At the beginning of my python development road, I've got familiar with Python Virtual Environment which was a very useful discovery for me. Before this finding I had a real mess in my development environment: every single library for every project was installed in one place. And when I asked my self: "What dependencies do I have for this project?", I could not give the exact answer.

If you are a beginner in Python and you don't know what is the virtual environment you definitely should go and start using it in your day to day development life.

Shortly speaking, the python virtual environment allows you to isolate all python library dependencies for your specific project. So if you have several python projects that use various versions of the library, e.g. your old university project uses Django 1.1.0 and your recent Django blog uses Django 3.0.7. For each project, you create a separate virtual environment and install your old or new shit in there.

When I started the development of this blog I've renamed my project one time: "djangoblog" to "www.salvicode.com". And at some point, I've started noticing that all installed libraries via pip install my_library got installed to my general python folder ignoring virtual environment lib folder (venv/lib/python3.7/site-packages) and pip freeze was giving me the result as if I called it outside of the virtual environment. I've spent a day trying to find out what had happened. My google searches like "python virtual env does not work", "python virtual env pip freeze issues", etc. did not bring many results. I thought I've broken everything and decided to quit development to remove the virtual environment and create it from scratch. Everything started working. The issue fix is from the type of the issues like "try to turn on and turn off your PC, mister Sergei. Thank you for your call".

Let's try to break down what happened there with the python virtual environment.

Prerequisites

At the moment of writing this article, I'm using Python 3.7.3

Also, pip should be installed.

To work with a virtual environment you can use python3 -m venv command or install virtualenv.

I'm using Mac OS 10.15.5. So if you are on Windows there will be some difference in activating virtual environment(Can't say you more what is there on Windows).

Repro steps

Create a python project folder somewhere and go into it.

mkdir venv_test
cd venv_test

Let's create a virtual environment and name it venv:

python3 -m venv venv

Or if you use virtualenv:

virtualenv venv

To activate the virtual environment run:

source venv/bin/activate

(venv) means you've activated a virtual environment. And everything you are going to do will be done within that virtual environment. Let's check what libraries do we have installed in our virtual env:

pip freeze

Nothing, right? Let's install Django

pip install django

And now let's check what we have installed:

pip freeze

In my case it shows django library and its dependencies that were installed along with django:

And now. Let's break our system.

Deactivate virtual environment and move out of the folder venv_test:

deactivate
cd ..

And let's rename our folder venv_test to venv_test_new:

mv venv_test venv_test_new

Go into that new folder and try to activate our virtual environment:

cd venv_test_new
source venv/bin/activate

Now you should see that everything seems fine for now:

It shows (venv), no errors are shown, the weather is quiet. Let's try to run pip freeze.

Quite a lot, huh? Have you installed anything before? No? Neither did I. And where is our Django library?

Under the hood

Let's check why it has happened.

Python resolves all virtual environment stuff with few bash scripts that are located in venv/bin/ folder. Open activate, activate.csh and activate.fish files. Three of them contains ABSOLUTE path of our previous folder! In my case these lines are:

# activate
VIRTUAL_ENV="/Users/sergeialekseev/Work/salvicode/venv_test/venv"
export VIRTUAL_ENV

# activate.csh
setenv VIRTUAL_ENV "/Users/sergeialekseev/Work/salvicode/venv_test/venv"

# activate.fish
set -gx VIRTUAL_ENV "/Users/sergeialekseev/Work/salvicode/venv_test/venv"

That's the reason why our virtual environment does not work properly. And the funny thing is that nothing informs us about this.

Solution 1

Basic idea is to move our project to the previous folder(venv_test), activate the virtual environment, grab a list of the libraries we use via pip freeze > requirements.txt, deactivate virtual environment, remove your virtual environment(venv) folder, move your project to a new folder(venv_test_new) and create a new virtual environment with python3 -m venv venv and install all libraries via pip install -r requirements.txt.

Quick and easy solution. My favorite one. But it will work if you know your previous location or at least you have a good habit to update your dependencies file(requirements.txt) each time you add a new library.

Solution 2

Let's update these 3 scripts: activate, activate.csh, activate.fish

Find all old locations and replace them with your new path.

You think that's it? Nope. Also, you need to modify all other python scripts that you can find in that venv/bin folder. Usually, it is only the top line of the files:

# pip, pip3, etc. files
#!/Users/sergeialekseev/Work/salvicode/venv_test/venv/bin/python3
...

I have not really tested this solution. But at least when you call pip freeze after all path replacing stuff it shows correct libraries.

I don't recommend this way since it seems like an unstable one. Or if you don't have any other choice you can pip freeze > requirements.txt and then just in case recreate virtual environment as it was done in the previous solution.

Solution 3

I've found some discussion on this topic at stackoverflow. There is a mention of --relocatable param during the creation of a virtual environment via virtualenv. I have not tried this one. Maybe you can try it and let me know how it works.

Solution 4

Use Pipenv. I've never tried it but it seems it was created to simplify this mess with dependencies.

Conclusion

It is always better to have read good practices before starting doing something. I mean having requirements.txt file with all dependencies of your project is a good practice that can help you in the future. And also having isolated environment during development is also a good practice. 

I hope that will help you avoid wasting your time trying to find out WTF has happened to the virtual environment.