Environments#
Note
This notebook uses the bash kernel. Install it with python3 -m pip install bash_kernel or run these commands in a shell.
A “virtual environment” is a directory isolating a collection of packages.
The venv package in the standard library creates virtual environments.
There are various wrappers, such as virtualenvwrapper that can add some functionality if you use envs a lot.
to create an env:
# --clear means delete an env if there already was one at this location
!python3 -m venv --clear ./env-1
!./env-1/bin/pip install --upgrade pip
# tree will show us a peek at what's in the env
!tree -L 2 env-1
Requirement already satisfied: pip in ./env-1/lib/python3.10/site-packages (23.0.1)
Collecting pip
Using cached pip-23.3.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 23.0.1
Uninstalling pip-23.0.1:
Successfully uninstalled pip-23.0.1
Successfully installed pip-23.3.1
env-1
├── bin
│ ├── activate
│ ├── activate.csh
│ ├── activate.fish
│ ├── Activate.ps1
│ ├── pip
│ ├── pip3
│ ├── pip3.10
│ ├── python -> python3
│ ├── python3 -> /home/dokken/src/mambaforge/envs/UIO-IN3110/bin/python3
│ └── python3.10 -> python3
├── include
├── lib
│ └── python3.10
├── lib64 -> lib
└── pyvenv.cfg
5 directories, 11 files
I have a lot of packages in my main environment!
!python3 -m pip list | wc -l && which python3
[notice] A new release of pip is available: 23.1.2 -> 23.3.1
[notice] To update, run: pip install --upgrade pip
314
/home/dokken/src/mambaforge/envs/UIO-IN3110/bin/python3
If we consider the new environment, we barely have any packages
# If you want to run this in the terminal, replace `%env` with `export`
%env VIRTUAL_ENV_DISABLE_PROMPT=1
!source ./env-1/bin/activate && python3 -m pip list && which python3
env: VIRTUAL_ENV_DISABLE_PROMPT=1
Package Version
------------------------- ------------
altair 5.1.2
attrs 23.1.0
blinker 1.7.0
click 8.1.7
Flask 3.0.0
itsdangerous 2.1.2
Jinja2 3.1.2
jsonschema 4.19.2
jsonschema-specifications 2023.7.1
MarkupSafe 2.1.3
numpy 1.26.2
packaging 23.2
pandas 2.1.3
pip 23.3.1
python-dateutil 2.8.2
pytz 2023.3.post1
referencing 0.30.2
rpds-py 0.12.0
setuptools 65.5.0
six 1.16.0
toolz 0.12.0
typing_extensions 4.8.0
tzdata 2023.3
vega-datasets 0.9.0
Werkzeug 3.0.1
/home/dokken/Documents/src/UiO/UiO-IN3110.github.io/lectures/production/env-1/bin/python3
!cat requirements.txt
altair
flask
vega_datasets
# To run this command in a terminal, simply call the source command above, as it sets the path to the correct python3 installation
!./env-1/bin/python3 -m pip install -r requirements.txt
Requirement already satisfied: altair in ./env-1/lib/python3.10/site-packages (from -r requirements.txt (line 1)) (5.1.2)
Requirement already satisfied: flask in ./env-1/lib/python3.10/site-packages (from -r requirements.txt (line 2)) (3.0.0)
Requirement already satisfied: vega_datasets in ./env-1/lib/python3.10/site-packages (from -r requirements.txt (line 3)) (0.9.0)
Requirement already satisfied: jinja2 in ./env-1/lib/python3.10/site-packages (from altair->-r requirements.txt (line 1)) (3.1.2)
Requirement already satisfied: jsonschema>=3.0 in ./env-1/lib/python3.10/site-packages (from altair->-r requirements.txt (line 1)) (4.19.2)
Requirement already satisfied: numpy in ./env-1/lib/python3.10/site-packages (from altair->-r requirements.txt (line 1)) (1.26.2)
Requirement already satisfied: packaging in ./env-1/lib/python3.10/site-packages (from altair->-r requirements.txt (line 1)) (23.2)
Requirement already satisfied: pandas>=0.25 in ./env-1/lib/python3.10/site-packages (from altair->-r requirements.txt (line 1)) (2.1.3)
Requirement already satisfied: toolz in ./env-1/lib/python3.10/site-packages (from altair->-r requirements.txt (line 1)) (0.12.0)
Requirement already satisfied: typing-extensions>=4.0.1 in ./env-1/lib/python3.10/site-packages (from altair->-r requirements.txt (line 1)) (4.8.0)
Requirement already satisfied: Werkzeug>=3.0.0 in ./env-1/lib/python3.10/site-packages (from flask->-r requirements.txt (line 2)) (3.0.1)
Requirement already satisfied: itsdangerous>=2.1.2 in ./env-1/lib/python3.10/site-packages (from flask->-r requirements.txt (line 2)) (2.1.2)
Requirement already satisfied: click>=8.1.3 in ./env-1/lib/python3.10/site-packages (from flask->-r requirements.txt (line 2)) (8.1.7)
Requirement already satisfied: blinker>=1.6.2 in ./env-1/lib/python3.10/site-packages (from flask->-r requirements.txt (line 2)) (1.7.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./env-1/lib/python3.10/site-packages (from jinja2->altair->-r requirements.txt (line 1)) (2.1.3)
Requirement already satisfied: attrs>=22.2.0 in ./env-1/lib/python3.10/site-packages (from jsonschema>=3.0->altair->-r requirements.txt (line 1)) (23.1.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in ./env-1/lib/python3.10/site-packages (from jsonschema>=3.0->altair->-r requirements.txt (line 1)) (2023.7.1)
Requirement already satisfied: referencing>=0.28.4 in ./env-1/lib/python3.10/site-packages (from jsonschema>=3.0->altair->-r requirements.txt (line 1)) (0.30.2)
Requirement already satisfied: rpds-py>=0.7.1 in ./env-1/lib/python3.10/site-packages (from jsonschema>=3.0->altair->-r requirements.txt (line 1)) (0.12.0)
Requirement already satisfied: python-dateutil>=2.8.2 in ./env-1/lib/python3.10/site-packages (from pandas>=0.25->altair->-r requirements.txt (line 1)) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in ./env-1/lib/python3.10/site-packages (from pandas>=0.25->altair->-r requirements.txt (line 1)) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in ./env-1/lib/python3.10/site-packages (from pandas>=0.25->altair->-r requirements.txt (line 1)) (2023.3)
Requirement already satisfied: six>=1.5 in ./env-1/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas>=0.25->altair->-r requirements.txt (line 1)) (1.16.0)
!./env-1/bin/python3 -m pip list
Package Version
------------------------- ------------
altair 5.1.2
attrs 23.1.0
blinker 1.7.0
click 8.1.7
Flask 3.0.0
itsdangerous 2.1.2
Jinja2 3.1.2
jsonschema 4.19.2
jsonschema-specifications 2023.7.1
MarkupSafe 2.1.3
numpy 1.26.2
packaging 23.2
pandas 2.1.3
pip 23.3.1
python-dateutil 2.8.2
pytz 2023.3.post1
referencing 0.30.2
rpds-py 0.12.0
setuptools 65.5.0
six 1.16.0
toolz 0.12.0
typing_extensions 4.8.0
tzdata 2023.3
vega-datasets 0.9.0
Werkzeug 3.0.1
But what about reproducible builds? What if a new release of altair breaks my application?
Solution: Pin specific versions?
# requirements.txt
altair==4.1.0
vega-datasets==0.8.0
flask==1.1.2
Warning
💣 this is the worst possible thing!
it ensures that your direct dependencies are not updated (fine), but
it does not ensure that their dependencies are not updated
This guarantees that your env will break when a dependency is updated. If you had left everything unpinned, it is likely your code would not break, unless the APIs your code uses change. But partial pinning is a way to guarantee something will break, even if your code works fine with the latest version of everything.
If you are pinning dependencies, it should be all or nothing, never partial. Ideally, this should include Python itself!
!./env-1/bin/python3 -m pip freeze
altair==5.1.2
attrs==23.1.0
blinker==1.7.0
click==8.1.7
Flask==3.0.0
itsdangerous==2.1.2
Jinja2==3.1.2
jsonschema==4.19.2
jsonschema-specifications==2023.7.1
MarkupSafe==2.1.3
numpy==1.26.2
packaging==23.2
pandas==2.1.3
python-dateutil==2.8.2
pytz==2023.3.post1
referencing==0.30.2
rpds-py==0.12.0
six==1.16.0
toolz==0.12.0
typing_extensions==4.8.0
tzdata==2023.3
vega-datasets==0.9.0
Werkzeug==3.0.1
Pip-tools#
pip-tools is a collection of tools to solve the “loose vs pinned” dependency problem.
Instead of a single requirements.txt, you have a human-mananged requirements.in with only direct, loose dependencies, and an automatically managed requirements.txt with a fully pinned environment, including all dependencies.
!./env-1/bin/python3 -m pip install pip-tools
Collecting pip-tools
Using cached pip_tools-7.3.0-py3-none-any.whl.metadata (23 kB)
Collecting build (from pip-tools)
Using cached build-1.0.3-py3-none-any.whl.metadata (4.2 kB)
Requirement already satisfied: click>=8 in ./env-1/lib/python3.10/site-packages (from pip-tools) (8.1.7)
Requirement already satisfied: pip>=22.2 in ./env-1/lib/python3.10/site-packages (from pip-tools) (23.3.1)
Requirement already satisfied: setuptools in ./env-1/lib/python3.10/site-packages (from pip-tools) (65.5.0)
Collecting wheel (from pip-tools)
Using cached wheel-0.41.3-py3-none-any.whl.metadata (2.2 kB)
Collecting tomli (from pip-tools)
Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Requirement already satisfied: packaging>=19.0 in ./env-1/lib/python3.10/site-packages (from build->pip-tools) (23.2)
Collecting pyproject_hooks (from build->pip-tools)
Using cached pyproject_hooks-1.0.0-py3-none-any.whl (9.3 kB)
Using cached pip_tools-7.3.0-py3-none-any.whl (57 kB)
Using cached build-1.0.3-py3-none-any.whl (18 kB)
Using cached wheel-0.41.3-py3-none-any.whl (65 kB)
Installing collected packages: wheel, tomli, pyproject_hooks, build, pip-tools
Successfully installed build-1.0.3 pip-tools-7.3.0 pyproject_hooks-1.0.0 tomli-2.0.1 wheel-0.41.3
!cp requirements.txt pip-tools/requirements.in
!rm -f pip-tools/requirements.txt
We can now navigate into the folder with the requirements.in file, and call pip-compile (or if we want to be particular about the version, call path/to/python/installation -m piptools compile)
!cd pip-tools && ../env-1/bin/python3 -m piptools compile
WARNING: --strip-extras is becoming the default in version 8.0.0. To silence this warning, either use --strip-extras to opt into the new default or use --no-strip-extras to retain the existing behavior.
#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
# pip-compile
#
altair==5.1.2
# via -r requirements.in
attrs==23.1.0
# via
# jsonschema
# referencing
blinker==1.7.0
# via flask
click==8.1.7
# via flask
flask==3.0.0
# via -r requirements.in
itsdangerous==2.1.2
# via flask
jinja2==3.1.2
# via
# altair
# flask
jsonschema==4.19.2
# via altair
jsonschema-specifications==2023.7.1
# via jsonschema
markupsafe==2.1.3
# via
# jinja2
# werkzeug
numpy==1.26.2
# via
# altair
# pandas
packaging==23.2
# via altair
pandas==2.1.3
# via
# altair
# vega-datasets
python-dateutil==2.8.2
# via pandas
pytz==2023.3.post1
# via pandas
referencing==0.30.2
# via
# jsonschema
# jsonschema-specifications
rpds-py==0.12.0
# via
# jsonschema
# referencing
six==1.16.0
# via python-dateutil
toolz==0.12.0
# via altair
typing-extensions==4.8.0
# via altair
tzdata==2023.3
# via pandas
vega-datasets==0.9.0
# via -r requirements.in
werkzeug==3.0.1
# via flask
When you start, there is no difference between
python3 -m pip install -r requirements.in
and
python3 -m pip install -r requirements.txt
The distinction is: requirements.in will install different things over time, while requirements.txt will always install the same exact things until you change it, e.g. by running pip-compile again to upgrade packages, or after editing requirements.in.
!cat pip-tools/requirements.in
altair
flask
vega_datasets
!cat pip-tools/requirements.txt
#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
# pip-compile
#
altair==5.1.2
# via -r requirements.in
attrs==23.1.0
# via
# jsonschema
# referencing
blinker==1.7.0
# via flask
click==8.1.7
# via flask
flask==3.0.0
# via -r requirements.in
itsdangerous==2.1.2
# via flask
jinja2==3.1.2
# via
# altair
# flask
jsonschema==4.19.2
# via altair
jsonschema-specifications==2023.7.1
# via jsonschema
markupsafe==2.1.3
# via
# jinja2
# werkzeug
numpy==1.26.2
# via
# altair
# pandas
packaging==23.2
# via altair
pandas==2.1.3
# via
# altair
# vega-datasets
python-dateutil==2.8.2
# via pandas
pytz==2023.3.post1
# via pandas
referencing==0.30.2
# via
# jsonschema
# jsonschema-specifications
rpds-py==0.12.0
# via
# jsonschema
# referencing
six==1.16.0
# via python-dateutil
toolz==0.12.0
# via altair
typing-extensions==4.8.0
# via altair
tzdata==2023.3
# via pandas
vega-datasets==0.9.0
# via -r requirements.in
werkzeug==3.0.1
# via flask
pipenv#
pipenv is a tool built around pipfiles that does a similar thing to pip-tools, but one step removed.
uses Pipfile format instead of requirements.txt
Pipfile.lock for pinned versions
includes specifying the Python version itself, and additional informat about how to install pacakges
manages environments as well
!cd pipfile && cat Pipfile
[requires]
python_version = "3.10"
[packages]
altair = "*"
flask = "*"
vega-datasets = "*"
If we are using a terminal were we have activate the env-1 environment with the source ./env-1/bin/activate command, we can deactivate it by calling deactivate.
!python3 -m pip install pipenv
Collecting pipenv
Downloading pipenv-2023.10.24-py3-none-any.whl (3.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 MB 17.2 MB/s eta 0:00:0000:0100:01
?25hRequirement already satisfied: certifi in /home/dokken/src/mambaforge/envs/UIO-IN3110/lib/python3.10/site-packages (from pipenv) (2023.7.22)
Requirement already satisfied: setuptools>=67 in /home/dokken/src/mambaforge/envs/UIO-IN3110/lib/python3.10/site-packages (from pipenv) (68.1.2)
Collecting virtualenv>=20.24.2 (from pipenv)
Downloading virtualenv-20.24.6-py3-none-any.whl (3.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.8/3.8 MB 29.2 MB/s eta 0:00:0000:0100:01
?25hCollecting distlib<1,>=0.3.7 (from virtualenv>=20.24.2->pipenv)
Using cached distlib-0.3.7-py2.py3-none-any.whl (468 kB)
Collecting filelock<4,>=3.12.2 (from virtualenv>=20.24.2->pipenv)
Downloading filelock-3.13.1-py3-none-any.whl (11 kB)
Collecting platformdirs<4,>=3.9.1 (from virtualenv>=20.24.2->pipenv)
Downloading platformdirs-3.11.0-py3-none-any.whl (17 kB)
Installing collected packages: distlib, platformdirs, filelock, virtualenv, pipenv
Attempting uninstall: distlib
Found existing installation: distlib 0.3.6
Uninstalling distlib-0.3.6:
Successfully uninstalled distlib-0.3.6
Attempting uninstall: platformdirs
Found existing installation: platformdirs 2.6.2
Uninstalling platformdirs-2.6.2:
Successfully uninstalled platformdirs-2.6.2
Attempting uninstall: filelock
Found existing installation: filelock 3.12.0
Uninstalling filelock-3.12.0:
Successfully uninstalled filelock-3.12.0
Attempting uninstall: virtualenv
Found existing installation: virtualenv 20.21.1
Uninstalling virtualenv-20.21.1:
Successfully uninstalled virtualenv-20.21.1
Successfully installed distlib-0.3.7 filelock-3.13.1 pipenv-2023.10.24 platformdirs-3.10.0 virtualenv-20.24.6
[notice] A new release of pip is available: 23.1.2 -> 23.3.1
[notice] To update, run: pip install --upgrade pip
!python3 -m pipenv lock
Creating a virtualenv for this project...
Pipfile: /home/dokken/Documents/src/UiO/UiO-IN3110.github.io/lectures/production/Pipfile
Using default python from /home/dokken/src/mambaforge/envs/UIO-IN3110/bin/python3 (3.10.12) to create virtualenv...
⠸ Creating virtual environment.....created virtual environment CPython3.10.12.final.0-64 in 639ms
creator CPython3Posix(dest=/home/dokken/.local/share/virtualenvs/production-2vyAGA3H, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/dokken/.local/share/virtualenv)
added seed packages: pip==23.3.1, setuptools==68.2.2, wheel==0.41.2
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
✔ Successfully created virtual environment!
⠸ Creating virtual environment...
Virtualenv location: /home/dokken/.local/share/virtualenvs/production-2vyAGA3H
requirements.txt found in
/home/dokken/Documents/src/UiO/UiO-IN3110.github.io/lectures/production instead
of Pipfile! Converting...
✔ Success! Importing requirements.....
⠸ Importing requirements...
Warning: Your Pipfile now contains pinned versions, if your requirements.txt
did.
We recommend updating your Pipfile to specify the "*" version, instead.
Locking [packages] dependencies...
?25lBuilding requirements...
Resolving dependencies...
✔ Success! Locking...
⠦ Locking...
Locking [dev-packages] dependencies...
Updated Pipfile.lock (644e57b07835e87b285d8338676a8bde3d24b8412120203e96731a83b770b772)!
!python3 -m pipenv install
Installing dependencies from Pipfile.lock (70b772)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
!python3 -c "import sys; print(sys.prefix)"
/home/dokken/src/mambaforge/envs/UIO-IN3110
!python3 -m pipenv run python -c "import sys; print(sys.prefix)"
/home/dokken/.local/share/virtualenvs/production-2vyAGA3H
!python3 -m pipenv run pip list
Package Version
------------------------- ------------
altair 5.1.2
attrs 23.1.0
blinker 1.7.0
click 8.1.7
Flask 3.0.0
itsdangerous 2.1.2
Jinja2 3.1.2
jsonschema 4.19.2
jsonschema-specifications 2023.7.1
MarkupSafe 2.1.3
numpy 1.26.2
packaging 23.2
pandas 2.1.3
pip 23.3.1
python-dateutil 2.8.2
pytz 2023.3.post1
referencing 0.30.2
rpds-py 0.12.0
setuptools 68.2.2
six 1.16.0
toolz 0.12.0
typing_extensions 4.8.0
tzdata 2023.3
vega-datasets 0.9.0
Werkzeug 3.0.1
wheel 0.41.2
!head -n 30 Pipfile.lock
{
"_meta": {
"hash": {
"sha256": "644e57b07835e87b285d8338676a8bde3d24b8412120203e96731a83b770b772"
},
"pipfile-spec": 6,
"requires": {
"python_version": "3.10"
},
"sources": [
{
"name": "pypi",
"url": "https://pypi.org/simple",
"verify_ssl": true
}
]
},
"default": {
"altair": {
"hashes": [
"sha256:7219708ec33c152e53145485040f428954ed15fd09b2a2d89e543e6d111dae7f",
"sha256:e5f52a71853a607c61ce93ad4a414b3d486cd0d46ac597a24ae8bd1ac99dd460"
],
"index": "pypi",
"markers": "python_version >= '3.8'",
"version": "==5.1.2"
},
"attrs": {
"hashes": [
"sha256:1f28b4522cdc2fb4256ac1a020c78acf9cba2c6b461ccd2c126f3aa8e8335d04",