Structuring Python code: Modules and Packages#

../../_images/python_structure_options.svg

What is a Python module and what is it good for?#

A module is a file consisting of Python code. A module can define functions, classes and variables. A module can also include runnable code.

Use modules to organize your program logically

  • Split the code into several files for easier maintenance.

  • Group related code into a module.

  • Share common code between scripts.

  • Publish modules on the web for other people to use (even better: create a package, see below).

Using modules#

We have already seen some example of usage in the previous lecture part

import itertools
# Access function from the module
itertools.product

# Alias
import itertools as itools
itool.product

# The following is considered a bad practice
from itertools import *
# Easy to shadow existing variables (also hard for IDEs)

What makes itertools into a module and not a package?#

It is a single file! Python has a lot of built-in modules

import sys
print(sys.builtin_module_names)

Creating and using Python modules#

Creating own modules in Python is very simple:

  1. Put any code (variables, functions, classes) that should be part of the module in a Python file.

Let us consider a simple user-defined function, put in the file user_factorial.py

!head -n 20 user_factorial.py

def factorial(n: int) -> int:
    """Return the factorial of n, an exact integer >= 0.

    Args:
       n (int):  n!

    Returns:
       int.  The factorial value::

    >>> factorial(5)
    120
    >>> factorial(0)
    1

    """
    if n == 0:
        return 1
    return n * factorial(n - 1)

We can now import this function to our code by calling

from user_factorial import factorial

factorial(10)

3628800

How does Python find your modules? When importing a module (or package module), Python tries to find it in multiples places (in this order):

- The built-in modules shown above
- Your current working directory.
- Paths defined by the environment variable $PYTHONPATH.
- Some global paths, e.g. /usr/lib/python3.7/site-packages. This depends on your OS and Python installation.

This can be verified as follows via another useful module sys

import sys

# Notice the order
sys.path

['/home/dokken/Documents/src/UiO/UiO-IN3110.github.io/lectures/python',
 '/home/dokken/src/mambaforge/envs/UIO-IN3110/lib/python310.zip',
 '/home/dokken/src/mambaforge/envs/UIO-IN3110/lib/python3.10',
 '/home/dokken/src/mambaforge/envs/UIO-IN3110/lib/python3.10/lib-dynload',
 '',
 '/home/dokken/.local/lib/python3.10/site-packages',
 '/home/dokken/src/mambaforge/envs/UIO-IN3110/lib/python3.10/site-packages',
 '/home/dokken/Documents/src/UiO/UiO-IN3110.github.io/lectures/python/data/my-package',
 '/home/dokken/Documents/src/UiO/UiO-IN3110.github.io/lectures/python/data/my-package']

Test block in a module#

Module files can have a test/demo section at the end:

  • The block is executed only if the module file is run as a program (not if imported by another script)

  • The tests at the end of a module often serve as good examples on the usage of the module

For the problem above, we add a simple hard-coded test of the factorial function

!tail -n 11 user_factorial.py

if __name__ == "__main__":
    import math
    import sys

    N = int(sys.argv[1])
    print(f"Testing user defined factorial function for {N=}")
    user_n = factorial(N)
    ref_factorial = math.factorial(N)
    assert user_n == math.factorial(
        N
    ), f"Factorial function returning wrong answer {user_n}!={ref_factorial}"

!python3 user_factorial.py 6

Testing user defined factorial function for N=6

What is a package?#

A package is a hierarchical file directory structure that consists of modules and subpackages and sub-subpackages, and so on.

Example:

from scipy.optimize import minimize
#      ^      ^               ^
#      |      |               |
#   Package   |               |
#           Module            |
#                          Function

Packages allow to organize modules and scripts into single environment. These can then easily be distributed and imported by name.

Python comes with a set of powerful packages, e.g.

  • scipy Scientific Python

  • numpy Numerical Python

  • ipython Interactive Python

  • matplotlib Plotting

  • pandas Data analysis

  • scikit learn Machine learning

Several useful packages are included in Python distributions like Anaconda

Creating a package#

  • A set of modules can be collected in a package

  • A package is organized as module files in a directory tree

  • Each subdirectory must have a __init__.py file (can be empty)

  • More infos: Section 6 in the Python Tutorial

We have a sample package in the data/my-package directory. The package tree is as follows

!tree examples/my-package

examples/my-package
├── LICENSE
├── pyproject.toml
├── README.md
├── src
│   └── pkg
│       ├── analysis.py
│       ├── __init__.py
│       └── printing
│           ├── __init__.py
│           └── printing.py
└── test
    ├── test_analysis.py
    └── test_printing.py

4 directories, 9 files

Installing a Python package#

One could in theory append the path to a package to the environment variable PYTHONPATH. However, this is not recommened, as it does not check dependencies and is not portable across systems.

A better option is to reate a pyproject.toml file in your package root directory. We start by expecting this file

!head -n 25 examples/my-package/pyproject.toml

[build-system]
requires = ["setuptools>=64.4.0", "wheel", "pip>=22.3"]


[project]
name="pkg"
authors=[{"name"="Jørgen S. Dokken", "email"="dokken@simula.no"},
         {"name"="Miroslav Kuchta", "email"="miroslav.kuchta@gmail.com"}
]
readme="README.md"
license={"file"="LICENSE"}
version="0.1.0"
requires-python=">=3.8"
dependencies=["numpy>=1.20.0"]

[project.optional-dependencies]
test = [
    "pytest",
]
dev = [
    "pdbpp", "ipython"
]

We now consider each of the sections in this file in turn. First we consider the [...]-notation. This defines a heading in a table, and we can create sub-tables, such as [a] and [a.b]. Many Python packages support their own headings for configuring the repository, such as formatting, import sorting, type-checking etc.

build-system#

Installing a package in Python means taking a set of files, and do some or all of the following options

  • Compile files from foreign languages (such as C/C++)

  • Move files from the current root directory to an appropriate path for the current installation of Python We will use setuptools for this, a common Python packager.

With the file above, we can now call the following. Pip will always install in the current Python environment, which may be system-wide (usually requires root permissions), per-user, or in a virtual environment.

!python3 -m pip install ./examples/my-package

Processing ./examples/my-package
  Installing build dependencies ... ?25ldone
?25h  Getting requirements to build wheel ... ?25ldone
?25h  Preparing metadata (pyproject.toml) ... ?25ldone
?25hRequirement already satisfied: numpy>=1.20.0 in /home/dokken/.local/lib/python3.10/site-packages (from pkg==0.1.0) (1.26.1)
Building wheels for collected packages: pkg
  Building wheel for pkg (pyproject.toml) ... ?25ldone
?25h  Created wheel for pkg: filename=pkg-0.1.0-py3-none-any.whl size=3881 sha256=4a38e99084fdf3ea23d9d46ff357efe2aca6ec3d08a63c29f3ce6131fdaa89da
  Stored in directory: /home/dokken/.cache/pip/wheels/e7/be/ae/c7a1fe811e9515c9b17aa8a901178b4faa716117ff2d3aab87
Successfully built pkg
Installing collected packages: pkg
Successfully installed pkg-0.1.0

[notice] A new release of pip is available: 23.1.2 -> 23.3.1
[notice] To update, run: pip install --upgrade pip

We can check where the package now is located by calling

!python3 -c "import pkg; print(pkg.__path__)"

['/home/dokken/src/mambaforge/envs/UIO-IN3110/lib/python3.10/site-packages/pkg']

Editable installation#

As we have seen in the scripts above, the files are copied from the current location to the another place on disk specific to the Python installation. With rapid development, this would be a time-consuming and annoying task. Therefore, one can use editable installations that creates special .pth file in the Python-installations site package, that extends the Python-path to the package directory. This means that one do not have to re-install the package to reflect changes

!python3 -m pip uninstall -y pkg
!python3 -m pip install -e ./examples/my-package -v

Found existing installation: pkg 0.1.0
Uninstalling pkg-0.1.0:
  Successfully uninstalled pkg-0.1.0
Using pip 23.1.2 from /home/dokken/.local/lib/python3.10/site-packages/pip (python 3.10)
Obtaining file:///home/dokken/Documents/src/UiO/UiO-IN3110.github.io/lectures/python/examples/my-package
  Installing build dependencies ... ?25l  Running command pip subprocess to install build dependencies
  Collecting setuptools>=64.4.0
    Using cached setuptools-68.2.2-py3-none-any.whl (807 kB)
  Collecting wheel
    Using cached wheel-0.41.3-py3-none-any.whl (65 kB)
  Collecting pip>=22.3
    Using cached pip-23.3.1-py3-none-any.whl (2.1 MB)
  Installing collected packages: wheel, setuptools, pip
  ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
  tensorboard 2.11.1 requires markdown>=2.6.8, which is not installed.
  tensorboard 2.11.1 requires protobuf<4,>=3.9.2, which is not installed.
  tensorflow 2.11.0 requires protobuf<3.20,>=3.9.2, which is not installed.
  numba 0.56.4 requires numpy<1.24,>=1.18, but you have numpy 1.26.1 which is incompatible.
  Successfully installed pip-23.3.1 setuptools-68.2.2 wheel-0.41.3

  [notice] A new release of pip is available: 23.1.2 -> 23.3.1
  [notice] To update, run: pip install --upgrade pip
?25hdone
  Checking if build backend supports build_editable ... ?25l  Running command Checking if build backend supports build_editable
?25hdone
  Getting requirements to build editable ... ?25l  Running command Getting requirements to build editable
  running egg_info
  writing src/pkg.egg-info/PKG-INFO
  writing dependency_links to src/pkg.egg-info/dependency_links.txt
  writing requirements to src/pkg.egg-info/requires.txt
  writing top-level names to src/pkg.egg-info/top_level.txt
  reading manifest file 'src/pkg.egg-info/SOURCES.txt'
  adding license file 'LICENSE'
  writing manifest file 'src/pkg.egg-info/SOURCES.txt'
?25hdone
  Preparing editable metadata (pyproject.toml) ... ?25l  Running command Preparing editable metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-ckzb6ip2/pkg.egg-info
  writing /tmp/pip-modern-metadata-ckzb6ip2/pkg.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-ckzb6ip2/pkg.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-modern-metadata-ckzb6ip2/pkg.egg-info/requires.txt
  writing top-level names to /tmp/pip-modern-metadata-ckzb6ip2/pkg.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-ckzb6ip2/pkg.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-modern-metadata-ckzb6ip2/pkg.egg-info/SOURCES.txt'
  adding license file 'LICENSE'
  writing manifest file '/tmp/pip-modern-metadata-ckzb6ip2/pkg.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-ckzb6ip2/pkg-0.1.0.dist-info'
?25hdone
Requirement already satisfied: numpy>=1.20.0 in /home/dokken/.local/lib/python3.10/site-packages (from pkg==0.1.0) (1.26.1)
Building wheels for collected packages: pkg
  Building editable for pkg (pyproject.toml) ... ?25l  Running command Building editable for pkg (pyproject.toml)
  running editable_wheel
  creating /tmp/pip-wheel-d2tjy3ki/.tmp-9syep96b/pkg.egg-info
  writing /tmp/pip-wheel-d2tjy3ki/.tmp-9syep96b/pkg.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-wheel-d2tjy3ki/.tmp-9syep96b/pkg.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-wheel-d2tjy3ki/.tmp-9syep96b/pkg.egg-info/requires.txt
  writing top-level names to /tmp/pip-wheel-d2tjy3ki/.tmp-9syep96b/pkg.egg-info/top_level.txt
  writing manifest file '/tmp/pip-wheel-d2tjy3ki/.tmp-9syep96b/pkg.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-wheel-d2tjy3ki/.tmp-9syep96b/pkg.egg-info/SOURCES.txt'
  adding license file 'LICENSE'
  writing manifest file '/tmp/pip-wheel-d2tjy3ki/.tmp-9syep96b/pkg.egg-info/SOURCES.txt'
  creating '/tmp/pip-wheel-d2tjy3ki/.tmp-9syep96b/pkg-0.1.0.dist-info'
  creating /tmp/pip-wheel-d2tjy3ki/.tmp-9syep96b/pkg-0.1.0.dist-info/WHEEL
  running build_py
  running egg_info
  creating /tmp/tmpxgpioiwc.build-temp/pkg.egg-info
  writing /tmp/tmpxgpioiwc.build-temp/pkg.egg-info/PKG-INFO
  writing dependency_links to /tmp/tmpxgpioiwc.build-temp/pkg.egg-info/dependency_links.txt
  writing requirements to /tmp/tmpxgpioiwc.build-temp/pkg.egg-info/requires.txt
  writing top-level names to /tmp/tmpxgpioiwc.build-temp/pkg.egg-info/top_level.txt
  writing manifest file '/tmp/tmpxgpioiwc.build-temp/pkg.egg-info/SOURCES.txt'
  reading manifest file '/tmp/tmpxgpioiwc.build-temp/pkg.egg-info/SOURCES.txt'
  adding license file 'LICENSE'
  writing manifest file '/tmp/tmpxgpioiwc.build-temp/pkg.egg-info/SOURCES.txt'

          Editable install will be performed using .pth file to extend `sys.path` with:
          ['src']

  Options like `package-data`, `include/exclude-package-data` or
  `packages.find.exclude/include` may have no effect.

  adding '__editable__.pkg-0.1.0.pth'
  creating '/tmp/pip-wheel-d2tjy3ki/.tmp-9syep96b/pkg-0.1.0-0.editable-py3-none-any.whl' and adding '/tmp/tmpl7nuej9ppkg-0.1.0-0.editable-py3-none-any.whl' to it
  adding 'pkg-0.1.0.dist-info/LICENSE'
  adding 'pkg-0.1.0.dist-info/METADATA'
  adding 'pkg-0.1.0.dist-info/WHEEL'
  adding 'pkg-0.1.0.dist-info/top_level.txt'
  adding 'pkg-0.1.0.dist-info/RECORD'
?25hdone
  Created wheel for pkg: filename=pkg-0.1.0-0.editable-py3-none-any.whl size=2854 sha256=369cc98d72e82f2aa28a6bed58a6836e4f47328788679836e8c916cfd51ea238
  Stored in directory: /tmp/pip-ephem-wheel-cache-2456l8n7/wheels/e7/be/ae/c7a1fe811e9515c9b17aa8a901178b4faa716117ff2d3aab87
Successfully built pkg
Installing collected packages: pkg
Successfully installed pkg-0.1.0

[notice] A new release of pip is available: 23.1.2 -> 23.3.1
[notice] To update, run: pip install --upgrade pip

!python3 -c "import pkg; print(pkg.__path__)"

['/home/dokken/Documents/src/UiO/UiO-IN3110.github.io/lectures/python/examples/my-package/src/pkg']

Project specification#

!head -n15 ./examples/my-package/pyproject.toml | tail -n11

[project]
name="pkg"
authors=[{"name"="Jørgen S. Dokken", "email"="dokken@simula.no"},
         {"name"="Miroslav Kuchta", "email"="miroslav.kuchta@gmail.com"}
]
readme="README.md"
license={"file"="LICENSE"}
version="0.1.0"
requires-python=">=3.8"
dependencies=["numpy>=1.20.0"]

Package name#

As seen in the file above, we observe that we do not need to specify alot in the project description. We give the package a name, which should be reflected in /src/pkg. We use the source layout as it avoids some issues with a flat package structure, see Python packaging guide for more details.

Author list#

We create a list of authors with their name and email.

We will now work on the package with an aim to cover some of the existing functionality by testing and add new functionality by practicing test-drive-development. First some crash-course in Python unit testing.

README#

A package should have a description. This should be placed in a README file. The string should contain the local path (relative to the pyproject.toml file) to the description file.

Licensing#

It is very important to have a license for your published code. This is how you instruct anyone that wants to use it on the terms and conditions of copying or modifying the code. ChooseYourLicense is a good source of information. A summary of recommened licenses:

  • MIT: Permissive - Others can use your code in any way, and you will not be sued if the software doesn’t work (recommended in most cases)

  • GPL: Copyleft - derivative work must use the same license - good way to embrace open source but often problematic for commercial companies

  • LGPL: Similar to GPL but software can be used under different license

  • CC-BY-4.0 - Typically used for creative work (more journals use this)

Version#

Given a version number MAJOR.MINOR.PATCH, increment the:

  1. MAJOR version when you make incompatible API changes

  2. MINOR version when you add functionality in a backward compatible manner

  3. PATCH version when you make backward compatible bug fixes

This is called semantic versioning, see semver.org for details.

Python versioning#

As Python evolves and introduces new features and deprecates versions, your software should do the same. Decide on a minimal requirement for Python (and in some cases a maximum version if a package you are using is not supporting the latest Python version).

Dependencies#

As our package might depend on external software, we create a list of packages such as ["numpy>=1.21", "sklearn==1.3.0"] etc. These will be installed if not found on the system at the time of installation.

Optional dependencies#

Sometimes, we require dependencies for testing or development that are not requirements of the source code. We call these optional dependencies and list them as below

!tail -8 ./examples/my-package/pyproject.toml

[project.optional-dependencies]
test = [
    "pytest",
]
dev = [
    "pdbpp", "ipython"
]

We install these by calling

python3 -m pip install .... path/to/package[dev]

or

python3 -m pip install .... path/to/package[test]

or

python3 -m pip install .... path/to/package[dev,test]

!python3 -m pip install -e ./examples/my-package[dev,test]

Obtaining file:///home/dokken/Documents/src/UiO/UiO-IN3110.github.io/lectures/python/examples/my-package
  Installing build dependencies ... ?25ldone
?25h  Checking if build backend supports build_editable ... ?25ldone
?25h  Getting requirements to build editable ... ?25ldone
?25h  Preparing editable metadata (pyproject.toml) ... ?25ldone
?25hRequirement already satisfied: numpy>=1.20.0 in /home/dokken/.local/lib/python3.10/site-packages (from pkg==0.1.0) (1.26.1)
Requirement already satisfied: pytest in /home/dokken/.local/lib/python3.10/site-packages (from pkg==0.1.0) (7.3.1)
Collecting pdbpp (from pkg==0.1.0)
  Using cached pdbpp-0.10.3-py2.py3-none-any.whl (23 kB)
Requirement already satisfied: ipython in /home/dokken/.local/lib/python3.10/site-packages (from pkg==0.1.0) (8.6.0)
Requirement already satisfied: backcall in /home/dokken/.local/lib/python3.10/site-packages (from ipython->pkg==0.1.0) (0.2.0)
Requirement already satisfied: decorator in /home/dokken/.local/lib/python3.10/site-packages (from ipython->pkg==0.1.0) (5.1.1)
Requirement already satisfied: jedi>=0.16 in /home/dokken/.local/lib/python3.10/site-packages (from ipython->pkg==0.1.0) (0.18.2)
Requirement already satisfied: matplotlib-inline in /home/dokken/.local/lib/python3.10/site-packages (from ipython->pkg==0.1.0) (0.1.6)
Requirement already satisfied: pickleshare in /home/dokken/.local/lib/python3.10/site-packages (from ipython->pkg==0.1.0) (0.7.5)
Requirement already satisfied: prompt-toolkit<3.1.0,>3.0.1 in /home/dokken/.local/lib/python3.10/site-packages (from ipython->pkg==0.1.0) (3.0.39)
Requirement already satisfied: pygments>=2.4.0 in /home/dokken/.local/lib/python3.10/site-packages (from ipython->pkg==0.1.0) (2.15.1)
Requirement already satisfied: stack-data in /home/dokken/.local/lib/python3.10/site-packages (from ipython->pkg==0.1.0) (0.6.1)
Requirement already satisfied: traitlets>=5 in /home/dokken/.local/lib/python3.10/site-packages (from ipython->pkg==0.1.0) (5.8.0)
Requirement already satisfied: pexpect>4.3 in /home/dokken/src/mambaforge/envs/UIO-IN3110/lib/python3.10/site-packages (from ipython->pkg==0.1.0) (4.8.0)
Collecting fancycompleter>=0.8 (from pdbpp->pkg==0.1.0)
  Using cached fancycompleter-0.9.1-py3-none-any.whl (9.7 kB)
Collecting wmctrl (from pdbpp->pkg==0.1.0)
  Downloading wmctrl-0.5-py2.py3-none-any.whl (4.3 kB)
Requirement already satisfied: iniconfig in /home/dokken/.local/lib/python3.10/site-packages (from pytest->pkg==0.1.0) (2.0.0)
Requirement already satisfied: packaging in /home/dokken/.local/lib/python3.10/site-packages (from pytest->pkg==0.1.0) (23.1)
Requirement already satisfied: pluggy<2.0,>=0.12 in /home/dokken/.local/lib/python3.10/site-packages (from pytest->pkg==0.1.0) (1.0.0)
Requirement already satisfied: exceptiongroup>=1.0.0rc8 in /home/dokken/.local/lib/python3.10/site-packages (from pytest->pkg==0.1.0) (1.0.4)
Requirement already satisfied: tomli>=1.0.0 in /home/dokken/.local/lib/python3.10/site-packages (from pytest->pkg==0.1.0) (2.0.1)
Collecting pyrepl>=0.8.2 (from fancycompleter>=0.8->pdbpp->pkg==0.1.0)
  Using cached pyrepl-0.9.0-py3-none-any.whl
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /home/dokken/.local/lib/python3.10/site-packages (from jedi>=0.16->ipython->pkg==0.1.0) (0.8.3)
Requirement already satisfied: ptyprocess>=0.5 in /home/dokken/src/mambaforge/envs/UIO-IN3110/lib/python3.10/site-packages (from pexpect>4.3->ipython->pkg==0.1.0) (0.7.0)
Requirement already satisfied: wcwidth in /home/dokken/.local/lib/python3.10/site-packages (from prompt-toolkit<3.1.0,>3.0.1->ipython->pkg==0.1.0) (0.2.5)
Requirement already satisfied: executing>=1.2.0 in /home/dokken/.local/lib/python3.10/site-packages (from stack-data->ipython->pkg==0.1.0) (1.2.0)
Requirement already satisfied: asttokens>=2.1.0 in /home/dokken/.local/lib/python3.10/site-packages (from stack-data->ipython->pkg==0.1.0) (2.1.0)
Requirement already satisfied: pure-eval in /home/dokken/.local/lib/python3.10/site-packages (from stack-data->ipython->pkg==0.1.0) (0.2.2)
Requirement already satisfied: attrs in /home/dokken/src/mambaforge/envs/UIO-IN3110/lib/python3.10/site-packages (from wmctrl->pdbpp->pkg==0.1.0) (23.1.0)
Requirement already satisfied: six in /home/dokken/src/mambaforge/envs/UIO-IN3110/lib/python3.10/site-packages (from asttokens>=2.1.0->stack-data->ipython->pkg==0.1.0) (1.16.0)
Building wheels for collected packages: pkg
  Building editable for pkg (pyproject.toml) ... ?25ldone
?25h  Created wheel for pkg: filename=pkg-0.1.0-0.editable-py3-none-any.whl size=2854 sha256=efbde18e0330484f94cd6f9cad380eb1166038c8641ccb2376635597a4fdb116
  Stored in directory: /tmp/pip-ephem-wheel-cache-tu8kamji/wheels/e7/be/ae/c7a1fe811e9515c9b17aa8a901178b4faa716117ff2d3aab87
Successfully built pkg
Installing collected packages: pyrepl, wmctrl, pkg, fancycompleter, pdbpp
  Attempting uninstall: pkg
    Found existing installation: pkg 0.1.0
    Uninstalling pkg-0.1.0:
      Successfully uninstalled pkg-0.1.0
Successfully installed fancycompleter-0.9.1 pdbpp-0.10.3 pkg-0.1.0 pyrepl-0.9.0 wmctrl-0.5

[notice] A new release of pip is available: 23.1.2 -> 23.3.1
[notice] To update, run: pip install --upgrade pip