Advanced topics in Python programming

Advanced topics in Python programming#

This notebook explores (slightly) more advanced elements of programming in Python. Among other things we will look into

functional programming
generators/iterators
documenting code
error handling

One aspect not addressed here is writting code following the PEP8 style guide, e.g. indentations, class/function names. There are tools to format your code that way (often given as extension to IDEs), for example flake8.

Bits of functional programming#

We have seen how to define a function, i.e. give it a name

def identity(x): return x

However, sometimes we might have use for nameless function - enter Anonymous functions. For example in sorting

import random

# Let's generate random pairs with the idea of sorting them later
random_tuples = []
for i in range(10):
    random_tuples.append((random.random(), random.random()))
random_tuples

[(0.9511176424926535, 0.9742813887530748),
 (0.5776776069602639, 0.10447364475487353),
 (0.37608222708654293, 0.6486109614387363),
 (0.30945796282871496, 0.07954468498467404),
 (0.4996430964021168, 0.32581504743166245),
 (0.24952665159729226, 0.7523088865961022),
 (0.7093848623063937, 0.016854321726398114),
 (0.9572218713564041, 0.3209356903789915),
 (0.1092176594392319, 0.020611646926582128),
 (0.9071444097710661, 0.8585649655113832)]

# By default tuples are sorted by considering first the first elements, then comparing the rest, i.e.
sorted(random_tuples)

[(0.1092176594392319, 0.020611646926582128),
 (0.24952665159729226, 0.7523088865961022),
 (0.30945796282871496, 0.07954468498467404),
 (0.37608222708654293, 0.6486109614387363),
 (0.4996430964021168, 0.32581504743166245),
 (0.5776776069602639, 0.10447364475487353),
 (0.7093848623063937, 0.016854321726398114),
 (0.9071444097710661, 0.8585649655113832),
 (0.9511176424926535, 0.9742813887530748),
 (0.9572218713564041, 0.3209356903789915)]

# Treating them as points we might want to consider their l^2 norm
sorted(random_tuples, key=lambda t: (t[0] ** 2 + t[1] ** 2) ** 0.5)

[(0.1092176594392319, 0.020611646926582128),
 (0.30945796282871496, 0.07954468498467404),
 (0.5776776069602639, 0.10447364475487353),
 (0.4996430964021168, 0.32581504743166245),
 (0.7093848623063937, 0.016854321726398114),
 (0.37608222708654293, 0.6486109614387363),
 (0.24952665159729226, 0.7523088865961022),
 (0.9572218713564041, 0.3209356903789915),
 (0.9071444097710661, 0.8585649655113832),
 (0.9511176424926535, 0.9742813887530748)]

min(random_tuples, key=lambda t: (t[0] ** 2 + t[1] ** 2) ** 0.5)

(0.1092176594392319, 0.020611646926582128)

Here lambda is a key word used for defining anonymous functions. It is followed by arguments. Above the function accepts one argument (referred to as t). The function body follows after :. Side note, \(\lambda\)-calculus and its inventor Alonzo Church

In capturing variables beware of late binding

# The idea is that foos[1](x) should return x+1
foos = [lambda x: x + n for n in range(5)]
# But ...
for f in foos:
    print(f(0))

Definition is evaluated at runtime (then n = 4) and not at definition time

# Solution [referred to as currying]
foos = [lambda x, n=n: x + n for n in range(5)]
for f in foos:
    print(f(0))

Anonymous functions are often used to build generator. Here the idea is that we want to compute on demand and not all the answers at once.

selected = filter(lambda p: p[0] < 0.5, random_tuples)
# Not the answers but ...
selected

<filter at 0x105bd28f0>

# Generator are iterated over
next(selected)

(0.37608222708654293, 0.6486109614387363)

# or consumed entirely
for item in selected:
    print(item)

(0.30945796282871496, 0.07954468498467404)
(0.4996430964021168, 0.32581504743166245)
(0.24952665159729226, 0.7523088865961022)
(0.1092176594392319, 0.020611646926582128)

# Note that we have now exhausted the iterator so that the following attempt to get the next item fails
next(selected)

---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
Cell In[10], line 2
      1 # Note that we have now exhausted the iterator so that the following attempt to get the next item fails
----> 2 next(selected)

StopIteration: 

Iterators can be combined to build processing pipelines

# Keep only the elements in iterable for which the function is true
selected = filter(lambda p: p[0] < 0.5, random_tuples)
# Apply sum function to all the elements in iterable
processed = map(sum, selected)
processed

<map at 0x106cc46a0>

What is the sum of such elements ? One option is

sum(list(processed))

Also sum(processed) would work but we want to showcase a nice module from the standard library, namely, functools.

# Option 1) to comsume and turn into a list
from functools import reduce

# combine first two items of iterable to make the input
# for next round while the other argument is the next item in iterable
reduce(lambda x, y: x + y, processed)

3.370818824731656

Food for thought:

Could we use reduce(sum, processed) above ?
What does functools.partial do?
Which digits are common to all entries obtained by summing tuples in random_tuples?

import operator


def digits(num):
    """Set of digits of the number"""
    return set(filter(str.isdecimal, str(num)))


# To find the answer we reduce by set intersection
reduce(operator.and_, (map(lambda t: digits(t[0] + t[1]), random_tuples)))

{'1', '3', '8'}

Many useful iterators can be constructed using standard library module itertools. Let’s do cartesian coordinates

from itertools import product

x = range(1, 5)
y = range(4, 12)
grid = product(x, y)
# Get them all
print(list(grid))

[(1, 4), (1, 5), (1, 6), (1, 7), (1, 8), (1, 9), (1, 10), (1, 11), (2, 4), (2, 5), (2, 6), (2, 7), (2, 8), (2, 9), (2, 10), (2, 11), (3, 4), (3, 5), (3, 6), (3, 7), (3, 8), (3, 9), (3, 10), (3, 11), (4, 4), (4, 5), (4, 6), (4, 7), (4, 8), (4, 9), (4, 10), (4, 11)]

Using itertools.permutations, itertools.combinations we can save ourselves writing a lot to nested for loop. For example: Which number 1 <= a < b < c <= 100 for Pythagorean trippets?

tripplets = []
for a in range(1, 101):
    for b in range(a, 101):
        for c in range(b, 101):
            # Short circuit
            # arg0 and arg1 evals arg1 only if arg0 is True because
            # arg0 False determined the boolean value of the expression
            # Alternative to if
            a**2 + b**2 == c**2 and tripplets.append((a, b, c))

Contrast with

from itertools import combinations

tripplets2 = []
for a, b, c in combinations(range(1, 101), 3):
    a**2 + b**2 == c**2 and tripplets2.append((a, b, c))
# Or even shorter with list comprehensions, see later
set(tripplets) == set(tripplets2)

True

Another example of ondemand/lazy computations are generators. They are introduced by yield keyword.

def even_numbers():
    k = 1
    while True:
        yield 2 * k
        k += 1

evens = even_numbers()

We can start consuming the generator. For example take the first items

# NOTE: zip - pairs iterables into tuples, terminating when one of them is exhaused [range(5) determines this above]
# We can run this many times.
for v, _ in zip(evens, range(5)):  # And some more ...
    print(v)

Here we use itertools’s count which is counts from argument upwards

from itertools import count  # range()


def odd_numbers():
    for v in count(1):
        yield 2 * v - 1

for v, _ in zip(odd_numbers(), range(5)):
    print(v)

In the previous example we were consuming other iterator which was built in count. Would the definition of count below accomplish the same?

def count(n):
    yield n
    yield from count(n + 1)


numbers = count(-10)
for i in range(10_000):  # How about counting longer (1_000) ?
    print(next(numbers))

---------------------------------------------------------------------------
RecursionError                            Traceback (most recent call last)
Cell In[22], line 8
      6 numbers = count(-10)
      7 for i in range(10_000):  # How about counting longer (1_000) ?
----> 8     print(next(numbers))

Cell In[22], line 3, in count(n)
      1 def count(n):
      2     yield n
----> 3     yield from count(n + 1)

Cell In[22], line 3, in count(n)
      1 def count(n):
      2     yield n
----> 3     yield from count(n + 1)

    [... skipping similar frames: count at line 3 (2971 times)]

Cell In[22], line 3, in count(n)
      1 def count(n):
      2     yield n
----> 3     yield from count(n + 1)

RecursionError: maximum recursion depth exceeded

The issue with our definition is that it is recursive and the calls go to stack

import sys

sys.getrecursionlimit()
# sys.setrecursionlimit(3_000)

However, increasing the stack size doesn’t always cut it. The recursive implementation often yields to exponentially growing execution time (and memory). The notorious example of this is (doubly recursive) Fibonacci numbers.

from itertools import count


def fib(k):
    assert k > 0
    if k == 1:
        return 1
    if k == 2:
        return 1
    return fib(k - 1) + fib(k - 2)


def fibs():
    """Generator for all of them"""
    for k in count(1):
        yield fib(k)

Let’s look at peformance

from itertools import islice  # Iterator slice
from time import perf_counter

t0 = perf_counter()
for i, k in enumerate(islice(fibs(), 40)):
    t1 = perf_counter()
    print(f"{i} -> {k} in {(t1 - t0):.5f}s")
    t0 = t1

-> 1 in 0.00006s
-> 1 in 0.00003s
-> 2 in 0.00001s
-> 3 in 0.00001s
-> 5 in 0.00000s
-> 8 in 0.00000s
-> 13 in 0.00000s
-> 21 in 0.00001s
-> 34 in 0.00001s
-> 55 in 0.00001s
-> 89 in 0.00002s
-> 144 in 0.00002s
-> 233 in 0.00004s
-> 377 in 0.00006s
-> 610 in 0.00010s
-> 987 in 0.00016s
-> 1597 in 0.00025s
-> 2584 in 0.00041s
-> 4181 in 0.00065s
-> 6765 in 0.00100s
-> 10946 in 0.00151s
-> 17711 in 0.00237s
-> 28657 in 0.00325s
-> 46368 in 0.00535s
-> 75025 in 0.00856s
-> 121393 in 0.01399s
-> 196418 in 0.02200s
-> 317811 in 0.03579s
-> 514229 in 0.05784s
-> 832040 in 0.09766s
-> 1346269 in 0.15621s
-> 2178309 in 0.24385s
-> 3524578 in 0.40004s
-> 5702887 in 0.64227s
-> 9227465 in 1.03693s
-> 14930352 in 1.67304s
-> 24157817 in 2.70860s
-> 39088169 in 4.40471s
-> 63245986 in 7.18042s
-> 102334155 in 11.64312s

After a while the time till next number behaves like the Fibonacci sequence itself! We are better off with iterations:

def fibs():
    """Generate Fibonacci numbers"""
    a, b = 0, 1
    while True:
        a, b = a + b, a
        yield a  # Yield keyword makes this function a generator


numbers = fibs()

# Let get first ten
for i, num in zip(range(50), numbers):
    print(i, num)

1
1
2
3
5
8
13
21
34
55
89
144
233
377
610
987
1597
2584
4181
6765
10946
17711
28657
46368
75025
121393
196418
317811
514229
832040
1346269
2178309
3524578
5702887
9227465
14930352
24157817
39088169
63245986
102334155
165580141
267914296
433494437
701408733
1134903170
1836311903
2971215073
4807526976
7778742049
12586269025

When we care about all results of pipeline it might be better/more explicit/readbable to use comprehensions. Here we consider list and dictionary comprehensions.

with open("./data/file.txt") as stream:
    # NOTE: with invokes a context manager. We want to manage resources;
    # here open a file and then make sure that it is correctly closed no matter what
    # will happen during manipulation, e.g. some error
    lines = [float(line.strip()) for i, line in enumerate(stream) if i % 2]

    # A Dictionary comprehension, create dict mapping row number to value
    d = {i: float(line.strip()) for i, line in enumerate(stream) if i % 2}


# To be compared with
with open("./data/file.txt") as stream:
    iterator = map(
        lambda p: float(p[1].strip()), filter(lambda p: p[0] % 2, enumerate(stream))
    )
    lines_ = list(iterator)
# Check that they are the same. We will come back to the `assert` statement shortly
assert lines == lines_
(d, bool(lines))

({}, True)

Food for thought:

Why is the dictionary empty while we clearly have lines as a non-empty list?
What is the performance of building list by for-loop versus list comprehensions? [Consider %%timeit magic]

%%timeit

def f(string):
    return sum(map(ord, string))

f("IN3110")

276 ns ± 1.33 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

While there are set comprehensions, one has to be careful with “tuple” comprehensions

{l for l in "aqowngeåonewvewuw"}

{'a', 'e', 'g', 'n', 'o', 'q', 'u', 'v', 'w', 'å'}

(v for v in islice(fibs(), 20) if v % 2)

<generator object <genexpr> at 0x105bb6190>

tuple(v for v in islice(fibs(), 20) if v % 2)

(1, 1, 3, 5, 13, 21, 55, 89, 233, 377, 987, 1597, 4181, 6765)

The with keyword is an entry point for contextmanager Its functionality is useful when there is some setup and tear-down needed before and after computations. Typical example is writing to file.

# Only write in caps


class CapsFile:
    # __enter__ and __exit__ special for context manager protocol
    def __init__(self, file_name, mode):
        self.file_obj = open(file_name, mode)
        write = self.file_obj.write
        # Here we replace the file object's write method
        self.file_obj.write = lambda w, write=write: write(w.upper())

    def __enter__(self):
        return self.file_obj

    def __exit__(self, type, value, traceback):
        self.file_obj.close()


with CapsFile("tes.t", "w") as out:
    out.write("Hello weltx")
    # raise ValueError

See about the result

cat tes.t

HELLO WELTX

Writing cleaner functions#

(Personal opinion) A good function 1) does what it is supposed to do, 2) does it quickly, 3) is user/developer friendly. Here we will focus on friendlines

Python uses so called duck-typing but we can express our intensions of the input arguments and function output by type annotations. These can be checked by mypy (but are not enforced)

# Following is code included in factorial.py.
def factorial(n: int) -> int:
    if n == 0:
        return 1
    return n*factorial(n-1)

factorial('works?')

We run type analysis by

(in3110) mirok@evalApply:data|$ mypy factorial.py 

Role of arguments should be clarified in a docstring of a function (or class). Type can be part of the docstring. We can also include tests via doctest. Documentation in the form of for example HTML pages can be generated by shinx. The following illustates a doctring with some nonexhaustive tests. For examples of Google-style docstrings see here

# Part of factorial_doctest.py
def factorial(n: int) -> int:
    '''Return the factorial of n, an exact integer >= 0.

    Args:
       n (int):  n!

    Returns:
       int.  The factorial value::

    >>> factorial(5)
    120
    >>> factorial(0)
    1

    '''
    if n == 0:
        return 1
    return n*factorial(n-1)

To run the doctest we execute

(in3110) mirok@evalApply:data|$ python factorial_doctest.py -v

We will come back to testing when discussing Python package development in the next part.

Enforcing the behaviour via exceptions (and their handling). There are several predifined exception types: eg. ValueError, AssertionError, MethodError. We can also define our own type.

class MyError(BaseException):
    def __init__(self, msg):
        self.msg = msg

    def __str__(self):
        return f'MyError occured with error message "{self.msg}"'

# This is contrived to illustrate the custom exceptions in action.


def factorial(n: int) -> int:
    """Return the factorial of n, an exact integer >= 0.

    Args:
       n (int):  n!

    Returns:
       int.  The factorial value::

    >>> factorial(5)
    120
    >>> factorial(0)
    1
    >>> factorial(-1)
    Traceback (most recent call last):
        ...
    ValueError: Only non-negative inputs are expected
    """
    # Raise AssertionError if the type is wrong
    assert isinstance(n, int)
    # Raise a different exception for negative integers
    if n < 0:
        raise ValueError("Only non-negative inputs are expected")

    if n == 42:
        raise MyError("This is not meant to be")

    if n == 0:
        return 1
    return n * factorial(n - 1)

Handling the raised exceptions. There are several predifined expection types: eg. ValueError, AssertionError, MethodError. We can also define our own type.

val = -4  #  32

try:
    f = factorial(val)
# We will try with the integer value
except AssertionError:
    from math import ceil

    n = ceil(val)
    print(f"Calling instead with {n}")
    f = factorial(n)

# Let's say that for negative we flip the sign
except ValueError:
    n = -val
    print(f"Calling instead with {n}")
    f = factorial(n)

except MyError as e:
    print("42!")

finally:
    # Sieve through here
    pass

Calling instead with 4

Modifying function behavior#

By now we have written function, we have seen functions that take in functions. What we want to do now is to write functions that return modified functions. In our first example we want to write a function which modifies the input function with timing information.

import time
from functools import wraps


def timeit(foo):
    """Return exacution time"""

    @wraps(foo)
    def wrapper(*args, **kwargs):
        then = time.perf_counter()
        result = foo(*args, **kwargs)
        now = time.perf_counter()
        print(f"{foo.__name__} executed in {now-then} s")

        return result

    return wrapper

Here we use the @wraps in order to preserve metadata of foo (see below). Let’s write the function to be timed.

def one_second():
    time.sleep(1)


print(one_second())

timed = timeit(one_second)
# As a **Food for thought** omit the @wraps decorator above and consider what happens with timed.__name__
timed.__name__

None

'one_second'

print(timed())

one_second executed in 1.0050931249988935 s
None

A syntactic sugar for applying timeit is via @

@timeit
def one_second():
    time.sleep(1)


print(one_second())

one_second executed in 1.0040542499991716 s
None

As another example of decorators consider memoization is a technique for caching the function’s return value for given input such that it does not need to be computed again. We can test the idea with the functools.lru_cache decorator.

from functools import lru_cache


def slow_factorial(n):
    """Factorial by recursion"""
    if n == 0:
        return 1
    return n * slow_factorial(n - 1)


@lru_cache
def faster_factorial(n):
    return slow_factorial(n)

Let’s see about the speed

%timeit slow_factorial(10)

567 ns ± 6.84 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

%timeit faster_factorial(10)

37.9 ns ± 0.171 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

As an exercise let’s write the cache decorator ourselves. However, unlike in @timeit we want to make a decorator which takes in an argument which is the cache size. Note that @lru_cache has this behavior too. We base our cache on a dictionary

class Cache(dict):
    def __init__(self, size):
        self.size = size

    def __setitem__(self, key, value):
        # Make room
        if len(self) >= self.size:
            # Grab some key
            key = next(iter(self))
            # and remove the entry
            self.pop(key)
        # Set it via parent (dict class)
        super().__setitem__(key, value)

Recall that decorator with arguments is applied as decorator(arguments)(function). That is decorator(arguments) must return a function

from functools import wraps


def cache(size):
    """Memoize"""

    def decorate(foo):
        memory = Cache(size)

        @wraps(foo)
        def wrapper(*args):
            # Lookup arguments. NOTE: here we only assumed positional arguments
            if args in memory:
                return memory[args]
            # Compute and remember
            result = foo(*args)
            memory[args] = result
            return result

        return wrapper

    return decorate


@cache(10)
def faster_factorial2(n):
    return slow_factorial(n)

faster_factorial2(3)

Let’s now speed up Fibinacci with cache

# Speed up Fibinachi with cache


@cache(10_000)
def fib(k):
    assert k > 0
    if k == 1:
        return 1
    if k == 2:
        return 1
    return fib(k - 1) + fib(k - 2)


def fibs():
    """Generator for all of them"""
    for k in count(1):
        yield fib(k)

from time import perf_counter

t0 = perf_counter()
for i, k in enumerate(islice(fibs(), 100)):
    t1 = perf_counter()
    print(f"{i} -> {k} in {(t1 - t0):.5f}s")
    t0 = t1

-> 1 in 0.00023s
-> 1 in 0.00035s
-> 2 in 0.00003s
-> 3 in 0.00002s
-> 5 in 0.00002s
-> 8 in 0.00002s
-> 13 in 0.00002s
-> 21 in 0.00002s
-> 34 in 0.00007s
-> 55 in 0.00002s
-> 89 in 0.00002s
-> 144 in 0.00002s
-> 233 in 0.00002s
-> 377 in 0.00001s
-> 610 in 0.00001s
-> 987 in 0.00001s
-> 1597 in 0.00001s
-> 2584 in 0.00003s
-> 4181 in 0.00002s
-> 6765 in 0.00001s
-> 10946 in 0.00001s
-> 17711 in 0.00002s
-> 28657 in 0.00001s
-> 46368 in 0.00001s
-> 75025 in 0.00001s
-> 121393 in 0.00001s
-> 196418 in 0.00001s
-> 317811 in 0.00002s
-> 514229 in 0.00001s
-> 832040 in 0.00001s
-> 1346269 in 0.00001s
-> 2178309 in 0.00001s
-> 3524578 in 0.00001s
-> 5702887 in 0.00002s
-> 9227465 in 0.00001s
-> 14930352 in 0.00001s
-> 24157817 in 0.00001s
-> 39088169 in 0.00001s
-> 63245986 in 0.00001s
-> 102334155 in 0.00002s
-> 165580141 in 0.00001s
-> 267914296 in 0.00001s
-> 433494437 in 0.00002s
-> 701408733 in 0.00001s
-> 1134903170 in 0.00001s
-> 1836311903 in 0.00001s
-> 2971215073 in 0.00001s
-> 4807526976 in 0.00002s
-> 7778742049 in 0.00001s
-> 12586269025 in 0.00001s
-> 20365011074 in 0.00001s
-> 32951280099 in 0.00001s
-> 53316291173 in 0.00001s
-> 86267571272 in 0.00001s
-> 139583862445 in 0.00001s
-> 225851433717 in 0.00002s
-> 365435296162 in 0.00001s
-> 591286729879 in 0.00001s
-> 956722026041 in 0.00001s
-> 1548008755920 in 0.00003s
-> 2504730781961 in 0.00001s
-> 4052739537881 in 0.00001s
-> 6557470319842 in 0.00001s
-> 10610209857723 in 0.00001s
-> 17167680177565 in 0.00001s
-> 27777890035288 in 0.00001s
-> 44945570212853 in 0.00001s
-> 72723460248141 in 0.00002s
-> 117669030460994 in 0.00001s
-> 190392490709135 in 0.00001s
-> 308061521170129 in 0.00001s
-> 498454011879264 in 0.00001s
-> 806515533049393 in 0.00001s
-> 1304969544928657 in 0.00001s
-> 2111485077978050 in 0.00001s
-> 3416454622906707 in 0.00001s
-> 5527939700884757 in 0.00001s
-> 8944394323791464 in 0.00001s
-> 14472334024676221 in 0.00001s
-> 23416728348467685 in 0.00001s
-> 37889062373143906 in 0.00001s
-> 61305790721611591 in 0.00002s
-> 99194853094755497 in 0.00002s
-> 160500643816367088 in 0.00001s
-> 259695496911122585 in 0.00001s
-> 420196140727489673 in 0.00002s
-> 679891637638612258 in 0.00001s
-> 1100087778366101931 in 0.00001s
-> 1779979416004714189 in 0.00001s
-> 2880067194370816120 in 0.00001s
-> 4660046610375530309 in 0.00001s
-> 7540113804746346429 in 0.00001s
-> 12200160415121876738 in 0.00001s
-> 19740274219868223167 in 0.00001s
-> 31940434634990099905 in 0.00001s
-> 51680708854858323072 in 0.00001s
-> 83621143489848422977 in 0.00001s
-> 135301852344706746049 in 0.00001s
-> 218922995834555169026 in 0.00001s
-> 354224848179261915075 in 0.00001s

Some other useful decorators are @property in class definitions.

class UnixName:
    """Max 8 characters"""

    def __init__(self, name):
        self.name = name  # NOTE: here were're calling the setter

    # Get
    @property
    def name(self):
        return self._name

    # Set
    @name.setter
    def name(self, name):
        if len(name) > 8:
            name = name[:8]
        self._name = name


(UnixName("Miro").name, UnixName("MiroslavKuchta").name)

('Miro', 'Miroslav')

Some extras for class: we have seen special methods that hook up e.g. to arithmetic ‘+’ (__add__) or interact with with (__enter__). What if you want to make your class iterable? In the example below we would like to iterate over a word in a special way where each letter is (shifter)[https://en.wikipedia.org/wiki/Caesar_cipher]

# __iter__ method


class Ceasar:
    def __init__(self, word, shift=2):
        assert word.isalpha()
        assert word.lower() == word
        self.word = word
        self.shift = shift

    def __iter__(self):
        for ch in self.word:
            yield chr(ord(ch) + self.shift)

for l, c in zip("helloworld", Ceasar("helloworld")):
    print(f"{l} -> {c}")

h -> j
e -> g
l -> n
l -> n
o -> q
w -> y
o -> q
r -> t
l -> n
d -> f

Note that many things can be iterated over - list produces its items, str the individual characters. Then it makes sense to ask for the first element. We know we list and str we are good

a = [2, 3, 4]
assert a[0] == 2

How would we get first element of a say a set?

things = set(map(chr, range(35, 40)))
things[1]

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[53], line 2
      1 things = set(map(chr, range(35, 40)))
----> 2 things[1]

TypeError: 'set' object is not subscriptable

# See Peter Norvig's https://github.com/norvig/pytudes

def first(iterable):
    # By iter we make sure we have something for which `next` can be called
    return next(iter(iterable))

first(things)

"'"

See more at Peter Norvig’s pytudes

Some review projects#

The let block#

In Julia language the let keyword introduces a new scope, i.e. we can e.g. have local variables named as global ones without interference. We would like to do something similar in python. Of course one option is a function - function body is a new scope.

import importlib

a = 2

def let(np=importlib.import_module("numpy")):
    a = 4
    return np.zeros((a, a))

print(let(), a)

[[0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]] 2

But this let is quite cumbersome. We can do [better](# Inspired by https://stackoverflow.com/questions/61371134/python-let-with-local-scopes-debug-printing-and-temporary-variables )

from contextlib import contextmanager
from inspect import currentframe, getouterframes

@contextmanager
def let(**bindings):
    # 2 because first frame in `contextmanager` is the decorator      
    frame = getouterframes(currentframe(), 2)[-1][0] 
    locals_ = frame.f_locals
    original = {var: locals_.get(var) for var in bindings}
    locals_.update(bindings)
    yield
    locals_.update(original)
    # Cleanup with keeping in scope the vars that were there originally
    for var in set(bindings).difference(locals_):
        del locals_[var]

We can now do something like this

import importlib
    
b = 3
with let(np=importlib.import_module('numpy'), x=2, y=4, b=5):
    ans = np.zeros((x, y))

print(ans)
print('b is as before let', b)
try:
    print(np, x, y)
except NameError:
    print('np x y not visible in this scope')

Fun with annotations#

Putting some of the lessons learned today and last time together: the following is heavily inspired by talk by David Beazley, see (YouTube)[https://www.youtube.com/watch?v=js_0wjzuMfc&t=3s] for original. We have seen that type annotations can be used for static type checking. Here we would like to “hook” into the type system with (i) our own types and (ii) at runtime. Think for example a function set_month that should take only non-empty strings containing [A-Z][a-z]

def set_month(x : AlphaString):
    return x

We begin by getting out custom string type by (multiple) inheritance

class Contract:
    @classmethod
    def check(cls, val):
        pass


class Typed(Contract):
    type = None

    @classmethod
    def check(cls, val):
        assert isinstance(val, cls.type)
        super().check(val)


class String(Typed):
    type = str


class NonEmpty(Contract):
    @classmethod
    def check(cls, val):
        assert len(val) > 0
        super().check(val)


class AlphaString(String, NonEmpty):
    @classmethod
    def check(cls, val):
        assert val.isalpha()
        super().check(val)


AlphaString.check("mkds6")

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[56], line 34
     30         assert val.isalpha()
     31         super().check(val)
---> 34 AlphaString.check("mkds6")

Cell In[56], line 30, in AlphaString.check(cls, val)
     28 @classmethod
     29 def check(cls, val):
---> 30     assert val.isalpha()
     31     super().check(val)

AssertionError: 

Now we would like to declare the argument type and enforce it. Of course we could put all in type assertions in the function’s body but we have in mind using this pattern a lot and don’t want to have these hand-written checks at the beginning of our code

def set_month(x : AlphaString):
    return x

For modifying function behaviour we have before used decorators …

import inspect
from functools import wraps


def checked(func):
    signature = inspect.signature(func)
    annotations = func.__annotations__

    @wraps(func)
    def wrapper(*args, **kwargs):
        # Bind arguments(names) to their values
        bound = signature.bind(*args, **kwargs)
        for varname, value in bound.arguments.items():
            if varname in annotations:
                # For each annotated argument check the bound value
                typed = annotations[varname]
                typed.check(value)
        # If everything is okay we can call func
        return func(*args, **kwargs)

    return wrapper

@checked
def set_month(x: AlphaString):
    return x


set_month("janua7ry")

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[58], line 6
      1 @checked
      2 def set_month(x: AlphaString):
      3     return x
----> 6 set_month("janua7ry")

Cell In[57], line 17, in checked.<locals>.wrapper(*args, **kwargs)
     14     if varname in annotations:
     15         # For each annotated argument check the bound value
     16         typed = annotations[varname]
---> 17         typed.check(value)
     18 # If everything is okay we can call func
     19 return func(*args, **kwargs)

Cell In[56], line 30, in AlphaString.check(cls, val)
     28 @classmethod
     29 def check(cls, val):
---> 30     assert val.isalpha()
     31     super().check(val)

AssertionError: 

More fun with annotations#

In C++ we can have functions with the same name but differing by arguments (types or count) and the right function to be called will be determined by the arguments at compile time. In python, this idea can be accomplished e.g. by

def foo(a, b=None):
    if b is None:
        b = 2
    if isinstance(b, float):
        return a*b
    return a+b

def foo(a, b=None):
    if b is None:
        b = 2
    if isinstance(b, float):
        return a * b
    return a + b


(foo(2, 3), foo(2, 3.0), foo(2))

(5, 6.0, 4)

While this works the if statements make the code somewhat convoluted. Since we have the type annotations, wouldn’t it be nicer if something like this was possible?

def foo(a: int, b: int):
    return a + b

def foo(a: int, b: float):
    return a*b

def foo(a: int):
    return foo(2, a)

We take some inspiration from Python’s founder Guido van Rossum’s recipe

class MultiMethod:
    def __init__(self, name):
        self.name = name
        self.typemap = {}

    def __call__(self, *args):
        """Dispatch to method based on type of the arguments"""
        types = tuple(arg.__class__ for arg in args)
        function = self.typemap.get(types, None)
        if function is None:
            raise TypeError("no match")
        return function(*args)

    def register(self, types, function):
        """Add"""
        if types in self.typemap:
            raise TypeError("duplicate registration")
        self.typemap[types] = function

import inspect

_FUNC_TABLE_ = {}


def multimethod(func):
    signature = inspect.signature(func)
    annotations = func.__annotations__
    # Pull out type annotations to define the type signature
    type_info = tuple(annotations[arg] for arg in signature.parameters)

    name = func.__name__
    mm = _FUNC_TABLE_.get(name)
    if mm is None:
        mm = _FUNC_TABLE_[name] = MultiMethod(name)
    # Register the implementation for that name and the signature
    mm.register(type_info, func)
    return mm

# In action


@multimethod
def bar(a: int, b: int):
    return a + b


@multimethod
def bar(a: int, b: float):
    return a * b


@multimethod
def bar(a: int):
    return bar(2, a)

(bar(2, 3), bar(2, 3.0), bar(2))

(5, 6.0, 4)

Back to \(\lambda\)-calculus#

In (Scheme)(https://en.wikipedia.org/wiki/Scheme_(programming_language)) and other Lisp there is are functions cons, car and cdr that (for the purpose of this project) behave such that

car(cons(a, b)) == a
cdr(cond(a, b)) == b

If we were to implement the 3 functions based only on this specification we could reach out to list or tuples, i.e. rely on some data structure

cons = lambda a, b: (a, b)

car = lambda p: p[0]
cdr = lambda p: p[1]

a, b = 2, 3

car(cons(a, b)) == a
cdr(cons(a, b)) == b

True

In this spirit of \(\lambda\) calculus we can as whether we need data structures? Afterall, everyhing should be possible with just functions

def cons(a, b):
    return lambda f: f(a, b)


def car(thing):
    return thing(lambda p, q: p)


def cdr(thing):
    return thing(lambda p, q: q)

a, b = 2, 3

car(cons(a, b)) == a
cdr(cons(a, b)) == b

True

See Structured and Interpretetion of Computer Programs for more inspiration.