Close modal

Blog Post

Python3 - What you're missing

Development
Tue 10 April 2018
0 Comments

Python 2.7 is convenient, it's already installed on many linux and unix machines (including mac) out of the box, and is therefore portable. If you haven't looked at Python 3 (3.6 as of this article as stable non-beta), well you are missing out. I am going to cover off just a few of the features that versions since 3.0 have released and that may be most interesting to everyday usage. Let us stroll down release lane.

What's new in Python 3.6:

PEP 498: Formatted string literals

Those of you familiar with Swift 3+ will love this one. Like swift variables can be substituted within the string directly, unlike swift you need to prefix the string literal with f, such as f'{variable}', but you done't need a special '\' as in '(variable)'.

::: python
>>> user, id = ('Johnny Appleseed', 100)
>>> f'User is {user} with id {id}'
'User is Johnny Appleseed with id 100'
>>> users = [dict(name='Bob', id=100), dict(name='Jane', id=101), dict(name='Tom', id=102)]
>>> print(f'Users are: {users}')
Users are: [{'name': 'Bob', 'id': 100}, {'name': 'Jane', 'id': 101}, {'name': 'Tom', 'id': 102}]

We can do even more interesting things though:

>>> print(f'Users are: {[user["name"] for user in users]}')
Users are: ['Bob', 'Jane', 'Tom']

As you can see, it will evaluate expressions within {}, note of course you'll need to alternate ' and " as per the string's delimiting.

PEP 487: Simpler customization of class creation. This is awesome.

Ever wondered if there was a way to grab any classes in your namespace who inherited from something? Yes, now there is thanks to PEP-487. The magic happens in the init_subclass method, whenever a subclass is initialialised (we're talking static type initialiser, not instance as that already exists when super().init(self) is called). The most obvious application of this comes to a pluggable plugin architecture, observe a simple example.

:::python import shutil import psutil

class StatPlugin: subclasses: list = []

  def __init_subclass__(cls, **kwargs):
      super().__init_subclass__(**kwargs)
      cls.subclasses.append(cls)
      print(f'[I] Loaded plugin {cls.__name__}')

  def scan(self) -> str:
      pass

class MemoryPlugin(StatPlugin): def scan(self) -> str: total, free = psutil.virtual_memory()[:2] return f'Available Memory: {free} bytes. {int((100 * free)/total)}%.'

class DiskPlugin(StatPlugin): def scan(self) -> str: total, _, free = shutil.disk_usage(file) return f'Available Disk: {free} bytes. {int((100 * free)/total)}%.'

print("\nHere are your system stats:") for p in StatPlugin.subclasses: print('-', p().scan())

The output of running this is:

:::shell Mitchells-MacBook-Pro% python3 test_plugin.py [I] Loaded plugin MemoryPlugin [I] Loaded plugin DiskPlugin

Here are your system stats: - Available Memory: 5535215616 bytes. 32%. - Available Disk: 239523131392 bytes. 47%.

PEP 520: Preserving Class Attribute Definition Order

PEP 468: Preserving Keyword Argument Order

In CPython 3.6, kwargs is ordered, but holdtight because in 3.7 dict itself will be ordered (for insertion) and that's a language spec, not an implementation!

PEP 506 -- Adding A Secrets Module To The Standard Library

The secrets module provides access to the most secure source of randomness that your operating system provides.

Enhancements to existing modules:

contextlib.AbstractContextManager

contextlib.AbstractContextManager: Designed as simple starting point for context managers where __enter__() return self and abstract method __exit__() returns None.

Enum enhancements for Flag, IntFlag and auto.

enum.IntFlag, enum.Flag base classes and the enum.auto method: ...

What's new in Python 3.5:

PEP 492 - Coroutines with async and await syntax

Here's an example that lists the rank, date and title of reddit's top posts.

import aiohttp
import asyncio
from lxml import html

async def make_request():
    async with aiohttp.ClientSession() as session:
        async with session.get('https://www.reddit.com/top/') as resp:
            text = await resp.text()
            tree = html.fromstring(''.join(c for c in text if ord(c) < 128))
            topic_elements = tree.xpath('.//div[@class="top-matter"]')
            for index, topic in enumerate(topic_elements):
                info, title = topic.xpath('.//p')
                print(f'{index}: {info.text_content()}\n{title.text_content()}\n')

asyncio.get_event_loop().run_until_complete(make_request())

PEP 448 - Additional Unpacking Generalizations

If you can't think of a use case for this, imagine you have two sources of keyword arguments, perhaps one coming from os.envrion and one generated locally. You could quite easily combine them into a single dictionary, but that will add extra lines and detract from the clarity for such a mundane task. Let's try unpacking multiple keyword dictionaries to a function call, with Python 3.4 version.

Python 3.4.5 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:47:57)
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> args1 = {'name': 'Alice'}
>>> args2 = {'debug': True}
>>> def display(name, debug):
...     print(name, debug)
...
>>> display(**args1, **args2)
  File "<stdin>", line 1
    display(**args1, **args2)
                   ^
SyntaxError: invalid syntax

Dang, it won't let us do that... seems this isn't supported in Python, but wait, let's try this on Python 3.5

Python 3.5.5 |Anaconda, Inc.| (default, Mar 12 2018, 16:25:05)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> args1 = {'name': 'Alice'}
>>> args2 = {'debug': True}
>>> def display(name, debug):
...     print(name, debug)
...
>>> display(**args1, **args2)
Alice True

Simple. If you need to use this, it makes things much tidier.

PEP 461 - percent formatting support for bytes and bytearray

Need to generate bytes from a template (such a protocol or file header) but don't want to do the gymnastics back and forth from string to use formatting? Good news it can now be done as follows:

>>> from datetime import datetime
>>> from struct import pack
>>> b'HEADER:%b' % pack('q', int(datetime.now().timestamp()))
b'HEADER:\xbb\xfd\xcaZ\x00\x00\x00\x00'

In that way we've inserted bytes into another bytes just like string.

PEP 484 - Type Hints

The typing module (provisional) from the standard library is quite useful. You can use the Generic, Tuple and List... etc classes to provide more description than just primitive types, here's an example

from typing import List, Tuple
def factors(target: int) -> List[Tuple[int, int]]:
    return []

Python 3.4 would throw an expected error of ImportError: No module named 'typing', and we wouldn't be able to represent this comppsite return type to help with code analysis.

What's new in Python 3.4:

PEP 435 -- Adding an Enum type to the Python standard library

First availability of enums in Python 3, the application of these should be fairly apparent, for example representing a result or error state of HTTP client (extremely simplified):

>>> from enum import Enum
>>> class HTTPError(Enum):
...      ok = 200
...      not_found = 404
...      server_error = 500
...

That's quite a neat and succinct way to represent a known set of states, and creating or parsing them is actually very elegant too, you can use both the members of known names or even parse the code for example:

>>> HTTPError.ok
<HTTPError.ok: 200>
>>> HTTPError(404)
<HTTPError.not_found: 404>

What's new in Python 3.3:

unittest.mock — mock object library

This is available as a backport on pypi via pip install mock, and is a very useful function I elaborate more on in part in this post. Here's the simplest example:

import os
from mock import patch

# Simple way, declare it in the decorator - short and sweet
@patch('os.getcwd', return_value='/var')
def test_path_simple(mock_getcwd):
  result = os.getcwd()
  assert mock_getcwd.call_count == 1
  assert result== "/var"

This not only replaces the call for os.getcwd() (meaning you prevent the real operation from being invoked, and replace the value), but also lets you measure and asset the call count as part of the test case - in this case we require it to be called exactly once.

What's new in Python 3.2:

Enhancements to contextlib

This is pretty neat, essentially a context manager can be used as a decorator.

from contextlib import contextmanager
from datetime import datetime
import logging
import random

logging.basicConfig(level=logging.INFO)

@contextmanager
def log_time(name):
    start_time = datetime.now()
    yield
    duration = int((datetime.now() - start_time).total_seconds() * 1000)  # milliseconds
    logging.info('Duration: %s for %d', name, duration)


# Now, it can be used as a decorator as well:
@log_time('Generate numbers')
def generate():
    numbers = [random.random() for i in range(0,1000000)]

generate()

Run this, and you'll see something like: INFO:root:Duration: Generate numbers for 147. This is a simple example, but by decorating and applying to a number of utility of core functions, you can easily mix-in behaviour like custom timing of methods and so-on.

What's new in Python 3.0:

There not being much of note for such a considerably condensed article in Python 3.1, we're down the changes Python 3.0 introduced over 2.x

Gotchas

  • Print is and always has been a function, it's now enforced so use print(...) and not print ....
  • dict.iterkeys(), dict.iteritems() and dict.itervalues() are deprecated, use dict.keys(), dict.items() and dict.values() respectively which return 'views' not lists (more efficient using iterators).
  • Many other functions also work with generators/iterators instead of plain lists, you can use next() on these iterators.
New stuff
  • Annotations for function argument were first introduced here, but much expanded upon in later versions as alluded to above.
  • nonlocal keyword added to refer to a variable in outer (non-global) scope without creating an implicit item in the local scope.
  • Better unpacking of iterable: (a, *rest, b) = range(5) the rest list takes all items between the first and last (it can even be empty if there were only 2 items to unpack).

Summary

Wheh! that was a lot to cover... So if you're still eeking by on Python 2.7, update today! There's plenty not covered here that is also available. It's practically a different langauge and using the stock 2.7 installation that comes with most systems means you're missing out on some of the most modern and awesome features covered by Python 3.6.