tartley.com (Posts about software)

Code in Place

Jonathan Hartley — Mon, 07 Jul 2025 14:03:35 GMT

Code in Place 2025 has ended. I participated as a Section Leader, tutoring a small group of beginner programmers through learning basic coding in Python. We covered fundamental programming - variables and loops and conditionals and data-structures, and used them to cajole small on-screen robots into moving around and performing tasks, and to draw some graphics with lines and filled shapes.

The course was open to all, so there were many working members of the public alongside the students. While the course is a redrafting of the famous Stanford CS106A course content, these were not generally computer science students, but instead were studying some other subject, and wanted to add a little bit of programming as an extra skill on the side.

The interesting thing about Code in Place is that it aims to investigate scaling the teaching experience, without falling prey to the usual failure modes of Massive Open-Access Online Courses (MOOCs), of poor feedback on student work and high drop-out rates. They did this by recruiting 4,000 teaching volunteers like me to act as Section Leaders. We guided the 40,000 students through the material in small classes of ~10 students, and provided 1:1 guidance and question answering in forums.

The application for section leader involved some basic competency questions, and an interesting task to record a video of me teaching some basic programming concepts. This all helped me to understand that I need to scale down my expectations of what is "self-evident", reminding me that beginners need to be taught from the beginning.

While AI was used, it was in a thoughtful, advisory capacity. For students, providing explinations of terse technical errors in their own code, and giving feedback on successfully completed tasks, to improve their solutions.

Once the classes started, I heard complaints in the teachers' forums that other classes suffered from some absenteeism, or students simply not doing the work. But my students were absolutely fabulous, reliably showing up on time, doing all the exercises, and engaging honestly and curiously in the classes. I think that students were sorted into age groups that tend to be not too dissimilar to their Section Leader, so perhaps I had an older-than-average cohort, which might partly explain how I got so lucky.

Regardless of how it happened, it was a thrillingly rewarding experience, seeing people rapidly grow their abilities, and become confident where they had originally been fearful. I'm already looking forward to next year's course!

Integer Division With Recurring Decimals

Jonathan Hartley — Mon, 03 Mar 2025 18:42:50 GMT

I've been doing some programming tests and puzzles while job hunting lately. One quick challenge was quite nice, reminding me a bit of Project Euler questions, and I nerd sniped myself into doing a 2nd pass at it here.

Question

Produce a Python function which takes two integers, numerator and denominator, and returns the result of their division as a decimal fraction in a string. E.g:

divide(1, 4) -> "0.25"

If the decimal places contain an infinite recurring pattern of digits, then enclose the recurring digits in parentheses. E.g:

divide(1, 3) -> "0.(3)"
divide(1, 7) -> "0.(142857)"

Wrong approaches

Evaluating the division using normal floats is going to trip you up in several ways with the limited precision.

For one, a large enough denominator might have a recurring sequence which is longer than the number of decimal places you have available (more on this later), which makes it impossible to detect recurring sequences by examining the division's decimal digits.

Worse, the inherent imprecision of floating point, e.g. if a simple division like 10/3 comes back as 3.3333333333333335, then examining the trailing digits of this looking for recurring digits is going to be problematic.

Using the decimal module does markedly improve precision and control. But infinitely repeating sequences are still going to return results like Decimal(20) / Decimal(3) -> Decimal('6.666666666666666666666666667'), which is going to trip us up.

We can sidestep all these complexities if we see that the question is asking us to perform this division ourselves, longhand. We are going back to elementary school! Wheee!

Better

Let's just do basic division first, ignoring infinite or recurring digits:

def divide(numerator: int, denominator:int) -> str:
    # Accumulate parts of our result here
    results = []
    while True:
        int_part = str(numerator // denominator)
        remainder = numerator % denominator
        numerator = remainder * 10
        results.append(int_part)

        # If there is no remainder, we are done
        if remainder == 0:
            break

        # Add a decimal point after our first integer part
        if len(results) == 1:
            results.append(".")

    return ''.join(results)

The only confusing parts of this are that int_part might contain several digits on the first iteration, but is only ever one decimal digit thereafter. Plus we have to be careful to get the ordering right for our checks to exit the loop, versus appending the decimal point to the output, to avoid weird looking outputs like divide(6, 2) -> "3.".

Trying this out:

>>> divide(1, 4)
'0.25'

It works! But we haven't yet handled infinite decimals, they result in an infinite loop:

>>> divide(1, 3) # Hangs!

Recurring digits

Because we're dividing integers, we cannot get infinitely varying decimal places. If we have an infinite number of decimal places, it must be because of a cycle of one or more recurring digits. To detect such a cycle we have to notice a couple of things.

First, simply seeing a digit in the output which we have seen before is not enough. Looking at the three assignments at the start of the above while-loop, which capture our state:

int_part = str(numerator // denominator)
remainder = numerator % denominator
numerator = remainder * 10

Here, int_part gets the value of each successive decimal digit. However if it takes on the same value as in a previous iteration, the accompanying remainder might be different, and it is the remainder which is used to generate the numerator for the next iteration, and hence generate the sequence of digits after this.

So, as we already knew from common sense, two iterations with the same int_part may go on to produce different sequences of subsequent digits. However, The value of remainder is the only thing which determines the inputs to our next iteration:

int_part depends on numerator and on denominator (which is constant)
numerator depends on remainder.

Hence, two iterations might produce different digits, but produce the same remainder, and from that point onwards, they will be in lockstep. If we find two such iterations, then we have detected an infinite recurring cycle of digits.

So, before the loop begins, initialize a dict:

# Remainders seen to date, mapped to their position in the result
remainders = {}

Then inside the loop, after everything else, use our new dict to detect if we have seen the current remainder before:

# If we have seen this remainder before, we are now in exactly the
# same state as then, so we have found a recurring digit sequence.
if remainder in remainders:
    # We have found a cycle of decimal digits! Insert parens into our results,
    # from the last seen position of this remainder, up to the current digit.
    last_pos = remainders[remainder]
    results = (
        results[:last_pos] +
        ["("] +
        results[last_pos:] +
        [")"]
    )
    break
# Remember the position at which we saw this remainder
remainders[remainder] = len(results)

Trying this out:

>>> divide(1, 3)
0.(3)
>>> divide(1, 7)
0.(142857)

OMG it works!

Defensive coding

We're putatively done, but the grumpy old dev in me is uncomfortable leaving that while True in there. By deduction, we always must eventually hit the if <condition>: break to escape from it, so ostensibly it's fine. But if I have a bug in the code or my reasoning, then it might lead to an infinite loop, in some scenario I'm not thinking of. Can we limit the number of iterations in some other, foolproof way? Turns out we can.

We've seen already that a repeated value of remainder means we can break from the loop. Also, notice that remainder, given by:

remainder = numerator % denominator

can only take values from 0 to denominator - 1. So it can have denominator possible values, and this is the maximum number of iterations we will ever need.

Hence, we can safely replace the while(True) with:

for _ in range(denominator):
    ...

Splendid! Much less anxiety-inducing

The source is on github.

SVG Trees using recursive Python functions

Jonathan Hartley — Fri, 28 Feb 2025 17:10:27 GMT

Inspired by a woodland hike under the first blue skies we've seen this year, I got home and showed the kiddo how to draw an SVG tree with recursive functions in Python.

At first the generated shape looked kinda lumpy and uninspiring, but it did demonstrate the principle. We were thinking of calling it a day, but I did a little bit of tweaking on parameters to control how each branch differs in length and direction from its parent. Suddenly, the generated shape really came alive, and started to look a lot more like the trees we'd seen on our hike that afternoon.

This image uses a recursion depth of 18, yielding 2^18 twigs, i.e. 250,000, which generates a 100MB SVG file. This takes about ten seconds to generate, and another ten to display in an SVG viewer. Alternatively, I can convert the SVG to a lossy webp, as displayed here, which is only 280kB and displays instantly.

Pushing the generation to greater recursion depth makes my SVG viewer and conversion tools start to stutter and barf. Presumably I could be smarter about the SVG I generate -- maybe generating the outline of the tree as points on fewer, more complex polygons, instead of a polygon for each branch segment? No matter, the artifact is the thing here, and it's done now.

Source is at https://github.com/tartley/tree-art.

TIL: Shell environment variable tricks

Jonathan Hartley — Thu, 03 Oct 2024 20:37:27 GMT

envsubst is an executable you likely already have on your PATH (part of the gettext package, often installed with dev packages), which is a convenient way to replace $VAR or ${VAR} style environment variables with their values. This allows rendering templates without heavyweight tools like Ansible, Jinja, or embedding with heredocs. Usage is:

envsubst <template >rendered

For example:

$ envsubst <<<'Hello $USER'
Hello jonathan

(Note the use of single quotes so that $USER isn't expanded by our shell, as it wouldn't be in the file which <<< is emulating for us.)

If you'd like to use KEY=value declarations from a dotenv-style .env file, you can auto-export them by setting the -a Bash option:

set -a; source .env; set +a

Something I've managed to avoid ever realizing for 30 years, but now that I've seen it I can't imagine a week going by without using it. The kind of thing that should be part of everyone's "Week 1 in a terminal" training that formal education courses never include.

Illustrating Uses of IBM Cloud Security Groups

Jonathan Hartley — Wed, 23 Aug 2023 02:33:47 GMT

I wrote this high-level public-facing guide while employed by IBM, implementing the security groups feature for IBM Cloud in Go and PHP (don't ask). It used to reside on the IBM blog, but has recently been replaced by newer content, so I've preserved it here for posterity.

This article illustrates a few possible uses of IBM Cloud Security Groups, a per-instance firewall for IBM Cloud virtual instances.

Why Use Security Groups?

Security groups firewall your IBM Cloud applications from nefarious network traffic, protecting you and your company from the efforts of “industrious users” trying to bring down your application, or make off with your customer's credit card details. If those sound like sub-optimal outcomes for your situation, read on…

Allow Incoming SSH Connections

The simplest use of security groups is to allow a single type of network connection to your instances, blocking all other traffic. For example, to allow only incoming SSH connections, which are TCP connections on port 22. All other types of traffic, such as ICMP ‘ping' connections, or TCP connections on other ports, are blocked. Fig 1. A security group configured to allow incoming SSH

Fig 1. A security group configured to allow incoming SSH.

When instances 1 & 2 are added to your security group, firewalls are created directly on those instances,configured to allow or deny the corresponding traffic. Hence, your support engineer can create SSH connections, but cannot send arbitrary network traffic.

Allow SSH from a Specified IP Address

The above scenario allows SSH connection attempts from any IP address. To increase security, you might only allow connections from a particular instance. Fig 2. A security group configured to allow incoming SSH connections (TCP port 22) from a particular IP address.

Fig 2. A security group configured to allow incoming SSH connections (TCP port 22) from a particular IP address.

The security group has been configured with the IP address used by a support engineer – the single instance that is authorized to make a connection. Connections from other instances are blocked.

As well as allowing traffic from a single IP address, security groups can be configured with a CIDR block, to allow traffic from all instances on that subnet. Fig 3. A security group configured to allow incoming SSH connections (TCP port 22) from all instances on a given subnet.

Fig 3. A security group configured to allow incoming SSH connections (TCP port 22) from all instances on a given subnet.

The diagram shows instances on an authorized subnet (deploy1 & deploy2), representing a project CI/CD infrastructure, all being able to make SSH connections to our protected instances, for example to deploy updates to our application. Other instances, such as an enterprising hacker, are blocked.

Allow Application Instances to Access a Distributed Data Store

Another use case would be to allow application servers to access the nodes of a distributed data store. To do this, we'll make a few changes to the above security group configuration.

Firstly, we modify the open port from SSH's 22, to MongoDB's default query API port of 27017.

Secondly, security group 1 is now allowing the creation of outgoing network connections, from our application instances, where previously it was allowing incoming connections. Security group rules can manage traffic in either direction.

Thirdly, we don't want our data store instances to be unprotected, so we'll put them into a security group of their own. Since there's now two security groups, we'll give them names: “app” and “db”.

For now, security group “db” allows all incoming MongoDB queries (TCP connections on 27017), without restricting the IP addresses allowed to make connections. We'll fix that soon. Fig 4. Two security groups configured to allow application servers to send queries to MongoDB nodes on a subnet.

Fig 4. Two security groups configured to allow application servers to send queries to MongoDB nodes on a subnet.

For clarity, these diagrams don't show the many connections which are blocked by this setup - which are, of course, the whole point of security groups. On the above diagram, blocked connections would include:

On app instances:
- All incoming connections.
- Outgoing connections that aren't TCP, or are on the wrong port.
- Outgoing connections to anything other than a DB instance.
On DB instances:
- All outgoing connections.
- Incoming connections that aren't TCP, or are on the wrong port.

This setup does have a couple of problems. It relies on our MongoDB instances all residing on a single subnet. Worse, as mentioned earlier, it allows any IP address in the world to make queries to MongoDB. We'll fix both of these next.

Using Remote Groups to Specify Arbitrary IP Addresses

Specifying allowed instances using a CIDR block can be inappropriate. It's often preferable to specify a set of arbitrary IP addresses instead. We can do this by using a second security group – known as a remote group – to contain the set of allowed instances. Our first security group can then refer to the remote group to specify which instances are allowed.

In our example above, we would modify the “app” security group by dropping the CIDR block, and replacing it with a reference to “db” as a remote security group.

Similarly, we would modify the “db” group to use “app” as a remote group, only allowing connections from the members of that group.

Fig 5. Two security groups configured to allow application servers to send queries to MongoDB nodes using remote groups.

This has several advantages. Firstly, our data store no longer accepts malicious queries from hackers all over the internet – only from our app instances.

Secondly, our data store instances no longer need occupy a subnet, they can have arbitrary IP addresses.

Because the security groups now specify allowed hosts by referencing each other, when the members of either group changes, the instance level firewalling rules on all instances are updated automatically, to allow or deny traffic based on the new membership.

This configuration starts to show what makes security groups a flexible, dynamic, low-maintenance solution.

Accepting Web Requests

Our application instances aren't any use without a web front end. We put our web instances into their own security group (“web”), which allows incoming requests from users on the web, and outgoing API requests to our app instances, on port 61516.

Similarly, we need to add a second rule to the “app” security group, to allow incoming requests from “web”. Fig 6. A traditional three-tier application using remote security groups.

Fig 6. A traditional three-tier application using remote security groups.

The more types of server we add to our setup, the more benefit “remote” groups provide, by minimizing setup configuration, and by automatically keeping firewalling rules up to date when group membership changes.

A future blog post will discuss how to set up this three-tier scenario, and describe details such as how to use multiple network interfaces on an instance, such as the web instances which will use a public IP address exposed to users, versus a private IP exposed to the API instances.

Add a Bastion

We need some way to access our servers, so that we can, for example, deploy new versions of our application. Commonly this is achieved using a bastion server, which provides a single, carefully hardened point of access.

Here we add a bastion server, and modify all our security groups to allow SSH access from the bastion to all our instances. The bastion instance itself would be configured to only allow incoming connections from the appropriate points in a CI/CD infrastructure (not shown.) Fig 7. Adding a bastion server with SSH access to all other instances.

Fig 7. Adding a bastion server with SSH access to all other instances.

This is starting to look like a realistic setup for a modest but scalable multi-tier application.

Conclusion

Security groups are a flexible and powerful way to firewall network traffic to and from your system's instances. We've shown how they might be used in a few typical scenarios, and hopefully demonstrated that they are flexible enough to accommodate many others. For more information, see Getting Started with Security Groups.

Jonathan Hartley
Senior Cloud Developer

TIL: git push --force-with-lease

Jonathan Hartley — Fri, 04 Aug 2023 18:48:19 GMT

Don't ever type git push --force. Yes, there are times we have to hold our nose and do a force push. Maybe the project requires contributions to be rebased or squashed. Maybe we pushed the nuclear launch codes. But there are failure modes:

You might be accidentally pushing to or from the wrong branch, and hence are about to blow away valuable work at the remote. Yes, is unlikely, and can be fixed after the fact, but who knows how much embarrassing disruption and confusion you'll cause the team before you realize what you did.
Do you always remember to check the state of the remote, to make sure there isn't unexpected extra commits on the remote that you'll unknowingly blow away when you push? Do you enjoy always having to type those extra commands to pull and check the remote commits?
No matter how conscientious we are about checking the above, there is a race condition. We might check the remote, then someone else pushes valuable changes, then we force push and blow them away.

Although there are conventions that can help with all the above (e.g. only ever force pushing to your own fork, to which nobody else ever pushes), they aren't generally watertight. (e.g. you might have pushed something yourself, before vacation, and forgotten about it.)

So the generally agreed method to avoid the above failure modes is "be more careful", which sounds to me like the common prelude to failure. What we need are push's newer command-line options:

--force-with-lease: Like --force, but refuses to push if the remote ref doesn't point at the same commit that our local remote-tracking branch 'origin/mybranch' thinks it should. So if someone else pushes something to the remote's 'mybranch' just before we try to force push, our push will fail until we pull (and, in theory, inspect) those commits that we were about to blow away.

It turns out that this is inadequate. One might have fetched an up-to-date remote branch, but somehow or other ended up with our local HEAD on a divergent branch anyway:

C origin/mybranch
|
B¹   B² HEAD mybranch
 \ /
  A
  |

In this situation, --force-with-lease will allow us to push, not only blowing away the original commit B¹, as we intended, but also C, which was maybe pushed by someone else before we fetched.

To guard against this, we can use the even newer option:

--force-if-includes: This makes --force-with-lease even more strict about rejecting pushes, using clever heuristics on your local reflog, to check that the remote ref being updated doesn't include commits which have never been part of your local branch.

Upshot is, I plan to default to always replacing uses of --force with:

git push --force-with-lease --force-if-includes ...

That's a lot to type, the options don't have short versions, and it's easy to forget to do. Hence, shadow git to enforce it, and make it easy. In .bashrc or similar:

# Shadow git to warn againt the use of 'git push -f'
git() {
    is_push=false
    is_force=false
    for arg in "$@"; do
        [ "$arg" = "push" ] && is_push=true
        [ "$arg" = "-f" -o "$arg" = "--force" ] && is_force=true
    done
    if [ "$is_push" = true ] && [ "$is_force" = true ]; then
        # Suggest alternative commands.
        echo "git push -f: Consider 'git push --force-with-lease --force-if-includes' instead, which is aliased to 'gpf'"
        return 1
    fi
    # Run the given command, using the git executable instead of this function.
    $(which git) "$@"
}

# git push force: using the new, safer alternatives to --force
gpf() {
    # Older versions of git don't have --force-if-includes. Fallback to omitting it.
    if ! git push --quiet --force-with-lease --force-if-includes "$@" 2>/dev/null ; then
      git push --quiet --force-with-lease "$@"
    fi
}

Then trying to do it wrong tells you how to easily do it right:

$ git push -f
git push -f: Consider 'git push --force-with-lease --force-if-includes' instead, which is aliased to 'gpf'
[1]
$ gpf
$

(The [1] is my prompt telling me that the last command had an error exit value.)

Structured Pattern Matching in Python

Jonathan Hartley — Sat, 29 Jul 2023 23:15:26 GMT

I read through descriptions of structured pattern matching when it was added in Python 3.10 a couple of years ago, and have studiously avoided it ever since. It seemed like a language feature that's amazingly useful in one or two places, like writing a parser, say, and is a horrifically over-complicated mis-step just about everywhere else.

Update: A day after writing this I see that Guido van Rossum wrote exactly that, a parser, to showcase the feature. I'm guessing he writes a lot of parsers. I definitely don't write enough of them to think this language feature is worth the extra complexity it brings.

Regardless, I really ought to remember how it works, so this is my attempt to make the details stick, by writing about it.

If you're not me, you really ought to be reading about it from the source instead:

PEP 643: Specification.
PEP 635: Motivation and rationale.
PEP 636: A tutorial.

Basic structure

match EXPRESSION:
    case PATTERN1:
        ...
    case PATTERN2:
        ...
    case _:
        ...

This evaluates the match EXPRESSION, then tries to match it against each case PATTERN, executing the body of the first case that matches, falling back to the optional final _ default case. (match and case are not keywords, except in the context of a match...case block, so you can continue using them as variable names elsewhere.)

But what are PATTERNs, and how are they tested for a match?

Patterns

Patterns can be any of the following. As becomes increasingly obvious down the list, the real power of this feature comes from composing each of these patterns with the others. For complicated patterns, parentheses can be used to indicate order of operations.

Literals

Like other languages' traditional switch statement:

match mycommand:
    case 'start':
        ...
    case 'stop':
        ...
    case _:
        raise CommandNotFoundError(mycommand)

Such literal case patterns may be strings (including raw and byte-strings, but not f-strings), numbers, booleans or None.

Such cases are compared with equality:

match 123:
    case 123.0:
        # matches!

except for booleans and None, which are compared using is:

class Any:
    def __eq__(self, _):
        return True

myfalse = Any()

match myfalse:
    case False:
        # Doesn't match, even though myfalse == False
        assert False

Variable names

We can replace a literal with a variable name, to capture the value of the match expression.

match command:
    case 'start':
        ...
    case 'stop':
        ...
    case unknown:
        # New variable 'unknown' is assigned the value of command

The 'default' case pattern _ is just a special case variable name which binds no name.

Beware the common error of using "constants" as the case pattern:

NOT_FOUND = 404

match error:
    case NOT_FOUND: # bad
        handle_404()

The above case is intended to test for error == NOT_FOUND, but instead assigns the variable NOT_FOUND = error. The best defense is to always include a default catch-all case at the end, which causes the above NOT_FOUND case to produce a SyntaxError:

NOT_FOUND = 404

match error:
    case NOT_FOUND:
        handle_404()
    case _:
        pass

SyntaxError: name capture 'NOT_FOUND' makes remaining patterns unreachable

To use a 'constant' in a case pattern like this, qualify it with a dotted name, such as by using an enum.Enum:

match error
    case errors.NOT_FOUND:
        # correctly matches

Sequences

Using a list-like or tuple-like syntax, matches must have the right number of items. Like Python's existing iterable unpacking feature. Use * to match the rest of a sequence. Included variable names are set if a case matches by all other criteria.

match command:
    case ('start', name):
        # New variable name=command[1]
    case ('stop', name):
        # New variable name=command[1]
    case ('stop', name, delay):
        # New variables name=command[1], delay=command[2]
    case ('stop', name, delay, *extra):
        # New variables name=command[1], delay=command[2] & extra=command[3:]
    case _:
        raise BadCommand(command)

Mappings

Using a dict-like syntax. The match expression must must contain a corresponding mapping, and can contain other keys, too. Use ** to match the rest of a mapping.

match config:
    case {'host': hostname}:
        # 'config' must contain key 'host'. New variable hostname=config['host']
    case {'port': portnumber}:
        # 'config' must contain key 'port'. New variable portnumber=config['port']
        # Remember we only use the first matching case.
        # If 'config' contains 'host', then this 'port' case will not match.
    case {'scheme': scheme, **extras}:
        # new variables 'scheme' and 'extras' are assigned.

Case patterns may contain more than one key-value pair. The match expression must contain all of them to match.

    case {
        'host': hostname,
        'port': portnumber,
    }:
        ...

Objects and their attributes

Using class syntax, the value must match an isinstance check with the given class:

match event:
    case Click():
        # handle click
        ...
    case KeyPress():
        # handle key press
        ...

Beware the common error of omitting the parentheses:

match myval:
    case Click: # bad
        # handle clicks

The above case is intended to test for isinstance(myval, Click), but instead creates a new var, Click = myval. The best defence against this error is to always include a default catch-all at the end, which makes the Click catch-all produce an error by making subsequent patterns unreachable.

Attribute values for the class can be given, which must also match.

match event:
    case KeyPress(key_name='q', release=False):
        game.quit()
    case KeyPress():
        handle_keypress(event)

Values can also be passed as positional args to the class-like case syntax:

    case KeyPress('q', True)
        ...

If the class is a namedtuple or dataclass, then positional args to a class-like case pattern can automatically be handled using the unambiguous ordering of its attributes:

@dataclass
class Dog:
    name: str
    color: str

d = Dog('dash', 'golden')

match d:
    case Dog('dash', 'golden'):
        # matches

But for regular classes, the ordering of the class attributes is ambiguous. To fix this, add a __match_args__ attribute on the class, a tuple which specifies which class attributes, in which order, can be specified in a case pattern:

class KeyPress:
    __match_args__ = ('key_name', 'release')

event = KeyPress(key_name='q', release=False)

match event:
    case KeyPress('q', False):
        # matches!

As you might expect, the literal positional args can be replaced with variable names to capture attribute values instead:

match event:
    case KeyPress(k, r): # names unimportant, order matters
        handle_keypress(k, r)

Positional sub-patterns behave slightly differently for builtins bool, bytearray, bytes, dict, float, frozenset, int, list, set, str, and tuple. A positional value is matched by equality against the match expression itself, rather than an attribute on it:

match 123:
    case int(123):
        # matches
    case int(123.0):
        # would also match if it wasn't shadowed

Similarly, a positional variable is assigned the value of the match expression itself, not an attribute on that value:

match 123:
   case int(value):
        ...

assert value == 123

The values passed as keyword or positional args to class-like case patterns can be more than just literals or variable names. In fact they can use any of the listed pattern types. For example, they could be a nested instance of this class-like syntax:

class Location:
    def __init__(self, x, y):
        self.x = x
        self.y = y

class Car:
    def __init__(self, location):
        self.location = location

mycar = Car(Location(11, 22))

match mycar:
    case Car(location=(Location(x=x, y=y))):
        # matches, and captures 'x' and 'y'

assert x == 11
assert y == 22

Combine patterns using `|`

To match either one pattern or another:

    case 1 | True | 'true' | 'on' | 'yes':
        # matches any of those values

Capture sub-patterns using `as`

We've seen how we can either match against a value, or capture the value using a variable name. We can do both using as:

    case 'a' | 'b' as ab:
        # matches either value, captures what the value actually was

This might not be much use when capturing the whole match expression like that. If the match expression is just a variable, then we could instead simply refer to that variable. But using as can be useful when the match expression is lengthy or has side-effects:

match events.get_next():
    case KeyDown() as key_event:
        ...

or to capture just a component of the whole expression. Contrived example:

    case ('a' | 'b' as ab, 'c'):
        # matchs ['a', 'c'] or ['b', 'c'], and captures the first letter in 'ab'

An `if` guard clause

Add arbitrary conditions to the match:

    case int(i) if i < 100:
        # matches integers less than 100

Or, alternatively:

    case int() as i if i < 100:
        # matches integers less than 100

Complications

This feature seems rife with complexity. The flexible syntax of case patterns forms a new mini-language, embedded within Python. It has many similarities to Python, but also many initially unintuitive differences.

For example, a class-like case pattern such as case Click():. Anywhere else in the language, the expression like Click(...) would create an instance of the Click class. In a case statement, it instead is doing things like isinstance and hasattr checks.

Similarly, including variable names doesn't return the variable value as in ordinary Python. Instead it binds a value as that name. This is the source of the annoying gotcha described above, that bare "constants" like NOT_FOUND behave very unexpectedly when used as case expressions.

There are a few places in real-world code where structured pattern matching will produce nicer code than the equivalent using nested elifs. But equally, there are a lot of places where the elifs are a more natural match. Developers now get to choose which they're going to use, and then later disagree with each other about it, or simply change their mind, and end up converting code from one to the other.

If this was a simple feature, with low overheads, then I'd forgive its inclusion in the language, accepting the costs in return for the marginal and unevenly distributed benefits.

But it's really not simple. In addition to Python programmers all having to do an exercise like this post just to add it to their mental toolbox, it needs maintenance effort, not just in CPython but in other implementations too, and needs handling by tools such as syntax highlighters, type checkers. It really doesn't seem like a net win to me, unless you're writing way more parsers than the average programmer, which no doubt the champions of this feature are.

TIL: Format Python Snippets with Black.

Jonathan Hartley — Tue, 09 Jun 2020 19:36:58 GMT

Black, the opinionated Python code formatter, can easily be invoked from your editor to reformat a whole file. For example, from Vim:

" Black(Python) format the whole file
nnoremap <leader>b :1,$!black -q -<CR>

But often you'd like to reformat just a section of the file, while leaving everything else intact. In principle, it's easy to tell Vim to just send the current visual selection:

" Black(Python) format the visual selection
xnoremap <Leader>b :!black -q -<CR>

(Note that both the above Vim configuration snippets map the same key sequence -- leader (commonly comma) followed by lower case b. These can be defined simultaneously, because the second one uses 'xnoremap', meaning it is used only while a visual selection exists, while the first uses 'nnoremap', so is used all other times.)

But if the given code starts with an indent on the first line, for example if it comes from lines in the middle of a function, then this won't work. Black parses the given code into a Python abstract syntax tree (AST), and a leading indent is a syntax error - it's just not valid Python.

I filed a hopeful issue with Black, suggesting they could handle this case, but it was a long shot and hasn't gained much enthusiasm.

So, I present a tiny Python3 wrapper, enblacken, which:

Unindents the given code such that the first line has no indent.
Passes the result to Black.
Reindents Black's output, by the same amount as the original unindent.

See enblacken on github

LXD for Development Environments.

Jonathan Hartley — Mon, 20 Apr 2020 18:57:47 GMT

@hjwp asks:

I would be interested in seeing some example lxd config files, bash command history when creating, etc?

Here goes then.

I have one LXD container running for each nontrivial development project I'm working on.

$ lxc ls
|    NAME     |  STATE  |        IPV4         | IPV6 |   TYPE    | SNAPSHOTS |
| devicegw    | RUNNING | 10.44.99.228 (eth0) |      | CONTAINER | 0         |
| ident       | RUNNING | 10.44.99.4 (eth0)   |      | CONTAINER | 0         |
| revs        | RUNNING | 10.44.99.151 (eth0) |      | CONTAINER | 0         |
| siab        | RUNNING | 10.44.99.128 (eth0) |      | CONTAINER | 0         |
| tartley-com | RUNNING | 10.44.99.161 (eth0) |      | CONTAINER | 0         |

Out of the gate we see one source of confusion. "LXD", the daemon, is a newer project that builds on top of "LXC" the containers. However the user interface to all the new LXD-goodness is through a command-line called "lxc", which replaces the older command line tool called "lxd". :-/

To create a new one:

$ time lxc launch ubuntu:16.04 -p default -p jhartley demo
Creating demo
Starting demo
real    0m9.593s

Once created, they take about 3 seconds to stop and 0.5 seconds to start.

Those "-p" options cause the container to use two profiles. They are:

The default profile, which I've never touched. It's just doing whatever it always does.
The jhartley profile, I created in a one-off step by running a Bash script derived from instructions one of my colleagues passed around. I'll describe it at the end.

Once a new container is up, we can execute commands directly on it:

$ lxc exec demo hostname
demo
$ lxc exec demo whoami
root

Or SSH to them using their IP address:

jhartley@t460 $ lxc ls demo
| NAME |  STATE  |        IPV4         | IPV6 |   TYPE    | SNAPSHOTS |
| demo | RUNNING | 10.44.99.162 (eth0) |      | CONTAINER | 0         |
jhartley@t460 $ ssh 10.44.99.162
...
Warning: Permanently added '10.44.99.162' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 16.04.6 LTS (GNU/Linux 5.4.0-25-generic x86_64)
jhartley@demo $

Better than using IP addresses, you can run a DNS server to recognize {containername}.lxd hostnames. (This part is from here.)

Find your lxd bridge IPv4 address

lxc network show lxdbr0

Create file /etc/systemd/network/lxd.network:

[Match]
Name=lxdbr0

[Network]
Address=IPADDR/24
DNS=IPADDR
Domains=~lxd

Where IPADDR is the lxdbr0 IPv4 address.

sudo systemctl enable systemd-networkd
sudo reboot now

Then:

jhartley@t460 $ ssh demo.lxd
jhartley@demo $ # \o/

One nice thing is that DNS works both from the host and on the containers, so your services can be configured by default to talk to each other at SERVICE1.lxd, SERVICE2.lxd. Then running them in containers on your host they would just find each other. We don't actually do this, but it seems trivially easy to do. I should ask why we don't.

In practice I wrap up the ssh command with my accumulated foibles:

jhartley@demo $ type -a lssh
lssh is a function
lssh ()
{
    TERM=xterm-color ssh -A -t "$1.lxd" -- "cd $PWD && exec $SHELL -l";
}

I forget why -A and -t were required. The rest is mostly just to start the shell on the container in the same directory as I was in on the host. There is probably a simpler way.

The booooooring bits:

When we started the container, we mentioned a one-off setup script.

The script does a few things:

Creates a new key pair specifically to SSH to the container.
Creates the custom jhartley profile, which causes all containers started with it to:
Create a new user on the container with user and group ID mapped to those of my user on the host, presumably so that file permissions work for...
Mount my $HOME directory on the container. Might not always be what you want, but works for me right now.
Doubtless due to my own misunderstanding somewhere, in order to get working IPv4 connections to my containers, I had to disable IPv6 connections to them.

Vonnegut on software development teams.

Jonathan Hartley — Mon, 01 Jul 2019 02:28:04 GMT

So here's a thing. Spotted this in some of Kurt Vonnegut's personal correspondence, talking about an instructor of his named Slotkin:

What Slotkin said was this: no man who achieved greatness in the arts operated by himself; he was top man in a group of like-minded individuals. This works out fine for the cubists, and Slotkin had plenty of good evidence for its applying to Goethe, Thoreau, Hemingway, and just about anybody you care to name.

If this isn't 100% true, it's true enough to be interesting—and maybe helpful.

The school gives a man, Slotkin said, the fantastic amount of guts it takes to add to culture. It gives him morale, esprit de corps, the resources of many brains, and—maybe most important—one-sidedness with assurance.

Reminds me powerfully of my growing impressions of the environment a person needs to be in in order to do great things in software. There is no doubt some effectiveness in assembling great individuals to create a great team.

But in my personal experience, there is a whole lot more value in creating a great team by instilling the right values, and then watching the members visibly level each other up, producing a succession of great individuals, and only subsequently attracting more of the same.