Augmented assignment expressions

The language grammar would be adjusted to allow augmented assignments to appear as expressions, where the result of the augmented assignment expression is the same post-calculation reference as is being bound to the given target.

For example:

>>> n = 0
>>> n += 5
5
>>> n -= 2
3
>>> n *= 3
9
>>> n
9

For mutable targets, this means the result is always just the original object:

>>> seq = []
>>> seq_id = id(seq)
>>> seq += range(3)
[0, 1, 2]
>>> seq_id == id(seq)
True

Augmented assignments to attributes and container subscripts will be permitted, with the result being the post-calculation reference being bound to the target, just as it is for simple name targets:

def increment(self, step=1):
    return self._value += step

In these cases, __getitem__ and __getattribute__ will not be called after the assignment has already taken place (they will only be called as needed to evaluate the in-place operation).

Adding an inline assignment operator

Given only the addition of augmented assignment expressions, it would be possible to abuse a symbol like |= as a general purpose assignment operator by defining a Target wrapper type that worked as follows:

>>> class Target:
...     def __init__(self, value):
...         self.value = value
...     def __or__(self, other):
...         return Target(other)
...
>>> x = Target(10)
>>> x.value
10
>>> x |= 42
<__main__.Target object at 0x7f608caa8048>
>>> x.value
42

This is similar to the way that storing a single reference in a list was long used as a workaround for the lack of a nonlocal keyword, and can still be used today (in combination with operator.itemsetter) to work around the lack of expression level assignments.

Rather than requiring such workarounds, this PEP instead proposes that PEP 572’s “NAME := EXPR” syntax be adopted as a new inline assignment expression that uses the augmented assignment scoping rules described below.

This cleanly handles cases where only the new value is of interest, and the previously bound value (if any) can just be discarded completely.

Note that for both simple names and complex assignment targets, the inline assignment operator does not read the previous reference before assigning the new one. However, when used at function scope (either directly or inside a scoped expression), it does not implicitly define a new local variable, and will instead raise TargetNameError (as described for augmented assignments below).

Assignment operator precedence

To preserve the existing semantics of augmented assignment statements, inline assignment operators will be defined as being of lower precedence than all other operators, include the comma pseudo-operator. This ensures that when used as a top level expression the entire right hand side of the expression is still interpreted as the value to be processed (even when that value is a tuple without parentheses).

The difference this introduces relative to PEP 572 is that where (n := first, second) sets n = first in PEP 572, in this PEP it would set n = (first, second), and getting the first meaning would require an extra set of parentheses (((n := first), second)).

PEP 572 quite reasonably notes that this results in ambiguity when assignment expressions are used as function call arguments. This PEP resolves that concern a different way by requiring that assignment expressions be parenthesised when used as arguments to a function call (unless they’re the sole argument).

This is a more relaxed version of the restriction placed on generator expressions (which always require parentheses, except when they’re the sole argument to a function call).

Augmented assignment to names in block scopes

No target name binding changes are proposed for augmented assignments at module or class scope (this also includes code executed using “exec” or “eval”). These will continue to implicitly declare a new local variable as the binding target as they do today, and (if necessary) will be able to resolve the name from an outer scope before binding it locally.

At function scope, augmented assignments will be changed to require that there be either a preceding name binding or variable declaration to explicitly establish the target name as being local to the function, or else an explicit global or nonlocal declaration. TargetNameError, a new SyntaxError subclass, will be raised at compile time if no such binding or declaration is present.

For example, the following code would compile and run as it does today:

x = 0
x += 1 # Sets global "x" to 1

class C:
    x += 1 # Sets local "x" to 2, leaves global "x" alone

def local_target():
    x = 0
    x += 1 # Sets local "x" to 1, leaves global "x" alone

def global_target():
    global x
    x += 1 # Increments global "x" each time this runs

def nonlocal_target():
    x = 0
    def g():
        nonlocal x
        x += 1 # Increments "x" in outer scope each time this runs
        return x
    return g

The follow examples would all still compile and then raise an error at runtime as they do today:

n += 1 # Raises NameError at runtime

class C:
    n += 1 # Raises NameError at runtime

def missing_global():
    global n
    n += 1 # Raises NameError at runtime

def delayed_nonlocal_initialisation():
    def f():
        nonlocal n
        n += 1
    f() # Raises NameError at runtime
    n = 0

def skipped_conditional_initialisation():
    if False:
        n = 0
    n += 1 # Raises UnboundLocalError at runtime

def local_declaration_without_initial_assignment():
    n: typing.Any
    n += 1 # Raises UnboundLocalError at runtime

Whereas the following would raise a compile time DeprecationWarning initially, and eventually change to report a compile time TargetNameError:

def missing_target():
    x += 1 # Compile time TargetNameError due to ambiguous target scope
           # Is there a missing initialisation of "x" here? Or a missing
           # global or nonlocal declaration?

As a conservative implementation approach, the compile time function name resolution change would be introduced as a DeprecationWarning in Python 3.8, and then converted to TargetNameError in Python 3.9. This avoids potential problems in cases where an unused function would currently raise UnboundLocalError if it was ever actually called, but the code is actually unused - converting that latent runtime defect to a compile time error qualifies as a backwards incompatible change that requires a deprecation period.

When augmented assignments are used as expressions in function scope (rather than as standalone statements), there aren’t any backwards compatibility concerns, so the compile time name binding checks would be enforced immediately in Python 3.8.

Similarly, the new inline assignment expressions would always require explicit predeclaration of their target scope when used as part of a function, at least for Python 3.8. (See the design discussion section for notes on potentially revisiting that restriction in the future).

Augmented assignment to names in scoped expressions

Scoped expressions is a new collective term being proposed for expressions that introduce a new nested scope of execution, either as an intrinsic part of their operation (lambda expressions, generator expressions), or else as a way of hiding name binding operations from the containing scope (container comprehensions).

Unlike regular functions, these scoped expressions can’t include explicit global or nonlocal declarations to rebind names directly in an outer scope.

Instead, their name binding semantics for augmented assignment expressions would be defined as follows:

• augmented assignment targets used in scoped expressions are expected to either be already bound in the containing block scope, or else have their scope explicitly declared in the containing block scope. If no suitable name binding or declaration can be found in that scope, then TargetNameError will be raised at compile time (rather than creating a new binding within the scoped expression).
• if the containing block scope is a function scope, and the target name is explicitly declared as global or nonlocal, then it will be use the same scope declaration in the body of the scoped expression
• if the containing block scope is a function scope, and the target name is a local variable in that function, then it will be implicitly declared as nonlocal in the body of the scoped expression
• if the containing block scope is a class scope, than TargetNameError will always be raised, with a dedicated message indicating that combining class scopes with augmented assignments in scoped expressions is not currently permitted.
• if a name is declared as a formal parameter (lambda expressions), or as an iteration variable (generator expressions, comprehensions), then that name is considered local to that scoped expression, and attempting to use it as the target of an augmented assignment operation in that scope, or any nested scoped expression, will raise TargetNameError (this is a restriction that could potentially be lifted later, but is being proposed for now to simplify the initial set of compile time and runtime semantics that needs to be covered in the language reference and handled by the compiler and interpreter)

For example, the following code would work as shown:

>>> global_target = 0
>>> incr_global_target = lambda: global_target += 1
>>> incr_global_target()
1
>>> incr_global_target()
2
>>> global_target
2
>>> def cumulative_sums(data, start=0)
...    total = start
...    yield from (total += value for value in data)
...    return total
...
>>> print(list(cumulative_sums(range(5))))
[0, 1, 3, 6, 10]

While the following examples would all raise TargetNameError:

class C:
    cls_target = 0
    incr_cls_target = lambda: cls_target += 1 # Error due to class scope

def missing_target():
    incr_x = lambda: x += 1 # Error due to missing target "x"

def late_target():
    incr_x = lambda: x += 1 # Error due to "x" being declared after use
    x = 1

lambda arg: arg += 1 # Error due to attempt to target formal parameter

[x += 1 for x in data] # Error due to attempt to target iteration variable

As augmented assignments currently can’t appear inside scoped expressions, the above compile time name resolution exceptions would be included as part of the initial implementation rather than needing to be phased in as a potentially backwards incompatible change.

Allowing complex assignment targets

The initial drafts of this PEP kept PEP 572’s restriction to single name targets when augmented assignments were used as expressions, allowing attribute and subscript targets solely for the statement form.

However, enforcing that required varying the permitted targets based on whether or not the augmented assignment was a top level expression or not, as well as explaining why n += 1, (n += 1), and self.n += 1 were all legal, but (self.n += 1) was prohibited, so the proposal was simplified to allow all existing augmented assignment targets for the expression form as well.

Since this PEP defines TARGET := EXPR as a variant on augmented assignment, that also gained support for assignment and subscript targets.

Augmented assignment or name binding only?

PEP 572 makes a reasonable case that the potential use cases for inline augmented assignment are notably weaker than those for inline assignment in general, so it’s acceptable to require that they be spelled as x := x + 1, bypassing any in-place augmented assignment methods.

While this is at least arguably true for the builtin types (where potential counterexamples would probably need to focus on set manipulation use cases that the PEP author doesn’t personally have), it would also rule out more memory intensive use cases like manipulation of NumPy arrays, where the data copying involved in out-of-place operations can make them impractical as alternatives to their in-place counterparts.

That said, this PEP mainly exists because the PEP author found the inline assignment proposal much easier to grasp as “It’s like +=, only skipping the addition step”, and also liked the way that that framing provides an actual semantic difference between NAME = EXPR and NAME := EXPR at function scope.

That difference in target scoping behaviour means that the NAME := EXPR syntax would be expected to have two primary use cases:

• as a way of allowing assignments to be embedded as an expression in an if or while statement, or as part of a scoped expression
• as a way of requesting a compile time check that the target name be previously declared or bound in the current function scope

At module or class scope, NAME = EXPR and NAME := EXPR would be semantically equivalent due to the compiler’s lack of visibility into the set of names that will be resolvable at runtime, but code linters and static type checkers would be encouraged to enforce the same “declaration or assignment required before use” behaviour for NAME := EXPR as the compiler would enforce at function scope.

Postponing a decision on expression level target declarations

At least for Python 3.8, usage of inline assignments (whether augmented or not) at function scope would always require a preceding name binding or scope declaration to avoid getting TargetNameError, even when used outside a scoped expression.

The intent behind this requirement is to clearly separate the following two language design questions:

1. Can an expression rebind a name in the current scope?
2. Can an expression declare a new name in the current scope?

For module global scopes, the answer to both of those questions is unequivocally “Yes”, because it’s a language level guarantee that mutating the globals() dict will immediately impact the runtime module scope, and global NAME declarations inside a function can have the same effect (as can importing the currently executing module and modifying its attributes).

For class scopes, the answer to both questions is also “Yes” in practice, although less unequivocally so, since the semantics of locals() are currently formally unspecified. However, if the current behaviour of locals() at class scope is taken as normative (as PEP 558 proposes), then this is essentially the same scenario as manipulating the module globals, just using locals() instead.

For function scopes, however, the current answers to these two questions are respectively “Yes” and “No”. Expression level rebinding of function locals is already possible thanks to lexically nested scopes and explicit nonlocal NAME expressions. While this PEP will likely make expression level rebinding more common than it is today, it isn’t a fundamentally new concept for the language.

By contrast, declaring a new function local variable is currently a statement level action, involving one of:

• an assignment statement (NAME = EXPR, OTHER_TARGET = NAME = EXPR, etc)
• a variable declaration (NAME : EXPR)
• a nested function definition
• a nested class definition
• a for loop
• a with statement
• an except clause (with limited scope of access)

The historical trend for the language has actually been to remove support for expression level declarations of function local names, first with the introduction of “fast locals” semantics (which made the introduction of names via locals() unsupported for function scopes), and again with the hiding of comprehension iteration variables in Python 3.0.

Now, it may be that in Python 3.9, we decide to revisit this question based on our experience with expression level name binding in Python 3.8, and decide that we really do want expression level function local variable declarations as well, and that we want NAME := EXPR to be the way we spell that (rather than, for example, spelling inline declarations more explicitly as NAME := EXPR given NAME, which would permit them to carry type annotations, and also permit them to declare new local variables in scoped expressions, rather than having to pollute the namespace in their containing scope).

But the proposal in this PEP is that we explicitly give ourselves a full release to decide how much we want that feature, and exactly where we find its absence irritating. Python has survived happily without expression level name bindings or declarations for decades, so we can afford to give ourselves a couple of years to decide if we really want both of those, or if expression level bindings are sufficient.

Ignoring scoped expressions when determining augmented assignment targets

When discussing possible binding semantics for PEP 572’s assignment expressions, Tim Peters made a plausible case [1], [2], [3] for assignment expressions targeting the containing block scope, essentially ignoring any intervening scoped expressions.

This approach allows use cases like cumulative sums, or extracting the final value from a generator expression to be written in a relatively straightforward way:

total = 0
partial_sums = [total := total + value for value in data]

factor = 1
while any(n % (factor := p) == 0 for p in small_primes):
    n //= factor

Guido also expressed his approval for this general approach [4].

The proposal in this PEP differs from Tim’s original proposal in three main areas:

• it applies the proposal to all augmented assignment operators, not just a single new name binding operator
• as far as is practical, it extends the augmented assignment requirement that the name already be defined to the new name binding operator (raising TargetNameError rather than implicitly declaring new local variables at function scope)
• it includes lambda expressions in the set of scopes that get ignored for target name binding purposes, making this transparency to assignments common to all of the scoped expressions rather than being specific to comprehensions and generator expressions

With scoped expressions being ignored when calculating binding targets, it’s once again difficult to detect the scoping difference between the outermost iterable expressions in generator expressions and comprehensions (you have to mess about with either class scopes or attempting to rebind iteration Variables to detect it), so there’s also no need to tinker with that.

Treating inline assignment as an augmented assignment variant

One of the challenges with PEP 572 is the fact that NAME = EXPR and NAME := EXPR are entirely semantically equivalent at every scope. This makes the two forms hard to teach, since there’s no inherent nudge towards choosing one over the other at the statement level, so you end up having to resort to “NAME = EXPR is preferred because it’s been around longer” (and PEP 572 proposes to enforce that historical idiosyncrasy at the compiler level).

That semantic equivalence is difficult to avoid at module and class scope while still having if NAME := EXPR: and while NAME := EXPR: work sensibly, but at function scope the compiler’s comprehensive view of all local names makes it possible to require that the name be assigned or declared before use, providing a reasonable incentive to continue to default to using the NAME = EXPR form when possible, while also enabling the use of the NAME := EXPR as a kind of simple compile time assertion (i.e. explicitly indicating that the targeted name has already been bound or declared and hence should already be known to the compiler).

If Guido were to declare that support for inline declarations was a hard design requirement, then this PEP would be updated to propose that EXPR given NAME also be introduced as a way to support inline name declarations after arbitrary expressions (this would allow the inline name declarations to be deferred until the end of a complex expression rather than needing to be embedded in the middle of it, and PEP 8 would gain a recommendation encouraging that style).

Disallowing augmented assignments in class level scoped expressions

While modern classes do define an implicit closure that’s visible to method implementations (in order to make __class__ available for use in zero-arg super() calls), there’s no way for user level code to explicitly add additional names to that scope.

Meanwhile, attributes defined in a class body are ignored for the purpose of defining a method’s lexical closure, which means adding them there wouldn’t work at an implementation level.

Rather than trying to resolve that inherent ambiguity, this PEP simply prohibits such usage, and requires that any affected logic be written somewhere other than directly inline in the class body (e.g. in a separate helper function).

Comparison operators vs assignment operators

The OP= construct as an expression currently indicates a comparison operation:

x == y # Equals
x >= y # Greater-than-or-equal-to
x <= y # Less-than-or-equal-to

Both this PEP and PEP 572 propose adding at least one operator that’s somewhat similar in appearance, but defines an assignment instead:

x := y # Becomes

This PEP then goes much further and allows all 13 augmented assignment symbols to be uses as binary operators:

x +=  y # In-place add
x -=  y # In-place minus
x *=  y # In-place multiply
x @=  y # In-place matrix multiply
x /=  y # In-place division
x //= y # In-place int division
x %=  y # In-place mod
x &=  y # In-place bitwise and
x |=  y # In-place bitwise or
x ^=  y # In-place bitwise xor
x <<= y # In-place left shift
x >>= y # In-place right shift
x **= y # In-place power

Of those additional binary operators, the most questionable would be the bitshift assignment operators, since they’re each only one doubled character away from one of the inclusive ordered comparison operators.

Simplifying retry loops

There are currently a few different options for writing retry loops, including:

# Post-decrementing a counter
remaining_attempts = MAX_ATTEMPTS
while remaining_attempts:
    remaining_attempts -= 1
    try:
        result = attempt_operation()
    except Exception as exc:
        continue # Failed, so try again
    log.debug(f"Succeeded after {attempts} attempts")
    break # Success!
else:
    raise OperationFailed(f"Failed after {MAX_ATTEMPTS} attempts") from exc

# Loop-and-a-half with a pre-incremented counter
attempt = 0
while True:
    attempts += 1
    if attempts > MAX_ATTEMPTS:
        raise OperationFailed(f"Failed after {MAX_ATTEMPTS} attempts") from exc
    try:
        result = attempt_operation()
    except Exception as exc:
        continue # Failed, so try again
    log.debug(f"Succeeded after {attempts} attempts")
    break # Success!

Each of the available options hides some aspect of the intended loop structure inside the loop body, whether that’s the state modification, the exit condition, or both.

The proposal in this PEP allows both the state modification and the exit condition to be included directly in the loop header:

attempt = 0
while (attempt += 1) <= MAX_ATTEMPTS:
    try:
        result = attempt_operation()
    except Exception as exc:
        continue # Failed, so try again
    log.debug(f"Succeeded after {attempts} attempts")
    break # Success!
else:
    raise OperationFailed(f"Failed after {MAX_ATTEMPTS} attempts") from exc

Simplifying if-elif chains

if-elif chains that need to rebind the checked condition currently need to be written using nested if-else statements:

m = pattern.match(data)
if m:
    ...
else:
    m = other_pattern.match(data)
    if m:
        ...
    else:
        m = yet_another_pattern.match(data)
        if m:
            ...
        else:
            ...

As with PEP 572, this PEP allows the else/if portions of that chain to be condensed, making their consistent and mutually exclusive structure more readily apparent:

m = pattern.match(data)
if m:
    ...
elif m := other_pattern.match(data):
    ...
elif m := yet_another_pattern.match(data):
    ...
else:
    ...

Unlike PEP 572, this PEP requires that the assignment target be explicitly indicated as local before the first use as a := target, either by binding it to a value (as shown above), or else by including an appropriate explicit type declaration:

m: typing.re.Match
if m := pattern.match(data):
    ...
elif m := other_pattern.match(data):
    ...
elif m := yet_another_pattern.match(data):
    ...
else:
    ...

Capturing intermediate values from comprehensions

The proposal in this PEP makes it straightforward to capture and reuse intermediate values in comprehensions and generator expressions by exporting them to the containing block scope:

factor: int
while any(n % (factor := p) == 0 for p in small_primes):
    n //= factor

total = 0
partial_sums = [total += value for value in data]

Allowing lambda expressions to act more like re-usable code thunks

This PEP allows the classic closure usage example:

def make_counter(start=0):
    x = start
    def counter(step=1):
        nonlocal x
        x += step
        return x
    return counter

To be abbreviated as:

def make_counter(start=0):
    x = start
    return lambda step=1: x += step

While the latter form is still a conceptually dense piece of code, it can be reasonably argued that the lack of boilerplate (where the “def”, “nonlocal”, and “return” keywords and two additional repetitions of the “x” variable name have been replaced with the “lambda” keyword) may make it easier to read in practice.