comprehension assignment python

Python Enhancement Proposals

Python »
PEP Index »

PEP 572 – Assignment Expressions

The importance of real code, exceptional cases, scope of the target, relative precedence of :=, change to evaluation order, differences between assignment expressions and assignment statements, specification changes during implementation, _pydecimal.py, datetime.py, sysconfig.py, simplifying list comprehensions, capturing condition values, changing the scope rules for comprehensions, alternative spellings, special-casing conditional statements, special-casing comprehensions, lowering operator precedence, allowing commas to the right, always requiring parentheses, why not just turn existing assignment into an expression, with assignment expressions, why bother with assignment statements, why not use a sublocal scope and prevent namespace pollution, style guide recommendations, acknowledgements, a numeric example, appendix b: rough code translations for comprehensions, appendix c: no changes to scope semantics.

This is a proposal for creating a way to assign to variables within an expression using the notation NAME := expr .

As part of this change, there is also an update to dictionary comprehension evaluation order to ensure key expressions are executed before value expressions (allowing the key to be bound to a name and then re-used as part of calculating the corresponding value).

During discussion of this PEP, the operator became informally known as “the walrus operator”. The construct’s formal name is “Assignment Expressions” (as per the PEP title), but they may also be referred to as “Named Expressions” (e.g. the CPython reference implementation uses that name internally).

Naming the result of an expression is an important part of programming, allowing a descriptive name to be used in place of a longer expression, and permitting reuse. Currently, this feature is available only in statement form, making it unavailable in list comprehensions and other expression contexts.

Additionally, naming sub-parts of a large expression can assist an interactive debugger, providing useful display hooks and partial results. Without a way to capture sub-expressions inline, this would require refactoring of the original code; with assignment expressions, this merely requires the insertion of a few name := markers. Removing the need to refactor reduces the likelihood that the code be inadvertently changed as part of debugging (a common cause of Heisenbugs), and is easier to dictate to another programmer.

During the development of this PEP many people (supporters and critics both) have had a tendency to focus on toy examples on the one hand, and on overly complex examples on the other.

The danger of toy examples is twofold: they are often too abstract to make anyone go “ooh, that’s compelling”, and they are easily refuted with “I would never write it that way anyway”.

The danger of overly complex examples is that they provide a convenient strawman for critics of the proposal to shoot down (“that’s obfuscated”).

Yet there is some use for both extremely simple and extremely complex examples: they are helpful to clarify the intended semantics. Therefore, there will be some of each below.

However, in order to be compelling , examples should be rooted in real code, i.e. code that was written without any thought of this PEP, as part of a useful application, however large or small. Tim Peters has been extremely helpful by going over his own personal code repository and picking examples of code he had written that (in his view) would have been clearer if rewritten with (sparing) use of assignment expressions. His conclusion: the current proposal would have allowed a modest but clear improvement in quite a few bits of code.

Another use of real code is to observe indirectly how much value programmers place on compactness. Guido van Rossum searched through a Dropbox code base and discovered some evidence that programmers value writing fewer lines over shorter lines.

Case in point: Guido found several examples where a programmer repeated a subexpression, slowing down the program, in order to save one line of code, e.g. instead of writing:

they would write:

Another example illustrates that programmers sometimes do more work to save an extra level of indentation:

This code tries to match pattern2 even if pattern1 has a match (in which case the match on pattern2 is never used). The more efficient rewrite would have been:

Syntax and semantics

In most contexts where arbitrary Python expressions can be used, a named expression can appear. This is of the form NAME := expr where expr is any valid Python expression other than an unparenthesized tuple, and NAME is an identifier.

The value of such a named expression is the same as the incorporated expression, with the additional side-effect that the target is assigned that value:

There are a few places where assignment expressions are not allowed, in order to avoid ambiguities or user confusion:

This rule is included to simplify the choice for the user between an assignment statement and an assignment expression – there is no syntactic position where both are valid.

Again, this rule is included to avoid two visually similar ways of saying the same thing.

This rule is included to disallow excessively confusing code, and because parsing keyword arguments is complex enough already.

This rule is included to discourage side effects in a position whose exact semantics are already confusing to many users (cf. the common style recommendation against mutable default values), and also to echo the similar prohibition in calls (the previous bullet).

The reasoning here is similar to the two previous cases; this ungrouped assortment of symbols and operators composed of : and = is hard to read correctly.

This allows lambda to always bind less tightly than := ; having a name binding at the top level inside a lambda function is unlikely to be of value, as there is no way to make use of it. In cases where the name will be used more than once, the expression is likely to need parenthesizing anyway, so this prohibition will rarely affect code.

This shows that what looks like an assignment operator in an f-string is not always an assignment operator. The f-string parser uses : to indicate formatting options. To preserve backwards compatibility, assignment operator usage inside of f-strings must be parenthesized. As noted above, this usage of the assignment operator is not recommended.

An assignment expression does not introduce a new scope. In most cases the scope in which the target will be bound is self-explanatory: it is the current scope. If this scope contains a nonlocal or global declaration for the target, the assignment expression honors that. A lambda (being an explicit, if anonymous, function definition) counts as a scope for this purpose.

There is one special case: an assignment expression occurring in a list, set or dict comprehension or in a generator expression (below collectively referred to as “comprehensions”) binds the target in the containing scope, honoring a nonlocal or global declaration for the target in that scope, if one exists. For the purpose of this rule the containing scope of a nested comprehension is the scope that contains the outermost comprehension. A lambda counts as a containing scope.

The motivation for this special case is twofold. First, it allows us to conveniently capture a “witness” for an any() expression, or a counterexample for all() , for example:

Second, it allows a compact way of updating mutable state from a comprehension, for example:

However, an assignment expression target name cannot be the same as a for -target name appearing in any comprehension containing the assignment expression. The latter names are local to the comprehension in which they appear, so it would be contradictory for a contained use of the same name to refer to the scope containing the outermost comprehension instead.

For example, [i := i+1 for i in range(5)] is invalid: the for i part establishes that i is local to the comprehension, but the i := part insists that i is not local to the comprehension. The same reason makes these examples invalid too:

While it’s technically possible to assign consistent semantics to these cases, it’s difficult to determine whether those semantics actually make sense in the absence of real use cases. Accordingly, the reference implementation [1] will ensure that such cases raise SyntaxError , rather than executing with implementation defined behaviour.

This restriction applies even if the assignment expression is never executed:

For the comprehension body (the part before the first “for” keyword) and the filter expression (the part after “if” and before any nested “for”), this restriction applies solely to target names that are also used as iteration variables in the comprehension. Lambda expressions appearing in these positions introduce a new explicit function scope, and hence may use assignment expressions with no additional restrictions.

Due to design constraints in the reference implementation (the symbol table analyser cannot easily detect when names are re-used between the leftmost comprehension iterable expression and the rest of the comprehension), named expressions are disallowed entirely as part of comprehension iterable expressions (the part after each “in”, and before any subsequent “if” or “for” keyword):

A further exception applies when an assignment expression occurs in a comprehension whose containing scope is a class scope. If the rules above were to result in the target being assigned in that class’s scope, the assignment expression is expressly invalid. This case also raises SyntaxError :

(The reason for the latter exception is the implicit function scope created for comprehensions – there is currently no runtime mechanism for a function to refer to a variable in the containing class scope, and we do not want to add such a mechanism. If this issue ever gets resolved this special case may be removed from the specification of assignment expressions. Note that the problem already exists for using a variable defined in the class scope from a comprehension.)

See Appendix B for some examples of how the rules for targets in comprehensions translate to equivalent code.

The := operator groups more tightly than a comma in all syntactic positions where it is legal, but less tightly than all other operators, including or , and , not , and conditional expressions ( A if C else B ). As follows from section “Exceptional cases” above, it is never allowed at the same level as = . In case a different grouping is desired, parentheses should be used.

The := operator may be used directly in a positional function call argument; however it is invalid directly in a keyword argument.

Some examples to clarify what’s technically valid or invalid:

Most of the “valid” examples above are not recommended, since human readers of Python source code who are quickly glancing at some code may miss the distinction. But simple cases are not objectionable:

This PEP recommends always putting spaces around := , similar to PEP 8 ’s recommendation for = when used for assignment, whereas the latter disallows spaces around = used for keyword arguments.)

In order to have precisely defined semantics, the proposal requires evaluation order to be well-defined. This is technically not a new requirement, as function calls may already have side effects. Python already has a rule that subexpressions are generally evaluated from left to right. However, assignment expressions make these side effects more visible, and we propose a single change to the current evaluation order:

In a dict comprehension {X: Y for ...} , Y is currently evaluated before X . We propose to change this so that X is evaluated before Y . (In a dict display like {X: Y} this is already the case, and also in dict((X, Y) for ...) which should clearly be equivalent to the dict comprehension.)

Most importantly, since := is an expression, it can be used in contexts where statements are illegal, including lambda functions and comprehensions.

Conversely, assignment expressions don’t support the advanced features found in assignment statements:

Multiple targets are not directly supported: x = y = z = 0 # Equivalent: (z := (y := (x := 0)))
Single assignment targets other than a single NAME are not supported: # No equivalent a [ i ] = x self . rest = []
Priority around commas is different: x = 1 , 2 # Sets x to (1, 2) ( x := 1 , 2 ) # Sets x to 1
Iterable packing and unpacking (both regular or extended forms) are not supported: # Equivalent needs extra parentheses loc = x , y # Use (loc := (x, y)) info = name , phone , * rest # Use (info := (name, phone, *rest)) # No equivalent px , py , pz = position name , phone , email , * other_info = contact
Inline type annotations are not supported: # Closest equivalent is "p: Optional[int]" as a separate declaration p : Optional [ int ] = None
Augmented assignment is not supported: total += tax # Equivalent: (total := total + tax)

The following changes have been made based on implementation experience and additional review after the PEP was first accepted and before Python 3.8 was released:

for consistency with other similar exceptions, and to avoid locking in an exception name that is not necessarily going to improve clarity for end users, the originally proposed TargetScopeError subclass of SyntaxError was dropped in favour of just raising SyntaxError directly. [3]
due to a limitation in CPython’s symbol table analysis process, the reference implementation raises SyntaxError for all uses of named expressions inside comprehension iterable expressions, rather than only raising them when the named expression target conflicts with one of the iteration variables in the comprehension. This could be revisited given sufficiently compelling examples, but the extra complexity needed to implement the more selective restriction doesn’t seem worthwhile for purely hypothetical use cases.

Examples from the Python standard library

env_base is only used on these lines, putting its assignment on the if moves it as the “header” of the block.

Current: env_base = os . environ . get ( "PYTHONUSERBASE" , None ) if env_base : return env_base
Improved: if env_base := os . environ . get ( "PYTHONUSERBASE" , None ): return env_base

Avoid nested if and remove one indentation level.

Current: if self . _is_special : ans = self . _check_nans ( context = context ) if ans : return ans
Improved: if self . _is_special and ( ans := self . _check_nans ( context = context )): return ans

Code looks more regular and avoid multiple nested if. (See Appendix A for the origin of this example.)

Current: reductor = dispatch_table . get ( cls ) if reductor : rv = reductor ( x ) else : reductor = getattr ( x , "__reduce_ex__" , None ) if reductor : rv = reductor ( 4 ) else : reductor = getattr ( x , "__reduce__" , None ) if reductor : rv = reductor () else : raise Error ( "un(deep)copyable object of type %s " % cls )
Improved: if reductor := dispatch_table . get ( cls ): rv = reductor ( x ) elif reductor := getattr ( x , "__reduce_ex__" , None ): rv = reductor ( 4 ) elif reductor := getattr ( x , "__reduce__" , None ): rv = reductor () else : raise Error ( "un(deep)copyable object of type %s " % cls )

tz is only used for s += tz , moving its assignment inside the if helps to show its scope.

Current: s = _format_time ( self . _hour , self . _minute , self . _second , self . _microsecond , timespec ) tz = self . _tzstr () if tz : s += tz return s
Improved: s = _format_time ( self . _hour , self . _minute , self . _second , self . _microsecond , timespec ) if tz := self . _tzstr (): s += tz return s

Calling fp.readline() in the while condition and calling .match() on the if lines make the code more compact without making it harder to understand.

Current: while True : line = fp . readline () if not line : break m = define_rx . match ( line ) if m : n , v = m . group ( 1 , 2 ) try : v = int ( v ) except ValueError : pass vars [ n ] = v else : m = undef_rx . match ( line ) if m : vars [ m . group ( 1 )] = 0
Improved: while line := fp . readline (): if m := define_rx . match ( line ): n , v = m . group ( 1 , 2 ) try : v = int ( v ) except ValueError : pass vars [ n ] = v elif m := undef_rx . match ( line ): vars [ m . group ( 1 )] = 0

A list comprehension can map and filter efficiently by capturing the condition:

Similarly, a subexpression can be reused within the main expression, by giving it a name on first use:

Note that in both cases the variable y is bound in the containing scope (i.e. at the same level as results or stuff ).

Assignment expressions can be used to good effect in the header of an if or while statement:

Particularly with the while loop, this can remove the need to have an infinite loop, an assignment, and a condition. It also creates a smooth parallel between a loop which simply uses a function call as its condition, and one which uses that as its condition but also uses the actual value.

An example from the low-level UNIX world:

Rejected alternative proposals

Proposals broadly similar to this one have come up frequently on python-ideas. Below are a number of alternative syntaxes, some of them specific to comprehensions, which have been rejected in favour of the one given above.

A previous version of this PEP proposed subtle changes to the scope rules for comprehensions, to make them more usable in class scope and to unify the scope of the “outermost iterable” and the rest of the comprehension. However, this part of the proposal would have caused backwards incompatibilities, and has been withdrawn so the PEP can focus on assignment expressions.

Broadly the same semantics as the current proposal, but spelled differently.

Since EXPR as NAME already has meaning in import , except and with statements (with different semantics), this would create unnecessary confusion or require special-casing (e.g. to forbid assignment within the headers of these statements).

(Note that with EXPR as VAR does not simply assign the value of EXPR to VAR – it calls EXPR.__enter__() and assigns the result of that to VAR .)

Additional reasons to prefer := over this spelling include:

In if f(x) as y the assignment target doesn’t jump out at you – it just reads like if f x blah blah and it is too similar visually to if f(x) and y .
import foo as bar
except Exc as var
with ctxmgr() as var

To the contrary, the assignment expression does not belong to the if or while that starts the line, and we intentionally allow assignment expressions in other contexts as well.

NAME = EXPR
if NAME := EXPR

reinforces the visual recognition of assignment expressions.

This syntax is inspired by languages such as R and Haskell, and some programmable calculators. (Note that a left-facing arrow y <- f(x) is not possible in Python, as it would be interpreted as less-than and unary minus.) This syntax has a slight advantage over ‘as’ in that it does not conflict with with , except and import , but otherwise is equivalent. But it is entirely unrelated to Python’s other use of -> (function return type annotations), and compared to := (which dates back to Algol-58) it has a much weaker tradition.

This has the advantage that leaked usage can be readily detected, removing some forms of syntactic ambiguity. However, this would be the only place in Python where a variable’s scope is encoded into its name, making refactoring harder.

Execution order is inverted (the indented body is performed first, followed by the “header”). This requires a new keyword, unless an existing keyword is repurposed (most likely with: ). See PEP 3150 for prior discussion on this subject (with the proposed keyword being given: ).

This syntax has fewer conflicts than as does (conflicting only with the raise Exc from Exc notation), but is otherwise comparable to it. Instead of paralleling with expr as target: (which can be useful but can also be confusing), this has no parallels, but is evocative.

One of the most popular use-cases is if and while statements. Instead of a more general solution, this proposal enhances the syntax of these two statements to add a means of capturing the compared value:

This works beautifully if and ONLY if the desired condition is based on the truthiness of the captured value. It is thus effective for specific use-cases (regex matches, socket reads that return '' when done), and completely useless in more complicated cases (e.g. where the condition is f(x) < 0 and you want to capture the value of f(x) ). It also has no benefit to list comprehensions.

Advantages: No syntactic ambiguities. Disadvantages: Answers only a fraction of possible use-cases, even in if / while statements.

Another common use-case is comprehensions (list/set/dict, and genexps). As above, proposals have been made for comprehension-specific solutions.

This brings the subexpression to a location in between the ‘for’ loop and the expression. It introduces an additional language keyword, which creates conflicts. Of the three, where reads the most cleanly, but also has the greatest potential for conflict (e.g. SQLAlchemy and numpy have where methods, as does tkinter.dnd.Icon in the standard library).

As above, but reusing the with keyword. Doesn’t read too badly, and needs no additional language keyword. Is restricted to comprehensions, though, and cannot as easily be transformed into “longhand” for-loop syntax. Has the C problem that an equals sign in an expression can now create a name binding, rather than performing a comparison. Would raise the question of why “with NAME = EXPR:” cannot be used as a statement on its own.

As per option 2, but using as rather than an equals sign. Aligns syntactically with other uses of as for name binding, but a simple transformation to for-loop longhand would create drastically different semantics; the meaning of with inside a comprehension would be completely different from the meaning as a stand-alone statement, while retaining identical syntax.

Regardless of the spelling chosen, this introduces a stark difference between comprehensions and the equivalent unrolled long-hand form of the loop. It is no longer possible to unwrap the loop into statement form without reworking any name bindings. The only keyword that can be repurposed to this task is with , thus giving it sneakily different semantics in a comprehension than in a statement; alternatively, a new keyword is needed, with all the costs therein.

There are two logical precedences for the := operator. Either it should bind as loosely as possible, as does statement-assignment; or it should bind more tightly than comparison operators. Placing its precedence between the comparison and arithmetic operators (to be precise: just lower than bitwise OR) allows most uses inside while and if conditions to be spelled without parentheses, as it is most likely that you wish to capture the value of something, then perform a comparison on it:

Once find() returns -1, the loop terminates. If := binds as loosely as = does, this would capture the result of the comparison (generally either True or False ), which is less useful.

While this behaviour would be convenient in many situations, it is also harder to explain than “the := operator behaves just like the assignment statement”, and as such, the precedence for := has been made as close as possible to that of = (with the exception that it binds tighter than comma).

Some critics have claimed that the assignment expressions should allow unparenthesized tuples on the right, so that these two would be equivalent:

(With the current version of the proposal, the latter would be equivalent to ((point := x), y) .)

However, adopting this stance would logically lead to the conclusion that when used in a function call, assignment expressions also bind less tight than comma, so we’d have the following confusing equivalence:

The less confusing option is to make := bind more tightly than comma.

It’s been proposed to just always require parentheses around an assignment expression. This would resolve many ambiguities, and indeed parentheses will frequently be needed to extract the desired subexpression. But in the following cases the extra parentheses feel redundant:

Frequently Raised Objections

C and its derivatives define the = operator as an expression, rather than a statement as is Python’s way. This allows assignments in more contexts, including contexts where comparisons are more common. The syntactic similarity between if (x == y) and if (x = y) belies their drastically different semantics. Thus this proposal uses := to clarify the distinction.

The two forms have different flexibilities. The := operator can be used inside a larger expression; the = statement can be augmented to += and its friends, can be chained, and can assign to attributes and subscripts.

Previous revisions of this proposal involved sublocal scope (restricted to a single statement), preventing name leakage and namespace pollution. While a definite advantage in a number of situations, this increases complexity in many others, and the costs are not justified by the benefits. In the interests of language simplicity, the name bindings created here are exactly equivalent to any other name bindings, including that usage at class or module scope will create externally-visible names. This is no different from for loops or other constructs, and can be solved the same way: del the name once it is no longer needed, or prefix it with an underscore.

(The author wishes to thank Guido van Rossum and Christoph Groth for their suggestions to move the proposal in this direction. [2] )

As expression assignments can sometimes be used equivalently to statement assignments, the question of which should be preferred will arise. For the benefit of style guides such as PEP 8 , two recommendations are suggested.

If either assignment statements or assignment expressions can be used, prefer statements; they are a clear declaration of intent.
If using assignment expressions would lead to ambiguity about execution order, restructure it to use statements instead.

The authors wish to thank Alyssa Coghlan and Steven D’Aprano for their considerable contributions to this proposal, and members of the core-mentorship mailing list for assistance with implementation.

Appendix A: Tim Peters’s findings

Here’s a brief essay Tim Peters wrote on the topic.

I dislike “busy” lines of code, and also dislike putting conceptually unrelated logic on a single line. So, for example, instead of:

instead. So I suspected I’d find few places I’d want to use assignment expressions. I didn’t even consider them for lines already stretching halfway across the screen. In other cases, “unrelated” ruled:

is a vast improvement over the briefer:

The original two statements are doing entirely different conceptual things, and slamming them together is conceptually insane.

In other cases, combining related logic made it harder to understand, such as rewriting:

as the briefer:

The while test there is too subtle, crucially relying on strict left-to-right evaluation in a non-short-circuiting or method-chaining context. My brain isn’t wired that way.

But cases like that were rare. Name binding is very frequent, and “sparse is better than dense” does not mean “almost empty is better than sparse”. For example, I have many functions that return None or 0 to communicate “I have nothing useful to return in this case, but since that’s expected often I’m not going to annoy you with an exception”. This is essentially the same as regular expression search functions returning None when there is no match. So there was lots of code of the form:

I find that clearer, and certainly a bit less typing and pattern-matching reading, as:

It’s also nice to trade away a small amount of horizontal whitespace to get another _line_ of surrounding code on screen. I didn’t give much weight to this at first, but it was so very frequent it added up, and I soon enough became annoyed that I couldn’t actually run the briefer code. That surprised me!

There are other cases where assignment expressions really shine. Rather than pick another from my code, Kirill Balunov gave a lovely example from the standard library’s copy() function in copy.py :

The ever-increasing indentation is semantically misleading: the logic is conceptually flat, “the first test that succeeds wins”:

Using easy assignment expressions allows the visual structure of the code to emphasize the conceptual flatness of the logic; ever-increasing indentation obscured it.

A smaller example from my code delighted me, both allowing to put inherently related logic in a single line, and allowing to remove an annoying “artificial” indentation level:

That if is about as long as I want my lines to get, but remains easy to follow.

So, in all, in most lines binding a name, I wouldn’t use assignment expressions, but because that construct is so very frequent, that leaves many places I would. In most of the latter, I found a small win that adds up due to how often it occurs, and in the rest I found a moderate to major win. I’d certainly use it more often than ternary if , but significantly less often than augmented assignment.

I have another example that quite impressed me at the time.

Where all variables are positive integers, and a is at least as large as the n’th root of x, this algorithm returns the floor of the n’th root of x (and roughly doubling the number of accurate bits per iteration):

It’s not obvious why that works, but is no more obvious in the “loop and a half” form. It’s hard to prove correctness without building on the right insight (the “arithmetic mean - geometric mean inequality”), and knowing some non-trivial things about how nested floor functions behave. That is, the challenges are in the math, not really in the coding.

If you do know all that, then the assignment-expression form is easily read as “while the current guess is too large, get a smaller guess”, where the “too large?” test and the new guess share an expensive sub-expression.

To my eyes, the original form is harder to understand:

This appendix attempts to clarify (though not specify) the rules when a target occurs in a comprehension or in a generator expression. For a number of illustrative examples we show the original code, containing a comprehension, and the translation, where the comprehension has been replaced by an equivalent generator function plus some scaffolding.

Since [x for ...] is equivalent to list(x for ...) these examples all use list comprehensions without loss of generality. And since these examples are meant to clarify edge cases of the rules, they aren’t trying to look like real code.

Note: comprehensions are already implemented via synthesizing nested generator functions like those in this appendix. The new part is adding appropriate declarations to establish the intended scope of assignment expression targets (the same scope they resolve to as if the assignment were performed in the block containing the outermost comprehension). For type inference purposes, these illustrative expansions do not imply that assignment expression targets are always Optional (but they do indicate the target binding scope).

Let’s start with a reminder of what code is generated for a generator expression without assignment expression.

Original code (EXPR usually references VAR): def f (): a = [ EXPR for VAR in ITERABLE ]
Translation (let’s not worry about name conflicts): def f (): def genexpr ( iterator ): for VAR in iterator : yield EXPR a = list ( genexpr ( iter ( ITERABLE )))

Let’s add a simple assignment expression.

Original code: def f (): a = [ TARGET := EXPR for VAR in ITERABLE ]
Translation: def f (): if False : TARGET = None # Dead code to ensure TARGET is a local variable def genexpr ( iterator ): nonlocal TARGET for VAR in iterator : TARGET = EXPR yield TARGET a = list ( genexpr ( iter ( ITERABLE )))

Let’s add a global TARGET declaration in f() .

Original code: def f (): global TARGET a = [ TARGET := EXPR for VAR in ITERABLE ]
Translation: def f (): global TARGET def genexpr ( iterator ): global TARGET for VAR in iterator : TARGET = EXPR yield TARGET a = list ( genexpr ( iter ( ITERABLE )))

Or instead let’s add a nonlocal TARGET declaration in f() .

Original code: def g (): TARGET = ... def f (): nonlocal TARGET a = [ TARGET := EXPR for VAR in ITERABLE ]
Translation: def g (): TARGET = ... def f (): nonlocal TARGET def genexpr ( iterator ): nonlocal TARGET for VAR in iterator : TARGET = EXPR yield TARGET a = list ( genexpr ( iter ( ITERABLE )))

Finally, let’s nest two comprehensions.

Original code: def f (): a = [[ TARGET := i for i in range ( 3 )] for j in range ( 2 )] # I.e., a = [[0, 1, 2], [0, 1, 2]] print ( TARGET ) # prints 2
Translation: def f (): if False : TARGET = None def outer_genexpr ( outer_iterator ): nonlocal TARGET def inner_generator ( inner_iterator ): nonlocal TARGET for i in inner_iterator : TARGET = i yield i for j in outer_iterator : yield list ( inner_generator ( range ( 3 ))) a = list ( outer_genexpr ( range ( 2 ))) print ( TARGET )

Because it has been a point of confusion, note that nothing about Python’s scoping semantics is changed. Function-local scopes continue to be resolved at compile time, and to have indefinite temporal extent at run time (“full closures”). Example:

This document has been placed in the public domain.

Source: https://github.com/python/peps/blob/main/peps/pep-0572.rst

Last modified: 2023-10-11 12:05:51 GMT

Python Tutorial

File handling, python modules, python numpy, python pandas, python matplotlib, python scipy, machine learning, python mysql, python mongodb, python reference, module reference, python how to, python examples, python - list comprehension, list comprehension.

List comprehension offers a shorter syntax when you want to create a new list based on the values of an existing list.

Based on a list of fruits, you want a new list, containing only the fruits with the letter "a" in the name.

Without list comprehension you will have to write a for statement with a conditional test inside:

With list comprehension you can do all that with only one line of code:

The return value is a new list, leaving the old list unchanged.

The condition is like a filter that only accepts the items that valuate to True .

Only accept items that are not "apple":

The condition if x != "apple" will return True for all elements other than "apple", making the new list contain all fruits except "apple".

The condition is optional and can be omitted:

With no if statement:

The iterable can be any iterable object, like a list, tuple, set etc.

You can use the range() function to create an iterable:

Same example, but with a condition:

Accept only numbers lower than 5:

The expression is the current item in the iteration, but it is also the outcome, which you can manipulate before it ends up like a list item in the new list:

Set the values in the new list to upper case:

You can set the outcome to whatever you like:

Set all values in the new list to 'hello':

The expression can also contain conditions, not like a filter, but as a way to manipulate the outcome:

Return "orange" instead of "banana":

The expression in the example above says:

"Return the item if it is not banana, if it is banana return orange".

COLOR PICKER

Report Error

If you want to report an error, or if you want to make a suggestion, do not hesitate to send us an e-mail:

[email protected]

Python Flow Control

Python if...else
Python for Loop
Python while Loop
Python break and continue
Python Pass

Python Functions

Python Function
Function Argument
Python Recursion
Anonymous Function
Global, Local and Nonlocal
Python Global Keyword
Python Modules
Python Package

Python Datatypes

Python Numbers
Python List
Python Tuple
Python String
Python Dictionary

Python Files

Python File Operation
Python Directory
Python Exception
Exception Handling
User-defined Exception

Python Object & Class

Classes & Objects
Python Inheritance
Multiple Inheritance
Operator Overloading

Python Advanced Topics

Python Iterator
Python Generator
Python Closure
Python Decorators
Python Property
Python RegEx
Python Examples

Python Date and time

Python datetime Module
Python datetime.strftime()
Python datetime.strptime()
Current date & time
Get current time
Timestamp to datetime
Python time Module
Python time.sleep()

Python Tutorials

Python Dictionary Comprehension

Python Dictionary fromkeys()

Python Lambda/Anonymous Function

Python range() Function

Python List insert()

Python List Comprehension

List comprehension offers a concise way to create a new list based on the values of an existing list.

Suppose we have a list of numbers and we desire to create a new list containing the double value of each element in the list.

Syntax of List Comprehension

for every item in list , execute the expression if the condition is True .

Note: The if statement in list comprehension is optional.

for Loop vs. List Comprehension

List comprehension makes the code cleaner and more concise than for loop.

Let's write a program to print the square of each list element using both for loop and list comprehension.

List Comprehension

It's much easier to understand list comprehension once you know Python for loop() .

Conditionals in List Comprehension

List comprehensions can utilize conditional statements like if…else to filter existing lists.

Let's see an example of an if statement with list comprehension.

Here, list comprehension checks if the number from range(1, 10) is even or odd. If even, it appends the number in the list.

Note : The range() function generates a sequence of numbers. To learn more, visit Python range() .

Let's use if...else with list comprehension to find even and odd numbers.

Here, if an item in the numbers list is divisible by 2 , it appends Even to the list even_odd_list . Else, it appends Odd .

Let's use nested if with list comprehension to find even numbers that are divisible by 5 .

Here, list comprehension checks two conditions:

if y is divisible by 2 or not.
if yes, is y divisible by 5 or not.

If y satisfies both conditions, the number appends to num_list .

Example: List Comprehension with String

We can also use list comprehension with iterables other than lists.

Here, we used list comprehension to find vowels in the string 'Python' .

Video: Python List, Set & Dictionary Comprehension

Sorry about that.

Comprehensive Guide to Python List Comprehensions

June 28, 2021
Author - Chetan Ambi
Category - Python

Table of Contents

Introduction

Python list comprehensions provide a concise way of creating lists. For example, if you want to write code that squares all the numbers from 1 to 10, you would write as below (the traditional way). First, you need to create an empty list, then loop through all the elements, square each element and then append to the list.

You can achieve the same result using Python list comprehensions as shown below. As you can see, list comprehensions are readable, concise, and more Pythonic.

Understanding list comprehensions don’t stop here. But there is much to learn about list comprehensions. In this comprehensive guide, we’ll dig deeper and understand more about the list comprehensions. Once you go through this guide, you would have mastered list comprehensions in Python. I know you are excited!! Let’s get started.

Lists are one of the sequence types in Python. Refer to our in-depth article here for introduction to sequence types in Python.

List Comprehensions

As mentioned earlier, list comprehensions are concise ways of creating lists. The general syntax of list comprehension looks as below. It consists of three parts: iteration, transformation, and filtering .

Iteration: This is the first part that gets executed in a list comprehension that iterates through all the elements in the iterable.
Transformation : As the name says transformation will be applied to each element in the iterable. In the above example, we are squaring each element from the iterable.
Filter (optional) : This is optional and it filters out some of the elements based on the conditions. The above example loops through all the elements from 1 to 10, filters out odd numbers and considers only positive numbers before applying the transformation.

Let’s see couple more examples of list comprehensions. The below example converts all the strings in the list to upper case.

Here is another example that displays all the numbers less than 50 that are even and divisible by 7.

What happens under the hood

Now that you have understood the syntax and how to write list comprehensions, let’s try to understand what goes on under the hood.

Whenever Python encounters a list comprehension, it creates a temporary function that gets executed during the runtime. The result is stored at some memory location and then gets assigned the variable on the LHS. Refer to the diagram below.

You can confirm this by running the list comprehension through the dis module. In the below example, the built-in compile() function generates the bytecode of the list comprehension. Then when you run it through the dis module, it generates byte code instructions as below.

Refer to the output of dis below. The bytecode instruction has MAKE_FUNCTION , CALL_FUNCTION . This confirms that Python converts list comprehensions to temporary functions internally.

Since list comprehensions are internally treated as functions, they exhibit some of the properties of functions. List comprehensions also have local and global scopes . In the below example, num is a local variable and factor is a global variable. But you can still access the global variable factor inside the list comprehension.

Nested list comprehensions

It’s possible to write nested list comprehensions meaning you can include a list comprehension within another. One of the uses of nested list comprehensions is that they can be used to create n-dimensional matrices.

In the example below, we are able to create a 5×5 matrix using nested list comprehensions.

Nested loops in list comprehensions

It’s also possible to include multiple loops in list comprehensions. You can use as many loops as you want. But, as a best practice, you should limit the depth to two so as not to lose the readability. The below example uses two for loops.

One of the uses of nested loops is to vectorize the matrix (flattening the matrix). In the below example, my_matrix is a 2×2 matrix. Using nested loops, we are able to create a vectorized version.

Multi-line list comprehensions

Most of the examples you have seen are one-line list comprehensions. But note that you can also write multi-line list comprehensions and are preferred when you can’t fit all the instructions in one line. They just increase the readability and won’t offer any added benefit. See an example below.

The same multi-line list comprehension is written in the single line as below. As you can see this is not readable.

List comprehensions with conditionals (if-else)

You can also use if-else conditions with list comprehensions. An example of using the if condition was covered in the syntax section. However, let me provide two examples here one with if and another with if-else .

Using if conditional:

The below example uses the if statement to filter out the languages that start with the letter “P”. Notice that if statement comes after the loop. In general, if condition is used to filtering.

Using if-else conditional:

When the if-else is used with list comprehension, the syntax is a little different compared to comprehension with only if statement. The below example calculates the square even numbers and cube of odd numbers between 1 to 10 using an if-else statement.

Do you want to master Python for else and while else? Please read our detailed article here .

List comprehensions with walrus operator

You can create list comprehensions using the walrus operator := and is also called as “assignment expression”. The walrus operator was first introduced in Python 3.8. As mentioned in PEP 572, it was introduced to speedup the coding by writing short code.

The below example creates a list of even numbers between 0 to 10 using the walrus operator.

List comprehensions Vs map and filter

The map and filter function can also be used as alternatives to list comprehensions as both the functions help us in creating the list object.

map vs. list comprehension

The below example shows that the list comprehension function can be an alternative to the map function.

The below code also confirms that list comprehensions are faster than the map function.

filter vs. list comprehension

From the below example, it is evident that list comprehensions can be an alternative to the filter function.

Even in this case, list comprehensions are faster than filter functions. Refer to the below example.

Set Comprehensions

You can also create set comprehensions but the only difference is they are enclosed within curly brackets { } . The below example creates set comprehension from a list. It converts all the strings in the given list to upper case and creates a set (of course, by removing the duplicates)

Dictionary Comprehensions

Dictionary comprehensions are also enclosed with { } brackets but its contents follow key: value format as you expect. The below example creates a dictionary comprehension from a list. It creates a dictionary with string as key and length of the string as value.

Generator Expressions

The generator expressions look very similar to list comprehension with the difference is that generator expressions are enclosed within parenthesis ( ) . You could say generator comprehension a lazy version of list comprehension .

In the below example, we created a list of squares of numbers between 1 to 10 that are even-numbered using generator expression. Note that the output of generator expression is a generator object. To get the result, you need to pass the generator object to the list constructor (or to a tuple or set the constructor based on your need).

Generator Expressions Vs. List Comprehensions

a) Storage efficiency: A list comprehension stores the entire list in the memory whereas a generator yields one item at a time on-demand. Hence, generator expressions are more memory efficient than lists. As seen from the below example, the same code generator expression takes far less memory.

b) Speed efficiency: List comprehensions are faster than generator expressions. The same code takes 7 seconds for list comprehensions but 9 seconds with generator expressions.

c) Use list comprehensions if you need to iterate over the list multiple times. If not, consider using generator expressions. Because you won’t be able to iterate generator expression again once you already iterated over. See the example below. When you iterate over a second time, it returns an empty list.

So, which one should you consider? list comprehensions or generator expressions. There is always a trade-off between memory and speed efficiency so you need to choose based on the requirements.

Do we have tuple comprehensions?

No, there are no tuple comprehensions! As we have gone through earlier, list comprehensions loop through each item in the iterable and append to the list under the hood. You are mutating the list by appending elements to it. List, set and dictionary comprehensions exist because they are mutable. But tuple is an immutable object. Hence, we don’t have tuple comprehensions.

Refer to this detailed discussion on the same subject on Stack Overflow.

List comprehensions provide a concise way of creating lists.
Apart from list comprehensions , Python also has set comprehensions , dictionary comprehensions , and generator expressions .
Under the hood, list comprehensions converted to equivalent functions.
List comprehensions come in different formats — nested list comprehensions, with nested loop, with multi-lines, with walrus operator, with if and if-else conditionals, etc.
List comprehensions with if conditional is used for filtering while if-else is used when you want to perform different transformations based on the conditions.
List comprehensions can be a replacement for map, filter, reduce, and for loop . In most cases list comprehensions is faster than all these methods.
If you need to use more than two for loops in the comprehension then it’s better to use the traditional way of creating a list. Because using more than two for loops is not easy to readable.
Consider using generator expressions instead of list comprehension if you are dealing with large data and not iterating over the list multiple times.
Comprehensions are supposed to be concise , clear , and Pythonic . Don’t write complicated lengthy comprehensions that will defeat the very purpose of using them.

[1]. https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions

Chetan Ambi

List comprehensions in Python

In Python, you can create a new list using list comprehensions. It's simpler than using the for loop.

5. Data Structures - List Comprehensions — Python 3.11.3 documentation
6. Expressions — Python 3.11.3 documentation

Basics of list comprehensions

List comprehensions with if.

List comprehensions with conditional expressions (like if else)
List comprehensions with zip(), enumerate()

Nested list comprehensions

Set comprehensions, dictionary comprehensions, generator expressions.

See the following article for the basics of the for statement.

Python for loop (with range, enumerate, zip, etc.)
Convert a list of strings and a list of numbers to each other in Python

See the following article for examples of using list comprehensions.

Extract, replace, convert elements of a list in Python
Extract and replace elements that meet the conditions of a list of strings in Python

List comprehensions are written as follows:

Each element of iterable , such as a list or a tuple, is assigned to variable_name and evaluated with expression . A new list is created with the result evaluated by expression as an element.

An example of list comprehensions is provided alongside an equivalent for statement. Squares the elements of range .

How to use range() in Python

Although map() can do the same thing, list comprehension is preferred for simplicity and clarity.

Apply a function to items of a list with map() in Python

In list comprehensions, if can be used as follows:

Only elements of iterable that satisfy the condition are evaluated with expression . Elements that do not meet the condition are excluded from the resulting list.

Although filter() can do the same thing, list comprehension is preferred for simplicity and clarity.

Filter (extract/remove) items of a list with filter() in Python

List comprehensions with conditional expressions (like if else )

In the previous example, elements that do not satisfy the condition are excluded from the new list.

If you want to apply another operation to elements that do not meet condition like if else , use conditional expressions.

In Python, conditional expressions can be written as follows:

X is the value or expression when the condition is True , and Y is the value or expression when the condition is False .

Conditional expressions in Python

You can use it for the expression part of list comprehensions:

It is also possible to apply different operations to the original element depending on the condition as follows:

List comprehensions with zip() , enumerate()

In the for statement, zip() and enumerate() are often used. zip() aggregates elements from multiple iterables and enumerate() returns a value and its index.

zip() in Python: Get elements from multiple lists
How to start enumerate() at 1 in Python

Of course, you can also use zip() and enumerate() in list comprehensions.

List comprehensions with zip() :

List comprehensions with enumerate() :

You can also use if .

It is also possible to calculate a new value using each element.

List comprehensions can be nested just as for loops are nested.

Line breaks and indentation have been added for readability, but they are not required.

Another example:

If you change the square brackets [] in the list comprehension to the curly braces {} , a set ( set type object) is created.

For more information about set , see the following article.

Set operations in Python (union, intersection, symmetric difference, etc.)

Dictionaries ( dict type objects) can also be created with dictionary comprehensions.

5. Data Structures - Dictionaries — Python 3.11.3 documentation

Use {} and specify the key and the value in the expression part as key: value .

Any value or expression can be specified for key and value .

Use the zip() function to create a new dictionary from each list of keys and values.

For other ways to create dictionaries, see the following articles

Create a dictionary in Python ({}, dict(), dict comprehensions)

If the square bracket [] in the list comprehension is changed to the parenthesis () , the generator is returned instead of the tuple. This is called a generator expression.

6. Expressions - Generator expressions — Python 3.11.3 documentation

List comprehension:

Generator expression:

print() does not output the generator elements. You can get elements by using the for statement.

It is possible to use if and nest with generator expressions as well as list comprehensions.

For example, if a list with many elements is created with list comprehensions and used in the for loop, list comprehensions create a list containing all elements at first.

On the other hand, generator expressions generate elements one at a time during iteration, which can reduce memory usage.

You can omit the parentheses () if the generator expression is the only argument passed to the function.

When all elements are processed, list comprehensions are often faster than generator expressions.

python - List comprehension vs generator expression's weird timeit results? - Stack Overflow

However, in all() and any() , for example, a generator expression may be faster than list comprehensions because the result is determined when False or True is encountered.

How to use all() and any() in Python

There are no tuple comprehensions, but it is possible to create tuples by passing the generator expression to tuple() .

Related Categories

Comprehensions in Python

Comprehensions in Python provide us with a short and concise way to construct new sequences (such as lists, sets, dictionaries, etc.) using previously defined sequences. Python supports the following 4 types of comprehension:

List Comprehensions

Dictionary comprehensions, set comprehensions, generator comprehensions.

List Comprehensions provide an elegant way to create new lists. The following is the basic structure of list comprehension:

Syntax: output_list = [ output_exp for var in input_list if ( var satisfies this condition)]

Note that list comprehension may or may not contain an if condition. List comprehensions can contain multiple

Example 1: Generating an Even list WITHOUT using List comprehensions

Suppose we want to create an output list that contains only the even numbers which are present in the input list. Let’s see how to do this using loops, list comprehension, and list comprehension, and decide which method suits you better.

Example 2: Generating Even list using List comprehensions

Here we use the list comprehensions in Python. It creates a new list named list_using_comp by iterating through each element var in the input_list . Elements are included in the new list only if they satisfy the condition, which checks if the element is even. As a result, the output list will contain all even numbers.

Example 1: Squaring Number WITHOUT using List comprehensions

Suppose we want to create an output list which contains squares of all the numbers from 1 to 9. Let’s see how to do this using for loops and list comprehension.

Example 2: Squaring Number using List comprehensions

In This we use list comprehension to generate a new list. It iterates through the numbers in the range from 1 to 9 (inclusive). For each number var , it calculates the square of the number using the expression and adds the result to the new list. The printed output will contain the squares of numbers from 1 to 9.

Extending the idea of list comprehensions, we can also create a dictionary using dictionary comprehensions. The basic structure of a dictionary comprehension looks like below.

output_dict = {key:value for (key, value) in iterable if (key, value satisfy this condition)}

Example 1: Generating odd number with their cube values without using dictionary comprehension

Suppose we want to create an output dictionary which contains only the odd numbers that are present in the input list as keys and their cubes as values. Let’s see how to do this using for loops and dictionary comprehension.

Example 2: Generating odd number with their cube values with using dictionary comprehension

We are using dictionary comprehension in Python. It initializes an list containing numbers from 1 to 7. It then constructs a new dictionary using dictionary comprehension. For each odd number var in the list , it calculates the cube of the number and assigns the result as the value to the key var in the dictionary.

Example 1: Mapping states with their capitals without Using dictionary comprehension

Given two lists containing the names of states and their corresponding capitals, construct a dictionary which maps the states with their respective capitals. Let’s see how to do this using for loops and dictionary comprehension.

Example 2: Mapping states with their capitals with using dictionary comprehension

Here we will use dictionary comprehension to initializes two lists, state and capital , containing corresponding pairs of states and their capitals. It iterates through the pairs of state and capital using the zip() function, and for each pair, it creates a key-value pair in the dictionary. The key is taken from the state list, and the value is taken from the capital list. Finally, the printed output will contain the mapping of states to their capitals.

Set comprehensions are pretty similar to list comprehensions. The only difference between them is that set comprehensions use curly brackets { }

Let’s look at the following example to understand set comprehensions.

Example 1 : Checking Even number Without using set comprehension

Suppose we want to create an output set which contains only the even numbers that are present in the input list. Note that set will discard all the duplicate values. Let’s see how we can do this using for loops and set comprehension.

Example 2: Checking Even number using set comprehension

We will use set comprehension to initializes a list with integer values. The code then creates a new set using set comprehension. It iterates through the elements of the input_list , and for each element, it checks whether it’s even. If the condition is met, the element is added to the set. The printed output which will contain unique even numbers from the list.

Generator Comprehensions are very similar to list comprehensions. One difference between them is that generator comprehensions use circular brackets whereas list comprehensions use square brackets. The major difference between them is that generators don’t allocate memory for the whole list. Instead, they generate each value one by one which is why they are memory efficient. Let’s look at the following example to understand generator comprehension:

Please Login to comment...

python-basics
python-dict
python-list
Technical Scripter 2018
Technical Scripter
10 Best Zoho Vault Alternatives in 2024 (Free)
10 Best Twitch Alternatives in 2024 (Free)
Top 10 Best Cartoons of All Time
10 Best AI Instagram Post Generator in 2024
30 OOPs Interview Questions and Answers (2024)

Improve your Coding Skills with Practice

What kind of Experience do you want to share?

Python List Comprehension

A comprehension is a compact way of creating a Python data structure from iterators. With comprehensions, you can combine loops and conditional tests with a less verbose syntax.

Comprehension is considered more Pythonic and often useful in a variety of scenarios.

What is List Comprehension?

List comprehension sounds complex but it really isn’t.

List comprehension is a way to build a new list by applying an expression to each item in an iterable.

It saves you having to write several lines of code, and keeps the readability of your code neat.

Basic Example

Suppose you want to create a list of all integer square numbers from 0 to 4. You could build that list by appending one item at a time to an empty list:

Or, you could just use an iterator and the range() function:

Here both approaches produce the same result. However, a more Pythonic way to build a list is by using a list comprehension.

The general syntax for a list comprehension is:

Here’s how a list comprehension would build the above list:

In the example above, list comprehension has two parts.

The first part collects the results of an expression on each iteration and uses them to fill out a new list.

The second part is exactly the same as the for loop, where you tell Python which iterable to work on. Every time the loop goes over the iterable, Python will assign each individual element to a variable x.

More Examples

Below are few examples of list comprehension.

List comprehensions can iterate over any type of iterable such as lists, strings, files, ranges, and anything else that supports the iteration protocol .

Here’s a simple list comprehension that uses string as an iterable.

Following example applies abs() function to all the elements in a list.

Following example calls a built-in method strip() on each element in a list.

Following example creates a list of (number, square) tuples. Please note that, if a list comprehension is used to construct a list of tuples, the tuple values must be enclosed in parentheses.

Here’s why you should use list comprehension more often:

List comprehensions are more concise to write and hence they turn out to be very useful in many contexts.
Since a list comprehension is an expression, you can use it wherever you need an expression (e.g. as an argument to a function, in a return statement).
List comprehensions run substantially faster than manual for loop statements (roughly twice as fast). It offers a major performance advantage especially for larger data sets.

List Comprehension with if Clause

A list comprehension may have an optional associated if clause to filter items out of the result.

Iterable’s items are skipped for which the if clause is not true.

This list comprehension is the same as a for loop that contains an if statement :

Nested List Comprehensions

The initial expression in a list comprehension can be any expression, including another list comprehension.

For example, here’s a simple list comprehension that flattens a nested list into a single list of items.

Here’s another list comprehension that transposes rows and columns.

List Comprehension vs map() + lambda

When all you’re doing is calling an already-defined function on each element, map(f, L) is a little faster than the corresponding list comprehension [f(x) for x in L] . Following example collects the ASCII codes of all characters in an entire string.

However, when evaluating any other expression, [some_expr for x in L] is faster and clearer than map(lambda x: some_expr, L) , because the map incurs an extra function call for each element. Following example creates a list of all integer square numbers.

List Comprehension vs filter() + lambda

List comprehension with if clause can be thought of as analogous to the filter() function as they both skip an iterable’s items for which the if clause is not true. Following example filters a list to exclude odd numbers.

As with map() function, filter() is slightly faster if you are using a built-in function.

List Comprehensions and Variable Scope

In Python 2, the iteration variables defined within a list comprehension remain defined even after the list comprehension is executed.

For example, in [x for x in L] , the iteration variable x overwrites any previously defined value of x and is set to the value of the last item, after the resulting list is created.

so, remember to use variable names that won’t conflict with names of other local variables you have.

Fortunately, this is not the case in Python 3 where the iteration variable remains private, so you need not worry.

Every dunder method in Python

You've just made a class. You made a __init__ method. Now what?

Python includes tons of dunder methods ("double underscore" methods) which allow us to deeply customize how our custom classes interact with Python's many features. What dunder methods could you add to your class to make it friendly for other Python programmers who use it?

Let's take a look at every dunder method in Python , with a focus on when each method is useful.

Note that the Python documentation refers to these as special methods and notes the synonym "magic method" but very rarely uses the term "dunder method". However, "dunder method" is a fairly common Python colloquialism, as noted in my unofficial Python glossary .

You can use the links scattered throughout this page for more details on any particular dunder method. For a list of all of them, see the cheat sheet in the final section.

The 3 essential dunder methods 🔑

There are 3 dunder methods that most classes should have: __init__ , __repr__ , and __eq__ .

The __init__ method is the initializer (not to be confused with the constructor ), the __repr__ method customizes an object's string representation, and the __eq__ method customizes what it means for objects to be equal to one another.

The __repr__ method is particularly helpful at the the Python REPL and when debugging.

Equality and hashability 🟰

In addition to the __eq__ method, Python has 2 other dunder methods for determining the "value" of an object in relation to other objects.

Python's __eq__ method typically returns True , False , or NotImplemented (if objects can't be compared). The default __eq__ implementation relies on the is operator, which checks for identity .

The default implementation of __ne__ calls __eq__ and negates any boolean return value given (or returns NotImplemented if __eq__ did). This default behavior is usually "good enough", so you'll almost never see __ne__ implemented .

Hashable objects can be used as keys in dictionaries or values in sets. All objects in Python are hashable by default, but if you've written a custom __eq__ method then your objects won't be hashable without a custom __hash__ method. But the hash value of an object must never change or bad things will happen so typically only immutable objects implement __hash__ .

For implementing equality checks, see __eq__ in Python . For implementing hashability, see making hashable objects in Python .

Orderability ⚖️

Python's comparison operators ( < , > , <= , >= ) can all be overloaded with dunder methods as well. The comparison operators also power functions that rely on the relative ordering of objects, like sorted , min , and max .

If you plan to implement all of these operators in the typical way (where x < y would be the same as asking y > x ) then the total_ordering decorator from Python's functools module will come in handy.

Type conversions and string formatting ⚗️

Python has a number of dunder methods for converting objects to a different type.

The __bool__ function is used for truthiness checks, though __len__ is used as a fallback.

If you needed to make an object that acts like a number (like decimal.Decimal or fractions.Fraction ), you'll want to implement __int__ , __float__ , and __complex__ so your objects can be converted to other numbers. If you wanted to make an object that could be used in a memoryview or could otherwise be converted to bytes , you'll want a __bytes__ method.

The __format__ and __repr__ methods are different string conversion flavors. Most string conversions rely the __str__ method, but the default __str__ implementation simply calls __repr__ .

The __format__ method is used by all f-string conversions , by the str class's format method, and by the (rarely used) built-in format function. This method allows datetime objects to support custom format specifiers .

Context managers 🚪

A context manager is an object that can be used in a with block.

For more on context managers see, what is a context manager and creating a context manager .

Containers and collections 🗃️

Collections (a.k.a. containers) are essentially data structures or objects that act like data stuctures. Lists, dictionaries, sets, strings, and tuples are all examples of collections.

The __iter__ method is used by the iter function and for all forms of iteration: for loops , comprehensions , tuple unpacking , and using * for iterable unpacking .

While the __iter__ method is necessary for creating a custom iterable, the __next__ method is necessary for creating a custom iterator (which is much less common). The __missing__ method is only ever called by the dict class on itself, unless another class decides to implement __missing__ . The __length_hint__ method supplies a length guess for structures which do not support __len__ so that lists or other structures can be pre-sized more efficiently.

Also see: the iterator protocol , implementing __len__ , and implementing __getitem__ .

Callability ☎️

Functions, classes, and all other callable objects rely on the __call__ method.

When a class is called, its metaclass 's __call__ method is used. When a class instance is called, the class's __call__ method is used.

For more on callability, see Callables: Python's "functions" are sometimes classes .

Arithmetic operators ➗

Python's dunder methods are often described as a tool for "operator overloading". Most of this "operator overloading" comes in the form of Python's various arithmetic operators.

There are two ways to break down the arithmetic operators:

Mathematical (e.g. + , - , * , / , % ) versus bitwise (e.g. & , | , ^ , >> , ~ )
Binary (between 2 values, like x + y ) versus unary (before 1 value, like +x )

The mathematical operators are much more common than the bitwise ones and the binary ones are a bit more common than the unary ones.

These are the binary mathematical arithmetic operators:

Each of these operators includes left-hand and right-hand methods. If x.__add__(y) returns NotImplemented , then y.__radd__(x) will be attempted. See arithmetic dunder methods for more.

These are the binary bitwise arithmetic operators:

These are Python's unary arithmetic operators:

The unary + operator typically has no effect , though some objects use it for a specific operation. For example using + on collections.Counter objects will remove non-positive values.

Python's arithmetic operators are often used for non-arithmetic ends: sequences use + to concatenate and * to self-concatenate and sets use & for intersection, | for union, - for asymmetric difference, and ^ for symmetric difference. Arithmetic operators are sometimes overloaded for more creative uses too. For example, pathlib.Path objects use / to create child paths .

In-place arithmetic operations ♻️

Python includes many dunder methods for in-place operations. If you're making a mutable object that supports any of the arithmetic operations, you'll want to implement the related in-place dunder method(s) as well.

All of Python's binary arithmetic operators work in augmented assignment statements , which involve using an operator followed by the = sign to assign to an object while performing an operation on it.

Augmented assignments on mutable objects are expected to mutate the original object , thanks to the mutable object implementing the appropriate dunder method for in-place arithmetic.

When no dunder method is found for an in-place operation, Python performs the operation followed by an assignment. Immutable objects typically do not implement dunder methods for in-place operations , since they should return a new object instead of changing the original.

Built-in math functions 🧮

Python also includes dunder methods for many math-related functions, both built-in functions and some functions in the math library.

Python's divmod function performs integer division ( // ) and a modulo operation ( % ) at the same time. Note that, just like the many binary arithmetic operators, divmod will also check for an __rvidmod__ method if it needs to ask the second argument to handle the operation.

The __index__ method is for making integer-like objects. This method losslessly converts to an integer, unlike __int__ which may perform a "lossy" integer conversion (e.g. from float to int ). It's used by operations that require true integers, such as slicing , indexing, and bin , hex , and oct functions ( example ).

Attribute access 📜

Python even includes dunder methods for controlling what happens when you access, delete, or assign any attribute on an object!

The __getattribute__ method is called for every attribute access, while the __getattr__ method is only called after Python fails to find a given attribute. All method calls and attribute accesses call __getattribute__ so implementing it correctly is challenging (due to accidental recursion ).

The __dir__ method should return an iterable of attribute names (as strings). When the dir function calls __dir__ , it converts the returned iterable into a sorted list (like sorted does).

The built-in getattr , setattr , and delattr functions correspond to the dunder methods of the same name, but they're only intended for dynamic attribute access (not all attribute accesses).

Metaprogramming 🪄

Now we're getting into the really unusual dunder methods. Python includes many dunder methods for metaprogramming-related features.

The __prepare__ method customizes the dictionary that's used for a class's initial namespace. This is used to pre-populate dictionary values or customize the dictionary type ( silly example ).

The __instancecheck__ and __subclasscheck__ methods override the functionality of isinstance and issubclass . Python's ABCs use these to practice goose typing ( duck typing while type checking).

The __init_subclass__ method allows classes to hook into subclass initialization ( example ). Classes also have a __subclasses__ method (on their metaclass ) but it's not typically overridden.

Python calls __mro_entries__ during class inheritance for any base classes that are not actually classes. The typing.NamedTuple function uses this to pretend it's a class ( see here ).

The __class_getitem__ method allows a class to be subscriptable ( without its metaclass needing a __getitem__ method). This is typically used for enabling fancy type annotations (e.g. list[int] ).

Descriptors 🏷️

Descriptors are objects that, when attached to a class, can hook into the access of the attribute name they're attached to on that class.

The descriptor protocol is mostly a feature that exists to make Python's property decorator work, though it is also used by a number of third-party libraries.

Implementing a low-level memory array? You need Python's buffer protocol .

The __release_buffer__ method is called when the buffer that's returned from __buffer__ is deleted.

Python's buffer protocol is typically implemented in C, since it's meant for low level objects.

Asynchronous operations 🤹

Want to implement an asynchronous context manager? You need these dunder methods:

__aenter__ : just like __enter__ , but it returns an awaitable object
__aexit__ : just like __exit__ , but it returns an awaitable object

Need to support asynchronous iteration? You need these dunder methods:

__aiter__ : must return an asynchronous iterator
__anext__ : like __next__ or non-async iterators, but this must return an awaitable object and this should raise StopAsyncIteration instead of StopIteration

Need to make your own awaitable object? You need this dunder method:

__await__ : returns an iterator

I have little experience with custom asynchronous objects, so look elsewhere for more details.

Construction and finalizing 🏭

The last few dunder methods are related to object creation and destruction.

Calling a class returns a new class instance thanks to the __new__ method. The __new__ method is Python's constructor method , though unlike constructors in many programming languages, you should almost never define your own __new__ method. To control object creation, prefer the initializer ( __init__ ), not the constructor ( __new__ ). Here's an odd __new__ example .

You could think of __del__ as a "destructor" method, though it's actually called the finalizer method . Just before an object is deleted, its __del__ method is called ( example ). Files implement a __del__ method that closes the file and any binary file buffer that it may be linked to.

Library-specific dunder methods 🧰

Some standard library modules define custom dunder methods that aren't used anywhere else:

dataclasses support a __post_init__ method
abc.ABC classes have a __subclasshook__ method which abc.ABCMeta calls in its __subclasscheck__ method (more in goose typing )
Path-like objects have a __fspath__ method, which returns the file path as a string
Python's copy module will use the __copy__ and __deepcopy__ methods if present
Pickling relies on __getnewargs_ex__ or __getargs__ , though __getstate__ and __setstate__ can customize further and __reduce__ or __reduce_ex__ are even lower-level
sys.getsizeof relies on the __sizeof__ method to get an object's size (in bytes)

Dunder attributes 📇

In addition to dunder methods, Python has many non-method dunder attributes .

Here are some of the more common dunder attributes you'll see:

__name__ : name of a function, classes, or module
__module__ : module name for a function or class
__doc__ : docstring for a function, class, or module
__class__ : an object's class (call Python's type function instead)
__dict__ : most objects store their attributes here (see where are attributes stored? )
__slots__ : classes using this are more memory efficient than classes using __dict__
__match_args__ : classes can define a tuple noting the significance of positional attributes when the class is used in structural pattern matching ( match - case )
__mro__ : a class's method resolution order used when for attribute lookups and super() calls
__bases__ : the direct parent classes of a class
__file__ : the file that defined the module object (though not always present!)
__wrapped__ : functions decorated with functools.wraps use this to point to the original function
__version__ : commonly used for noting the version of a package
__all__ : modules can use this to customize the behavior of from my_module import *
__debug__ : running Python with -O sets this to False and disables Python's assert statements

Those are only the more commonly seen dunder attributes. Here are some more:

Functions have __defaults__ , __kwdefaults__ , __code__ , __globals__ , and __closure__
Both functions and classes have __qualname__ , __annotations__ , and __type_params__
Instance methods have __func__ and __self__
Modules may also have __loader__ , __package__ , __spec__ , and __cached__ attributes
Packages have a __path__ attribute
Exceptions have __traceback__ , __notes__ , __context__ , __cause__ , and __suppress_context__
Descriptors use __objclass__
Metaclasses use __classcell__
Python's weakref module uses __weakref__
Generic aliases have __origin__ , __args__ , __parameters__ , and __unpacked__
The sys module has __stdout__ and __stderr__ which point to the original stdout and stderr versions

Additionally, these dunder attributes are used by various standard library modules: __covariant__ , __contravariant__ , __infer_variance__ , __bound__ , __constraints__ . And Python includes a built-in __import__ function which you're not supposed to use ( importlib.import_module is preferred) and CPython has a __builtins__ variable that points to the builtins module (but this is an implementation detail and builtins should be explicitly imported when needed instead). Also importing from the __future__ module can enable specific Python feature flags and Python will look for a __main__ module within packages to make them runnable as CLI scripts.

And that's just most of the dunder attribute names you'll find floating around in Python. 😵

Every dunder method: a cheat sheet ⭐

This is every Python dunder method organized in categories and ordered very roughly by the most commonly seen methods first. Some caveats are noted below.

The above table has a slight but consistent untruth . Most of these dunder methods are not actually called on an object directly but are instead called on the type of that object: type(x).__add__(x, y) instead of x.__add__(y) . This distinction mostly matters with metaclass methods.

I've also purposely excluded library-specific dunder methods (like __post_init__ ) and dunder methods you're unlikely to ever define (like __subclasses__ ). See those below.

So, Python includes 103 "normal" dunder methods, 12 library-specific dunder methods, and at least 52 other dunder attributes of various types. That's over 150 unique __dunder__ names! I do not recommend memorizing these: let Python do its job and look up the dunder method or attribute that you need to implement/find whenever you need it.

Keep in mind that you're not meant to invent your own dunder methods . Sometimes you'll see third-party libraries that do invent their own dunder method, but this isn't encouraged and it can be quite confusing for users who run across such methods and assume they're " real " dunder methods.

A Python tip every week

Need to fill-in gaps in your Python skills?

Need to fill-in gaps in your Python skills ? I send weekly emails designed to do just that.

Improve the accuracy of output from inspect.getsource for expression-based code objects

This idea proposes adding the end line number, as well as the start and end column offsets, of the lines that define a function scope, to the line table entry of the RESUME bytecode that signals the start of the scope. This information will be used to improve the accuracy of the source code returned by inspect.getsource in order to reduce the complexity involved in runtime profiling and code transformations.

The primary motivation of this idea is to improve the end user’s ability to obtain the precise defining source code of a given code object at runtime.

Python currently offers the co_firstlineno attribute to a code object so that runtime inspection tools such as inspect.getsource can extract the code block that defines a given code object by tracking indentations with a tokenizer.

However, for code objects not defined by a code block, such as ones defined by a lambda or a generator expression, having just the co_firstlineno attribute is not enough to determine where it is defined because as an expression it can be defined anywhere in a line, possibly with multiple other lambda and/or generator expressions on the same lines.

Consider the following code of a runtime call profiler:

It currently produces the following output:

From the output with the same entire for statement being returned as the source code for all 3 calls that take place, it is impossible to determine which of the 3 reported calls corresponds to which of the three functions in the returned source code, including two lambda functions and one generator expression.

With the implementation of PEP-657 , each bytecode is now accompanied with a fine-grained position, including both the start and end column offsets, exposed via a public code object method, co_positions , which in theory can be used to aid the accuracy of inspect.getsource ’s output.

The problem is, however, that the start and end of a lambda function or a generator expression do not correspond to a bytecode with a meaningful position. For example, using co_positions on a lambda function’s code object:

produces the output:

From the output, all that can be determined is that the definition of the code object starts somewhere on line 2, and that the portion of its body that produces bytecode starts at column 4 of line 3 and ends at column 5 of line 3. The end line number of 4, the start colum of 4 and the end column of 1 are all missing from the output. It therefore remains tricky for inspect.getsource to extract the more precise source code that defines the lambda function, in this case:

As the output of dis.dis in the last code example shows, the position information of the RESUME bytecode, except its start line number, is currently meaningless, where the end line number is always equal to the start line number, the start and end column offsets always 0.

It would therefore be reasonable to store the precise position of the source code that covers the definition of the code object, in the line table entry for the RESUME bytecode, a no-op instruction meant to signal the start of a function when its arg is 0. That RESUME is always there at or near the start of a scope also means that inspect.getsource can efficiently obtain the position information of the scope in constant time without having to iterate through all the positions generated from the co_positions method.

Since the additional position information is already available internally in the AST for the compiler core, all that needs to be done is for the compiler to pass the additional position information when entering a scope , and to replace the hard-coded position of (lineno, lineno, 0, 0) with (lineno, end_lineno, col_offset, end_col_offset) for the RESUME instruction.

Specification

With the proposed change, the line table entry for the RESUME bytecode when its arg is 0, will cover the entire source code that defines the scope, except when the scope is of a module, in which case the entry would remain (1, 1, 0, 0) as it is now.

The position covering the entire source code that defines the scope will come from lineno , end_lineno , col_offset and end_col_offset of the statement and expression AST structs, after applicable decorators are taken into account for the start line number as they are now.

Backwards Compatibility

Code analysis and coverage tools that currently presume a no-op bytecode to have zero width in the line table should be refactored to special-case the RESUME instruction to take its position information with the new meaning.

Reference Implementation

A reference implementation can be found in the implementation fork, where the supporting functions for inspect.getsource are refactored to make use of the new semantics.

Performance Implications

The compiler will spend minimally more time to pass the 3 integers end_lineno , col_offset and end_col_offset when entering a scope. The efficiency gain for inspect.getsource from not having to reparse the source code to find the code block given a start line number should be significant.

Doesn’t code.co_positions() do what you need? See Lib/dis.py for an example how to use it.

Ah very cool. Thank you. Not sure how I missed or forgot about this exciting blurb about co_positions in the What’s New of Python 3.11.

This means that inspect.getsource can be trivially reimplemented with co_positions instead of the current approach of re-parsing the lines with a tokenizer, only to end up with inaccurate results for expression-based code objects. Will attempt at a reimplementation now. Thanks again.

Well, at first glance co_positions has all I needed, but as soon as I started experimenting with it I realized that its bytecode-driven output is missing crucial positions for what I would like to achieve, namely to generically obtain from any given code object its defining source code that can be directly recompiled back into the object that contains the same code object.

For example, given an assignment from a lambda expression:

My goal would be to extract:

that can be directly recompiled back into a function object containing the same code object.

However, the above code outputs:

Note the lack of a proper col_offset to pinpoint the start of the lambda expression, as well as the lack of end_lineno and end_col_offset to identify the end position.

Would it be reasonably feasible to include such information in the RESUME no-op instruction that apparently currently always has the same lineno and end_lineno , and always has 0s for both col_offset and end_col_offset ? In the example above, co_positions() should ideally yield (2, 4, 4, 1) for the RESUME bytecode.

Also note that if we are to enable inspect.getsource to trivially apply the line numbers and column offsets yielded by co_positions() to the source lines without any re-parsing efforts, it may be necessary to include comments in the AST to handle the corner case where the code block ends with a comment at the same indentation level:

This outputs:

Note the maximum end_lineno of all bytecodes being 5 rather than 6 here.

But adding a comment node to AST may be an overkill just to handle such a corner case. Perhaps inspect.getsource should continue to use a tokenizer to re-parse the source unless it’s determined that the code object is derived from an expression.

Adding more meaningful positional information to the RESUME bytecode as proposed in my previous post on the other hand feels like a much bigger bang for the buck.

You can get full location info from the AST.

Making resume appear as if it’s covering the whole function can confuse things like code coverage tools.

By AST do you mean the ast module? If so, the ast module can build an AST from source code but not from a live code object. Or if you mean AST for the compiler core, it just doesn’t currently expose the precise position of a code object to end users, which is the point of ths proposal.

My goal is to make inspect.getsource return a compilable piece of source code from a given code object for runtime reporting and transformation.

I understand that with the current defintion, a no-op bytecode such as RESUME is not supposed to cover any width. But if we document the change, code coverage tools can special-case RESUME accordingly.

RESUME is IMHO the best place to store such info, being a no-op and always at the top of a scope. Or would you help recommend a better approach to exposing the precise position of an expression-based code object?

I’ve gone ahead to prototype the necessary changes to make RESUME include precise positions and modified inspect.getsource -related functions accordingly (by applying the precise position from RESUME directly to source lines). Works well in my limited testing so far, where:

would output:

Anyone interested is welcome to try it out at:

The diff to f4cc77d494ee0e10ed84ce369f0910c70a2f6d44 is as follows:

I mean the AST of the source code. For your use case, you must have access to the source code, so you can calculate its AST.

Again, the issue is, given a code object at runtime, how do you extract its defining source code?

AST from source code doesn’t help because it bears no direct relationship to a live code object.

You can’t because it may not even have “defining source code”. Consider a function loaded from a .pyc file that has no corresponding .py file. Or just a function defined in the REPL. The alternative answer is inspect.getsource - in what situation does that not give an answer where it could do?

so, you want the code object to give you the location but presumably you have the source code. Otherwise how would you do this?

See this slightly modified example:

This prints

First one is correct, second one is completely wrong, and third includes extra code that should be stripped away.

@blhsing is talking about improving getsource to work more reliably.

The possibility of source-less code objects doesn’t stop the stdlib from offering inspect.getsource . The point of this proposal is to improve the accuracy of the output of inspect.getsource for the vast majority of cases where the source code is available.

I think my original post can use an example for better clarity indeed. Please see my updated original post.

I need a way for inspect.getsource to give me the exact source code of a given live code object. I don’t need it to give me an entire statement that contains many other tokens when it could really just give me the very expression that defines the code object. Please also see my updated original post for a better example.

Also note that with my prototype , the example code in the OP would then output:

You’re not responding to my question: why can’t you get exact locations from the AST?

How would you relate AST node and code object if you don’t have the exact location of the code object? Guesstimating is of-course possible, but is probably going to always have edge cases.

You have the line number.

Yes, but the line number is not enough information in cases of nested (or even just adjacent) codeblocks, like lambdas and/or generator expressions in the same line. That is the entire point of this suggestion, see the updated examples in OP.

You also have location information for what’s inside the body of the lambdas (which can be matched to locations of instructions in the code object), as well as the location of the lambda expression itself. For example:

In that example you are again working from the string source code. The question is can you get that information from the code object (or from the function object containing it).

comprehension assignment python

IMAGES

VIDEO

COMMENTS

Python Enhancement Proposals

PEP 572 – Assignment Expressions

Syntax and semantics

Examples from the Python standard library

Rejected alternative proposals

Frequently Raised Objections

Appendix A: Tim Peters’s findings

Python Tutorial

COLOR PICKER

Report Error

Top Tutorials

Popular Tutorials

Python Flow Control

Python Functions

Python Datatypes

Python Files

Python Object & Class

Python Advanced Topics

Python Date and time

Python Tutorials

Python List Comprehension

More on Python List Comprehension

Table of Contents

Video: Python List, Set & Dictionary Comprehension

Related Tutorials

Comprehensive Guide to Python List Comprehensions

Introduction

List Comprehensions

What happens under the hood

Nested list comprehensions

Nested loops in list comprehensions

Multi-line list comprehensions

List comprehensions with conditionals (if-else)

List comprehensions with walrus operator

List comprehensions Vs map and filter

Set Comprehensions

Dictionary Comprehensions

Generator Expressions

Generator Expressions Vs. List Comprehensions

Do we have tuple comprehensions?

Chetan Ambi

List comprehensions in Python

Basics of list comprehensions

Nested list comprehensions

List comprehensions with conditional expressions (like if else )

List comprehensions with zip() , enumerate()

Related Categories

Comprehensions in Python

List Comprehensions

Please Login to comment...

Improve your Coding Skills with Practice

What kind of Experience do you want to share?

Python List Comprehension

What is List Comprehension?

Basic Example

More Examples

List Comprehension with if Clause

Nested List Comprehensions

List Comprehension vs map() + lambda

List Comprehension vs filter() + lambda

List Comprehensions and Variable Scope

Every dunder method in Python

The 3 essential dunder methods 🔑

Equality and hashability 🟰

Orderability ⚖️

Type conversions and string formatting ⚗️

Context managers 🚪

Containers and collections 🗃️

Callability ☎️

Arithmetic operators ➗

In-place arithmetic operations ♻️

Built-in math functions 🧮

Attribute access 📜

Metaprogramming 🪄

Descriptors 🏷️

Asynchronous operations 🤹

Construction and finalizing 🏭

Library-specific dunder methods 🧰

Dunder attributes 📇

Every dunder method: a cheat sheet ⭐