python assignment not allowed to field in protocol message object

Search code, repositories, users, issues, pull requests...

Provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

• Notifications

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assignment not allowed to repeated field "conversions" in protocol message object #559

panukuulu commented Jan 11, 2022

sakkammadam commented Jan 11, 2022 • edited

Sorry, something went wrong.

panukuulu commented Jan 12, 2022

BenRKarl commented Jan 12, 2022

No branches or pull requests

dynamic generation of proto objects and the error: Assignment not allowed to composite field

Tom Lichtenberg

Anton Danilov

Python Generated Code Guide

Any differences between proto2 and proto3 generated code are highlighted - note that these differences are in the generated code as described in this document, not the base message classes/interfaces, which are the same in both versions. You should read the proto2 language guide and/or proto3 language guide before reading this document.

The Python Protocol Buffers implementation is a little different from C++ and Java. In Python, the compiler only outputs code to build descriptors for the generated classes, and a Python metaclass does the real work. This document describes what you get after the metaclass has been applied.

Compiler Invocation

The protocol buffer compiler produces Python output when invoked with the --python_out= command-line flag. The parameter to the --python_out= option is the directory where you want the compiler to write your Python output. The compiler creates a .py file for each .proto file input. The names of the output files are computed by taking the name of the .proto file and making two changes:

• The extension ( .proto ) is replaced with _pb2.py .
• The proto path (specified with the --proto_path= or -I command-line flag) is replaced with the output path (specified with the --python_out= flag).

So, for example, let’s say you invoke the compiler as follows:

The compiler will read the files src/foo.proto and src/bar/baz.proto and produce two output files: build/gen/foo_pb2.py and build/gen/bar/baz_pb2.py . The compiler will automatically create the directory build/gen/bar if necessary, but it will not create build or build/gen ; they must already exist.

Protoc can generate Python stubs ( .pyi ) using the --pyi_out parameter.

Note that if the .proto file or its path contains any characters which cannot be used in Python module names (for example, hyphens), they will be replaced with underscores. So, the file foo-bar.proto becomes the Python file foo_bar_pb2.py .

The Python code generated by the protocol buffer compiler is completely unaffected by the package name defined in the .proto file. Instead, Python packages are identified by directory structure.

Given a simple message declaration:

The protocol buffer compiler generates a class called Foo , which subclasses google.protobuf.Message . The class is a concrete class; no abstract methods are left unimplemented. Unlike C++ and Java, Python generated code is unaffected by the optimize_for option in the .proto file; in effect, all Python code is optimized for code size.

If the message’s name is a Python keyword, then its class will only be accessible via getattr() , as described in the Names which conflict with Python keywords section.

You should not create your own Foo subclasses. Generated classes are not designed for subclassing and may lead to "fragile base class" problems. Besides, implementation inheritance is bad design.

Python message classes have no particular public members other than those defined by the Message interface and those generated for nested fields, messages, and enum types (described below). Message provides methods you can use to check, manipulate, read, or write the entire message, including parsing from and serializing to binary strings. In addition to these methods, the Foo class defines the following static methods:

• FromString(s) : Returns a new message instance deserialized from the given string.

Note that you can also use the text_format module to work with protocol messages in text format: for example, the Merge() method lets you merge an ASCII representation of a message into an existing message.

Nested Types

A message can be declared inside another message. For example:

In this case, the Bar class is declared as a static member of Foo , so you can refer to it as Foo.Bar .

Well Known Types

Protocol buffers provides a number of well-known types that you can use in your .proto files along with your own message types. Some WKT messages have special methods in addition to the usual protocol buffer message methods, as they subclass both google.protobuf.Message and a WKT class.

For Any messages, you can call Pack() to pack a specified message into the current Any message, or Unpack() to unpack the current Any message into a specified message. For example:

Unpack() also checks the descriptor of the passed-in message object against the stored one and returns False if they don’t match and does not attempt any unpacking; True otherwise.

You can also call the Is() method to check if the Any message represents the given protocol buffer type. For example:

Use the TypeName() method to retrieve the protobuf type name of an inner message.

Timestamp messages can be converted to/from RFC 3339 date string format (JSON string) using the ToJsonString() / FromJsonString() methods. For example:

You can also call GetCurrentTime() to fill the Timestamp message with current time:

To convert between other time units since epoch, you can call ToNanoseconds(), FromNanoseconds(), ToMicroseconds(), FromMicroseconds(), ToMilliseconds(), FromMilliseconds(), ToSeconds() , or FromSeconds() . The generated code also has ToDatetime() and FromDatetime() methods to convert between Python datetime objects and Timestamps. For example:

Duration messages have the same methods as Timestamp to convert between JSON string and other time units. To convert between timedelta and Duration, you can call ToTimedelta() or FromTimedelta . For example:

FieldMask messages can be converted to/from JSON string using the ToJsonString() / FromJsonString() methods. In addition, a FieldMask message has the following methods:

• IsValidForDescriptor(message_descriptor) : Checks whether the FieldMask is valid for Message Descriptor.
• AllFieldsFromDescriptor(message_descriptor) : Gets all direct fields of Message Descriptor to FieldMask.
• CanonicalFormFromMask(mask) : Converts a FieldMask to the canonical form.
• Union(mask1, mask2) : Merges two FieldMasks into this FieldMask.
• Intersect(mask1, mask2) : Intersects two FieldMasks into this FieldMask.
• MergeMessage(source, destination, replace_message_field=False, replace_repeated_field=False) : Merges fields specified in FieldMask from source to destination.

Struct messages let you get and set the items directly. For example:

To get or create a list/struct, you can call get_or_create_list() / get_or_create_struct() . For example:

A ListValue message acts like a Python sequence that lets you do the following:

To add a ListValue/Struct, call add_list() / add_struct() . For example:

For each field in a message type, the corresponding class has a property with the same name as the field. How you can manipulate the property depends on its type.

As well as a property, the compiler generates an integer constant for each field containing its field number. The constant name is the field name converted to upper-case followed by _FIELD_NUMBER . For example, given the field optional int32 foo_bar = 5; , the compiler will generate the constant FOO_BAR_FIELD_NUMBER = 5 .

If the field’s name is a Python keyword, then its property will only be accessible via getattr() and setattr() , as described in the Names which conflict with Python keywords section.

Singular Fields (proto2)

If you have a singular (optional or required) field foo of any non-message type, you can manipulate the field foo as if it were a regular field. For example, if foo ’s type is int32 , you can say:

Note that setting foo to a value of the wrong type will raise a TypeError .

If foo is read when it is not set, its value is the default value for that field. To check if foo is set, or to clear the value of foo , you must call the HasField() or ClearField() methods of the Message interface. For example:

Singular Fields (proto3)

If you have a singular field foo of any non-message type, you can manipulate the field foo as if it were a regular field. For example, if foo ’s type is int32 , you can say:

If foo is read when it is not set, its value is the default value for that field. To clear the value of foo and reset it to the default value for its type, you call the ClearField() method of the Message interface. For example:

Singular Message Fields

Message types work slightly differently. You cannot assign a value to an embedded message field. Instead, assigning a value to any field within the child message implies setting the message field in the parent. You can also use the parent message’s HasField() method to check if a message type field value has been set.

So, for example, let’s say you have the following .proto definition:

You cannot do the following:

Instead, to set bar , you simply assign a value directly to a field within bar , and - presto! - foo has a bar field:

Similarly, you can set bar using the Message interface’s CopyFrom() method. This copies all the values from another message of the same type as bar .

Note that simply reading a field inside bar does not set the field:

If you need the "has" bit on a message that does not have any fields you can or want to set, you may use the SetInParent() method.

Repeated Fields

Repeated fields are represented as an object that acts like a Python sequence. As with embedded messages, you cannot assign the field directly, but you can manipulate it. For example, given this message definition:

You can do the following:

The ClearField() method of the Message interface works in addition to using Python del .

When using the index to retrieve a value, you can use negative numbers, such as using -1 to retrieve the last element in the list. If your index goes out of bounds, you’ll get an IndexError: list index out of range .

Repeated Message Fields

Repeated message fields work similar to repeated scalar fields. However, the corresponding Python object also has an add() method that creates a new message object, appends it to the list, and returns it for the caller to fill in. Also, the object’s append() method makes a copy of the given message and appends that copy to the list. This is done so that messages are always owned by the parent message to avoid circular references and other confusion that can happen when a mutable data structure has multiple owners. Similarly, the object’s extend() method appends an entire list of messages, but makes a copy of every message in the list.

For example, given this message definition:

Unlike repeated scalar fields, repeated message fields don’t support item assignment (i.e. __setitem__ ). For example:

Groups (proto2)

Note that groups are deprecated and should not be used when creating new message types – use nested message types instead.

A group combines a nested message type and a field into a single declaration, and uses a different wire format for the message. The generated message has the same name as the group. The generated field’s name is the lowercased name of the group.

For example, except for wire format, the following two message definitions are equivalent:

A group is either required , optional , or repeated . A required or optional group is manipulated using the same API as a regular singular message field. A repeated group is manipulated using the same API as a regular repeated message field.

For example, given the above SearchResponse definition, you can do the following:

Given this message definition:

The generated Python API for the map field is just like a Python dict :

As with embedded message fields , messages cannot be directly assigned into a map value. Instead, to add a message as a map value you reference an undefined key , which constructs and returns a new submessage:

You can find out more about undefined keys in the next section.

Referencing undefined keys

The semantics of Protocol Buffer maps behave slightly differently to Python dict s when it comes to undefined keys. In a regular Python dict , referencing an undefined key raises a KeyError exception:

However, in Protocol Buffers maps, referencing an undefined key creates the key in the map with a zero/false/empty value. This behavior is more like the Python standard library defaultdict .

This behavior is especially convenient for maps with message type values, because you can directly update the fields of the returned message.

Note that even if you don’t assign any values to message fields, the submessage is still created in the map:

This is different from regular embedded message fields , where the message itself is only created once you assign a value to one of its fields.

As it may not be immediately obvious to anyone reading your code that m.message_map[10] alone, for example, may create a submessage, we also provide a get_or_create() method that does the same thing but whose name makes the possible message creation more explicit:

Enumerations

In Python, enums are just integers. A set of integral constants are defined corresponding to the enum’s defined values. For example, given:

The constants VALUE_A , VALUE_B , and VALUE_C are defined with values 0, 5, and 1234, respectively. You can access SomeEnum if desired. If an enum is defined in the outer scope, the values are module constants; if it is defined within a message (like above), they become static members of that message class.

For example, you can access the values in the three following ways for the following enum in a proto:

An enum field works just like a scalar field.

If the enum’s name (or an enum value) is a Python keyword, then its object (or the enum value’s property) will only be accessible via getattr() , as described in the Names which conflict with Python keywords section.

The values you can set in an enum depend on your protocol buffers version:

• In proto2 , an enum cannot contain a numeric value other than those defined for the enum type. If you assign a value that is not in the enum, the generated code will throw an exception.
• proto3 uses open enum semantics: enum fields can contain any int32 value.

Enums have a number of utility methods for getting field names from values and vice versa, lists of fields, and so on - these are defined in enum_type_wrapper.EnumTypeWrapper (the base class for generated enum classes). So, for example, if you have the following standalone enum in myproto.proto :

…you can do this:

For an enum declared within a protocol message, such as Foo above, the syntax is similar:

If multiple enum constants have the same value (aliases), the first constant defined is returned.

In the above example, myproto_pb2.SomeEnum.Name(5) returns "VALUE_B" .

Given a message with a oneof:

The Python class corresponding to Foo will have properties called name and serial_number just like regular fields . However, unlike regular fields, at most one of the fields in a oneof can be set at a time, which is ensured by the runtime. For example:

The message class also has a WhichOneof method that lets you find out which field (if any) in the oneof has been set. This method returns the name of the field that is set, or None if nothing has been set:

HasField and ClearField also accept oneof names in addition to field names:

Note that calling ClearField on a oneof just clears the currently set field.

Names which conflict with Python keywords

If the name of a message, field, enum, or enum value is a Python keyword , then the name of its corresponding class or property will be the same, but you’ll only be able to access it using Python’s getattr() and setattr() built-in functions, and not via Python’s normal attribute reference syntax (i.e. the dot operator).

For example, if you have the following .proto definition:

You would access those fields like this:

By contrast, trying to use obj.attr syntax to access these fields results in Python raising syntax errors when parsing your code:

Extensions (proto2 only)

Given a message with an extension range:

The Python class corresponding to Foo will have a member called Extensions , which is a dictionary mapping extension identifiers to their current values.

Given an extension definition:

The protocol buffer compiler generates an "extension identifier" called bar . The identifier acts as a key to the Extensions dictionary. The result of looking up a value in this dictionary is exactly the same as if you accessed a normal field of the same type. So, given the above example, you could do:

Note that you need to specify the extension identifier constant, not just a string name: this is because it’s possible for multiple extensions with the same name to be specified in different scopes.

Analogous to normal fields, Extensions[...] returns a message object for singular messages and a sequence for repeated fields.

The Message interface’s HasField() and ClearField() methods do not work with extensions; you must use HasExtension() and ClearExtension() instead. To use the HasExtension() and ClearExtension() methods, pass in the field_descriptor for the extension you are checking for the existence of.

If the .proto file contains the following line:

Then the protocol buffer compiler will generate code based on the service definitions found in the file as described in this section. However, the generated code may be undesirable as it is not tied to any particular RPC system, and thus requires more levels of indirection that code tailored to one system. If you do NOT want this code to be generated, add this line to the file:

If neither of the above lines are given, the option defaults to false , as generic services are deprecated. (Note that prior to 2.4.0, the option defaults to true )

RPC systems based on .proto -language service definitions should provide plugins to generate code appropriate for the system. These plugins are likely to require that abstract services are disabled, so that they can generate their own classes of the same names. Plugins are new in version 2.3.0 (January 2010).

The remainder of this section describes what the protocol buffer compiler generates when abstract services are enabled.

Given a service definition:

The protocol buffer compiler will generate a class Foo to represent this service. Foo will have a method for each method defined in the service definition. In this case, the method Bar is defined as:

The parameters are equivalent to the parameters of Service.CallMethod() , except that the method_descriptor argument is implied.

These generated methods are intended to be overridden by subclasses. The default implementations simply call controller.SetFailed() with an error message indicating that the method is unimplemented, then invoke the done callback. When implementing your own service, you must subclass this generated service and implement its methods as appropriate.

Foo subclasses the Service interface. The protocol buffer compiler automatically generates implementations of the methods of Service as follows:

• GetDescriptor : Returns the service’s ServiceDescriptor .
• CallMethod : Determines which method is being called based on the provided method descriptor and calls it directly.
• GetRequestClass and GetResponseClass : Returns the class of the request or response of the correct type for the given method.

The protocol buffer compiler also generates a "stub" implementation of every service interface, which is used by clients wishing to send requests to servers implementing the service. For the Foo service (above), the stub implementation Foo_Stub will be defined.

Foo_Stub is a subclass of Foo . Its constructor takes an RpcChannel as a parameter. The stub then implements each of the service’s methods by calling the channel’s CallMethod() method.

The Protocol Buffer library does not include an RPC implementation. However, it includes all of the tools you need to hook up a generated service class to any arbitrary RPC implementation of your choice. You need only provide implementations of RpcChannel and RpcController .

Plugin Insertion Points

Code generator plugins which want to extend the output of the Python code generator may insert code of the following types using the given insertion point names.

• imports : Import statements.
• module_scope : Top-level declarations.

Sharing Messages Between Python and C++

Prior to the 4.21.0 version of the Protobuf Python API, Python apps could share messages with C++ using a native extension. Starting in the 4.21.0 API version, sharing messages between Python and C++ is not supported by the default install. To enable this capability when working with the 4.x and later versions of the Protobuf Python API, define the environment variable, PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp , and ensure that the Python/C++ extension is installed.