python assignment not allowed to field in protocol message object

dynamic generation of proto objects and the error: Assignment not allowed to composite field

Tom Lichtenberg

Anton Danilov

google.protobuf.message ¶

Contains an abstract base class for protocol messages.

Exception raised when deserializing messages.

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

Exception raised when serializing messages.

Base error type for this module.

Abstract base class for protocol messages.

Protocol message classes are almost always generated by the protocol compiler. These generated types subclass Message and implement the methods shown below.

Returns the serialized size of this message.

Recursively calls ByteSize() on all contained messages.

The number of bytes required to serialize this message.

Clears all data that was set in the message.

Clears the contents of a given extension.

extension_handle – The handle for the extension to clear.

Clears the contents of a given field.

Inside a oneof group, clears the field set. If the name neither refers to a defined field or oneof group, ValueError is raised.

field_name ( str ) – The name of the field to check for presence.

ValueError – if the field_name is not a member of this message.

Copies the content of the specified message into the current message.

The method clears the current message and then merges the specified message using MergeFrom.

other_msg ( Message ) – A message to copy into the current one.

The google.protobuf.descriptor.Descriptor for this message type.

Clears all fields in the UnknownFieldSet .

This operation is recursive for nested message.

Checks if a certain extension is present for this message.

Extensions are retrieved using the Extensions mapping (if present).

extension_handle – The handle for the extension to check.

Whether the extension is present for this message.

KeyError – if the extension is repeated. Similar to repeated fields, there is no separate notion of presence: a “not present” repeated extension is an empty list.

Checks if a certain field is set for the message.

For a oneof group, checks if any field inside is set. Note that if the field_name is not defined in the message descriptor, ValueError will be raised.

Whether a value has been set for the named field.

Checks if the message is initialized.

The method returns True if the message is initialized (i.e. all of its required fields are set).

Returns a list of (FieldDescriptor, value) tuples for present fields.

A message field is non-empty if HasField() would return true. A singular primitive field is non-empty if HasField() would return true in proto2 or it is non zero in proto3. A repeated field is non-empty if it contains at least one element. The fields are ordered by field number.

field descriptors and values for all fields in the message which are not empty. The values vary by field type.

list [ tuple ( FieldDescriptor , value)]

Merges the contents of the specified message into current message.

This method merges the contents of the specified message into the current message. Singular fields that are set in the specified message overwrite the corresponding fields in the current message. Repeated fields are appended. Singular sub-messages and groups are recursively merged.

other_msg ( Message ) – A message to merge into the current message.

Merges serialized protocol buffer data into this message.

When we find a field in serialized that is already present in this message:

If it’s a “repeated” field, we append to the end of our list.

Else, if it’s a scalar, we overwrite our field.

Else, (it’s a nonrepeated composite), we recursively merge into the existing composite.

serialized ( bytes ) – Any object that allows us to call memoryview(serialized) to access a string of bytes using the buffer interface.

The number of bytes read from serialized . For non-group messages, this will always be len(serialized) , but for messages which are actually groups, this will generally be less than len(serialized) , since we must stop when we reach an END_GROUP tag. Note that if we do stop because of an END_GROUP tag, the number of bytes returned does not include the bytes for the END_GROUP tag information.

DecodeError – if the input cannot be parsed.

Parse serialized protocol buffer data into this message.

Like MergeFromString() , except we clear the object first.

message.DecodeError if the input cannot be parsed. –

Serializes the protocol message to a binary string.

This method is similar to SerializeToString but doesn’t check if the message is initialized.

deterministic ( bool ) – If true, requests deterministic serialization of the protobuf, with predictable ordering of map keys.

A serialized representation of the partial message.

A binary string representation of the message if all of the required fields in the message are set (i.e. the message is initialized).

EncodeError – if the message isn’t initialized (see IsInitialized() ).

Mark this as present in the parent.

This normally happens automatically when you assign a field of a sub-message, but sometimes you want to make the sub-message present while keeping it empty. If you find yourself using this, you may want to reconsider your design.

Returns the UnknownFieldSet.

The unknown fields stored in this message.

UnknownFieldSet

Returns the name of the field that is set inside a oneof group.

If no field is set, returns None.

oneof_group ( str ) – the name of the oneof group to check.

The name of the group that is set, or None.

str or None

ValueError – no group with the given name exists

Table of Contents

• google.protobuf
• google.protobuf.any_pb2
• google.protobuf.descriptor
• google.protobuf.descriptor_database
• google.protobuf.descriptor_pb2
• google.protobuf.descriptor_pool
• google.protobuf.duration_pb2
• google.protobuf.empty_pb2
• google.protobuf.field_mask_pb2
• google.protobuf.internal.containers
• google.protobuf.json_format
• google.protobuf.message
• google.protobuf.message_factory
• google.protobuf.proto_builder
• google.protobuf.reflection
• google.protobuf.service
• google.protobuf.service_reflection
• google.protobuf.struct_pb2
• google.protobuf.symbol_database
• google.protobuf.text_encoding
• google.protobuf.text_format
• google.protobuf.timestamp_pb2
• google.protobuf.type_pb2
• google.protobuf.unknown_fields
• google.protobuf.wrappers_pb2

Related Topics

• Previous: google.protobuf.json_format
• Next: google.protobuf.message_factory
• Show Source

Protocol Buffer Basics: Python

This tutorial provides a basic Python programmer’s introduction to working with protocol buffers. By walking through creating a simple example application, it shows you how to

• Define message formats in a .proto file.
• Use the protocol buffer compiler.
• Use the Python protocol buffer API to write and read messages.

This isn’t a comprehensive guide to using protocol buffers in Python. For more detailed reference information, see the Protocol Buffer Language Guide (proto2) , the Protocol Buffer Language Guide (proto3) , the Python API Reference , the Python Generated Code Guide , and the Encoding Reference .

The Problem Domain

The example we’re going to use is a very simple “address book” application that can read and write people’s contact details to and from a file. Each person in the address book has a name, an ID, an email address, and a contact phone number.

How do you serialize and retrieve structured data like this? There are a few ways to solve this problem:

• Use Python pickling. This is the default approach since it’s built into the language, but it doesn’t deal well with schema evolution, and also doesn’t work very well if you need to share data with applications written in C++ or Java.
• You can invent an ad-hoc way to encode the data items into a single string – such as encoding 4 ints as “12:3:-23:67”. This is a simple and flexible approach, although it does require writing one-off encoding and parsing code, and the parsing imposes a small run-time cost. This works best for encoding very simple data.
• Serialize the data to XML. This approach can be very attractive since XML is (sort of) human readable and there are binding libraries for lots of languages. This can be a good choice if you want to share data with other applications/projects. However, XML is notoriously space intensive, and encoding/decoding it can impose a huge performance penalty on applications. Also, navigating an XML DOM tree is considerably more complicated than navigating simple fields in a class normally would be.

Instead of these options, you can use protocol buffers. Protocol buffers are the flexible, efficient, automated solution to solve exactly this problem. With protocol buffers, you write a .proto description of the data structure you wish to store. From that, the protocol buffer compiler creates a class that implements automatic encoding and parsing of the protocol buffer data with an efficient binary format. The generated class provides getters and setters for the fields that make up a protocol buffer and takes care of the details of reading and writing the protocol buffer as a unit. Importantly, the protocol buffer format supports the idea of extending the format over time in such a way that the code can still read data encoded with the old format.

Where to Find the Example Code

The example code is included in the source code package, under the “examples” directory. Download it here.

Defining Your Protocol Format

To create your address book application, you’ll need to start with a .proto file. The definitions in a .proto file are simple: you add a message for each data structure you want to serialize, then specify a name and a type for each field in the message. Here is the .proto file that defines your messages, addressbook.proto .

As you can see, the syntax is similar to C++ or Java. Let’s go through each part of the file and see what it does.

The .proto file starts with a package declaration, which helps to prevent naming conflicts between different projects. In Python, packages are normally determined by directory structure, so the package you define in your .proto file will have no effect on the generated code. However, you should still declare one to avoid name collisions in the Protocol Buffers name space as well as in non-Python languages.

Next, you have your message definitions. A message is just an aggregate containing a set of typed fields. Many standard simple data types are available as field types, including bool , int32 , float , double , and string . You can also add further structure to your messages by using other message types as field types – in the above example the Person message contains PhoneNumber messages, while the AddressBook message contains Person messages. You can even define message types nested inside other messages – as you can see, the PhoneNumber type is defined inside Person . You can also define enum types if you want one of your fields to have one of a predefined list of values – here you want to specify that a phone number can be one of the following phone types: PHONE_TYPE_MOBILE , PHONE_TYPE_HOME , or PHONE_TYPE_WORK .

The " = 1", " = 2" markers on each element identify the unique “tag” that field uses in the binary encoding. Tag numbers 1-15 require one less byte to encode than higher numbers, so as an optimization you can decide to use those tags for the commonly used or repeated elements, leaving tags 16 and higher for less-commonly used optional elements. Each element in a repeated field requires re-encoding the tag number, so repeated fields are particularly good candidates for this optimization.

Each field must be annotated with one of the following modifiers:

• optional : the field may or may not be set. If an optional field value isn’t set, a default value is used. For simple types, you can specify your own default value, as we’ve done for the phone number type in the example. Otherwise, a system default is used: zero for numeric types, the empty string for strings, false for bools. For embedded messages, the default value is always the “default instance” or “prototype” of the message, which has none of its fields set. Calling the accessor to get the value of an optional (or required) field which has not been explicitly set always returns that field’s default value.
• repeated : the field may be repeated any number of times (including zero). The order of the repeated values will be preserved in the protocol buffer. Think of repeated fields as dynamically sized arrays.
• required : a value for the field must be provided, otherwise the message will be considered “uninitialized”. Serializing an uninitialized message will raise an exception. Parsing an uninitialized message will fail. Other than this, a required field behaves exactly like an optional field.

You’ll find a complete guide to writing .proto files – including all the possible field types – in the Protocol Buffer Language Guide . Don’t go looking for facilities similar to class inheritance, though – protocol buffers don’t do that.

Compiling Your Protocol Buffers

Now that you have a .proto , the next thing you need to do is generate the classes you’ll need to read and write AddressBook (and hence Person and PhoneNumber ) messages. To do this, you need to run the protocol buffer compiler protoc on your .proto :

If you haven’t installed the compiler, download the package and follow the instructions in the README.

Now run the compiler, specifying the source directory (where your application’s source code lives – the current directory is used if you don’t provide a value), the destination directory (where you want the generated code to go; often the same as $SRC_DIR ), and the path to your .proto . In this case, you…:

Because you want Python classes, you use the --python_out option – similar options are provided for other supported languages.

Protoc is also able to generate python stubs ( .pyi ) with --pyi_out .

This generates addressbook_pb2.py (or addressbook_pb2.pyi ) in your specified destination directory.

The Protocol Buffer API

Unlike when you generate Java and C++ protocol buffer code, the Python protocol buffer compiler doesn’t generate your data access code for you directly. Instead (as you’ll see if you look at addressbook_pb2.py ) it generates special descriptors for all your messages, enums, and fields, and some mysteriously empty classes, one for each message type:

The important line in each class is __metaclass__ = reflection.GeneratedProtocolMessageType . While the details of how Python metaclasses work is beyond the scope of this tutorial, you can think of them as like a template for creating classes. At load time, the GeneratedProtocolMessageType metaclass uses the specified descriptors to create all the Python methods you need to work with each message type and adds them to the relevant classes. You can then use the fully-populated classes in your code.

The end effect of all this is that you can use the Person class as if it defined each field of the Message base class as a regular field. For example, you could write:

Note that these assignments are not just adding arbitrary new fields to a generic Python object. If you were to try to assign a field that isn’t defined in the .proto file, an AttributeError would be raised. If you assign a field to a value of the wrong type, a TypeError will be raised. Also, reading the value of a field before it has been set returns the default value.

For more information on exactly what members the protocol compiler generates for any particular field definition, see the Python generated code reference .

Enums are expanded by the metaclass into a set of symbolic constants with integer values. So, for example, the constant addressbook_pb2.Person.PhoneType.PHONE_TYPE_WORK has the value 2.

Standard Message Methods

Each message class also contains a number of other methods that let you check or manipulate the entire message, including:

• IsInitialized() : checks if all the required fields have been set.
• __str__() : returns a human-readable representation of the message, particularly useful for debugging. (Usually invoked as str(message) or print message .)
• CopyFrom(other_msg) : overwrites the message with the given message’s values.
• Clear() : clears all the elements back to the empty state.

These methods implement the Message interface. For more information, see the complete API documentation for Message .

Parsing and Serialization

Finally, each protocol buffer class has methods for writing and reading messages of your chosen type using the protocol buffer binary format . These include:

• SerializeToString() : serializes the message and returns it as a string. Note that the bytes are binary, not text; we only use the str type as a convenient container.
• ParseFromString(data) : parses a message from the given string.

These are just a couple of the options provided for parsing and serialization. Again, see the Message API reference for a complete list.

Writing a Message

Now let’s try using your protocol buffer classes. The first thing you want your address book application to be able to do is write personal details to your address book file. To do this, you need to create and populate instances of your protocol buffer classes and then write them to an output stream.

Here is a program which reads an AddressBook from a file, adds one new Person to it based on user input, and writes the new AddressBook back out to the file again. The parts which directly call or reference code generated by the protocol compiler are highlighted.

Reading a Message

Of course, an address book wouldn’t be much use if you couldn’t get any information out of it! This example reads the file created by the above example and prints all the information in it.

Extending a Protocol Buffer

Sooner or later after you release the code that uses your protocol buffer, you will undoubtedly want to “improve” the protocol buffer’s definition. If you want your new buffers to be backwards-compatible, and your old buffers to be forward-compatible – and you almost certainly do want this – then there are some rules you need to follow. In the new version of the protocol buffer:

• you must not change the tag numbers of any existing fields.
• you must not add or delete any required fields.
• you may delete optional or repeated fields.
• you may add new optional or repeated fields but you must use fresh tag numbers (that is, tag numbers that were never used in this protocol buffer, not even by deleted fields).

(There are some exceptions to these rules, but they are rarely used.)

If you follow these rules, old code will happily read new messages and simply ignore any new fields. To the old code, optional fields that were deleted will simply have their default value, and deleted repeated fields will be empty. New code will also transparently read old messages. However, keep in mind that new optional fields will not be present in old messages, so you will need to either check explicitly whether they’re set with has_ , or provide a reasonable default value in your .proto file with [default = value] after the tag number. If the default value is not specified for an optional element, a type-specific default value is used instead: for strings, the default value is the empty string. For booleans, the default value is false. For numeric types, the default value is zero. Note also that if you added a new repeated field, your new code will not be able to tell whether it was left empty (by new code) or never set at all (by old code) since there is no has_ flag for it.

Advanced Usage

Protocol buffers have uses that go beyond simple accessors and serialization. Be sure to explore the Python API reference to see what else you can do with them.

One key feature provided by protocol message classes is reflection . You can iterate over the fields of a message and manipulate their values without writing your code against any specific message type. One very useful way to use reflection is for converting protocol messages to and from other encodings, such as XML or JSON. A more advanced use of reflection might be to find differences between two messages of the same type, or to develop a sort of “regular expressions for protocol messages” in which you can write expressions that match certain message contents. If you use your imagination, it’s possible to apply Protocol Buffers to a much wider range of problems than you might initially expect!

Reflection is provided as part of the Message interface .

Protobuf: Question: How to set oneof fields in python?

message Test { oneof test_oneof { MessageA a = 1; MessageB b = 2; } message MessageA {…} message MessageB {…} } test =Test() test.a = Test.MessageA() it always complains:Assignment not allowed to field "a" in protocol message object.

Most helpful comment

"Assignment not allowed to field "a" in protocol message object." is not a oneof problem, it is complain about assignment for message field.

There are two ways you can do:

test.a.CopyFrom(...) or test.a.sub_field = ...

All 5 comments

HI @anandolee , I have the same problem. I have tried test.a.CopyFrom(...) , it works, but how to use test.a.sub_field = ?

I want to ask some questions too.

test =Test() test.c = "abcd" test.a = ... I want to know why test.a = ... cannot work? The official tutorial explain how to use oneof only with string and int32 types. I have searched on google, and it also lack of answers about how to fill the message fields in oneof. https://developers.google.com/protocol-buffers/docs/reference/python-generated#oneof

For "sub_field", it means a sub scalar field of MessageA. For example: https://github.com/google/protobuf/blob/master/python/google/protobuf/internal/message_test.py#L795

Protobuf python does not support assignment for message field, even it is a normal message field instead of oneof, "test.a =" is still not allowed. "You cannot assign a value to an embedded message field. Instead, assigning a value to any field within the child message implies setting the message field in the parent. " : https://developers.google.com/protocol-buffers/docs/reference/python-generated#embedded_message

Thanks for your reply @anandolee I want to ask the other question, I tried to read the unittest code, but I don't know where does the message_module parameter come from? I think it is the unittest_pb2 or unittest_proto3_arena_pb2 . In my opinion, it is generated by some .proto file, where is the .proto file? What's the meaning of the golden_message? @_parameterized.named_parameters( ('_proto2', unittest_pb2), ('_proto3', unittest_proto3_arena_pb2))

Yes, it is unittest_pb2 or unittest_proto3_arena_pb2 (we tested both): https://github.com/google/protobuf/blob/master/src/google/protobuf/unittest.proto#L64 https://github.com/google/protobuf/blob/master/src/google/protobuf/unittest_proto3_arena.proto#L41

"Golden data" is the wire format for a message and we put the wire format to a "golden data file". This is because we can write wire format in the test directly.

Related issues

AttributeError: Assignment not allowed to composite field "task" in protocol message object

The error message you're encountering, "AttributeError: Assignment not allowed to composite field 'task' in protocol message object," typically occurs when you try to directly assign a value to a field of a Protocol Buffers (protobuf) message that is marked as a composite field. Composite fields are fields that have a message type.

In protobuf, you cannot directly assign values to composite fields like you would with simple fields (e.g., integers or strings). Instead, you should create an instance of the composite message type and then set its fields individually before assigning it to the parent message.

Here's an example of how to set a composite field correctly:

In this example, we create instances of both the parent message ( ParentMessage ) and the composite message ( CompositeMessage ). We then set the fields of the composite message ( field1 and field2 ) and finally use CopyFrom to assign the composite message to the parent message's task field.

Make sure to replace your_protobuf_module_pb2 with the actual module generated by protoc for your protobuf definition. Additionally, replace ParentMessage , CompositeMessage , and field names with the actual names from your protobuf definition.

roles matlab jsch change-password primefaces firebase-authentication form-data git-config kendo-ui-grid pkcs#11

More Python Questions

• Making square axes plot with log2 scales in matplotlib
• Django Invalid HTTP_HOST header: 'testserver'. You may need to add u'testserver' to ALLOWED_HOSTS
• Typical AngularJS workflow and project structure (with Python Flask)
• Python: How to get values of an array at certain index positions?
• Tensorflow mean squared error loss function
• Fill cells with colors using openpyxl?
• How to write a confusion matrix in python
• Python: Converting string into decimal number
•   Previous
• Next  

• Apache Beam documentation »
• Module code »

Quick search

Source code for google.protobuf.internal.python_message.

Navigation Menu

Search code, repositories, users, issues, pull requests..., provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

• Notifications

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attribute error: Assignment not allowed (no field "force_gpu_compatible" in protocol message object) #23

comaniac commented May 30, 2017

cwhipkey commented May 31, 2017 via email

Sorry, something went wrong.

tfboyd commented Jun 1, 2017

No branches or pull requests