Access Protobuf message field of unknown type in Python

Question

Let's say I have 2 Protobuf messages, A and B. Their general structure is similar, but not identical. Therefore, we moved the general material to a separate post, which we called Common. It works great.

However, now I am facing the following problem: there is a special case when I have to process a serialized message, but I do not know if it is a message of type A or type B. I have a working solution in C ++ (shown below), but I could not find a way to do the same in Python.

Example:

// file: Common.proto // contains some kind of shared struct that is used by all messages: message Common { ... } // file: A.proto import "Common.proto"; message A { required int32 FormatVersion = 1; optional bool SomeFlag [default = true] = 2; optional Common CommonSettings = 3; ... A-specific Fields ... } // file: B.proto import "Common.proto"; message B { required int32 FormatVersion = 1; optional bool SomeFlag [default = true] = 2; optional Common CommonSettings = 3; ... B-specific Fields ... } 

Working solution in C ++

In C ++, I use the reflection API to access the CommonSettings field as follows:

 namespace gp = google::protobuf; ... Common* getCommonBlock(gp::Message* paMessage) { gp::Message* paMessage = new gp::Message(); gp::FieldDescriptor* paFieldDescriptor = paMessage->GetDescriptor()->FindFieldByNumber(3); gp::Reflection* paReflection = paMessage->GetReflection(); return dynamic_cast<Common&>(paReflection->GetMessage(*paMessage,paFieldDescriptor)); } 

The getCommonBlock method uses FindFieldByNumber () to get the handle of the field I'm trying to get. He then uses reflection to obtain evidence. getCommonBlock can process messages of type A, B or any future type if the Common field remains at index 3.

My question is: is there a way to do a similar thing to Python? I looked at the Protobuf documentation, but couldn't figure out how to do this.

Answer 1

I know this is an old thread, but I will still be responsible for the offspring:

Firstly, as you know, it is impossible to determine the type of protocol buffer message only from its serialized form. The only information in serialized form that you have access to is field numbers and their serialized values.

Secondly, the “right” way to do this should consist of a proto that contains both, for example

 message Parent { required int32 FormatVersion = 1; optional bool SomeFlag [default = true] = 2; optional Common CommonSettings = 3; oneof letters_of_alphabet { A a_specific = 4; B b_specific = 5; } } 

So there is no ambiguity: you just parse the same proto ( Parent ) every time.

---

In any case, if it is too late to change this, then what I recommend to you is to define a new message with only common fields, for example

 message Shared { required int32 FormatVersion = 1; optional bool SomeFlag [default = true] = 2; optional Common CommonSettings = 3; } 

Then you can pretend that the message (either A or B ) is actually Shared , and analyze it accordingly. Unknown fields will be irrelevant.

Answer 2

One of the advantages of Python over a statically typed language, such as C ++, is that you do not need to use any special reflection code to get an attribute of an object of an unknown type: you simply request the object. The built-in function that does this is getattr , so you can do:

 settings_value = getattr(obj, 'CommonSettings') 

Answer 3

I had a similar problem.

I did this to create a new post with an enumeration indicating the type:

 enum TYPE { A = 0; B = 1; } message Base { required TYPE type = 1; ... Other common fields ... } 

Then create specific message types:

 message A { required TYPE type = 1 [default: A]; ... other A fields ... } 

and

 message B { required TYPE type = 1 [default: B]; ... other B fields ... } 

Remember to correctly define the "Base" message, or you will not be binary compatible if you add fields recently (since you will also have to migrate the inheriting message fields).

So you can get a general message:

 msg = ... receive message from net ... # detect message type packet = Base() packet.ParseFromString(msg) # check for type if packet.type == TYPE.A: # parse message as appropriate type packet = A() packet.ParseFromString(msg) else: # this is a B message... or whatever # ... continue with your business logic ... 

Hope this helps.

Answer 4

How about "concatenating" two protocol buffers in a header + payload format, for example. How does the general data follow either from message A or B, as suggested by the protobuf methods ?

Here's how I did it with various types of payloads like blob in the mqtt post.