Is there a way to mark the end of each protobuf-net entry

Question

I save a number of protobuf-net objects in the database cell as a byte [] from protobuf-net prefix objects with a length:

//retrieve existing protobufs from database and convert to Byte[] object q = sql_agent_cmd.ExecuteScalar(); older-pbfs = (Byte[])q; // serialize the new pbf to add into MemoryStream m //now write p and the new pbf-net Byte[] into a memory stream and retrieve the sum var s = new System.IO.MemoryStream(); s.Write(older-pbfs, 0, older-pbfs.Length); s.Write(m.GetBuffer(), 0, m.ToArray().Length); // append new bytes at the end of old Byte[] sum-pbfs = s.ToArray(); //sum-pbfs = old pbfs + new pbf. Insert sum-pbfs into database 

It works great. My concern is what happens if there is little corruption. Now it is no longer possible to find out which byte is the length prefix, and all the contents of the cells would have to be discarded. Is it not recommended to use some kind of indicator of the end-pbf-object (kind of like indicators \ n or EOF used in text files). Thus, even if the record is damaged, other records will be restored.

If so, what is the recommended way to add end-of-record indicators at the end of each pbf.

Using protobuf-netv2 and C # in Visual Studio 2010.

Thanks Manish

Answer 1

If you use the vanilla message via Serialize / Deserialize , then no: this is not part of the specification (since the format is for adding).

If, however, you use SerializeWithLengthPrefix , it will unload the length at the beginning of the message; he will know in advance how much data is expected. You deserialize with DeserializeWithLengthPrefix and it will complain loudly if it doesn't have enough data. However! He will not complain if you have additional data, as this is also intended to be added.

In terms of John’s answer, the use of the *WithLengthPrefix method by default refers to data stored exactly identical to what John suggests; he pretends that there is a shell object and behaves accordingly. The differences are as follows:

• there is actually no wrapper object.
• the "withlengthprefix" methods explicitly stop after one event, and do not combine any later data into the same object (useful, for example, for sending several discrete objects to a single file or down one socket).

The difference in the two "incremental" here is that the first means "merge into one object", where - as the second means "I expect several records."

Unrelated sentence:

 s.Write(m.GetBuffer(), 0, m.ToArray().Length); 

it should be:

 s.Write(m.GetBuffer(), 0, (int)m.Length); 

(no need to create an additional buffer)

Answer 2

(Note: I don’t know much about protobuf-net itself, but this is usually applicable to protocol Buffer messages.)

Usually, if you want to record several messages, you just need to put the message "wrapper" - to make the "real" message a repeating field inside it. Each “real” message will have a length prefixed using the natural format of the protocol buffer wires.

This will not reveal corruption, of course, but to be honest, if the database gets corrupted, you will have big problems. You can detect corruption, for example. saving the hash along with each entry ... but you need to consider the possibility of corruption in the length prefix or inside the hash itself. Think about what you are actually trying to achieve here - which scenarios you are trying to protect and what level of recovery you need.