Encoding / serialization issues when using ICacheClient and protobuf in ServiceStack

I am using the current ServiceStack with protobuff serialization.

When adding ICacheClient to cache my responses, the binary response sent from the cache client has a different encoding / binary serialization than the original response without the cache client.

This leads to client-side deserialization issues, where I use a pre-compiled deserializer.

For example, this protobuf exception occurs on the client side when using ICacheClient on the ServerStack server side:

OverflowException: Number overflow. ProtoBuf.ProtoReader.TryReadUInt32VariantWithoutMoving (Boolean trimNegative, System.UInt32& value) ProtoBuf.ProtoReader.ReadUInt32Variant (Boolean trimNegative) ProtoBuf.ProtoReader.ReadUInt32 () MyModelSerializer.Read (My.Models.MyDataModel , ProtoBuf.ProtoReader ) 

(I think this is a random exception, quite certain that other exceptions are possible).

This behavior is the same with the MemoryCacheClient and Redis client.

This is how I initialize protobuf:

 ContentTypeFilters.Register(ContentType.ProtoBuf, (reqCtx, res, stream) => ProtoBuf.Serializer.NonGeneric.Serialize(stream, res), ProtoBuf.Serializer.NonGeneric.Deserialize); 

The cache client is initialized as follows:

 container.Register<ICacheClient>(new MemoryCacheClient()); 

(Or Redis after this guide ).

Thus, the cached data transmitted over the proto-buffer, which pass through the wire, look and differ from the unavailable: Binary diff Large image

The rolled back response seems to be encoded / serialized differently.

Can I do anything with this encoding problem to make Redis or Memory-ICacheClient compatible with protobuf?


Update: I studied a little, these are my findings:

  • To save the cache in the provider (be it Redis os Memory), the protobuf binary stream is "converted" to a string via StreamReader in HttpResponseFilter-> SerializeToString using "UTF8 without specification" -Encoding, which itself is called through CacheClientExtensions-> Cache ().

  • First problem : CacheClientExtensions-> Cache () then returns a DTO with string serialization, which makes protobuf deserialization still impossible. First decision . Return the original DTO to CacheClientExtensions-> Cache (). But this will only work for the first uncached response, since the cached response will not be properly de-serialized yet. It will lead us to

  • The second problem. . Retrieving data from the cache requires proper serialization of the protobuf binary data for string serialization in order to put it in the cache first. I know that it will work with base64.

  • The third problem . Currently, there seems to be no way to replace the current stream to string conversion via StreamReader to HttpResponseFilter-> SerializeToString at runtime, right?

  • The fourth problem . When you return data from the cache, it should be base64 decoded again.

+4
source share
1 answer

I was digging the ServiceStack code, but did not find any way to implement my requirement (I really hope that I have not forgotten anything).

So, I introduced some changes to ServiceStack that allow you to use custom TextSerializers, which are then used to serialize to the bevor string by pushing DTO to cache clients.

See my diff here: https://github.com/derFunk/ServiceStack/commit/93f62c7d7e44a303c88a81c3096b9757daba4c7c

It works for my purposes, although it may not be fully tested and focused on the protobuff.

I would appreciate it if some ServiceStack ninja could check what I'm doing, maybe this will end up coming out of the request.

+1
source

All Articles