Binary (de) serialization [u8; 128]

Question

I have a struct containing an array of bytes that I would like to serialize and deserialize to and from the binary. But it only works for arrays up to 32 elements .

Here is my minimal code example

main.rs :

 #[macro_use] extern crate serde_derive; extern crate serde; extern crate bincode; use bincode::{serialize, deserialize, Infinite}; const BYTECOUNT: usize = 32; // 33 and more does not work, I need 128 type DataArr = [u8; BYTECOUNT]; #[derive(Serialize, Deserialize, Debug)] struct Entry { number: i64, data: DataArr } fn main() { let mut my_entry = Entry { number: 12345, data: [0; BYTECOUNT] }; my_entry.data[4] = 42; // Convert the Entry to binary. let serialized: Vec<u8> = serialize(&my_entry, Infinite).unwrap(); println!("serialized = {:?}", serialized); // Convert the binary representation back to an Entry. let deserialized: Entry = deserialize(&serialized).unwrap(); println!("deserialized = {:?}", deserialized); } 

Cargo.toml :

 [package] name = "array_serialization_test" version = "0.1.0" [dependencies] serde = "*" serde_derive = "*" bincode = "*" 

exit:

 serialized = [57, 48, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 42, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] deserialized = Entry { number: 12345, data: [0, 0, 0, 0, 42, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 

How can I make it work for 128 elements in an array? Can I expand array_impls! somehow manually in my user code with this?

 128 => (0 a 1 b 2 c 3 d 4 e 5 f 6 g 7 h 8 i 9 j 10 k 11 l 12 m 13 n 14 o 15 p 16 q 17 e 18 s 19 t 20 u 21 v 22 w 23 x 24 y 25 z 26 aa 27 ab 28 ac 29 ad 30 ae 31 af 32 ag 33 ah 34 ai 35 aj 36 ak 37 al 38 am 39 an 40 ao 41 ap 42 aq 43 ar 44 as 45 at 46 au 47 av 48 aw 49 ax 50 ay 51 az 52 ba 53 bb 54 bc 55 bd 56 be 57 bf 58 bg 59 bh 60 bi 61 bj 62 bk 63 bl 64 bm 65 bn 66 bo 67 bp 68 bq 69 br 70 bs 71 bt 72 bu 73 bv 74 bw 75 bx 76 by 77 bz 78 ca 79 cb 80 cc 81 cd 82 ce 83 cf 84 cg 85 ch 86 ci 87 cj 88 ck 89 cl 90 cm 91 cn 92 co 93 cp 94 cq 95 cr 96 cs 97 ct 98 cu 99 cv 100 cw 101 cx 102 cy 103 cz 104 da 105 db 106 dc 107 dd 108 de 109 df 110 dg 111 dh 112 di 113 dj 114 dk 115 dl 116 dm 117 dn 118 do 119 dp 120 dq 121 dr 122 ds 123 dt 124 du 125 dv 126 dw 127 dx) 

Or is there an alternative approach?

NOTE: I think this question is different from. How do I map the C structure to over 32 bytes using serde and bincode? because I really need the contents of the array, as it is not just used for filling. Also I would like to know if I can expand array_impls! in your code.

Answer 1

Currently, Serde does not provide Serialize and Deserialize impls features that work for each array size. This is blocked on const generics , which we are working on, and we hope to land at night in 2018.

Now you can define your own "large array", which can serialize and deserialize arrays of any specific sizes used in your box. Fields for which you want to use the large array helper must be marked with #[serde(with = "BigArray")] , otherwise the Heart will look for nonexistent Serialize and Deserialize impls.

 #[macro_use] extern crate serde_derive; extern crate serde; extern crate bincode; mod big_array; use big_array::BigArray; const BYTECOUNT: usize = 128; type DataArr = [u8; BYTECOUNT]; #[derive(Serialize, Deserialize)] struct Entry { number: i64, #[serde(with = "BigArray")] data: DataArr } fn main() { let mut my_entry = Entry { number: 12345, data: [0; BYTECOUNT] }; my_entry.data[4] = 42; // Convert the Entry to binary. let serialized: Vec<u8> = bincode::serialize(&my_entry).unwrap(); println!("serialized = {:?}", serialized); // Convert the binary representation back to an Entry. let deserialized: Entry = bincode::deserialize(&serialized).unwrap(); println!("deserialized = {} {:?}", deserialized.number, &deserialized.data[..]); } 

A large array helper can be defined in src/big_array.rs as follows. Perhaps this would make a good box on its own if you want to own it!

 use std::fmt; use std::marker::PhantomData; use serde::ser::{Serialize, Serializer, SerializeTuple}; use serde::de::{Deserialize, Deserializer, Visitor, SeqAccess, Error}; pub trait BigArray<'de>: Sized { fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error> where S: Serializer; fn deserialize<D>(deserializer: D) -> Result<Self, D::Error> where D: Deserializer<'de>; } macro_rules! big_array { ($($len:expr,)+) => { $( impl<'de, T> BigArray<'de> for [T; $len] where T: Default + Copy + Serialize + Deserialize<'de> { fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error> where S: Serializer { let mut seq = serializer.serialize_tuple(self.len())?; for elem in &self[..] { seq.serialize_element(elem)?; } seq.end() } fn deserialize<D>(deserializer: D) -> Result<[T; $len], D::Error> where D: Deserializer<'de> { struct ArrayVisitor<T> { element: PhantomData<T>, } impl<'de, T> Visitor<'de> for ArrayVisitor<T> where T: Default + Copy + Deserialize<'de> { type Value = [T; $len]; fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result { formatter.write_str(concat!("an array of length ", $len)) } fn visit_seq<A>(self, mut seq: A) -> Result<[T; $len], A::Error> where A: SeqAccess<'de> { let mut arr = [T::default(); $len]; for i in 0..$len { arr[i] = seq.next_element()? .ok_or_else(|| Error::invalid_length(i, &self))?; } Ok(arr) } } let visitor = ArrayVisitor { element: PhantomData }; deserializer.deserialize_tuple($len, visitor) } } )+ } } big_array! { 40, 48, 50, 56, 64, 72, 96, 100, 128, 160, 192, 200, 224, 256, 384, 512, 768, 1024, 2048, 4096, 8192, 16384, 32768, 65536, }