Source code
Revision control
Copy as Markdown
Other Tools
//! # Deserialization Flavors
//!
//! "Flavors" in `postcard` are used as modifiers to the serialization or deserialization
//! process. Flavors typically modify one or both of the following:
//!
//! 1. The source medium of the deserialization, e.g. whether the data is serialized from a `[u8]` slice, or some other container
//! 2. The format of the deserialization, such as if the original data is encoded in a COBS format, contains a CRC32 checksum
//! appended to the message, etc.
//!
//! Flavors are implemented using the [`Flavor`] trait, which acts as a "middleware" for retrieving the bytes before they
//! are passed to `serde` for deserialization
//!
//! Multiple flavors may be combined to obtain a desired combination of behavior and storage.
//! When flavors are combined, it is expected that the storage flavor (such as [`Slice`]) is the innermost flavor.
//!
//! Custom flavors may be defined by users of the `postcard` crate, however some commonly useful flavors have been provided in
//! this module. If you think your custom flavor would be useful to others, PRs adding flavors are very welcome!
//!
//! ## Usability
//!
//! Flavors may not always be convenient to use directly, as they may expose some implementation details of how the
//! inner workings of the flavor behaves. It is typical to provide a convenience method for using a flavor, to prevent
//! the user from having to specify generic parameters, setting correct initialization values, or handling the output of
//! the flavor correctly. See `postcard::from_bytes()` for an example of this.
//!
//! ## When to use (multiple) flavors
//!
//! Combining flavors are nice for convenience, as they perform potentially multiple steps of
//! serialization at one time.
//!
//! This can often be more memory efficient, as intermediate buffers are not typically required.
//!
//! ## When NOT to use (multiple) flavors
//!
//! The downside of passing deserialization through multiple steps is that it is typically slower than
//! performing each step serially. Said simply, "cobs decoding while deserializing" is often slower
//! than "cobs decode then deserialize", due to the ability to handle longer "runs" of data in each
//! stage. The downside is that if these stages can not be performed in-place on the buffer, you
//! will need additional buffers for each stage.
//!
//! Additionally, deserializating flavors can be more restrictive or difficult to work with than
//! serialization flavors, as deserialization may require that the deserialized types borrow some
//! portion of the original message.
//!
//! ## Examples
//!
//! ### Using a single flavor
//!
//! In the first example, we use the `Slice` flavor, to retrieve the serialized output from a `[u8]` slice.
//! No other modification is made to the serialization process.
//!
//! ```rust
//! use postcard::{
//! de_flavors::Slice,
//! Deserializer,
//! };
//! use serde::Deserialize;
//!
//! #[derive(Deserialize, Debug, PartialEq)]
//! struct Tup(u8, u8, u8);
//!
//! let msg = [0x04, 0x00, 0x04, 0x01, 0x02, 0x03];
//! let slice = Slice::new(&msg);
//! let mut deserializer = Deserializer::from_flavor(slice);
//! let t = Tup::deserialize(&mut deserializer).unwrap();
//! assert_eq!(t, Tup(4, 0, 4));
//! let remainder = deserializer.finalize().unwrap();
//! assert_eq!(remainder, &[1, 2, 3]);
//! ```
use crate::{Error, Result};
use core::marker::PhantomData;
/// The deserialization Flavor trait
///
/// This is used as the primary way to decode serialized data from some kind of buffer,
/// or modify that data in a middleware style pattern.
///
/// See the module level docs for an example of how flavors are used.
pub trait Flavor<'de>: 'de {
/// The remaining data of this flavor after deserializing has completed.
///
/// Typically, this includes the remaining buffer that was not used for
/// deserialization, and in cases of more complex flavors, any additional
/// information that was decoded or otherwise calculated during
/// the deserialization process.
type Remainder: 'de;
/// The source of data retrieved for deserialization.
///
/// This is typically some sort of data buffer, or another Flavor, when
/// chained behavior is desired
type Source: 'de;
/// Obtain the next byte for deserialization
fn pop(&mut self) -> Result<u8>;
/// Returns the number of bytes remaining in the message, if known.
///
/// # Implementation notes
///
/// It is not enforced that this number is exactly correct.
/// A flavor may yield less or more bytes than the what is hinted at by
/// this function.
///
/// `size_hint()` is primarily intended to be used for optimizations such as
/// reserving space for deserialized items, but must not be trusted to
/// e.g., omit bounds checks in unsafe code. An incorrect implementation of
/// `size_hint()` should not lead to memory safety violations.
///
/// That said, the implementation should provide a correct estimation,
/// because otherwise it would be a violation of the trait’s protocol.
///
/// The default implementation returns `None` which is correct for any flavor.
fn size_hint(&self) -> Option<usize> {
None
}
/// Attempt to take the next `ct` bytes from the serialized message.
///
/// This variant borrows the data from the input for zero-copy deserialization. If zero-copy
/// deserialization is not necessary, prefer to use `try_take_n_temp` instead.
fn try_take_n(&mut self, ct: usize) -> Result<&'de [u8]>;
/// Attempt to take the next `ct` bytes from the serialized message.
///
/// This variant does not guarantee that the returned value is borrowed from the input, so it
/// cannot be used for zero-copy deserialization, but it also avoids needing to potentially
/// allocate a data in a temporary buffer.
///
/// This variant should be used instead of `try_take_n`
/// if zero-copy deserialization is not necessary.
///
/// It is only necessary to implement this method if the flavor requires storing data in a
/// temporary buffer in order to implement the borrow semantics, e.g. the `std::io::Read`
/// flavor.
fn try_take_n_temp<'a>(&'a mut self, ct: usize) -> Result<&'a [u8]>
where
'de: 'a,
{
self.try_take_n(ct)
}
/// Complete the deserialization process.
///
/// This is typically called separately, after the `serde` deserialization
/// has completed.
fn finalize(self) -> Result<Self::Remainder>;
}
/// A simple [`Flavor`] representing the deserialization from a borrowed slice
pub struct Slice<'de> {
// This string starts with the input data and characters are truncated off
// the beginning as data is parsed.
pub(crate) cursor: *const u8,
pub(crate) end: *const u8,
pub(crate) _pl: PhantomData<&'de [u8]>,
}
impl<'de> Slice<'de> {
/// Create a new [Slice] from the given buffer
pub fn new(sli: &'de [u8]) -> Self {
let range = sli.as_ptr_range();
Self {
cursor: range.start,
end: range.end,
_pl: PhantomData,
}
}
}
impl<'de> Flavor<'de> for Slice<'de> {
type Remainder = &'de [u8];
type Source = &'de [u8];
#[inline]
fn pop(&mut self) -> Result<u8> {
if self.cursor == self.end {
Err(Error::DeserializeUnexpectedEnd)
} else {
// SAFETY: `self.cursor` is in-bounds and won't be incremented past `self.end` as we
// have checked above.
unsafe {
let res = Ok(*self.cursor);
self.cursor = self.cursor.add(1);
res
}
}
}
#[inline]
fn size_hint(&self) -> Option<usize> {
Some((self.end as usize) - (self.cursor as usize))
}
#[inline]
fn try_take_n(&mut self, ct: usize) -> Result<&'de [u8]> {
let remain = (self.end as usize) - (self.cursor as usize);
if remain < ct {
Err(Error::DeserializeUnexpectedEnd)
} else {
// SAFETY: `self.cursor` is valid for `ct` elements and won't be incremented past `self.end` as we
// have checked above.
unsafe {
let sli = core::slice::from_raw_parts(self.cursor, ct);
self.cursor = self.cursor.add(ct);
Ok(sli)
}
}
}
/// Return the remaining (unused) bytes in the Deserializer
fn finalize(self) -> Result<&'de [u8]> {
let remain = (self.end as usize) - (self.cursor as usize);
// SAFETY: `self.cursor` is valid for `remain` elements
unsafe { Ok(core::slice::from_raw_parts(self.cursor, remain)) }
}
}
/// Support for [`std::io`] or `embedded-io` traits
#[cfg(any(
feature = "embedded-io-04",
feature = "embedded-io-06",
feature = "use-std"
))]
pub mod io {
use crate::{Error, Result};
use core::marker::PhantomData;
struct SlidingBuffer<'de> {
cursor: *mut u8,
end: *const u8,
_pl: PhantomData<&'de [u8]>,
}
impl<'de> SlidingBuffer<'de> {
pub fn new(sli: &'de mut [u8]) -> Self {
let range = sli.as_mut_ptr_range();
Self {
cursor: range.start,
end: range.end,
_pl: PhantomData,
}
}
#[inline]
fn take_n(&mut self, ct: usize) -> Result<&'de mut [u8]> {
let remain = (self.end as usize) - (self.cursor as usize);
let buff = if remain < ct {
return Err(Error::DeserializeUnexpectedEnd);
} else {
// SAFETY: `self.cursor` is valid for `ct` elements and won't be incremented
// past `self.end` as we have checked above.
unsafe {
let sli = core::slice::from_raw_parts_mut(self.cursor, ct);
self.cursor = self.cursor.add(ct);
sli
}
};
Ok(buff)
}
#[inline]
fn take_n_temp(&mut self, ct: usize) -> Result<&mut [u8]> {
let remain = (self.end as usize) - (self.cursor as usize);
let buff = if remain < ct {
return Err(Error::DeserializeUnexpectedEnd);
} else {
unsafe {
let sli = core::slice::from_raw_parts_mut(self.cursor, ct);
sli
}
};
Ok(buff)
}
fn complete(self) -> Result<&'de mut [u8]> {
let remain = (self.end as usize) - (self.cursor as usize);
// SAFETY: `self.cursor` is valid for `remain` elements
unsafe { Ok(core::slice::from_raw_parts_mut(self.cursor, remain)) }
}
}
/// Support for [`embedded_io`](crate::eio::embedded_io) traits
#[cfg(any(feature = "embedded-io-04", feature = "embedded-io-06"))]
pub mod eio {
use super::super::Flavor;
use super::SlidingBuffer;
use crate::{Error, Result};
/// Wrapper over a [`embedded_io`](crate::eio::embedded_io)::[`Read`](crate::eio::Read) and a sliding buffer to implement the [`Flavor`] trait
pub struct EIOReader<'de, T>
where
T: crate::eio::Read,
{
reader: T,
buff: SlidingBuffer<'de>,
}
impl<'de, T> EIOReader<'de, T>
where
T: crate::eio::Read,
{
/// Create a new [`EIOReader`] from a reader and a buffer.
///
/// `buff` must have enough space to hold all data read during the deserialisation.
pub fn new(reader: T, buff: &'de mut [u8]) -> Self {
Self {
reader,
buff: SlidingBuffer::new(buff),
}
}
}
impl<'de, T> Flavor<'de> for EIOReader<'de, T>
where
T: crate::eio::Read + 'de,
{
type Remainder = (T, &'de mut [u8]);
type Source = &'de [u8];
#[inline]
fn pop(&mut self) -> Result<u8> {
let mut val = [0; 1];
self.reader
.read_exact(&mut val)
.map_err(|_| Error::DeserializeUnexpectedEnd)?;
Ok(val[0])
}
#[inline]
fn size_hint(&self) -> Option<usize> {
None
}
#[inline]
fn try_take_n(&mut self, ct: usize) -> Result<&'de [u8]> {
let buff = self.buff.take_n(ct)?;
self.reader
.read_exact(buff)
.map_err(|_| Error::DeserializeUnexpectedEnd)?;
Ok(buff)
}
#[inline]
fn try_take_n_temp<'a>(&'a mut self, ct: usize) -> Result<&'a [u8]>
where
'de: 'a,
{
let buff = self.buff.take_n_temp(ct)?;
self.reader
.read_exact(buff)
.map_err(|_| Error::DeserializeUnexpectedEnd)?;
Ok(buff)
}
/// Return the remaining (unused) bytes in the Deserializer
fn finalize(self) -> Result<(T, &'de mut [u8])> {
let buf = self.buff.complete()?;
Ok((self.reader, buf))
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_pop() {
let mut reader = EIOReader::new(&[0xAA, 0xBB, 0xCC][..], &mut []);
assert_eq!(reader.pop(), Ok(0xAA));
assert_eq!(reader.pop(), Ok(0xBB));
assert_eq!(reader.pop(), Ok(0xCC));
assert_eq!(reader.pop(), Err(Error::DeserializeUnexpectedEnd));
}
#[test]
fn test_try_take_n() {
let mut buf = [0; 8];
let mut reader = EIOReader::new(&[0xAA, 0xBB, 0xCC, 0xDD, 0xEE][..], &mut buf);
assert_eq!(reader.try_take_n(2), Ok(&[0xAA, 0xBB][..]));
assert_eq!(reader.try_take_n(2), Ok(&[0xCC, 0xDD][..]));
assert_eq!(reader.try_take_n(2), Err(Error::DeserializeUnexpectedEnd));
}
}
}
/// Support for [`std::io`] traits
#[allow(clippy::module_inception)]
#[cfg(feature = "use-std")]
pub mod io {
use super::super::Flavor;
use super::SlidingBuffer;
use crate::{Error, Result};
/// Wrapper over a [`std::io::Read`] and a sliding buffer to implement the [Flavor] trait
pub struct IOReader<'de, T>
where
T: std::io::Read,
{
reader: T,
buff: SlidingBuffer<'de>,
}
impl<'de, T> IOReader<'de, T>
where
T: std::io::Read,
{
/// Create a new [`IOReader`] from a reader and a buffer.
///
/// `buff` must have enough space to hold all data read during the deserialisation.
pub fn new(reader: T, buff: &'de mut [u8]) -> Self {
Self {
reader,
buff: SlidingBuffer::new(buff),
}
}
}
impl<'de, T> Flavor<'de> for IOReader<'de, T>
where
T: std::io::Read + 'de,
{
type Remainder = (T, &'de mut [u8]);
type Source = &'de [u8];
#[inline]
fn pop(&mut self) -> Result<u8> {
let mut val = [0; 1];
self.reader
.read_exact(&mut val)
.map_err(|_| Error::DeserializeUnexpectedEnd)?;
Ok(val[0])
}
#[inline]
fn size_hint(&self) -> Option<usize> {
None
}
#[inline]
fn try_take_n(&mut self, ct: usize) -> Result<&'de [u8]> {
let buff = self.buff.take_n(ct)?;
self.reader
.read_exact(buff)
.map_err(|_| Error::DeserializeUnexpectedEnd)?;
Ok(buff)
}
#[inline]
fn try_take_n_temp<'a>(&'a mut self, ct: usize) -> Result<&'a [u8]>
where
'de: 'a,
{
let buff = self.buff.take_n_temp(ct)?;
self.reader
.read_exact(buff)
.map_err(|_| Error::DeserializeUnexpectedEnd)?;
Ok(buff)
}
/// Return the remaining (unused) bytes in the Deserializer
fn finalize(self) -> Result<(T, &'de mut [u8])> {
let buf = self.buff.complete()?;
Ok((self.reader, buf))
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_pop() {
let mut reader = IOReader::new(&[0xAA, 0xBB, 0xCC][..], &mut []);
assert_eq!(reader.pop(), Ok(0xAA));
assert_eq!(reader.pop(), Ok(0xBB));
assert_eq!(reader.pop(), Ok(0xCC));
assert_eq!(reader.pop(), Err(Error::DeserializeUnexpectedEnd));
}
#[test]
fn test_try_take_n() {
let mut buf = [0; 8];
let mut reader = IOReader::new(&[0xAA, 0xBB, 0xCC, 0xDD, 0xEE][..], &mut buf);
assert_eq!(reader.try_take_n(2), Ok(&[0xAA, 0xBB][..]));
assert_eq!(reader.try_take_n(2), Ok(&[0xCC, 0xDD][..]));
assert_eq!(reader.try_take_n(2), Err(Error::DeserializeUnexpectedEnd));
}
}
}
}
////////////////////////////////////////
// CRC
////////////////////////////////////////
/// This Cyclic Redundancy Check flavor applies [the CRC crate's `Algorithm`](https://docs.rs/crc/latest/crc/struct.Algorithm.html) struct on
/// the serialized data.
///
/// The flavor will check the CRC assuming that it has been appended to the bytes.
/// CRCs are used for error detection when reading data back.
/// Requires the `crc` feature.
///
#[cfg(feature = "use-crc")]
#[cfg_attr(docsrs, doc(cfg(feature = "use-crc")))]
pub mod crc {
use core::convert::TryInto;
use crc::Digest;
use crc::Width;
use serde::Deserialize;
use super::Flavor;
use super::Slice;
use crate::Deserializer;
use crate::Error;
use crate::Result;
/// Manages CRC modifications as a flavor.
pub struct CrcModifier<'de, B, W>
where
B: Flavor<'de>,
W: Width,
{
flav: B,
digest: Digest<'de, W>,
}
impl<'de, B, W> CrcModifier<'de, B, W>
where
B: Flavor<'de>,
W: Width,
{
/// Create a new Crc modifier Flavor.
pub fn new(bee: B, digest: Digest<'de, W>) -> Self {
Self { flav: bee, digest }
}
}
macro_rules! impl_flavor {
($int:ty, $from_bytes:ident, $take_from_bytes:ident) => {
impl<'de, B> Flavor<'de> for CrcModifier<'de, B, $int>
where
B: Flavor<'de>,
{
type Remainder = B::Remainder;
type Source = B::Source;
#[inline]
fn pop(&mut self) -> Result<u8> {
match self.flav.pop() {
Ok(byte) => {
self.digest.update(&[byte]);
Ok(byte)
}
e @ Err(_) => e,
}
}
#[inline]
fn size_hint(&self) -> Option<usize> {
self.flav.size_hint()
}
#[inline]
fn try_take_n(&mut self, ct: usize) -> Result<&'de [u8]> {
match self.flav.try_take_n(ct) {
Ok(bytes) => {
self.digest.update(bytes);
Ok(bytes)
}
e @ Err(_) => e,
}
}
fn finalize(mut self) -> Result<Self::Remainder> {
match self.flav.try_take_n(core::mem::size_of::<$int>()) {
Ok(prev_crc_bytes) => match self.flav.finalize() {
Ok(remainder) => {
let crc = self.digest.finalize();
let le_bytes = prev_crc_bytes
.try_into()
.map_err(|_| Error::DeserializeBadEncoding)?;
let prev_crc = <$int>::from_le_bytes(le_bytes);
if crc == prev_crc {
Ok(remainder)
} else {
Err(Error::DeserializeBadCrc)
}
}
e @ Err(_) => e,
},
Err(e) => Err(e),
}
}
}
/// Deserialize a message of type `T` from a byte slice with a Crc. The unused portion (if any)
/// of the byte slice is not returned.
pub fn $from_bytes<'a, T>(s: &'a [u8], digest: Digest<'a, $int>) -> Result<T>
where
T: Deserialize<'a>,
{
let flav = CrcModifier::new(Slice::new(s), digest);
let mut deserializer = Deserializer::from_flavor(flav);
let r = T::deserialize(&mut deserializer)?;
let _ = deserializer.finalize()?;
Ok(r)
}
/// Deserialize a message of type `T` from a byte slice with a Crc. The unused portion (if any)
/// of the byte slice is returned for further usage
pub fn $take_from_bytes<'a, T>(
s: &'a [u8],
digest: Digest<'a, $int>,
) -> Result<(T, &'a [u8])>
where
T: Deserialize<'a>,
{
let flav = CrcModifier::new(Slice::new(s), digest);
let mut deserializer = Deserializer::from_flavor(flav);
let t = T::deserialize(&mut deserializer)?;
Ok((t, deserializer.finalize()?))
}
};
}
impl_flavor!(u8, from_bytes_u8, take_from_bytes_u8);
impl_flavor!(u16, from_bytes_u16, take_from_bytes_u16);
impl_flavor!(u32, from_bytes_u32, take_from_bytes_u32);
impl_flavor!(u64, from_bytes_u64, take_from_bytes_u64);
impl_flavor!(u128, from_bytes_u128, take_from_bytes_u128);
}