FsPickler

A fast, general-purpose binary serializer for .NET written in F# that doubles as a pickler combinator library.

• Based on the notion of pickler combinators.
• Provides an automated, strongly typed, pickler generation framework.
• Full support for .NET types, including classes and open hierarchies.
• Fully backwards compatible with .Net serialization, including the ISerializable interface.
• Highly optimized for F# core types.
• Performance about 5-100x faster than the default .NET serializers.
• Full support for the mono framework.
• It is fast.

Get the NuGet package here.

Build Status

Head (branch master), Mono 3.x, OSX, build & unit tests (Travis)

Basic Usage

The following snippet presents the basic serialization/deserialization API for FsPickler:

open Nessos.FsPickler

let fsp = new FsPickler()

// typed serialization
fsp.Serialize<int list option>(stream, Some [1; 2; 3])
fsp.Deserialize<int list option>(stream)

// typed serialization with explicit pickler use
let pickler : Pickler<int list option> = fsp.GeneratePickler<int list option> ()

fsp.Serialize(pickler, stream, Some [1; 2; 3])
fsp.Deserialize(pickler, stream) : int list option

// untyped serialization
fsp.Serialize(typeof<int>, stream, 42)
fsp.Deserialize(typeof<int>, stream) : obj

// untyped serialization with explicit pickler use
let pickler' : Pickler = fsp.GeneratePickler typeof<int>

fsp.Serialize(pickler', stream, 42)
fsp.Deserializer(pickler', stream) : obj

All generated picklers are strongly typed; pickling is performed efficiently without intermediate boxings. Picklers are generated at runtime using reflection and expression trees that are aggressively cached for future use.

Pickler Combinators

FsPickler offers experimental support for generating user-defined picklers using combinators:

open Nessos.FsPickler
open Nessos.FsPickler.Combinators

let p : Pickler<int * string option> = 
    Pickler.string 
    |> Pickler.option 
    |> Pickler.pair Pickler.int
    
let data : byte [] = pickle p (42, Some "")
unpickle p data

The combinator library includes all primitives as described in Andrew Kennedy's Pickler Combinators such as wrap and alt. Fixpoint combinators for declaring recursive picklers are also available:

// val Pickler.fix : (Pickler<'T> -> Pickler<'T>) -> Pickler<'T>

type Peano = Zero | Succ of Peano

let pp : Pickler<Peano> =
    Pickler.fix(fun peano ->
        peano
        |> Pickler.option
        |> Pickler.wrap (function None -> Zero | Some p -> Succ p)
                        (function Zero -> None | Succ p -> Some p))
                        
Succ (Succ Zero) |> pickle pp |> unpickle pp

The library comes with array, array2D, list, set and map combinators that are used to build picklers for the corresponding generic types. The Pickler.seq combinator generates picklers for sequences using eager evaluation of elements. Finally, Pickler.func<'a,'b> : Pickler<('a -> 'b)> instantiates picklers for F# functions.

When it comes to generic types, picklers can be defined using user-defined combinators:

type Tree<'T> = Leaf | Node of 'T * Tree<'T> list

let tree (ep : Pickler<'T>) =
    Pickler.fix(fun tree ->
        tree
        |> Pickler.list
        |> Pickler.pair ep
        |> Pickler.option
        |> Pickler.wrap (function None -> Leaf | Some (t,c) -> Node(t,c))
                        (function Leaf -> None | Node (t,c) -> Some(t,c)))
                        
Node(2,[]) |> pickle (tree Pickler.int)

or it could be done using automatic resolution of type parameters:

let tree<'T> =
    Pickler.fix(fun tree ->
        tree
        |> Pickler.list
        |> Pickler.pair Pickler.auto<'T>
        |> Pickler.option
        |> Pickler.wrap (function None -> Leaf | Some (t,c) -> Node(t,c))
                        (function Leaf -> None | Node (t,c) -> Some(t,c)))


Node([1],[Leaf ; Leaf]) |> pickle tree

Experimental N-way Sum and Product Combinators

N-way sum and product combinators provide an alternative pretty syntax for defining picklers over arbitrary discriminated unions and records. Unfortunately at the moment the performance of resulting picklers is sub-optimal, this might improve in the future.

The types involved in the examples are not fit for human consumption, but thankfully F# infers them automatically. The implementation is total and purely functional, see this gist for some Coq code used to model these combinators.

Records / Product Types

type Person =
    {
        Address : string
        Age : int
        Name : string
    }

let makePerson name age address =
    {
        Address = address
        Age = age
        Name = name
    }

let personPickler =
    Pickler.product makePerson
    ^+ Pickler.field (fun p -> p.Name) Pickler.string
    ^+ Pickler.field (fun p -> p.Age) Pickler.int
    ^. Pickler.field (fun p -> p.Address) Pickler.string

Unions / Sum Types

type U =
| Case1
| Case2 of int
| Case3 of string * int

let uPickler =
    Pickler.sum (fun x k1 k2 k3 ->
        match x with
        | Case1 -> k1 ()
        | Case2 x -> k2 x
        | Case3 (x, y) -> k3 (x, y))
    ^+ Pickler.variant Case1
    ^+ Pickler.case Case2 Pickler.int
    ^. Pickler.case Case3 (Pickler.pair Pickler.string Pickler.int)

Custom Pickler Declarations

Since F# lacks mechanisms such as type classes or implicits, pickler resolution can only be performed by means of reflection. This is done using the following design pattern:

[<CustomPickler>]
type CustomClass<'T> (x : 'T) =

    member __.Value = x

    static member CreatePickler (resolver : IPicklerResolver) =
        let ep = resolver.Resolve<'T> ()
        ep |> Pickler.wrap (fun x -> CustomClass(x)) (fun c -> c.Value)

This tells the pickler generator to yield a pickler for the given type using that particular factory method. The IPicklerResolver argument provides a handle to the pickler generator and can be used for recursive types:

[<CustomPickler>]
type RecursiveClass(?nested : RecursiveClass) =

    member __.Value = nested

    static member CreatePickler (resolver : IPicklerResolver) =
        let self = resolver.Resolve<RecursiveClass> ()
        self 
        |> Pickler.option 
        |> Pickler.wrap (fun x -> RecursiveClass(?nested = x)) (fun rc -> rc.Value)


let p = Pickler.auto<RecursiveClass>

RecursiveClass(RecursiveClass()) |> pickle p |> unpickle p

Pluggable Picklers

When in need to define custom serialization rules for elsewhere-defined types, a pluggable pickler mechanism is provided. Supposing the following custom Pickler<int option>:

let customOptional = Pickler.FromPrimitives((fun _ -> Some 42), fun _ _ -> ())

how could one plug it into the pickler generator?

let registry = new CustomPicklerRegistry("pickler cache with custom optionals")

do registry.RegisterPickler customOptional

let fsp' = new FsPickler(registry)

The above initializes a new pickler cache; all picklers generated from that cache will adhere to the custom rules:

let optionlist = fsp'.GeneratePickler<int option list> ()

// outputs a list of Some 42's
[ for i in 1 .. 100 -> Some i ] |> fsp.Pickle optionlist |> fsp.UnPickle optionlist

Note that auto-generated picklers do not support interop between caches:

let pickler = fsp.GeneratePickler<int list> ()

fsp'.Pickler pickler [1] // runtime error

Care should be taken that custom FsPickler instances are treated as singletons; generating arbitrarily many instances might put strain on the runtime, since each will initialize a unique cache of picklers and emitted code.

Pluggable Generic Picklers

Defining pluggable picklers for generic types is a bit more involved, and requires the use of special factory methods that need to be implemented as instances of the IPicklerFactory interface. The interface itself does not include any methods but instances need to have implemented a factory method with type signature

Create<'T1,...,'Tn when [constraints]> : IPicklerResolver -> Pickler

For instance, supposing I need to define a custom pickler for F# sets:

type CustomSet () =
    interface IPicklerFactory

    member __.Create<'T when 'T : comparison> (resolver : IPicklerResolver) =
        let ep = resolver.Resolve<'T> ()

        ep |> Pickler.wrap (fun x -> set [x]) (fun s -> Seq.head s)
        
do customPicklers.RegisterPicklerFactory (new CustomSet())

let fsp'' = new FsPickler(customPicklers)

I can now combine this rather bad pickler implementation as expected:

let p = fsp''.GeneratePickler<Set<int> option> ()

// will return singleton set
Some(set [ 1 .. 1000 ]) |> fsp.Pickle p |> fsp.UnPickle p

Structural Hashcodes

FsPickler now offers experimental support for structural, non-cryptographic, hashcode generation:

> fsp.ComputeHash [1 .. 100000] ;;
val it : HashResult =
  {Algorithm = "MurMur3";
   Length = 400008L;
   Hash =
    [|52uy; 70uy; 141uy; 214uy; 3uy; 231uy; 11uy; 100uy; 94uy; 250uy; 231uy;
      97uy; 188uy; 215uy; 70uy; 0uy|];}

This will generate a 128-bit structural hashcode based on the MurMurHash algorithm. Implementation is memory efficient, since the hashing algorithm is integrated with the underlying stream implementation. It is possible for users to define their own hashing algorithms by inheriting the special HashStream class.

Hashing functionality offered by FsPickler is an ideal replacement to .GetHashCode() for large objects or complex object graphs, but it is not recommended for small values or primitives.

If a hashcode is not required, the size of an object alone can be computed as follows:

> fsp.ComputeSize [1 .. 1000000] ;;
val it : int64 = 4000008L

Future work

• Support for multiple pickle formats.
• A C# friendly API.