Skip to content

Add forwards compatibility for changes to broadcast messages #90

Description

@rfratto

As noted in #89, nodes will silently ignore unrecognized fields when rebroadcasting a received state message.

If new fields are added to state messages, this can cause convergence issues, and would rely on full state syncs between nodes that do recognize the fields to finally converge, which could take several minutes.

In the short term, this issue is addressed by adding new messages instead of expanding existing messages. However, this may become burdensome over time and difficult to remember the constraints when adding new functionality.

We should address this and make sure that there's forwards compatibility between messages. This can be done by upgrading to github.com/hashicorp/go-msgpack/v2 (from v1), which supports the MissingFielder interface, used during decoding and encoding.

There are two potential ways this could be used, described below.

Option 1: implement interface per-message

Each message type could implement this interface to provide forwards compatibility:

// State represents a State change broadcast from a node.
type State struct {
	// Name of the node this state change is for.
	NodeName string
	// New State of the node.
	NewState peer.State
	// Time the state was generated.
	Time lamport.Time

        unknownFields map[string]any 
}

// ... 


func (s *State) CodecMissingField(field []byte, value any) bool {
  s.unknownFields[string(byte)] = value 
  return true 
}

func (s *State) CodecMissingFields() map[string]any {
  return s.unknownFields 
}

The messages.Message interface would also be updated to ensure that MissingFielder is always implemented for broadcastable messages.

All code in ckit would need to take care to copy these unknown fields when constructing a copy of a peer's message.

Option 2: handled by messages.Decode, messages.Encode, and messages.Broadcast

Alternatively, messages.Decode and messages.Broadcast could be updated to expose unknown fields without needing to reflect them in messages:

package messages

// Decode decodes a message from Parse into m. Unknown fields are returned
// and can be redelivered with [Encode].  
func Decode(buf []byte, m Message) (unknown map[string]any, err error) 

// Encode encodes m into a byte slice that can be broadcast to other peers. 
// The unknown map permits broadcasting additional fields not known to m, 
// usually retrieved from [Decode]. 
//
// Encode will panic if the Type of m is invalid or unknown. 
func Encode(m Messages, unknown map[string]any) (raw []byte, err error) 

// Broadcast converts m into a memberlist Broadcast. m should not change once
// being converted into a Broadcast.
//
// The unknown map permits broadcasting additional fields not known to m,
// usually retrieved from [Decode].
//
// onDone will be called once the message has been broadcasted or invalidated.
//
// Queueing the resulting Broadcast will invalidate all previous broadcasts
// for messages with the same name; this means that callers should only queue
// newer messages.
func Broadcast(m Message, unknown map[string]any, onDone func()) (memberlist.Broadcast, error) 

Option 2 is the stronger API if there's a way to implement it correctly (as it would require wrapping the message to decode to catch unknown fields).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions