-
Notifications
You must be signed in to change notification settings - Fork 1
2) Packet Formats
Before covering the packet format, some there's some key points to go over. The N64NL network stack uses UDP communication for everything (with the exception of file downloads from the master server), and this presents a problem for reliability. So the N64NL packet system implements two solutions for this: Sequence numbers and packet flags.
In networking, there's a famous hypothetical called the "Two Generals' Problem". In essence, you have two generals need to coordinate the attack on an enemy territory, but they are only able to communicate over an unreliable channel using a messenger. If only one army attacks, the battle is lost. But if both attack at the same time, they win.
If general A sends general B a message with the date and time of the attack, there's a risk that the messenger is killed during the delivery and B never receives the information, meaning A attacks alone and loses. So one easy solution is to require B to send an acknowledgement message. But if B sends the acknowledgement and that message is lost, B attacks alone and loses. So you could require A to send an ack of the ack, but that has the same pitfall...
The approach used in this networking library is to use the concept of "Sequence Numbers". In essence, every packet needs an acknowledgement, and for every few seconds it doesn't receive an acknowledgement it will resend the packet until the acknowledgement arrives. When a connection is established the client and server will have a "Sequence Number" which is local to them (for instance, the client could start at 1 and the server at 100). Both the client and the server will also have a "Remote Sequence Number", which is the highest sequence number of all received packets thus far. When a packet is sent, the local sequence number and the remote sequence number is written on the packet, and afterwards the local sequence number is incremented.
Here's an example communication. In this example, the Client starts with LSN at 1 and RSN at 0, while the Server starts with LSN at 100 and RSN at 0:
- A client sends a packet to the server with Local Sequence Number 1, and after the packet is sent, they increment their LSN to 2.
- The server receives this packet and checks its Remote Sequence Number (which is initialized to 0). Since the received packet has a LSN larger than the server's RSN, the server's RSN value is set to 1 (the value in the packet).
- The server replies back by sending a packet with LSN 100 and RSN 1. The server's SN is incremented to 101.
- The client receives this packet and checks its RSN (which is initialized to 0). Since the received packet has a LSN larger than the client's RSN, the client's RSN value is set to 100.
- The client replies back by sending a packet with LSN 2 and RSN 100 (incrementing the LSN to 3 afterwards).
- The server receives this packet and checks its RSN (which is 1). Since the received packet has a SN larger than the server's RSN, the server's RSN value is set to 2.
- The server replies back by sending a packet with LSN 101 and RSN 2. The server's LSN is incremented to 102.
This system works fine if every time we send a packet, we receive exactly one back. But sometimes a server can send three or four packets at once, and we might only need to send one back. So the solution to this is that both the client and the server will keep a copy of all packets that were received, and when sending a packet, it will also append the list of all previously received numbers. However, since each sequence number is 16-bits, we'll be quickly wasting space listing old sequence numbers.
Because the sequence numbers are sent in sequential order, we can leverage this by storing this list in a bitfield. The Remote Sequence Number always contains the last largest received packet number, so we can use a bitfield to represent the last N sequence numbers we have received. For example, let's say a client received 4 packets with sequence number 100, 101, 103, and 104. When the client finally gets the chance to reply back, it will send a packet with the remote sequence number 104 and a bitfield with 0000 1101 (In this example we are using an 8-bit bitfield for legibility):
- The first bit (the right-most one) is
1, meaning that the client also acks packet 103 (104 - 1). - The second bit is
0, meaning that the client did not receive packet 102 (104 - 2) - The third bit is
1, meaning that the client also acks packet 101 (104 - 3). - The fourth bit is
1, meaning that the client also acks packet 100 (104 - 4).
When the server receives this packet from the client, it will know which packets were successfully acknowledged and resend the packet that wasn't.
When the client receives a new packet from the server and updates their RSN, the bitfield also needs to be updated. Using the previous example, if the client receives a packet with the number 105, we update the RSN first, and then update the bitfield to 0001 1011, since:
- The first bit (the right-most one) is
1, meaning that the client also acks packet 104 (105 - 1). - The second bit is
1, meaning that the client also acks packet 103 (105 - 2). - The third bit is
0, meaning that the client still has not received packet 102 (105 - 3). - etc...
If the client receives the missing packet 102 from the server, the RSN is not updated (since 102 < 105), but the bitfield for the next packet to be sent should be updated to set the bit corresponding to 102 to 1.
The sequence numbers used by this networking stack are 16 bit. This means that only 17 packets can be acked by a single packet, so if too many packets aren't acked back (or they are resent too many times without an acknowledgement), the packet is considered completely lost.
The browser application and example server code all use a "UDPHandler" object which deals with all of this sequence number stuff for you.
The following flags can be appended to packets:
-
UNRELIABLE - 0x01- This packet does not require an ack, and will never be resent. -
EXPLICITACK - 0x02- This packet requires an immediate reply back in the form of an actual acknowledgement packet (as opposed to using the sequence number system). More on this in a bit.
There are two types of packets used in the N64NL stack: Server64 (S64 for short) and NetLib packets.
These packets are used by the NetLib Browser in order for users to be able to find and search for servers. It is exclusively for master server on server communication, client on master server communication, and exceptionally for clients performing a discovery ping on a server. These packets will never be used in an N64 ROM.
Because the pinging of the master server (and subsequently of servers we want to discover) does not require a constant ping pong of packets, S64 packets tend to use the EXPLICITACK and UNRELIABLE flags as opposed to using the sequence number system described above.
The packet format is as follows:
Packet header - 3 bytes containing 'S', '6', '4'
Packet version - 1 byte (currently 1)
Flags - 1 byte
Local sequence number - 2 bytes
Remote sequence number - 2 bytes
Ack bitfield - 2 bytes
Type string length - 1 byte
Type string - N bytes (N was provided by the type string length field)
Data length - 2 bytes
Data - M bytes (M was provided by the data length field)
The following is a list of all used packet types. When writing a server, you will only need to worry about the ACK, REGISTER, HEARTBEAT and DISCOVER packet types. Despite that, all packet types are shown here for completeness sake.
Works as an explicit ack for packets with EXPLICITACK flags.
Used by servers to register themselves with the master server. Should contain:
Port number - 4 bytes
ROM name length - 4 bytes
ROM name (string) - N bytes (N was provided by the ROM name length field).
ROM hash length - 4 bytes
ROM hash (SHA-256) - M bytes (M was provided by the ROM hash length field).
The master server can also send this packet type back to a server if it needs the server to register itself again. In this scenario, there will be no data appended to the packet.
Sent by servers to the master server to remind the master server that they exist.
Sent by clients to the server so that they can check if the server exists. The packet should contain:
Identifier length - 4 bytes
Identifier - N bytes (N was provided by the identifier length field).
The server will echo back a packet with the same type. This packet will contain the following:
Identifier length - 4 bytes
Identifier (should be the same as the received packet) - N bytes (N was provided by the identifier length field).
Server name length - 4 bytes
Server name (string) - M bytes (M was provided by the server name length field).
Player count - 4 bytes
Max players supported - 4 bytes
Used by clients to request a list of servers from the master server.
Used by clients to request a file download from the master server. Should contain:
ROM hash - N bytes (N was provided by the data length field of the S64 packet). Should be a collection of bytes representing a SHA-256 hash.
When received by the master server, it will echo back a packet with the same type. If the ROM is not available for download, no data will be appended. Otherwise, the following data is appended:
File size (in bytes) - 4 bytes
Size of downloadable chunks (in bytes) - 4 bytes
Afterwards, the user is expected to allow a TCP socket to be opened to the master server for the file to be downloaded. The user should receive ceil(F/C) chunks of size C (where F is the filesize and C is the chunk size in the previous packet).
NetLib packets are used exclusively by communication between N64's and game servers. When writing a game ROM, this is pretty much everything you will need to worry about. You should never need to use the EXPLICITACK flag, as the client and server should be constantly heartbeating one another (unless you are doing one-sided communication, like downloading a file).
NetLib packets have a "Recipients bitfield", which is used to specify which clients should receive the packet. Upon joining a server, a player should be given a client number, and when sending packets, the intended recipient of the packet is stored in a bit field (So, if the packet is intended for the player with client number 1, the first bit is set. If the packet is intended for the player with client number 2, the second bit is set, etc...). Packets with the recipient number 0 are meant for the server only.
The packet format is as follows:
Packet header - 3 bytes containing 'N', 'L', 'P'
Packet version - 1 byte (currently 1)
Type number - 1 byte
Flags - 1 byte
Local sequence number - 2 bytes
Remote sequence number - 2 bytes
Ack bitfield - 2 bytes
Recipients bitfield - 4 bytes
Data length - 2 byte
Data - N bytes (N was provided by the data length field)
Packet types in NetLib packets are stored as a number, and are defined on a per-game basis. Packet type 0 is used exclusively for packets with EXPLICITACK flags / heartbeats (these packet will have no data appended), all other numbers from 1 to 255 are free to use.