Skip to content

README.md

Latest commit

 

History

History
243 lines (184 loc) · 9.93 KB

README.md

File metadata and controls

243 lines (184 loc) · 9.93 KB
Raw

Welcome to ⚡ Flash Buffer

Flash Buffer NPM Flash Buffer - MIT opensource

A lightning-fast, zero-copy binary data manipulation library for TypeScript.

⚡ FlashBuffer provides a powerful and convenient API for reading from and writing to ArrayBuffer and SharedArrayBuffer. It eliminates manual offset tracking, automatically manages buffer growth, and is designed for maximum performance by avoiding unnecessary memory copying wherever possible.

The library includes a suite of advanced tools: from bit-level operations and VarInt encoding to declarative object serialization.

About | Get Started | Samples | Comparison | Contact Me

About Flash Buffer

🚀 Flash Buffer Features

  • Zero-Copy Operations: Reading bytes returns a direct Uint8Array view into the underlying buffer, not a copy.

Smart Memory Management:

  • Automatic Offset Tracking: The internal cursor advances automatically on reads and writes.
  • Dynamic Buffer Growth: Buffers expand automatically when needed. Choose from exact, powerOfTwo, or fixed growth strategies.
  • Efficient Buffer Pool (FlashBufferPool): Reuses buffers of similar sizes to reduce GC pressure.
  • Resizable ArrayBuffer: Leverages the native ArrayBuffer.prototype.resize() for high-performance memory management.

Cross-Platform:

  • SharedArrayBuffer Support: Enables safe and efficient zero-copy data sharing between threads (Web Workers, Node.js Worker Threads).
  • Works Everywhere: Runs seamlessly in modern browsers, Node.js, Deno, and Bun.

Advanced Data Formats:

  • Automatic data serialization and deserialization: store different objects inside a buffer;
  • VarInt (LEB128) with ZigZag Encoding: Efficient storage for variable-length integers, commonly used in Protocol Buffers.
  • Bit-Level Operations (BitBuffer): Read and write arbitrary numbers of bits (1-32) for flags, compressed data, and cryptography.
  • C-Strings (Null-Terminated Strings): Convenient handling of strings found in system APIs and network protocols.
  • Streaming I/O: Adapters for seamless integration with Web Streams API (FlashReadableStream / FlashWritableStream).
  • Protobuf support: Added protobuf reader / writer based on Flash Buffer;

Schema-Based Serialization (experimental):

  • Declarative Schemas: Define binary structures using TypeScript decorators (@field).
  • Automatic (De)Serialization: The Schema class serializes and deserializes objects to and from binary buffers, eliminating manual errors.

Get Started

📦 Installation

From NPM:

npm install flash-buffer

Or from sources:

git clone https://github.com/devsdaddy/flash-buffer/flash-buffer.git
git cd ./flash-buffer/

💻 Usage Examples

Basic Reading and Writing

Using raw values:

import { FlashBuffer } from 'flash-buffer';

// Create a buffer (auto-growing, little-endian by default)
const buf = new FlashBuffer({ endianness: 'little' });

// Write data
buf.writeUint32(0xDEADBEEF);
buf.writeString('Hello, Flash!', 'utf-8', true); // true = prefix length as uint32

// Reset the offset for reading
buf.reset();

// Read data back
const value = buf.readUint32();
const strLength = buf.readUint32();
const str = buf.readString(strLength);

console.log(value.toString(16)); // 'deadbeef'
console.log(str);               // 'Hello, Flash!'

Using Branded Primitives:

import { FlashBuffer, Uint16LE, Uint32BE, Uint8, Float32LE } from 'flash-buffer';

const newBuf = new FlashBuffer();

// Create primitives
const magic = Uint16LE.create(0x4D42);
const fileSize = Uint32BE.create(102400);
const version = Uint8.create(1);
const compression = Float32LE.create(0.75);
const data = "Hello world";

// Write buffer
newBuf.write(Uint16LE, magic);
newBuf.write(Uint32BE, fileSize);
newBuf.write(Uint8, version);
newBuf.write(Float32LE, compression);
newBuf.writeString(data, "utf-16");

// Reset offset and read
newBuf.reset();
const readMagic = newBuf.read(Uint16LE);
const readSize = newBuf.read(Uint32BE);
const readVer = newBuf.read(Uint8);
const readComp = newBuf.read(Float32LE);
const readData = newBuf.readString(data.length, "utf-16");

Working with VarInt (Variable-Length Integers)

import { FlashBuffer } from 'flash-buffer';

const buf = new FlashBuffer();

// Write a large number (less than 128) in one byte instead of four
buf.writeVarUint(127);

// Write a negative number efficiently using ZigZag encoding
buf.writeVarInt(-15);

buf.reset();

console.log(buf.readVarUint()); // 127
console.log(buf.readVarInt());  // -15
console.log(buf.offset);        // 3 (1 byte for 127 + 2 bytes for -15)

Bit-Level Operations

import { FlashBuffer } from 'flash-buffer';

const buf = new FlashBuffer();
const bits = buf.bit();

// Write a sequence of bits that may cross byte boundaries
bits.writeBits(0b11111, 5);
bits.writeBits(0b10101, 5);
bits.flush(); // Important: flush to finalize the byte and align the offset

buf.reset();
const readBits = buf.bit();
console.log(readBits.readBits(5)); // 0b11111 (31)
console.log(readBits.readBits(5)); // 0b10101 (21)

Objects and Serialization:

import {FlashBufferSchema, field} from 'flash-buffer';

/* Example of serializable class */
class Player {
    @field({type: 'uint32'}) id: number = 0;
    @field({type: 'string'}) name: string = '';
    @field({type: 'float64'}) x: number = 0;
    @field({type: 'float64'}) y: number = 0;
    @field({type: 'varint'}) vint: number = 0;
}

/* Create and fill our class */
let player = new Player();
player.id = 1;
player.name = 'Elijah';
player.x = 55.2
player.y = 21.11;
player.vint = 5991;

/* Serialize and Deserialize */
const serialized = FlashBufferSchema.serialize(player);
console.log("Serialized:", serialized);

const deserialized = FlashBufferSchema.deserialize(Player, serialized.reset());
console.log("Deserialized:", deserialized)

Binary patch:

import { FlashBuffer } from 'flash-buffer';

// Binary patch
const oldBuf = new FlashBuffer();
oldBuf.writeString('The quick brown fox jumps over the lazy dog.');

const newBuf = new FlashBuffer();
newBuf.writeString('The quick brown cat jumps over the lazy dog.');

const patch = oldBuf.diff(newBuf);
const reconstructed = oldBuf.applyPatch(patch);
console.log(reconstructed.toUint8Array() === newBuf.toUint8Array()); // Must be equal with newBuf

NPM Commands

Command Usage
npm run build Build dist package
npm run test Run tests (43 tests included)
npm run bench Run benchmarks

Supported Types

FlashBuffer provides a large list of supported types for buffer:

Category Supported Types
Basic Int8,Uint8, Int16, Uint16, Int32, Uint32, BigInt64, BigUInt64, Float32, Float64, string, (raw) bytes
Advanced VarUint, VarUint64, VarInt, VarSint32, VarSint64, Fixed32, Fixed64, SFixed32, SFixed64

📊 Comparison with Alternatives

Several excellent libraries exist for binary data manipulation. Flash Buffer was designed to combine their strengths and offer unique capabilities.

Feature flash-buffer smart-buffer @hazae41/binary @jsonjoy.com/buffers
Zero-Copy ✅ ❌ ✅ ❌
**SharedArrayBuffer** ✅ ❌ ❌ ❌
Automatic Growth ✅ ✅ ❌ ✅
Streaming (Web Streams) ✅ ❌ ❌ ✅
Bit-Level Operations ✅ ❌ ❌ ❌
VarInt (LEB128) ✅ ❌ ❌ ❌
C-Strings ✅ ✅ ❌ ❌
Schema Serialization ✅ ❌ ✅ ❌
Buffer Pool ✅ ❌ ❌ ❌

Advantages of flash-buffer

  • Built-in SharedArrayBuffer Support: This is a key differentiator, unlocking high-performance multi-threading without unnecessary data copying. It is essential for resource-intensive client applications and high-load servers.
  • Modern and Intuitive API: The API is inspired by the native DataView, making it easy to learn and use.
  • Powerful Toolset: flash-buffer is not just a reader/writer. It is a comprehensive toolkit including VarInt, bit-level operations, and streaming adapters, allowing you to solve 99% of binary data tasks without pulling in additional libraries.

Flash Buffer library is distributed under the MIT license. You can use it however you like. I would appreciate any feedback and suggestions for improvement. Full license text can be found here

Have a questions? Contact me

About | Get Started | Comparison