Usenet

A .NET library for working with Usenet. It offers:

• an NNTP client
• an NZB document parser, builder and writer
• a yEnc encoder and decoder

It is built around a byte-oriented, System.IO.Pipelines transport that keeps allocations low: article bytes are framed off the wire without being transcoded to strings, yEnc parts decode into pooled buffers, and yEnc encoding streams straight into an IBufferWriter<byte>. The NNTP client is compliant with RFC 2980, RFC 3977, RFC 4643 and RFC 6048.

Install

dotnet add package Spottarr.Usenet

Usage

Connect and authenticate

Connection identity (host, port, SSL) lives on NntpConnectionOptions, so ConnectAsync reads it from configuration rather than taking it as arguments:

var options = new NntpConnectionOptions { Host = hostname, Port = port, UseSsl = useSsl };
var client = new NntpClient(new NntpConnection(options));
await client.ConnectAsync();
await client.AuthenticateAsync(username, password);

Retrieve an article

An article response owns a pooled buffer, so dispose it (using/await using). Read the body as raw bytes via Body, or as text lines on demand via ReadBodyLines():

await using var response = await client.ArticleAsync(messageId);
if (response.Success)
{
    foreach (var line in response.ReadBodyLines())
    {
        // ...
    }
}

To inspect headers without paying to transfer the body, use HeadAsync first and issue a conditional BodyAsync only when needed.

Stream an overview range

Unbounded scans (XOVER/OVER, HDR, LISTGROUP, NEWNEWS, …) stream typed rows as an IAsyncEnumerable<T>, so memory stays flat over arbitrarily large ranges. Enumerate the result fully, or dispose it, before issuing the next command on the connection:

await client.GroupAsync("alt.binaries.example");

// OverAsync (RFC 3977) and XoverAsync (legacy) stream the same typed rows; servers implement one or
// the other.
await using var overviews = await client.OverAsync(NntpArticleRange.Range(1000, 2000));
await foreach (var overview in overviews)
{
    Console.WriteLine($"{overview.Number}\t{overview.Subject}");
}

The by-message-id forms address a single article, so they return one record directly instead of a stream — OverByMessageIdAsync yields NntpArticleOverview? and HdrByMessageIdAsync/XhdrByMessageIdAsync yield NntpHeaderField? (null when the article is absent), with no enumerate-or-dispose contract to honour:

var overview = await client.OverByMessageIdAsync(messageId);
if (overview is not null)
{
    Console.WriteLine($"{overview.Subject}\t{overview.Bytes} bytes");
}

Compress the transport

Servers that support RFC 8054 COMPRESS DEFLATE can compress the whole session, which is a large saving on the headline workload — XOVER over millions of articles. It is enabled as a connection option rather than a command: set Compression on NntpConnectionOptions and the connection negotiates it after authentication, after which every command and response — overview and article alike — is transparently compressed in both directions. The streamed scans are unchanged — they just ride compressed bytes:

var options = new NntpConnectionOptions
{
    Host = hostname,
    Port = port,
    UseSsl = useSsl,
    Compression = NntpCompression.Deflate,
};

var client = new NntpClient(new NntpConnection(options));
await client.ConnectAsync();
await client.AuthenticateAsync(username, password); // compression is negotiated here

await using var overviews = await client.OverAsync(NntpArticleRange.Range(1, 1_000_000));
await foreach (var overview in overviews) { /* ... */ }

The pool re-applies the option on every transparent reconnect, so pooled clients stay compressed. A server that does not support compression fails the connection at setup (an NntpException) rather than silently serving plaintext.

Inspect server capabilities and metadata

The bounded CAPABILITIES and LIST commands return ready-to-use typed results rather than raw lines. CapabilitiesAsync yields an NntpCapabilities you can query directly, and the bounded LIST variants return typed models (NntpOverviewFormat, NntpGroups, NntpDistribution, …):

var capabilities = await client.CapabilitiesAsync();
if (capabilities.IsReader && capabilities.Supports("OVER", "MSGID"))
{
    var overview = await client.OverByMessageIdAsync(messageId);
    // ...
}

// The overview field layout, in order, including :metadata items and :full header-name fields.
var format = await client.ListOverviewFormatAsync();
foreach (var field in format.Fields)
{
    Console.WriteLine(field.IsMetadata ? $":{field.Name}" : field.Name);
}

The genuinely free-form HELP, LIST MOTD and LIST HEADERS commands return an NntpTextResponse exposing the raw Lines plus a convenience joined Text.

Decode a yEnc part

YencDecoder.Decode returns a YencPart that owns a buffer rented from ArrayPool. Its Data view is valid until the part is disposed, and the per-part pcrc32 (or crc32 for a single-part file) is verified during decoding:

// encoded holds the raw yEnc bytes of one part (ReadOnlyMemory<byte> or ReadOnlySequence<byte>)
using var part = YencDecoder.Decode(encoded);

YencHeader header = part.Header;
ReadOnlyMemory<byte> data = part.Data;

using var file = File.Open(header.FileName, FileMode.OpenOrCreate, FileAccess.Write, FileShare.ReadWrite);
file.Position = header.PartOffset;
await file.WriteAsync(data);

Download an NZB document

Parse the document, then for each segment retrieve the article, decode its body, and write the decoded part to the file at its offset. Only one article is held in memory at a time:

var nzbDocument = await NzbParser.ParseAsync(await File.ReadAllTextAsync(nzbPath));

foreach (var nzbFile in nzbDocument.Files)
{
    foreach (var segment in nzbFile.Segments)
    {
        await using var response = await client.BodyAsync(segment.MessageId);
        if (!response.Success)
            continue;

        using var part = YencDecoder.Decode(response.Body);
        var header = part.Header;

        using var file = File.Open(header.FileName, FileMode.OpenOrCreate, FileAccess.Write, FileShare.ReadWrite);
        if (file.Length == 0)
            file.SetLength(header.FileSize); // pre-allocate on first segment

        file.Position = header.PartOffset;
        await file.WriteAsync(part.Data);
    }
}

Build and post an article

var messageId = $"{Guid.NewGuid()}@example.net";

var article = new NntpArticleBuilder()
    .SetMessageId(messageId)
    .SetFrom("Random poster <randomposter@example.net>")
    .SetSubject("Random test post #1")
    .AddGroups("alt.test.clienttest", "alt.test")
    .AddLine("This is a message with id " + messageId)
    .AddLine("with multiple lines")
    .Build();

await client.PostAsync(article);

Encode yEnc into a buffer

The encoder reads the source in blocks and writes straight into an IBufferWriter<byte> through a precomputed escape table:

var writer = new ArrayBufferWriter<byte>();
await YencEncoder.EncodeAsync(header, stream, writer);
ReadOnlyMemory<byte> encodedBytes = writer.WrittenMemory;

Build and write an NZB document

var fileProvider = new PhysicalFileProvider(Path.GetFullPath("testdata"));

var builder = new NzbBuilder()
    .AddGroups("alt.test.clienttest")
    .SetMessageBase("random.local")
    .SetPartSize(50_000)
    .SetPoster("random poster <random.poster@random.com>")
    .AddMetaData("title", "Testing upload Pictures.rar");

foreach (var fileName in fileNames)
{
    builder.AddFile(fileProvider.GetFileInfo(fileName));
}

var nzbDocument = builder.Build();

using var file = File.Create("Pictures.nzb");
await using var writer = new StreamWriter(file, UsenetEncoding.Default);
await writer.WriteNzbDocumentAsync(nzbDocument);

Connection pooling

NntpClientPool manages a set of authenticated, connected clients and hands them out as disposable leases. Connecting and authenticating happen lazily the first time a client is borrowed, and idle clients are disconnected automatically:

using var pool = new NntpClientPool(
    new NntpPoolOptions
    {
        MaxPoolSize = 10,
        Username = username,
        Password = password,
        Connection = new NntpConnectionOptions { Host = hostname, Port = port, UseSsl = useSsl },
    });

using var lease = await pool.GetLease();
await using var response = await lease.Client.ArticleAsync(messageId);
// the client is returned to the pool when the lease is disposed

Logging

Logging is optional and flows through Microsoft.Extensions.Logging. Hand the components an ILoggerFactory directly, or register them with DI and they resolve one from the container:

// direct
var connection = new NntpConnection(options, loggerFactory);
var client = new NntpClient(connection, loggerFactory);

// dependency injection — register an NntpConnectionOptions to configure the connection
services.AddUsenet();

Close the connection

await client.QuitAsync();

Architecture

The library is split into independent layers — Connection (transport), Client (the RFC command API), Pool (reusable authenticated clients), and the independent yEnc and NZB codecs. The streaming/buffering model and the reasoning behind it are documented for maintainers in docs/architecture.md, backed by the ADRs.

License

This project is licensed under the MIT License - see the LICENSE.md file for details.

Acknowledgments

This is a standalone library. It descends from keimpema/Usenet, which was in turn based on Kristian Hellang's work:

• khellang/yEnc
• khellang/NntpLib.Net
• khellang/Nzb