[8/15] Docs: rewrite the files and websockets guides, add the v4-to-v5 migration guide

docs/files.md

@@ -1,115 +1,199 @@
 # File Systems
 
-**OUT OF DATE**
+Last updated July 2025 (`version-5`)
 
-Storing, serving, and using user-provided files is a common requirement, and as such, is built-in directly.
+Storing, serving, and using user-provided files is a common requirement, so Lightning Server provides a file-system
+abstraction and a set of ready-made upload endpoints. The abstraction lives in the service-abstractions library
+(`com.lightningkite.services.files`), and the HTTP endpoints that wrap it live in the `files` / `files-shared`
+modules.
 
-Valid file backends that have been built so far are Local, S3, and Azure Blob Storage. SFTP is also partially
-supported - it doesn't support public URLs.
+Available backends include a local (KotlinX-IO) file system and AWS S3. The API is modeled loosely on Kotlin's
+built-in file functions.
 
-The API is roughly based on Kotlin's built-in file functions.
+## Declaring the need for a file system
 
-## Declaring the need for a file sysstem
+Add a setting whose value is a `PublicFileSystem`:
 
-Add a setting as follows:
+```kotlin
+import com.lightningkite.services.files.PublicFileSystem
+
+object Server : ServerBuilder() {
+    val files = setting("files", PublicFileSystem.Settings())
+}
+```
+
+To make additional backends available, reference them in an `init` block so their loaders register themselves:
 
 ```kotlin
-object Server {
-    //...
-    val files = setting(name = "files", default = FilesSettings())
-    //...
+import com.lightningkite.services.files.s3.S3PublicFileSystem
+
+object Server : ServerBuilder() {
+    init { S3PublicFileSystem }
+    val files = setting("files", PublicFileSystem.Settings())
 }
 ```
 
+Like every service, the setting resolves to a `Runtime<PublicFileSystem>`. You invoke it inside a
+`ServerRuntime` context (for example, within a handler) to get the live `PublicFileSystem`.
+
 ## Accessing files
 
+`FileObject` is the internal handle used to read and write files. Get one from the file system's `root` and
+navigate with `then`:
+
 ```kotlin
-val rootFolder = Server.files().root
+import com.lightningkite.services.data.TypedData
+import com.lightningkite.MediaType
 
-val testFile = rootFolder.resolve("some/path/file.txt")
+val root = files().root
 
-// Files are written in terms of `HttpContent`
-// There's no mkdir.  If a folder does not exist it will be created.
-testFile.put(HttpContent.Text("Hello world!", ContentType.Text.Plain))
+val testFile = root.then("some/path/file.txt")
 
-// Files are read in terms of `HttpContent` as well
-testFile.get()!!.stream().use { it: InputStream ->
-    it.readAllBytes()
-}
+// Files are written and read as TypedData. Parent folders are created implicitly.
+testFile.put(TypedData.text("Hello world!", MediaType.Text.Plain))
 
-// You can just get metadata too
-testFile.head()
+// get() returns TypedData?, null if the file does not exist.
+val text = testFile.get()?.text()
 
-// Generates a file reference with a large random identifier in it.
-val newFile = rootFolder.resolveRandom("test", "txt")
+// Metadata only (media type, size) without downloading the body.
+val info = testFile.head()
 
-// You can list files too.
-rootFolder.list().forEach {
-    println(it)
-}
+// Remove a file.
+testFile.delete()
 ```
 
 ## Serving files
 
-These files have URLs that are signed for retrieval. Performing an HTTP GET will result in the file.
+Files can be served over signed URLs. Performing an HTTP GET against a signed URL returns the file:
 
 ```kotlin
-// Duration of the signature is determined by the file system settings
+// The signature duration is determined by the file system settings.
 println(testFile.signedUrl)
 ```
 
-## Uploading files from a client
+### `FileSystemEndpoints`
 
-You can sign an upload URL like so:
+`FileSystemEndpoints` exposes HEAD/GET/PUT handlers for a `PublicFileSystem`, including HTTP Range support on GET
+for partial/resumable downloads. GET streams the bytes, HEAD returns metadata, and PUT accepts an upload to a
+signed upload URL (only the local `KotlinxIoPublicFileSystem` accepts server-generated uploads this way; S3/Azure
+sign their own upload URLs).
 
 ```kotlin
-// Include the expiration duration
-alt.uploadUrl(Duration.ofMinutes(10))
+val fileServing = path.path("files") include FileSystemEndpoints(files)
 ```
 
-Performing an HTTP PUT with the file's contents will overwrite that file.
+## Uploading files from a client
 
-## Serialization
+You can sign a PUT upload URL for any file reference:
 
-`FileObject` is a file-system resolve object which can be used to read and write files. They are purely internal to the
-server.
+```kotlin
+import kotlin.time.Duration.Companion.minutes
 
-`ServerFile` is a wrapper around a string that contains a public URL for an object. They are used in APIs and
-serialization.
+val uploadUrl = root.then("uploads/photo.jpg").uploadUrl(10.minutes)
+```
 
-You can switch between the two using `FileObject.serverFile` and `ServerFile.fileObject`.
+Performing an HTTP PUT of the file's contents to that URL writes the file. Upload URLs never grant read access.
 
-### Security
+## `ServerFile` and serialization
 
-When a `ServerFile` is sent to a client, the url is automatically signed for reading. Therefore, if you wish to keep a
-file in your file system secure, only serialize references to it for the people you want to read it.
+Two types work together:
 
-## Default File Upload Endpoints
+- `FileObject` is the internal read/write handle. It never leaves the server.
+- `ServerFile` (`com.lightningkite.services.files.ServerFile`) is a serializable wrapper around a URL string.
+  It is what you store in models and send over your API.
 
-There is a pre-built upload endpoint for uploading files to use in subsequent requests. It requires a reference to the
-intended file system to use, a database to track whether the file has been used (if it's unused, it is garbage
-collected), and a `JwtSigner` setting to secure file reuse.
+```kotlin
+import com.lightningkite.services.files.ServerFile
+
+@GenerateDataClassPaths
+@Serializable
+data class Post(
+    override val _id: Uuid = Uuid.random(),
+    val title: String,
+    @MimeType("image/*") val coverImage: ServerFile? = null,
+) : HasId<Uuid>
+```
 
-This endpoint prevents abuse of your file system by returning two URLs: a `uploadUrl` and a `futureCallToken`.
+You move between the two by resolving the `ServerFile`'s location against the file system:
+
+```kotlin
+val obj = files().root.then(post.coverImage!!.location) // ServerFile -> FileObject
+val ref = ServerFile(obj.url)                            // FileObject -> ServerFile
+```
+
+### Security
 
-`uploadUrl` is the URL which the client should PUT their file to.
+When a `ServerFile` is serialized out to a client, its URL is automatically signed for reading. Consequently, a
+file stays private as long as you only serialize references to it for people you want to be able to read it. Do not
+build a public file-sharing feature on top of this behavior by accident.
 
-`futureCallToken` is a URL that can be used as a `ServerFile` in a subsequent request.
+## The `UploadEarlyEndpoint` (recommended upload flow)
 
-Neither URL will allow reading of the file, and thus, you cannot abuse this endpoint as a file-sharing system.
+`UploadEarlyEndpoint` is an opinionated group of endpoints for the common "upload now, reference later" pattern. It
+lets a client upload a file before the request that actually uses it, without turning your file system into an open
+file-sharing service.
 
-It is *strongly* recommended that you use this endpoint for handling files in your API rather than attempting to
-implement it yourself.
+Construct it with the file system, a database (used to track pending uploads for garbage collection), and a list of
+`FileScanner`s (often empty):
 
 ```kotlin
-val upload = UploadEarlyEndpoint(path("early-upload"), files, database, signer)
+import com.lightningkite.lightningserver.files.UploadEarlyEndpoint
+import com.lightningkite.lightningserver.definition.Runtime
+
+object Server : ServerBuilder() {
+    val files = setting("files", PublicFileSystem.Settings())
+    val database = setting("database", Database.Settings())
+
+    // Register ONCE. See the warning below.
+    val uploadEarly = path.path("upload") module UploadEarlyEndpoint(
+        files = files,
+        database = database,
+        fileScanner = Runtime.Constant(listOf()),
+    )
+}
 ```
 
+### The flow
+
+1. The client calls the `endpoint` (GET) to obtain an `UploadInformation`:
+    - `uploadUrl` — a presigned PUT URL to upload the bytes to.
+    - `futureCallToken` — a token that can be sent as a serialized `ServerFile` in a later request.
+2. The client PUTs the file to `uploadUrl`.
+3. The client includes `futureCallToken` as the `ServerFile` value in a subsequent API call.
+
+Neither URL grants read access, so the endpoint cannot be abused as a file host. A daily `cleanupSchedule` deletes
+uploads that were prepared but never used before their `expiration` (default one day).
+
+It is strongly recommended that you use this endpoint for API file handling rather than implementing the flow
+yourself.
+
+### Quarantine (jail) and scanning
+
+If you pass a non-empty list of `FileScanner`s (for example, the ClamAV scanner from the `files-clamav` module),
+uploads are first written to a jailed location (`jailFilePath`, default `upload-jail`) instead of the ready
+location (`filePath`, default `uploaded`). Before the
+file can be used it must be scanned and moved out of jail. The client does this by calling the `verify` (POST)
+endpoint with the token; if the file is safe it is moved to the ready location and a reusable reference is returned.
+Verifying up front also makes the subsequent request faster, since the scan has already happened.
+
+With an empty scanner list, uploads go straight to the ready location and are certified as already scanned.
+
+### ⚠️ Only one `UploadEarlyEndpoint` per server
+
+`UploadEarlyEndpoint` registers a **contextual serializer for `ServerFile`** (via its `externalSerialization`) so
+that upload tokens can be decoded as `ServerFile` values. If you instantiate `UploadEarlyEndpoint` more than once
+(for example, a separate instance created only for a test), the multiple instances register **conflicting
+`ServerFile` serializers**. This does not fail at compile time — it surfaces at runtime as serialization errors,
+typically appearing as `500 Internal Server Error` responses.
+
+**Instantiate `UploadEarlyEndpoint` exactly once in your server definition and reference that single instance
+everywhere, including from tests.**
+
 ## Available Backends
 
 ### Local
 
-Simply use a local filesystem folder.
+Use a local filesystem folder:
 
 ```json5
 // settings.json
@@ -122,9 +206,8 @@ Simply use a local filesystem folder.
 
 ```kotlin
 // Server.kt
-object Server: ServerPathGroup(ServerPath.root) {
-    // Adds S3FileSystem to the possible file system loaders
-    init { S3FileSystem }
+object Server : ServerBuilder() {
+    init { S3PublicFileSystem }
 }
 ```
 
@@ -134,20 +217,3 @@ object Server: ServerPathGroup(ServerPath.root) {
   "files": { "url": "s3://[user]:[password]@[bucket].[region].amazonaws.com" }
 }
 ```
-
-### Azure Blob Storage
-
-```kotlin
-// Server.kt
-object Server: ServerPathGroup(ServerPath.root) {
-    // Adds AzureFileSystem to the possible file system loaders
-    init { AzureFileSystem }
-}
-```
-
-```json5
-// settings.json
-{
-  "files": { "url": "azbs://key@account/container" }
-}
-```