docs: introduce Connect section, connect-string reference, QWP stubs#444
docs: introduce Connect section, connect-string reference, QWP stubs#444bluestreak01 wants to merge 78 commits into
Conversation
Consolidate how-applications-talk-to-QuestDB content under a single Connect supersection. Rewrite the ingestion overview as QWP-native "Connect to QuestDB", add a comprehensive connect-string reference at documentation/client-configuration/, and scaffold the Wire Protocols sub-section under documentation/protocols/. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
🤖 Component Converter ReminderA component in We are creating markdown correspondents of every path (e.g. questdb.com/docs/quick-start/ → questdb.com/docs/quick-start.md) for LLM consumption. Quick Check
💡 This is a friendly reminder, not a blocker. Ignore if not applicable. |
|
🚀 Build success! Latest successful preview: https://preview-444--questdb-documentation.netlify.app/docs/ Commit SHA: 15d4277
|
UDP documentation is deferred. Removes the stub page, sidebar entry, and links from ingress, egress, overview, and connect-string pages.
- Add "Why implement a QWP client" pitch and "Client lifecycle" narrative so new implementers can orient before diving into encoding details. - Spell out sequence numbering (server-assigned by receive order, not in the wire header), Gorilla first-DoD anchor, decimal scale formula (value = unscaled / 10^scale), and VARCHAR offset endianness — closes silent-wrong-guess risks for one-shot client generation. - Collapse Symbol section to WebSocket-only (per-table dict is UDP) and drop the now-stranded per-table example. - Document the practical WebSocket frame cap: http.recv.buffer.size (default 2 MiB) is the real ceiling, not the 16 MB protocol limit; exceeding it returns close code 1009 MESSAGE_TOO_BIG. - Fill out durable-ack semantics: watermark trails OK, empty messages trivially durable, reconnects discard in-flight tracking. - Note X-QWP-Client-Id may influence version selection. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding # Conflicts: # documentation/protocols/qwp-ingress-websocket.md
- Add "Why implement a QWP query client" pitch and "Client lifecycle" narrative paralleling the ingress doc; surfaces the java-questdb-client reference impl link upfront. - Document the practical WebSocket frame cap on /read/v1: client-to-server frames (QUERY_REQUEST in particular) are bounded by http.recv.buffer.size (default 2 MiB), not the 16 MiB protocol limit; oversized frames are rejected with close code 1009 MESSAGE_TOO_BIG. - Clarify X-QWP-Max-Batch-Rows only asks for smaller batches than the server default (clamps to server's hard limit). - Tighten NULL sentinel docs: FLOAT/DOUBLE sentinel is *any* NaN (incl. 0.0/0.0); IPv4 0.0.0.0 and all-ones GEOHASH cannot round-trip as non-null. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…sage Brokers up to Connect - Add a "Use QWP for new clients" tip callout to ilp/overview.md naming QWP's wins (binary, type-rich, faster, failover, store-and-forward) and framing ILP as the path for InfluxDB / Telegraf / Kafka / Flink users who already emit ILP. - Shorter callout on ilp/columnset-types.md reframing the page as "extensions on top of the InfluxDB type model" and noting QWP exposes the full QuestDB type system natively (no suffix encoding, no casts). - Operator-facing callout on ilp/advanced-settings.md flagging this page as the legacy ILP tuning surface and pointing new deployments at QWP. - Sidebar: lift Java Embedded and Message Brokers out from under ILP (they're protocol-agnostic delivery mechanisms, not ILP sub-pages). Final Connect order: Overview, Connect string, Date to Timestamp, Client Libraries, Message Brokers, Compatibility Protocols, Java Embedded, Wire Protocols. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eader) The egress endpoint /read/v1 is asymmetric on the wire: server-to-client frames carry the 12-byte QWP header, but client-to-server frames start directly with msg_kind — no QWP header. Including the header makes the server read 0x51 (the ASCII 'Q' of "QWP1") as an unknown msg_kind and close the WebSocket with code 1006, partway through send. Verified against server (QwpEgressUpgradeProcessor.dispatchEgressMessage calls peekMsgKind at offset 0 of the WS frame body) and Java reference client (QwpEgressIoThread.sendQueryRequest writes msg_kind as the first byte, no header). The upstream wire-egress.md spec is wrong on this point and should be filed separately. - Rewrite Message structure section with two ASCII diagrams (server-to- client with header, client-to-server without) and a warning callout naming the symptom and the reason (server keeps the header for RESULT_BATCH's flags + payload_length; client control frames have no analogous need). - Fix Example 1: drop the bogus 12-byte header from the QUERY_REQUEST hex dump; RESULT_BATCH / RESULT_END below unchanged. - Client lifecycle step 4: inline note that the binary frame body starts directly with msg_kind for client-to-server frames. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding
New page (documentation/connect/agents.md) covering how AI agents operate QuestDB across three angles: protocols, tooling, and operational practices. Positioning: - QWP egress is the recommended path for SQL execution (DDL + streaming SELECT), with native client libraries when available and the protocol spec for clean-room implementations. - QWP ingress is the recommended path for all writes (bulk and sustained), including local-file uploads — the recipe explicitly calls out the failure mode where agents reach for read_parquet/read_csv/COPY, which require server-side filesystem access. - REST is positioned for schema discovery and small ad-hoc queries that fit in a single HTTP response. - PGWire and /imp are intentionally not recommended (superseded by QWP). - No MCP framing: an MCP server would just wrap REST + QWP without adding capability, so the page tells agents to use the underlying protocols directly. Includes a Recipes section seeded with the local-file upload recipe; links to the existing Getting Started > AI Coding Agents page for the tooling quickstart and the QuestDB / TSBS Claude skills. Sidebar: - Add Connect > Agents (between Client Libraries and Message Brokers). - Move Date to Timestamp inside Client Libraries (cross-cutting reference for all language clients). - Move Connect string inside Client Libraries as the first item (config schema shared by every QWP client). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding
Verified against c-questdb-client ia_qwp_ws branch. Adds warning that sf_dir is strongly recommended for multi-host (flush blocks without it), env_logger example for observing reconnect events, and full tested failover example with retry pattern around flush() and must_close().
Adds storeAndForwardDir and senderId to the ingestion builder example. Updates trailing comments to explain SF vs memory-only behavior during outages.
…on caution Java: adds store-and-forward to the full enterprise example with builder API, connection events, and egress reconnect-on-failure pattern. Removes outdated "requires QuestDB 9.2.0" caution on decimals (QWP itself requires QuestDB 10). Rust: adds SF recommendation for multi-host, env_logger hint for observing reconnect events, and full tested failover example with flush() retry pattern and must_close().
…ample Verified by agent testing against Go client. Documents egress terminal state after failover exhaustion with recreate-on-failure pattern. Adds single-endpoint failover warning. Notes ingress/egress reconnect asymmetry. Full tested example: ingestion with SF + options API + observability counters, and querying with QwpFailoverReset handling. Enterprise features marked with inline comments.
… fan-out Wire spec adds X-QWP-Max-Batch-Size and X-QWP-Durable-Ack response headers, plus null-sentinel byte patterns for IPv4/UUID/LONG256 and an updated reference-implementation null-strategy table. Connect-string reference documents the new auto_flush_bytes=8m default and the server-side handshake clamp. Java client page documents ipv4Column, geoHashColumn, binaryColumn, and LongArray setters, plus the synchronous LineSenderException surfacing when a batch would exceed the server's advertised cap. review-client skill gains a Parallelization subsection for multi-file fan-out and a Doublecheck rule for negative findings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…try=on The QWP/WebSocket ingress reconnect knobs (reconnect_max_duration_millis, reconnect_initial_backoff_millis, reconnect_max_backoff_millis) now implicitly promote initial_connect_retry to `on` at parse time. The default fail-fast behaviour on first connect is preserved when the user explicitly sets initial_connect_retry=off. Updates: - Per-key descriptions for each reconnect_* knob note the implicit promotion and link to the explainer. - initial_connect_retry entry gets a bolded "Implicit promotion" paragraph documenting when the auto-on fires and how to opt out. - Recipes table replaces the two-row "Tune budget / Retry initial connect" pair with three sharper rows. - Key index entry for initial_connect_retry notes the auto-promotion in the default column. - Common patterns gains a "Tolerate a slow or restarting server at startup" example showing the one-line conf.
35 files relocated so on-disk paths mirror their slug frontmatter:
- documentation/client-configuration/connect-string.md
→ documentation/connect/clients/connect-string.md
- documentation/ingestion/{overview,java-embedded}.md
→ documentation/connect/
- documentation/ingestion/clients/*.md
→ documentation/connect/clients/
- documentation/ingestion/ilp/*.md
→ documentation/connect/compatibility/ilp/
- documentation/ingestion/import-csv.md, documentation/query/{rest-api,export-parquet}.md
→ documentation/connect/compatibility/
- documentation/ingestion/message-brokers/*.md
→ documentation/connect/message-brokers/
- documentation/protocols/*.md
→ documentation/connect/wire-protocols/
- documentation/query/pgwire/*.md
→ documentation/connect/compatibility/pgwire/
MDX relative imports rewritten for files whose depth changed by +1
(ilp/overview, wire-protocols/qwp-*, compatibility/rest-api, and the 10
compatibility/pgwire/* files with partial imports). sidebars.js updated
with the new doc IDs. Empty source directories removed. The legacy
documentation/ingestion/clients/configuration-string.md (no /connect/
slug, no inbound references) was left in place pending a separate
decision.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address findings from the QWP/WS C/C++ doc review:
- auth_timeout -> auth_timeout_ms as the canonical key (matches rust.md
and the conf parser); auth_timeout kept as a compat alias.
- Multi-host full example: only encode a row when the buffer is empty
(line_sender_buffer_row_count check), so failed flushes no longer
stack rows in the retained buffer.
- Manual-progress C example: replace the empty park comment with a
nanosleep() (1 ms) and the matching _POSIX_C_SOURCE define so the
loop does not spin on CPU. C++ side uses std::this_thread::sleep_for.
- Flushing prose: "local sender queue" instead of "replay queue", with
the in-memory vs sf_dir distinction spelled out.
- tls_verify=unsafe_off: note that this requires the
insecure-skip-verify Cargo build feature.
- Spell out line_sender_flush_and_keep for C readers.
- Standardize connect-string schemas on qwpws:: / qwpwss:: throughout
(ws / wss documented as aliases). Updates auth, TLS, env-var, full
example, and migration table.
- Buffer concurrency wording: each encoder thread owns its own buffer;
drop the "encode rows in parallel" phrasing that implied parallel
writes to a single buffer.
- Rename qdb::opts opts{...} to options{...} to avoid shadowing the
type name.
- Remove em-dashes from prose.
All snippets re-validated: 14 standalone C/C++ files compile against
c-questdb-client jh_qwp_followups; the 11 runnable samples were
executed against questdb-arrays vi_qwip_fix (rebuilt jar includes
PR #7108 IPv4 plus BINARY) and rows confirmed via /exec.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cross-page findings — QWP client docs (
|
| Concept | Divergence |
|---|---|
trades.amount type |
DOUBLE in go.md:69/680, rust.md:57, c-and-cpp.md:97, dotnet.md:75 — but LONG in java.md:673/704 (and read as LONG at java.md:106). java.md is also internally contradictory: doubleColumn("amount", 0.00044) at :84/:90 vs longColumn("amount", …) at :345/:480. |
| Instrument column | symbol in go.md SQL/DDL (104, 572, 679, 710) vs sym in java.md SQL/DDL (106, 581, 672, 704); both ingest symbol. |
trades timestamp precision |
microseconds / server-assigned in go.md, java.md, dotnet.md vs nanoseconds in rust.md:58 and c-and-cpp.md:98 — the same notional table gets a timestamp_ns designated column on two pages and a micro TIMESTAMP on three. |
| Order-book table | book in go.md, java.md, c-and-cpp.md vs fx_order_book in rust.md:259 (rust.md itself also uses book at :794). |
| Order-book instrument col | ticker in go/java/rust/c-and-cpp vs pair in dotnet.md:248 (fx_prices table). |
Separately, the Go canonical query example —
<RemoteRepoExample name="qwp-query" /> — renders
go-questdb-client/examples/qwp/basic-query/main.go, which uses placeholder
schema qwp_query_example / v LONG. That's the primary runnable query
example a reader copies, and it isn't capital-markets at all.
Fix shape: pin one canonical schema (e.g. trades =
ts TIMESTAMP, symbol SYMBOL, side SYMBOL, price DOUBLE, amount DOUBLE;
order book = book with ticker; one timestamp precision per notional
event) in a README under documentation/ingestion/clients/ (or the
review-client skill), then align the five pages plus the Go basic-query
example to it.
2. connect-string.md — OIDC silent (item 19)
The shared reference (in scope for every QWP client page) documents token
(216-219) and explicitly states mTLS is unsupported (224-228), but says
nothing about how an application acquires an OIDC token or what happens when
it expires. The equivalent paragraph was added to go.md in 97e25af; the
same gap remains here and applies to every client page that links to this
reference. Fix: mirror that static-token / OIDC-acquisition / no-refresh
paragraph in the Authentication section.
3. connect-string.md — stale Java-specific on_*_error caveat
Lines 646-651 / 719-722 state the on_*_error keys are "not yet recognised
by the Java connect-string parser … wired only via the fluent builder API."
For the Go client that's inaccurate — the Go connect-string parser does
accept them (go-questdb-client/conf_parse.go:384-413). And go.md:810
already refers to "the connect-string on_*_error keys" in its error-policy
precedence prose without documenting or linking them anywhere. Fix:
generalize the caveat to "client support varies (the Go parser accepts these
in the connect string)" and give go.md:810 a link target.
Nit
connect-string.md:58 and :81 illustrate syntax/escaping with http::
inside a QWP-only reference doc — harmless (the grammar is schema-agnostic)
but off-message; consider ws::.
- use ws::/wss:: as primary connect-string schemas across all examples; document qwpws/qwpwss as long-form aliases, matching rust/java/go - add Null values subsection with C and C++ omit-the-setter examples - add Unsupported auth paths subsection (OIDC, mTLS, token rotation) - align the trades hello-world schema with sibling clients (add side) - correct tls_ca default in config-reference table: webpki -> webpki_roots - name the C and C++ error-handler APIs correctly in migration prose - drop the incorrect "per-request overhead" framing on token auth connect-string.md: document qwpws/qwpwss aliases under the protocols table. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Hello world (C): caution that QDB_*_LITERAL is sizeof-based and only works on string literals; for runtime strings use line_sender_utf8_init (and the matching table_name/column_name _init forms) - Build the library: pointer to the runnable line_sender_c_example* / line_sender_cpp_example* binaries produced under build/ - Hello world (post-Tabs): clarify that .so needs no extra libraries but the static .a additionally requires -lpthread -ldl -lm (+ TLS deps); call out LD_LIBRARY_PATH / -Wl,-rpath for in-tree-build binaries - General usage pattern: 10-line ingest-loop example showing sender + buffer reuse, with the "flush retains rows on failure" qualifier - No auto-flush caution: tighten wording to "does not implement auto-flushing at all" and frame auto_flush=off as a no-op for HTTP/ILP compatibility, not a default value Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
agents would dismiss QWP as 'noise' or 'hallucination' (they dont have it in training data)
Reviewing PR #444's Go QWP client documentation surfaced one actively misleading statement and two coverage gaps. Fixes in documentation/ingestion/clients/go.md: - Flush: the page said it is a synchronous barrier that returns after the server acknowledges the flushed frames. The client does the opposite: Flush returns once rows are published into the cursor engine and never waits for the server ack. The old wording contradicted the page's own async-error model and invited a false-durability bug. Replaced with the real behavior and a pointer to FlushAndGetSequence + AwaitAckedFsn. - Query failover: added a warning that accumulating rows across batches without handling *QwpFailoverReset yields duplicate rows, since the server replays the whole result set from the beginning after a reconnect. - Token auth: noted the bearer token is a static credential the client never refreshes; it must be acquired out of band (OIDC flow) and rotated by rebuilding the client. plugins/remote-repo-example/index.js is intentionally left unstaged: it self-documents as a temporary local-preview hack to revert before commit/merge (it points at an unmerged branch), so it must not land in this commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DATE (0x0B) is encoded differently on the two QWP wire directions, and the asymmetry is deliberate. On egress DATE is grouped with TIMESTAMP/TIMESTAMP_NANOS: it carries the 1-byte encoding flag and can be Gorilla-encoded. On ingress DATE is a plain fixed-width int64 column written exactly like LONG: no encoding flag, never Gorilla, even under FLAG_GORILLA. The ingress spec previously used generic "timestamp columns" wording with no warning that DATE is excluded, so an implementor who assumes symmetry with egress would shift every DATE value by one byte (a clean x256) and break Gorilla-encoded DATE entirely. Add prominent :::warning admonitions on both sides: ingress Column types and Timestamp encoding sections, and the egress FLAG_GORILLA section, each cross-referencing the other direction. Also tighten the ingress "timestamp columns" wording to name TIMESTAMP and TIMESTAMP_NANOS explicitly and note DATE is always plain int64. Cross-links use the canonical frontmatter slug form. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- document the new on_failover_progress callback: add a "Failover progress phases" subsection with phase semantics, accessor surface, and side-by-side C / C++ examples; note that the callback coexists with on_failover_reset (Reset phase fires immediately before the reset hook) and that installing either one clears the silent-duplicate guard. - rewrite Connection-state observability: lead with on_failover_progress for mid-query lifecycle (disconnected / retrying / reset / gave-up); keep polling guidance for the genuine remaining gaps (initial-connect failures, between-queries idle). - add -Wl,-rpath to the hello-world compile commands so the resulting binary loads without LD_LIBRARY_PATH; link to upstream DEPENDENCY.md for production CMake setups. - surface line_sender_buffer_row_count alongside ..._buffer_size in the Flushing section so both batching-threshold idioms are visible. review-client skill: add an agent one-shot suitability bar to verdicts (items 23-30) so reviewers flag agent-specific failures (indirection chains, section non-self-containedness) with equal weight as human-skim gaps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…' into docs/connect-and-qwp-scaffolding # Conflicts: # documentation/ingestion/clients/c-and-cpp.md
2. adjust c/c++ api docs
…/questdb/documentation into docs/connect-and-qwp-scaffolding
Rewrite all 11 <Tabs> blocks from the multi-line
<Tabs defaultValue="..." values={[ ... ]}> JSX-expression form to the
modern <Tabs defaultValue="..."> with per-TabItem label= attributes.
Docusaurus 3.8.1 auto-derives the values list from TabItem children, so
the rendered output is identical.
WebStorm's HTML parser does not understand the values={[ ... ]} JSX
expression. With the old syntax every Tabs opening was treated as an
unrecognised tag, cascading into "Closing tag matches nothing" errors on
every </TabItem> and </Tabs> below. The IDE MCP confirms those errors
disappear entirely with the new syntax.
No content change. The diff is structural only (-33 lines, three
mechanical replace_all operations: <Tabs ... values={[...]}> -> <Tabs ...>,
<TabItem value="c"> -> <TabItem value="c" label="C">,
<TabItem value="cpp"> -> <TabItem value="cpp" label="C++">).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add OIDC/mTLS/token-rotation "unsupported" table mirroring the .NET pattern, document null/omit equivalence with the first-row schema- inference caveat, explain what surfaces when the reconnect budget elapses (Rust has no ConnectionListener so the budget-exhaustion path must be spelled out), align the arrays example table name with sibling clients (fx_order_book -> book), and fix the production full example: remove tls_verify=unsafe_off from the production-labelled snippet, and rewrite the inline "// comment" text inside the multi-line Rust string literal that was actually being concatenated into the connect string at runtime. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Quick-start ingest now catches LineSenderServerException (inspecting SenderError for status / category / server message) and LineSenderException, instead of bubbling untyped exceptions. Quick-start query aligns column names with the ingest example (sym -> symbol) and catches IllegalStateException for terminal-client state. amount is now DOUBLE consistently across ingest, full example, DDL, and bind-parameters SQL -- previously drifted between LONG and DOUBLE on the same page. New "Awaiting acknowledgements" subsection documents flushAndGetSequence / awaitAckedFsn / getAckedFsn and the FSN that correlates with SenderError.getFromFsn / getToFsn. Add request-id-aware onError / onEnd / onExecDone / onFailoverReset overloads with a "Diagnostics surface" table covering request id, failing SQL availability, bind-index policy, message stability, and PII safety. Full example: typed IllegalStateException / LineSenderException catches instead of bare catch (Exception e). Also tighten connect-string.md buffer_pool_size type to "int (>= 1)" to match the runtime minimum (resolves the pull-time conflict).
…/questdb/documentation into docs/connect-and-qwp-scaffolding # Conflicts: # documentation/connect/clients/connect-string.md
Summary
Consolidates how-applications-talk-to-QuestDB content under a single Connect supersection. The native QWP clients, the compatibility protocols (ILP / PGWire / REST), and the wire-protocol specifications now share one home. The connect-string reference at
documentation/client-configuration/connect-string.mdanchors the section as the comprehensive knob doc.What's new
Sidebar restructure
Connectsection absorbs Client Libraries, Compatibility Protocols, and Wire Protocols.Connect → Client Libraries.Connect → Compatibility Protocols.Connect → Wire Protocolsfor the client-implementer audience.Connect → Overview (
documentation/ingestion/overview.md)Connect → Connect string reference (
documentation/client-configuration/connect-string.md, new)*Applies to: ingress.*/*Applies to: ingress and egress.*).sf_dirpath handling (no shell expansion, parent must exist).sender_idvalidation (allowed characters, lock-collision semantics).Wire Protocols scaffolding (
documentation/protocols/, new)Touch-ups
#first-party-clients→#client-librariesincapacity-planning.md,monitoring-alerting.md,datatypes/overview.md, andsrc/components/Resources/index.tsx.date-to-timestamp-conversionfrontmatter description from a bare language list to an actual summary.Project handoff (
ONBOARDING.md)Bundle B / A / C team handoff document captured at repo root — assignments, coordination rules, first-prompt templates for the remaining work streams (Wire-protocol content, Client failover, Store-and-forward concepts). Reviewers can opt to keep or remove from this PR.
Out of scope (intentionally)
Connect → Compatibility Protocols → ILP. The connect-string reference is QWP-native; ILP-specific knobs (protocol_version,retry_timeout,request_timeout,request_min_throughput) are not duplicated.Known follow-ups
sf_durability=flush|append, theon_*_errorfamily) so LLM auto-completion is less likely to surface them as available.ingestion/clients/*→clients/*) and HTTP-side URL redirects in a separate restructure PR.Test plan
yarn build— 353 pages generated, no broken links / anchorstarget/zoneframing matches the QuestDB Enterprise feature storyONBOARDING.mdshould remain at repo root🤖 Generated with Claude Code