[FEATURE] `docs`: consider migrating to `Zensical` for `v1`

[Haleshot]

What is the use case?

Following-up on the discord thread about whether CocoIndex should have a separate docs repo, I came across Zensical and sent it to @prrao87, and we started talking about how it could be a good fit for CocoIndex. It's a modern SSG built by the same team behind mkdocs.

My takeaways from the thread (primarily from Prashanth's takes)

... is that Docusaurus forces a 1D layout where all concepts get crammed into a single sidebar. Too much nesting and you end up with docs where everything is buried levels deep and impossible to find (duckdb example). Tools like Zensical use two-dimensional layouts where concepts spread across both a sidebar and tabs along the top. This aligns better with diataxis (https://diataxis.fr/) principles and makes documentation more discoverable.

There's been a shift towards SSGs that specialize in docs rather than trying to be general-purpose site builders. Mintlify is popular but not open source. Zensical is.

With v1 works underway, this could be a nice opportunity for a fresh look alongside the conceptual rework happening for the framework.

Describe the solution you'd like

Zensical is written primarily in Rust (about 85% of the codebase) with some Python, distributed as a Python package via PyPI.

Rebuild times during dev are fast thanks to the ZRX build engine. There's a new search engine (Disco) with improved ranking and filtering that runs entirely client-side. The design is modern and more easily brandable.

Additional context

The GitHub repo already shows 159 dependent projects and 2.6k stars, and the HN Thread had generally positive reception. I also believe we could reach out to the team to ask questions re: how a potential migration from docusaurus might look like (there exists a webpage for mkdocs).

---

❤️ Contributors, please refer to 📙Contributing Guide.
Unless the PR can be sent immediately (e.g. just a few lines of code), we recommend you to leave a comment on the issue like I'm working on it or Can I work on this issue? to avoid duplicating work. Our Discord server is always open and friendly.

[Haleshot]

I imagine I'd need help w/ what might end up being quite the migration for a PR (if folks want to chime in). Happy to do the heavy-lifting for it though.

[badmonster0]

Great suggestion - and love the discussion! adding a few things to consider and evaluate - before we commit to this:

as a static site:

• we would like to examples/blogs/docs on same stack and have visual consistency on all sites (check - /docs, /examples, /blog and with home page too) we use markdown for everything. so one thing to potentially take a look is does the new framework support it? for us keeping it easy to maintain is an important factor.
• do we need additional horizontal tabs - we already have "examples / blogs at top nav" and we can configure anything to place there, we currently don't have the requirement and breakdown. and would take time to iterate on what's needed there.
• what kind of visual component does the new framework have, how easy is it to support all exiting docusarus visual components we have already? maybe share a few static site built with zensical would help.
• what's the ways to host it and what's the release look like? we currently host all static pages via gh pages.

Thanks a lot for creating this issue @Haleshot !

[prrao87]

This is great idea, thanks for writing it up @Haleshot! But I agree with @badmonster0, in that before committing to a new docs backend, it's worth building a simple PoC that shows it'll work rather than migrating it all at once. I also think that with the Zensical PoC, it's important to showcase the following:

• Clear visual consistency (colors in light/dark mode) that align with the CocoIndex brand
• A clear structure/scaffolding of what the sidebar headers and the top tabs would be. Because we'd move from a 1D to a 2D layout, more content types open up: see the lance.org docs for how this might be done. In the home page the sidebar could be a main "User Guide" section that showcases the "how", and second tab on top could be "Concepts", that focuses on the "what" and the "why". The "Examples" could also be entirely hosted in Zensical docs as a separate tab, rather than being part of the docs site. The blog could eventually stand on its own entirely and be deployed alongside the main cocoindex.io website, on a separate backend that's React-based.
• Demonstrate visual elements that exceed the capabilities of Docusaurus, see the authoring section of Zensical's docs. It's very feature rich and frankly, contains some outright beautiful UI elements for docs. It should be a superset of everything Docusaurus provides, and it has everything that mkdocs users expect, but with better performance. So from a docs writing and features standpoint, nothing would be lacking.

To answer @badmonster0's question about deployment, yes, it's trivial to deploy it via GH pages and point it via a CNAME to a custom URL (docs.cocoindex.io would be a good one) at a future point. My recommendation is for @Haleshot to allocate some spare time and build a PoC on his own personal GitHub and deploy it via GH pages, and then get it reviewed for feedback. But to start, it would just be on your own GH pages (haleshot.github.io/cocoindex-docs) linked to your account to demonstrate and showcase its value.

Hope that makes sense, and happy to participate in reviewing this PoC! I estimate it shouldn't take more than a few hours of brainstorming on a content layout and another couple hours to code it up and deploy to showcase.

[prrao87]

Just another point: {blogs + website} and {docs + examples} are typically two different stacks in most tools/companies these days. I don't think Zensical can help with the former (it specializes in the docs). So if a migration is considered, the {blogs + website} will likely be on a different stack than the Zensical stack. Everything uses Markdown (that part isn't in question), it's just the way components are specified are different in either - docusaurus sems to require a lot of custom components written in React to look nice, but Zensical provides pretty much everything off the bat (no need to maintain custom components for docs), and the syntax for calling them would differ between the blog and the docs.

As the company grows in size and the user community grows, I think moving to a separate stack for the {website + blog} and {docs + examples} is more or less a given, for reasons of branding. The docs are forever developer-focused, but the website and blog are a mix of engineering and marketing/outreach. So I think it's worth thinking about a longer term horizon while also keeping the path to migration clearly focused and forward-looking. Happy to discuss more!

[Haleshot]

Thanks for the detailed response @badmonster0 and @prrao87 (and for addressing LJ's questions)!

On the blog support point that LJ raised, I did some digging in their discord and found this message. The short answer is that blogging support isn't there yet but is likely to be supported later this year. One user mentioned it's a blocker for enterprise adoption on their end, and the Zensical team acknowledged they might prioritize it if Spark members need it. So it's a known gap.

docusaurus seems to require a lot of custom components written in React to look nice, but Zensical provides pretty much everything off the bat

I've noticed this too. There have been quite a few docs PRs (styling, customization, navbar, etc.) from LJ involving React and CSS tweaks that a good SSG would (should?) otherwise handle out of the box imo.

So if a migration is considered, the {blogs + website} will likely be on a different stack than the Zensical stack

I've seen the {docs + examples} combo in quite a few devtools/frameworks as well (and landing page + blogs separately), sometimes with examples renamed as "user guides", "references", or "gallery", etc.

One thing I'm curious about is timing. With v1 updates being made and quite a few new concepts being introduced (state, context, app, lifespan, etc.), I wonder if it makes more sense to wait until the v1 docs (atleast the v1-preview) stabilize before doing a PoC. That way I'm not chasing a moving target... Either way works for me though, happy to wire something up whenever it makes sense.

Also minor thing: Prashanth, the authoring hyperlink in your comment seems to be broken; I believe it was supposed to point to authoring.

[prrao87]

Good points! Let's wait for v1 docs and examples to stabilize and then you and I can try and take things for a spin. Good catch, this is the page I meant to link for authoring:
https://zensical.org/docs/authoring/markdown/

[badmonster0]

sg, thanks a lot for the in depth research and recommendation @prrao87 and @Haleshot ! sg to wait for v1 a bit!