Make slashdown conform to unist spec#5
Open
nickisnoble wants to merge 12 commits into
Open
Conversation
SlashDown now conforms to the Universal Syntax Tree (unist) specification, making it compatible with the broader unified ecosystem. Changes: - Add unist-compliant type definitions (Node, Position, Point) - Implement position tracking in lexer (line/column with 1-indexed values) - Add position information to all AST nodes in parser - Update all test files to include position tracking - 30/31 tests passing (1 pre-existing parser issue) All nodes now include optional `position` field with start/end points conforming to unist spec. This enables integration with remark, rehype, and other unist-compatible tools. Resolves the unist conformance future goal from README.
Create EXAMPLES.md and ARCHITECTURE.md to design the integration of SlashDown with the unified/remark/mdast ecosystem. Key design decisions: - Hybrid AST (slast): SlashDown nodes contain mdast children - Replace Markdown/Text nodes with proper mdast nodes - Configurable markdown handler for different parsers - Support for slashdown ↔ HTML, markdown, and other formats - Plugin-compatible architecture using unified processors This sets the foundation for v1.0.0 with breaking AST changes and full unified ecosystem compatibility.
BREAKING CHANGE: Complete AST structure overhaul for unified ecosystem integration
This is a major architectural change that makes SlashDown fully compatible
with the unified/remark/mdast ecosystem by implementing a hybrid AST where
SlashDown tags contain mdast nodes as children.
## Key Changes
### 1. Hybrid AST Structure
- AST is now a Root node with children array (not a direct array)
- SlashDown tags are now `slashdownTag` nodes (not `Tag`)
- Markdown content is parsed into proper mdast nodes (heading, paragraph, list, etc.)
- Inline text becomes mdast `text` nodes (not custom `Text` nodes)
- Code fences become mdast `code` nodes with lang/meta/value
### 2. New Components
- **MarkdownHandler**: Parses markdown strings into mdast nodes
- Uses mdast-util-from-markdown with GFM support
- Handles inline text, markdown blocks, and code fences
- Preserves position information
### 3. Updated Type System
- Import and extend unist Node types
- `SlashDownTag`: Hybrid node containing mdast children
- `Root`: Top-level node wrapping entire document
- `MarkdownHandlerOptions`: Configuration for markdown parsing
### 4. Parser Updates
- Integrates MarkdownHandler for parsing markdown content
- Returns Root node instead of array
- Spreads mdast nodes into children arrays
- Maintains position tracking throughout
### 5. HTML Renderer Updates
- Handles both slashdownTag and mdast nodes
- Uses mdast-util-to-hast + hast-util-to-html for mdast rendering
- Fixed attribute unpacking for boolean attributes
### 6. Test Updates
- Added comprehensive hybrid-ast.test.ts with 9 new tests
- Updated all existing tests for new AST structure
- All 40 tests passing ✅
## Migration Guide
### Before (v0.x):
```javascript
{
type: "Tag",
tagName: "article",
children: [
{ type: "Markdown", content: "# Hello\n\nWorld" }
]
}
```
### After (v1.x):
```javascript
{
type: "root",
children: [{
type: "slashdownTag",
tagName: "article",
children: [
{ type: "heading", depth: 1, children: [{ type: "text", value: "Hello" }] },
{ type: "paragraph", children: [{ type: "text", value: "World" }] }
]
}]
}
```
## Benefits
- ✅ Full unist/mdast compatibility
- ✅ Use unist-util-visit for tree traversal
- ✅ Apply remark plugins to markdown content
- ✅ GFM support (tables, strikethrough, etc.)
- ✅ Foundation for JSX/MDX support
- ✅ Interoperable with rehype for HTML transforms
## Dependencies Added
- mdast-util-from-markdown
- mdast-util-to-markdown
- mdast-util-to-hast
- mdast-util-gfm
- hast-util-to-html
Version: 1.0.0-alpha
Document the architectural decision to use type: 'slashdownTag' rather than type: 'element' (hast) or tag name as type. Key rationale: - Clear distinction between SlashDown and mdast nodes in hybrid tree - Maintains ergonomic classes/ids arrays vs hast properties - Follows hast's pattern (generic type + tagName field) - Easy to convert to hast when needed for unified pipeline This decision keeps the current implementation optimal while providing a path forward for full unified/rehype integration.
…own' Use consistent naming throughout the codebase: - Node type: 'slashdownTag' → 'element' (matches hast convention) - Project name: 'SlashDown' → 'Slashdown' (title case, not camel case) - Package/CLI: Keep as 'slashdown' (lowercase) This makes the AST more intuitive - we're representing HTML elements, so using type: 'element' + tagName follows the same pattern as hast. Changed files: - src/types.d.ts: SlashDownTag → Element - src/parser.ts: Use type: 'element' - src/renderers/html.ts: renderSlashDownTag → renderElement - All tests: Update expectations to use 'element' - All docs: Consistent 'Slashdown' naming All 40 tests passing ✅
- Fix slashdown.ts: use Root node, reuse instances, throw errors - Add HTML sanitization (escapeHtml) to prevent XSS - Pre-compile regex patterns to avoid recompilation - Improve error handling (remove console.error, add token type) - Add test:coverage script
- Add input validation: type checking, max line length, depth limits - Add recursive depth protection (MAX_NESTING_DEPTH = 100) - Update dependencies to latest versions - Export SD types, Lexer, Parser, MarkdownHandler for users - Add comprehensive JSDoc comments to public APIs - Expose MarkdownHandler configuration via SlashdownOptions - Extract constants: MAX_DEPTH, MAX_LINE_LENGTH, MAX_NESTING_DEPTH
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.