work.log

Done for 2025:
 - Transactions
 - Encoding which is compatible with some other database.
 - Protocol which can be benchmarked.
 - No dependencies
 // keep thinking about this

Want:
 - Error tracing.
 - A dedicated log package.
 - WAL backing for the store.

2025-04-21
 - Update to normal data types, trudging the unknown path should only be taken once the road understood is taken
 - Can I get sqllogictest and sysbench to work
   - Probably yes to sqllogictest, probably no to sysbench
   - I imagine that sysbench depends on being able to setup databases, initialize transactions etc.
 - A makefile so that I can
   - Install dependencies
   - Build / run / test
   - Generate the AST code
   - Run sysbench

 
For sysbench to work, the database needs prepared statements.
Might just be enough to do the sqllogictesting functionality and go from there.

sysbench \
--pgsql-host=127.0.0.1 \
--pgsql-port=5432 \
--db-driver=pgsql \
--table_size=100000 \
--tables=3  \
--threads=3 \
oltp_common prepare

sysbench \
--pgsql-host=127.0.0.1 \
--pgsql-port=5432 \
--db-driver=pgsql \
--table_size=100000 \
--tables=3  \
--threads=3 \
oltp_read_write run

 - Reorganize directories
2025-04-14
 - Finish out the query plumbing.
 - Happy path
 - Errors
 - Remove pointers from everything.

2025-04-13
 - I need to prevent users from modifying system tables. (do I?)
   - let's say they know what they're doing, perhaps this isn't necessarily an issue.
 - Okay, today I can choose to do joins or sqllogictest
 - Let's do sqllogictest

2025-04-12
 - Query planning (finish planning selects with filters)
 - Query execution (finish planning selects with filters)
 - Prove selection works on system tables
 - Prove selection works on user tables


  id SERIAL,
  k INTEGER DEFAULT '0' NOT NULL,
  c CHAR(120) DEFAULT '' NOT NULL,
  pad CHAR(60) DEFAULT '' NOT NULL,
  PRIMARY KEY (id)

Do I have a rows plan type? Does it screw everything up?

How do I make it work with the cursors? How do I have scans not be stateful?

And what if the plans were stateful? Would I not be able to cache them?

Could caching work in a way where the stateful nature of them was less of an issue?

Seems like in a lot of queries, there's a notion of a source.

I like the idea of iteratively executing a query plan. That for each node, there's simple "Next" functionality which can be executed individually.

That being said, the insert, update usecase is a little trickier.

For scanning into a put, this should be fine, but how does the executor keep track of which of the tuples it's inserted when the tuples are part of the plan?

There's state the executor is responsible for, but it needs to be known within the context of a single plan node.

"We're on insert 30 for this insert node."

"The cursor is here for this scan node, and here for this other scan node."

Plans should be reusable, and if I mutate the state on the nodes, I should be able to reuse the plan.

There should be iterators, which are responsible for "doing the next" and knowing a node.

This brings me to the next challenge, suppose I can create ids for the nodes in the plan. Should all node types have an id?

We only need the id whenever a node should be stateful.

Look into toydb, how does it manage iterating over executing a plan? Does it do it all at once? Does it execute in batch?

If plan execution is stateful, how is this state managed?

Insert executes all at once for toydb, but select has iterators.

2025-04-06
 - Try to implement some select, and rendering functionality
 - Try to get selecting to work.
 - After watching query execution, it's worth thinking about how to do more complex operations.
   - Projection (requires context?)
   - Revisit scoping in general.
 - Might not be the worst thing to have a server implemenetation.


Handling multiple queries? Handling semicolons?

Now that selecting is working, I need to begin thinking about how projection works. Rather, I need to begin thinking about scoping.

I suppose scoping won't be strictly necessary, but I do need to evaluate expressions within the context of query.

For example, given the query:

SELECT * from foo where bar = 1;

The Eval() call should be able to evaluate the expression bar = 1, within the context of the table foo.

Thanks GPT

For rendering, how should I render the header of some output?

If the rows themselves are devoid of header information, how d othey render at the end? How does toydb do it?

How should I render tabular result data?

insert planning and execution mostly works. most of the edges of all of this are untested, and the functionality itself seems reallly shakey.

that being said it looks good when it's running, so it's a positive start.

In insert execution, I'm already missing some pretty basic functionality, that being the validation of column values, primary key setting (when the user hasn't set a primary key), and some other things.

Debugging is getting a little hairy. There are times when I only want to debug parts, the keys in storage or the nodes of the parser. Might make debugging a bit more involved in the future.

Organizing the project has been challenging, especially since lots of packages share common themes (sequence, table, create table, etc)

Transactions, especially transaction execution with sequences has me a bit befuddled. I'm not sure how to all or one things in this particular layout.

2025-04-06
 - Lots of additional tests for the parser.

2025-04-05
 - implement the insert functionality
 - Remove references from the codebase (for now)

2025-04-01
 - Create a plan for insert that works
 - Create a debugger for plans

2025-03-30
 - DescriptorBuilders
 - Fix where sequence generation occurs

2025-03-28
 - Generic Manager
 - Bootstrapping
  - Reorganize logic to bootstrap from system objects

2025-03-27
 - Keys
  - Key part to bytes
    - This is quite doable
  - Delimiter
    - Null byte should work if delimited by bytes
  - Prefixes
    - Worth having a prefix class?
    - Might not be
  - Encoding
  - Decoding
  - Escaping unique values
  - How is prefix end handled?
  - Referring to Spans rather than ranges


having a debug store wrapper

type DebugStore interface {
  
}


type Prefix struct {
 Table *desc.Table
}

type Key []byte

func (k Key) Pretty (sc *schema.Schema) {
 
}

 I really do like the idea of being able to put registers. Things which have Key(), Value() functionality.

Even in relational databases, keys refer to two things, the absolute key of the item, stored in the keyspace, and its relative / index key, the tail of an absolute key.

type Register interface {
  Key() *keys.Key
}

interface Key {
 Key() string
}

type Datum struct {
 Values []any
 Key []int
}

keys.Prefix(Table)
prefix.Key(Keyable)

If I were to think about using the keys, whatever would I think?

What do I want the surface to look like?
Should store take a register?

What of the concept of a register?

TableRegister
SequenceRegister

2025-03-26
 - Test all new code.

2025-03-23
 - Clean up manager package.
 - Get everything wired up to create new tables.
 - Reorganize packages for schema management / description.