Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,11 @@ tools/python_bind/
Cypher → ANTLR Parser → Binder → Logical Plan → gopt Converter → Physical Plan → Execution
```

## Known Limitations

- **List literals require explicit CAST**: In `CREATE`, `SET`, and `MERGE` clauses, list values must be wrapped with `CAST(..., 'TYPE[]')` — bare list literals like `[1, 2, 3]` are rejected. Use `CAST([1, 2, 3], 'INT64[]')` instead.
- **List types cannot be primary keys**: Declaring a `PRIMARY KEY` on a `T[]` column is rejected.

## Code Style

- **C++**: C++20, clang-format (style=file)
Expand Down
51 changes: 51 additions & 0 deletions doc/source/cypher_manual/ddl_clause.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ The following table lists the recommended syntax for defining default values for
| `TIMESTAMP` | `prop TIMESTAMP DEFAULT TIMESTAMP('1970-01-01')` | `TIMESTAMP('1970-01-01')` |
| `INTERVAL` | `prop INTERVAL DEFAULT INTERVAL('0 year 0 month 0 day')` | `INTERVAL('0 year 0 month 0 day')` |
| `ARRAY` | `prop INT32[3] DEFAULT [1, 2, 3]` | child defaults repeated to the fixed length, for example `[0, 0, 0]` for `INT32[3]` |
| `LIST` | `prop INT64[] DEFAULT [1, 2, 3]` | `[]` (empty list) |

List types (`T[]`) are variable-length and can hold any number of elements, including zero. Array types (`T[N]`) have a fixed length `N` that must be a positive integer.

Please refer to the following examples for more usages.

Expand Down Expand Up @@ -124,6 +127,54 @@ CREATE NODE TABLE Matrix(

If an array property is omitted during `CREATE`, or explicitly set to `NULL` during `CREATE`, NeuG stores the declared default for that array. If no explicit default is declared, the system default repeats the child type's default value; for `INT32[3]`, that default is `[0, 0, 0]`. Setting an existing array property to `NULL` with `SET` is not supported yet.

## List Properties

Use `T[]` to declare a variable-length list property, where `T` is the child type. Unlike fixed-size arrays (`T[N]`), the number of elements is not constrained at the schema level.

```cypher
CREATE NODE TABLE Person(
id INT64,
tags STRING[],
scores INT64[],
PRIMARY KEY(id)
);

CREATE REL TABLE Knows(
FROM Person TO Person,
ratings DOUBLE[]
);
```

Nested lists are declared by chaining `[]`. `STRING[][]` is a list of lists of strings; `STRING[][2][]` is a list of fixed-size-2 arrays of variable-length string lists:

```cypher
CREATE NODE TABLE Matrix(
id INT64,
grid INT64[][],
PRIMARY KEY(id)
);
```

List values must be explicitly cast in `CREATE`, `SET`, and `MERGE` clauses — bare list literals are rejected:

```cypher
-- Correct: explicit CAST
CREATE (p:Person {id: 1, tags: CAST(['a', 'b'], 'STRING[]')});

-- Error: bare list literal is not accepted
CREATE (p:Person {id: 2, tags: ['a', 'b']});
```

List properties are supported in CSV and JSON bulk loading via `COPY FROM`. List values in CSV use bracket syntax `[1, 2, 3]`, and nested lists nest the brackets:

```
id|tags
1|[a,b,c]
2|[]
```

For more details on array vs list type distinctions, refer to the [Array Properties](#array-properties) section above.

## Drop Node Type

Delete a specified Node type. Use IF EXISTS to avoid errors when the type doesn't exist.
Expand Down
51 changes: 51 additions & 0 deletions include/neug/storages/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,57 @@ Unlike the property table of vertices, the property table of edges is not column
Fow now, only one property is supported for edges, but developers can define a struct with multiple fields to store multiple properties.


## 5. List Property Storage

Variable-length list properties (`T[]`) are stored using [ListPropertyColumn](../utils/property/list_property_column.h),
which decomposes each list into three separate sub-columns:

- **offsets** (`ULongColumn`): `offsets_[i]` is the starting index of row `i`'s elements within the elements column.
- **lengths** (`ULongColumn`): `lengths_[i]` is the number of elements in row `i`'s list.
- **elements** (`ColumnBase`): a contiguous column holding all list elements across all rows.

```
Row 0: offsets_[0] = 0, lengths_[0] = 3 → elements_[0], elements_[1], elements_[2]
Row 1: offsets_[1] = 3, lengths_[1] = 0 → (empty list)
Row 2: offsets_[2] = 3, lengths_[2] = 2 → elements_[3], elements_[4]
```

This design allows each sub-column to use the most efficient storage format for its own type
(e.g., `TypedColumn<T>` for POD elements, `StringColumn` for string elements), and enables
sequential access patterns during checkpoint dump.

### 5.1 Write Path

When setting a list value at row `i` (`set_any`):

1. If the new element count equals the existing `lengths_[i]`, the elements are overwritten
in place at `offsets_[i]` — no offset or length update needed.
2. If the element count differs, the new elements are **appended** to the tail of the elements
column, and `offsets_[i]` / `lengths_[i]` are updated to point to the new region. The old
elements become dead space and are reclaimed during checkpoint dump.

### 5.2 Checkpoint Dump (Compaction)

During `Dump`, the column is compacted: all live elements are written contiguously into a new
compact elements column, eliminating dead space from in-place updates that changed list lengths.
The offsets and lengths columns are rewritten to reflect the compacted layout.

### 5.3 Nested Lists

For nested list types (e.g., `STRING[][]`), the `elements` sub-column is itself a
`ListPropertyColumn`, creating a recursive storage structure. Each level of nesting adds another
layer of offset/length/elements decomposition.

For list-of-array types (e.g., `STRING[][2][]`), the elements column is an `ArrayColumn`,
which stores fixed-size arrays inline.

### 5.4 Thread Safety

The `insert_safe` parameter controls resize behavior, inherited from the `ColumnBase` interface:
- When `false`, throws on insufficient space in the elements column.
- When `true`, the elements column is resized as needed; the caller must provide external
synchronization during resize.


## 6. Durability

Expand Down
83 changes: 83 additions & 0 deletions include/neug/utils/property/list_property_column.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
/** Copyright 2020 Alibaba Group Holding Limited.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#pragma once

#include <memory>
#include <string>

#include "neug/common/types.h"
#include "neug/common/types/value.h"
#include "neug/utils/property/column.h"

namespace neug {

class ListPropertyColumn : public ColumnBase {
public:
ListPropertyColumn() : size_(0) {}
explicit ListPropertyColumn(const DataType& list_type);
~ListPropertyColumn() override = default;

void Open(Checkpoint& ckp, const ModuleDescriptor& desc,
MemoryLevel level) override;
void Open(Checkpoint& ckp, const CheckpointManifest& manifest,
const ModuleDescriptor& desc, MemoryLevel level) override;

void Dump(Checkpoint& ckp, CheckpointManifest& meta,
const std::string& key) override;

size_t size() const override { return size_; }
void resize(size_t size) override;
void resize(size_t size, const Value& default_value) override;

DataTypeId type() const override { return DataTypeId::kList; }
void set_any(size_t index, const Value& value, bool insert_safe) override;
Value get_any(size_t index) const override;

std::unique_ptr<Module> Clone() const override;
void Detach(Checkpoint& ckp, MemoryLevel level) override;

std::string ModuleTypeName() const override { return type_name(); }
static std::string type_name() { return "column<list>"; }

const DataType& list_type() const { return list_type_; }
const DataType& child_type() const { return child_type_; }

private:
void openInternal(Checkpoint& ckp, const CheckpointManifest* manifest,
const ModuleDescriptor& desc, MemoryLevel level);
ModuleDescriptor dumpSelfDescriptor() const;

DataType list_type_;
DataType child_type_;
size_t size_;
std::unique_ptr<ULongColumn> offsets_;
std::unique_ptr<ULongColumn> lengths_;
std::unique_ptr<ColumnBase> elements_;
};

class ListPropertyRefColumn : public RefColumnBase {
public:
explicit ListPropertyRefColumn(const ListPropertyColumn& column)
: column_(column) {}

Value get_any(size_t index) const override { return column_.get_any(index); }
DataTypeId type() const override { return DataTypeId::kList; }
ColType col_type() const override { return ColType::kInternal; }

private:
const ListPropertyColumn& column_;
};

} // namespace neug
7 changes: 7 additions & 0 deletions include/neug/utils/property/types.h
Original file line number Diff line number Diff line change
Expand Up @@ -646,11 +646,18 @@ struct convert<neug::DataType> {
: neug::STRING_DEFAULT_MAX_LENGTH;
} else if (id == neug::DataTypeId::kDate) {
node["temporal"]["date"] = "";
} else if (id == neug::DataTypeId::kTimestampMs) {
node["temporal"]["timestamp"] = "";
} else if (id == neug::DataTypeId::kInterval) {
node["temporal"]["interval"] = "";
} else if (id == neug::DataTypeId::kArray) {
auto child_type = neug::ArrayType::GetChildType(type);
uint64_t array_size = neug::ArrayType::GetNumElements(type);
node["array"]["component_type"] = encode(child_type);
node["array"]["max_length"] = array_size;
} else if (id == neug::DataTypeId::kList) {
node["list"]["component_type"] =
encode(neug::ListType::GetChildType(type));
} else {
LOG(ERROR) << "Unrecognized property type: " << type.ToString();
}
Expand Down
7 changes: 6 additions & 1 deletion src/compiler/binder/expression_binder.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,12 @@ std::shared_ptr<Expression> ExpressionBinder::bindExpression(
"bindExpression(" + ExpressionTypeUtil::toString(expressionType) +
").");
}
if (ConstantExpressionVisitor::needFold(*expression)) {
// Keep ANY-typed constants unfolded (e.g. the empty list literal `[]`):
// CAST binding retypes the list-creation function in place, which is only
// reachable while the argument is still a function expression, not a
// folded literal.
if (ConstantExpressionVisitor::needFold(*expression) &&
!expression->getDataType().containsAny()) {
return foldExpression(expression);
}
return expression;
Expand Down
30 changes: 20 additions & 10 deletions src/compiler/function/cast/cast_array.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,10 @@ bool CastArrayHelper::checkCompatibleNestedTypes(DataTypeId sourceTypeID,
return true;
}
case DataTypeId::kList:
case DataTypeId::kArray:
case DataTypeId::kArray: {
return targetTypeID == DataTypeId::kList ||
targetTypeID == DataTypeId::kArray;
}
case DataTypeId::kMap:
case DataTypeId::kStruct: {
return sourceTypeID == targetTypeID;
Expand All @@ -62,12 +65,12 @@ const DataType& getListLikeChildType(const DataType& type) {

bool CastArrayHelper::requiresArrayEntryValidation(const DataType& srcType,
const DataType& dstType) {
if (srcType.id() == DataTypeId::kArray &&
dstType.id() == DataTypeId::kArray) {
return true;
}

if (checkCompatibleNestedTypes(srcType.id(), dstType.id())) {
if (dstType.id() == DataTypeId::kArray &&
(srcType.id() == DataTypeId::kList ||
srcType.id() == DataTypeId::kArray)) {
return true;
}
switch (getPhysicalType(srcType.id())) {
case PhysicalTypeID::LIST: {
return requiresArrayEntryValidation(getListLikeChildType(srcType),
Expand Down Expand Up @@ -109,14 +112,19 @@ void CastArrayHelper::validateArrayEntries(ValueVector* inputVector,

switch (getPhysicalType(resultType.id())) {
case PhysicalTypeID::ARRAY: {
if (getPhysicalType(inputType.id()) != PhysicalTypeID::ARRAY ||
ArrayType::GetNumElements(inputType) !=
ArrayType::GetNumElements(resultType)) {
auto input_physical_type = getPhysicalType(inputType.id());
if (input_physical_type != PhysicalTypeID::ARRAY &&
input_physical_type != PhysicalTypeID::LIST) {
THROW_CONVERSION_EXCEPTION(
stringFormat("Unsupported casting function from {} to {}.",
inputType.ToString(), resultType.ToString()));
}
auto listEntry = inputVector->getValue<list_entry_t>(pos);
if (listEntry.size != ArrayType::GetNumElements(resultType)) {
THROW_CONVERSION_EXCEPTION(
stringFormat("Unsupported casting function from {} to {}.",
inputType.ToString(), resultType.ToString()));
}
auto inputChildVector = ListVector::getDataVector(inputVector);
for (auto i = listEntry.offset; i < listEntry.offset + listEntry.size;
i++) {
Expand All @@ -125,7 +133,9 @@ void CastArrayHelper::validateArrayEntries(ValueVector* inputVector,
}
} break;
case PhysicalTypeID::LIST: {
if (getPhysicalType(inputType.id()) == PhysicalTypeID::LIST) {
auto input_physical_type = getPhysicalType(inputType.id());
if (input_physical_type == PhysicalTypeID::LIST ||
input_physical_type == PhysicalTypeID::ARRAY) {
auto listEntry = inputVector->getValue<list_entry_t>(pos);
auto inputChildVector = ListVector::getDataVector(inputVector);
for (auto i = listEntry.offset; i < listEntry.offset + listEntry.size;
Expand Down
Loading
Loading