Cannot create Table with Array of Timestamp type

[yunzheng]

I'm trying to avoid pyarrow (due to it's size) and use arro3, but creating a Table with datetime objects doesn't seem to work yet:

from datetime import datetime, timezone

from arro3.core import Array, DataType, Table

a = Array([datetime.now(tz=timezone.utc)], type=DataType.timestamp("ns", tz="UTC"))
table = Table.from_pydict({"dt": a})

will raise the error:

NotImplementedError: Array constructor for Timestamp(Nanosecond, Some("UTC")) not yet implemented.

So it seems it's not implemented yet. But reading data with timestamps from disk (created using pyarrow) works just fine. So I was wondering if there are any workarounds?

[kylebarron]

It's not the table; it's the array. This line fails:

Array([datetime.now(tz=timezone.utc)], type=DataType.timestamp("ns", tz="UTC"))

It's true, right now in the array constructor, we support primitive, binary, and string types only:

arro3/pyo3-arrow/src/array.rs

Lines 215 to 286 in b9985f2

	let array: ArrayRef = match field.data_type() {
	DataType::Float32 => impl_primitive!(f32, Float32Type),
	DataType::Float64 => impl_primitive!(f64, Float64Type),
	DataType::UInt8 => impl_primitive!(u8, UInt8Type),
	DataType::UInt16 => impl_primitive!(u16, UInt16Type),
	DataType::UInt32 => impl_primitive!(u32, UInt32Type),
	DataType::UInt64 => impl_primitive!(u64, UInt64Type),
	DataType::Int8 => impl_primitive!(i8, Int8Type),
	DataType::Int16 => impl_primitive!(i16, Int16Type),
	DataType::Int32 => impl_primitive!(i32, Int32Type),
	DataType::Int64 => impl_primitive!(i64, Int64Type),
	DataType::Boolean => {
	let values: Vec<Option<bool>> = obj.extract()?;
	Arc::new(BooleanArray::from(values))
	}
	DataType::Binary => {
	let values: Vec<Option<PyBackedBytes>> = obj.extract()?;
	let slices = values
	.iter()
	.map(|maybe_vec| maybe_vec.as_ref().map(|vec| vec.as_ref()))
	.collect::<Vec<_>>();
	Arc::new(BinaryArray::from(slices))
	}
	DataType::LargeBinary => {
	let values: Vec<Option<PyBackedBytes>> = obj.extract()?;
	let slices = values
	.iter()
	.map(|maybe_vec| maybe_vec.as_ref().map(|vec| vec.as_ref()))
	.collect::<Vec<_>>();
	Arc::new(LargeBinaryArray::from(slices))
	}
	DataType::BinaryView => {
	let values: Vec<Option<PyBackedBytes>> = obj.extract()?;
	let slices = values
	.iter()
	.map(|maybe_vec| maybe_vec.as_ref().map(|vec| vec.as_ref()))
	.collect::<Vec<_>>();
	Arc::new(BinaryViewArray::from(slices))
	}
	DataType::FixedSizeBinary(size) => {
	let values: Vec<Option<PyBackedBytes>> = obj.extract()?;
	Arc::new(FixedSizeBinaryArray::try_from_sparse_iter_with_size(
	values
	.iter()
	.map(|maybe_vec| maybe_vec.as_ref().map(|vec| vec.as_ref())),
	*size,
	)?)
	}
	DataType::Utf8 => {
	let values: Vec<Option<PyBackedStr>> = obj.extract()?;
	let slices = values
	.iter()
	.map(|maybe_str| maybe_str.as_ref().map(|s| s.as_ref()))
	.collect::<Vec<_>>();
	Arc::new(StringArray::from(slices))
	}
	DataType::LargeUtf8 => {
	let values: Vec<Option<PyBackedStr>> = obj.extract()?;
	let slices = values
	.iter()
	.map(|maybe_str| maybe_str.as_ref().map(|s| s.as_ref()))
	.collect::<Vec<_>>();
	Arc::new(LargeStringArray::from(slices))
	}
	DataType::Utf8View => {
	let values: Vec<Option<PyBackedStr>> = obj.extract()?;
	let slices = values
	.iter()
	.map(|maybe_str| maybe_str.as_ref().map(|s| s.as_ref()))
	.collect::<Vec<_>>();
	Arc::new(StringViewArray::from(slices))
	}

We've implemented the reverse direction now, going from Arrow to Python datetime objects, but we need to also implement the reverse of going from Python datetime objects into an Arrow array.

Let's move this into an issue instead of a discussion. If you're interested in contributing I can give you some pointers.

[kylebarron]

Moving to #459

[yunzheng]

Thanks for the answer. I currently have no bandwidth to contribute, but if I do I will ask any questions in the issue.