diff --git a/_data/search_data.json b/_data/search_data.json
index 76193f933e0..cec6e60dd80 100644
--- a/_data/search_data.json
+++ b/_data/search_data.json
@@ -1,67 +1,116 @@
{
"data": [
{
- "title": "HTTPFS",
- "text": "the httpfs extension is a loadable extension implementing a file system that allows reading remote writing remote files for pure http s only file reading is supported for object storage using the s3 api the httpfs extension supports reading writing globbing files http s with the httpfs extension it is possible to directly query files over http s this currently works for csv and parquet files select from https domain tld file extension for csv files files will be downloaded entirely in most cases due to the row-based nature of the format for parquet files duckdb can use a combination of the parquet metadata and http range requests to only download the parts of the file that are actually required by the query for example the query select column_a from https domain tld file parquet will only read the parquet metadata and the data for the column_a column in some cases even no actual data needs to be read at all as they only require reading the metadata select count from https domain tld file parquet scanning multiple files over http s is also supported select from parquet_scan https domain tld file1 parquet https domain tld file2 parquet s3 the httpfs extension supports reading writing globbing files on object storage servers using the s3 api requirements the httpfs filesystem is tested with aws s3 minio and google cloud other services that implement the s3 api should also work but not all features may be supported below is a list of which parts of the s3 api are required for each httpfs feature feature required s3 api features --- --- public file reads http range requests private file reads secret key or session token authentication file glob listobjectv2 file writes multipart upload configuration to be able to read or write from s3 the correct region should be set set s3_region us-east-1 optionally the endpoint can be configured in case a non-aws object storage server is used set s3_endpoint domain tld port switching between path-style and vhost-style urls see aws docs is possible using set s3_url_style path however note that this may also require updating the endpoint for example for aws s3 it is required to change the endpoint to s3 region amazonaws com after configuring the correct endpoint and region public files can be read to also read private files authentication credentials can be added set s3_access_key_id aws access key id set s3_secret_access_key aws secret access key alternatively session tokens are also supported and can be used instead set s3_session_token aws session token reading reading files from s3 is now as simple as select from s3 bucket file extension multiple files are also possible for example select from parquet_scan s3 bucket file1 parquet s3 bucket file2 parquet glob file globbing is implemented using the listobjectv2 api call and allows to use filesystem-like glob patterns to match multiple files for example select from parquet_scan s3 bucket parquet this query matches all files in the root of the bucket with the parquet extension several features for matching are supported such as to match any number of any character for any single character or 0-9 for a single character in a range of characters select count from parquet_scan s3 bucket folder 100 t 0-9 parquet a useful feature when using globs is the filename option which adds a column with the file that a row originated from select from parquet_scan s3 bucket parquet filename 1 could for example result in column_a column_b filename --- --- --- 1 examplevalue1 s3 bucket file1 parquet 2 examplevalue1 s3 bucket file2 parquet hive partitioning duckdb also offers support for the hive partitioning scheme in the hive partitioning scheme data is partitioned in separate files the columns by which the data is partitioned are not actually in the files but are encoded in the file path so for example let us consider three parquet files hive paritioned by year s3 bucket year 2012 file parquet s3 bucket year 2013 file parquet s3 bucket year 2014 file parquet if scanning these files with the hive_partitioning option enabled select from parquet_scan s3 bucket file parquet hive_partitioning 1 could result in column_a column_b year --- --- --- 1 examplevalue1 2012 2 examplevalue2 2013 3 examplevalue3 2014 note that the year column does not actually exist in the parquet files it is parsed from the filenames within duckdb however these columns behave just like regular columns for example filters can be applied on hive partition columns select from parquet_scan s3 bucket file parquet hive_partitioning 1 where year 2013 writing writing to s3 uses the multipart upload api this allows duckdb to robustly upload files at high speed writing to s3 works for both csv and parquet copy table_name to s3 bucket file extension configuration some additional configuration options exist for the s3 upload though the default values should suffice for most use cases setting description --- --- s3_uploader_max_parts_per_file used for part size calculation see aws docs s3_uploader_max_filesize used for part size calculation see aws docs s3_uploader_thread_limit maximum number of uploader threads",
- "category": "Extensions",
- "url": "../docs/extensions/httpfs",
- "blurb": "The httpfs extension is a loadable extension implementing a file system that allows reading remote/writing remote ..."
+ "title": "Insert Statements",
+ "text": "insert statements are the standard way of loading data into a relational database when using insert statements the values are supplied row-by-row while simple there is significant overhead involved in parsing and processing individual insert statements this makes them very inefficient for bulk insertion as a rule-of-thumb avoid insert statements when inserting more than a few rows i e avoid using insert statements as part of a loop if you must use insert statements to load data in a loop avoid executing the statements in auto-commit mode after every commit the database is required to sync the changes made to disk to ensure no data is lost in auto-commit mode every single statement will be wrapped in a separate transaction meaning fsync will be called for every statement this is typically unnecessary when bulk loading and will significantly slow down your program if you absolutely must use insert statements in a loop to load data wrap them in calls to begin transaction and commit syntax an example of using insert into to load data in a table is as follows create table people id integer name varchar insert into people values 1 mark 2 hannes a more detailed description together with syntax diagram can be found here",
+ "category": "Data",
+ "url": "../docs/data/insert",
+ "blurb": "Insert statements are the standard way of loading data into a relational database. When using insert statements, the..."
},
{
- "title": "SQLite Scanner",
- "text": "see here for usage",
- "category": "Extensions",
- "url": "../docs/extensions/sqlite_scanner",
- "blurb": "See here for usage"
+ "title": "Parquet",
+ "text": "parquet files are compressed columnar files that are efficient to load and process duckdb provides support for both reading and writing parquet files in an efficient manner as well as support for pushing filters and projections into the parquet file scans examples -- read a single parquet file select from test parquet -- figure out which columns types are in a parquet file describe select from test parquet -- create a table from a parquet file create table test as select from test parquet -- if the file does not end in parquet use the read_parquet function select from read_parquet test parq -- use list parameter to read 3 parquet files and treat them as a single table select from read_parquet file1 parquet file2 parquet file3 parquet -- read all files that match the glob pattern select from test parquet -- read all files that match the glob pattern and include a filename column that specifies which file each row came from select from read_parquet test parquet filename true -- use a list of globs to read all parquet files from 2 specific folders select from read_parquet folder1 parquet folder2 parquet -- query the metadata of a parquet file select from parquet_metadata test parquet -- query the schema of a parquet file select from parquet_schema test parquet -- write the results of a query to a parquet file copy select from tbl to result-snappy parquet format parquet -- export the table contents of the entire database as parquet export database target_directory format parquet single-file reads duckdb includes an efficient parquet reader in the form of the read_parquet function select from read_parquet test parquet if your file ends in parquet the read_parquet syntax is optional the system will automatically infer that you are reading a parquet file select from test parquet unlike csv files parquet files are structured and as such are unambiguous to read no parameters need to be passed to this function the read_parquet function will figure out the column names and column types present in the file and emit them multi-file reads and globs duckdb can also read a series of parquet files and treat them as if they were a single table note that this only works if the parquet files have the same schema you can specify which parquet files you want to read using a list parameter glob pattern matching syntax or a combination of both list parameter the read_parquet function can accept a list of filenames as the input parameter see the nested types documentation for more details on lists -- read 3 parquet files and treat them as a single table select from read_parquet file1 parquet file2 parquet file3 parquet glob syntax any file name input to the read_parquet function can either be an exact filename or use a glob syntax to read multple files that match a pattern wildcard description ------------ ----------------------------------------------------------- matches any number of any characters including none matches any single character abc matches one character given in the bracket a-z matches one character from the range given in the bracket here is an example that reads all the files that end with parquet located in the test folder -- read all files that match the glob pattern select from read_parquet test parquet list of globs the glob syntax and the list input parameter can be combined to scan files that meet one of multiple patterns -- read all parquet files from 2 specific folders select from read_parquet folder1 parquet folder2 parquet filenames the filename parameter can be passed into the read_parquet function to include an extra filename column that specifies for each row from which file it was read for example select from parquet_scan data parquet-testing glob filename true order by i i j filename 1 a data parquet-testing glob t1 parquet 2 b data parquet-testing glob t2 parquet partial reading duckdb supports projection pushdown into the parquet file itself that is to say when querying a parquet file only the columns required for the query are read this allows you to read only the part of the parquet file that you are interested in this will be done automatically by the system duckdb also supports filter pushdown into the parquet reader when you apply a filter to a column that is scanned from a parquet file the filter will be pushed down into the scan and can even be used to skip parts of the file using the built-in zonemaps note that this will depend on whether or not your parquet file contains zonemaps filter and projection pushdown provide significant performance benefits see our blog post on this for more information inserts and views you can also insert the data into a table or create a table from the parquet file directly this will load the data from the parquet file and insert it into the database -- insert the data from the parquet file in the table insert into people select from read_parquet test parquet -- create a table directly from a parquet file create table people as select from read_parquet test parquet if you wish to keep the data stored inside the parquet file but want to query the parquet file directly you can create a view over the read_parquet function you can then query the parquet file as if it were a built-in table -- create a view over the parquet file create view people as select from read_parquet test parquet -- query the parquet file select from people parquet metadata the parquet_metadata function can be used to query the metadata contained within a parquet file which reveals various internal details of the parquet file such as the statistics of the different columns this can be useful for figuring out what kind of skipping is possible in parquet files or even to obtain a quick overview of what the different columns contain select from parquet_metadata test parquet below is a table of the columns returned by parquet_metadata field type ------------------------- --------- file_name varchar row_group_id bigint row_group_num_rows bigint row_group_num_columns bigint row_group_bytes bigint column_id bigint file_offset bigint num_values bigint path_in_schema varchar type varchar stats_min varchar stats_max varchar stats_null_count bigint stats_distinct_count bigint stats_min_value varchar stats_max_value varchar compression varchar encodings varchar index_page_offset bigint dictionary_page_offset bigint data_page_offset bigint total_compressed_size bigint total_uncompressed_size bigint parquet schema the parquet_schema function can be used to query the internal schema contained within a parquet file note that this is the schema as it is contained within the metadata of the parquet file if you want to figure out the column names and types contained within a parquet file it is easier to use describe -- fetch the column names and column types describe select from test parquet -- fetch the internal schema of a parquet file select from parquet_schema test parquet below is a table of the columns returned by parquet_schema field type ----------------- --------- file_name varchar name varchar type varchar type_length varchar repetition_type varchar num_children bigint converted_type varchar scale bigint precision bigint field_id bigint logical_type varchar writing to parquet files duckdb also has support for writing to parquet files using the copy statement syntax you can specify which compression format should be used using the codec parameter options uncompressed snappy default zstd gzip -- write a query to a snappy compressed parquet file copy select from tbl to result-snappy parquet format parquet -- write tbl to a zstd compressed parquet file copy tbl to result-zstd parquet format parquet codec zstd -- write a csv file to an uncompressed parquet file copy test csv to result-uncompressed parquet format parquet codec uncompressed duckdb s export command can be used to export an entire database to a series of parquet files see the export statement documentation for more details -- export the table contents of the entire database as parquet export database target_directory format parquet",
+ "category": "Data",
+ "url": "../docs/data/parquet",
+ "blurb": "Parquet files are compressed columnar files that are efficient to load and process. DuckDB provides support for both..."
},
{
- "title": "Extensions",
- "text": "duckdb has a number of extensions available for use not all of them are included by default in every distribution but duckdb has a mechanism that allows for remote installation remote installation if a given extensions is not available with your distribution you can do the following to make it available install fts load fts if you are using the python api client you can install and load them with the load_extension name str and install_extension name str methods unsigned extensions all verified extensions are signed if you wish to load your own extensions or extensions from untrusted third-parties you ll need to enable the allow_unsigned_extensions flag to load unsigned extensions using the cli you ll need to pass the -unsigned flag to it on startup listing extensions you can check the list of core and installed extensions with the following query select from duckdb_extensions all available extensions extension name description ---------------- -------------------------------------------------------------------- fts adds support for full-text search indexes httpfs adds support for reading and writing files over a http s connection icu adds support for time zones and collations using the icu library json adds support for json operations parquet adds support for reading and writing parquet files postgres_scanner adds support for reading from a postgres database sqlite_scanner adds support for reading sqlite database files substrait adds support for the substrait integration tpcds adds tpc-ds data generation and query support tpch adds tpc-h data generation and query support pages in this section",
- "category": "Extensions",
- "url": "../docs/extensions/overview",
- "blurb": "DuckDB has a number of extensions available for use. Not all of them are included by default in every distribution, but..."
+ "title": "Appender",
+ "text": "the c appender can be used to load bulk data into a duckdb database the appender is tied to a connection and will use the transaction context of that connection when appending an appender always appends to a single table in the database file duckdb db connection con db create the table con query create table people id integer name varchar initialize the appender appender appender con people the appendrow function is the easiest way of appending data it uses recursive templates to allow you to put all the values of a single row within one function call as follows appender appendrow 1 mark rows can also be individually constructed using the beginrow endrow and append methods this is done internally by appendrow and hence has the same performance characteristics appender beginrow appender append int32_t 2 appender append string hannes appender endrow any values added to the appender are cached prior to being inserted into the database system for performance reasons that means that while appending the rows might not be immediately visible in the system the cache is automatically flushed when the appender goes out of scope or when appender close is called the cache can also be manually flushed using the appender flush method after either flush or close is called all the data has been written to the database system date time and timestamps while numbers and strings are rather self-explanatory dates times and timestamps require some explanation they can be directly appended using the methods provided by duckdb date duckdb time or duckdb timestamp they can also be appended using the internal duckdb value type however this adds some additional overheads and should be avoided if possible below is a short example con query create table dates d date t time ts timestamp appender appender con dates construct the values using the date time timestamp types - this is the most efficient appender appendrow date fromdate 1992 1 1 time fromtime 1 1 1 0 timestamp fromdatetime date fromdate 1992 1 1 time fromtime 1 1 1 0 construct duckdb value objects appender appendrow value date 1992 1 1 value time 1 1 1 0 value timestamp 1992 1 1 1 1 1 0",
+ "category": "Data",
+ "url": "../docs/data/appender",
+ "blurb": "The C++ Appender can be used to load bulk data into a DuckDB database. The Appender is tied to a connection, and will use..."
},
{
- "title": "Full Text Search",
- "text": "full text search is an extension to duckdb that allows for search through strings similar to sqlite s fts5 extension api the extension adds two pragma statements to duckdb one to create and one to drop an index additionally a scalar macro stem is added which is used internally by the extension pragma create_fts_index create_fts_index input_table input_id input_values stemmer porter stopwords english ignore a-z strip_accents true lower true overwrite false pragma that creates a fts index for the specified table name type description -- -- -- input_table varchar qualified name of specified table e g table_name or main table_name input_id varchar column name of document identifier e g document_identifier input_values varchar column names of the text fields to be indexed vararg e g text_field_1 text_field_2 text_field_n or for all columns in input_table of type varchar stemmer varchar the type of stemmer to be used one of arabic basque catalan danish dutch english finnish french german greek hindi hungarian indonesian irish italian lithuanian nepali norwegian porter portuguese romanian russian serbian spanish swedish tamil turkish or none if no stemming is to be used defaults to porter stopwords varchar qualified name of table containing a single varchar column containing the desired stopwords or none if no stopwords are to be used defaults to english for a pre-defined list of 571 english stopwords ignore varchar regular expression of patterns to be ignored defaults to a-z ignoring all escaped and non-alphabetic lowercase characters strip_accents boolean whether to remove accents e g convert á to a defaults to true lower boolean whether to convert all text to lowercase defaults to true overwrite boolean whether to overwrite an existing index on a table defaults to false this pragma builds the index under a newly created schema the schema will be named after the input table if an index is created on table main table_name then the schema will be named fts_main_table_name pragma drop_fts_index drop_fts_index input_table drops a fts index for the specified table name type description -- -- -- input_table varchar qualified name of input table e g table_name or main table_name match_bm25 match_bm25 input_id query_string fields null k 1 2 b 0 75 conjunctive 0 when an index is built this retrieval macro is created that can be used to search the index name type description -- -- -- input_id varchar column name of document identifier e g document_identifier query_string varchar the string to search the index for fields varchar comma-separarated list of fields to search in e g text_field_2 text_field_n defaults to null to search all indexed fields k double parameter k sub 1 sub in the okapi bm25 retrieval model defaults to 1 2 b double parameter b in the okapi bm25 retrieval model defaults to 0 75 conjunctive boolean whether to make the query conjunctive i e all terms in the query string must be present in order for a document to be retrieved stem stem input_string stemmer reduces words to their base used internally by the extension name type description -- -- -- input_string varchar the column or constant to be stemmed stemmer varchar the type of stemmer to be used one of arabic basque catalan danish dutch english finnish french german greek hindi hungarian indonesian irish italian lithuanian nepali norwegian porter portuguese romanian russian serbian spanish swedish tamil turkish or none if no stemming is to be used example usage -- create a table and fill it with text data create table documents document_identifier varchar text_content varchar author varchar doc_version integer insert into documents values doc1 the mallard is a dabbling duck that breeds throughout the temperate hannes mühleisen 3 doc2 the cat is a domestic species of small carnivorous mammal laurens kuiper 2 -- build the index make both the text_content and author columns searchable pragma create_fts_index documents document_identifier text_content author -- search the author field index for documents that are written by hannes - this retrieves doc1 select text_content score from select fts_main_documents match_bm25 document_identifier muhleisen fields author as score from documents sq where score is not null and doc_version 2 order by score desc -- search for documents about small cats - this retrieves doc2 select text_content score from select fts_main_documents match_bm25 document_identifier small cats as score from documents sq where score is not null order by score desc caveats note that the fts index will not update automatically when input table changes a workaround of this limitation can be recreating the index to refresh",
- "category": "Extensions",
- "url": "../docs/extensions/full_text_search",
- "blurb": "Full Text Search is an extension to DuckDB that allows for search through strings, similar to SQLite's FTS5 extension. ..."
+ "title": "CSV Loading",
+ "text": "csv loading is a very common and yet surprisingly tricky task while csvs seem simple on the surface there are a lot of inconsistencies found within csv files that can make loading them a challenge csv files exist with different delimiters they can contain quoted values have an optional header row or even multiple or even be completely deformed the csv reader needs to cope with all of these different situations the duckdb csv reader can automatically infer which configuration flags to use by analyzing the csv file this will work correctly in most situations and should be the first option attempted in rare situations where the csv reader cannot figure out the correct configuration it is possible to manually configure the csv reader to correctly parse the csv file we use the following csv file in our examples keep in mind that you can also use compressed csv files in the following examples e g a gzipped file such as test csv gz will work just fine test csv flightdate uniquecarrier origincityname destcityname 1988-01-01 aa new york ny los angeles ca 1988-01-02 aa new york ny los angeles ca 1988-01-03 aa new york ny los angeles ca examples -- read a csv file from disk auto-infer options select from test csv -- read_csv with custom options select from read_csv_auto test csv delim header true columns flightdate date uniquecarrier varchar origincityname varchar destcityname varchar -- read a csv file into a table create table ontime flightdate date uniquecarrier varchar origincityname varchar destcityname varchar copy ontime from test csv auto_detect true -- alternatively create a table without specifying the schema manually create table ontime as select from test csv -- write the result of a query to a csv file copy select from ontime to test csv with header 1 delimiter parameters name description --- --- auto_detect option for csv parsing if true the parser will attempt to detect the input format and data types automatically delim sep quote escape and header parameters become optional read_csv_auto defaults to true for this parameter read_csv defaults to false columns a struct that specifies the column names and column types contained within the csv file e g col1 integer col2 varchar if auto_detect is enabled these will be inferred sep or delim specifies the string that separates columns within each row line of the file the default value is a comma nullstr specifies the string that represents a null value the default is an empty string header specifies that the file contains a header line with the names of each column in the file quote specifies the quoting string to be used when a data value is quoted the default is double-quote escape specifies the string that should appear before a data character sequence that matches the quote value the default is the same as the quote value so that the quoting string is doubled if it appears in the data dateformat specifies the date format to use when parsing dates see date format timestampformat specifies the date format to use when parsing timestamps see date format sample_size option to define number of sample rows for automatic csv type detection chunks of sample rows will be drawn from different locations of the input file set to -1 to scan the entire input file note only the first max 1024 rows will be used for dialect detection all_varchar option to skip type detection for csv parsing and assume all columns to be of type varchar normalize_names boolean value that specifies whether or not column names should be normalized removing any non-alphanumeric characters from them compression the compression type for the file by default this will be detected automatically from the file extension e g t csv gz will use gzip t csv will use none options are none gzip zstd filename boolean value that specifies whether or not an extra filename column should be included in the result skip the number of lines at the top of the file to skip read_csv_auto function the read_csv_auto is the simplest method of loading csv files it automatically attempts to figure out the correct configuration of the csv reader it also automatically deduces types of columns if the csv file has a header it will use the names found in that header to name the columns otherwise the columns will be named column0 column1 column2 select from read_csv_auto test csv flightdate uniquecarrier origincityname destcityname --------- ------------ ---------------- -------------- 1988-01-01 aa new york ny los angeles ca 1988-01-02 aa new york ny los angeles ca 1988-01-03 aa new york ny los angeles ca the path can either be a relative path relative to the current working directory or an absolute path we can use read_csv_auto to create a persistent table as well create table ontime as select from read_csv_auto test csv describe ontime field type null key default extra ------------- ------ --- --- ------ ---- flightdate date yes null null null uniquecarrier varchar yes null null null origincityname varchar yes null null null destcityname varchar yes null null null select from read_csv_auto test csv sample_size 20000 if we set delim sep quote escape or header explicitly we can bypass the automatic detection of this particular parameter select from read_csv_auto test csv header true note read_csv_auto is an alias for read_csv auto_detect true copy statement the copy statement can be used to load data from a csv file into a table this statement has the same syntax as the copy statement supported by postgresql for the copy statement we must first create a table with the correct schema to load the data into we then specify the csv file to load from plus any configuration options separately create table ontime flightdate date uniquecarrier varchar origincityname varchar destcityname varchar copy ontime from test csv delimiter header select from ontime flightdate uniquecarrier origincityname destcityname --------- ------------ ---------------- -------------- 1988-01-01 aa new york ny los angeles ca 1988-01-02 aa new york ny los angeles ca 1988-01-03 aa new york ny los angeles ca if we want to use the automatic format detection we can set auto_detect to true and omit the otherwise required configuration options create table ontime flightdate date uniquecarrier varchar origincityname varchar destcityname varchar copy ontime from test csv auto_detect true select from ontime more on the copy statement can be found here",
+ "category": "Data",
+ "url": "../docs/data/csv",
+ "blurb": "CSV loading is a very common, and yet surprisingly tricky, task. While CSVs seem simple on the surface, there are a lot..."
},
{
- "title": "JSON",
- "text": "the json extension is a loadable extension that implements sql functions that are useful for reading values from existing json and creating new json data json type the json extension makes use of the json logical type the json logical type is interpreted as json i e parsed in json functions rather than interpreted as varchar i e a regular string all json creation functions return values of this type json functions the following scalar json functions can be used to gain information about the stored json values with the exception of json_valid json all json functions produce an error when invalid json is supplied we support two kinds of notations to describe locations within json json pointer and jsonpath function description --- --- json json parse and minify json json_valid json return whether json is valid json json_array_length json path return the number of elements in the json array json or 0 if it is not a json array if path is specified return the number of elements in the json array at the given path if path is a list the result will be list of array lengths json_type json path return the type of the supplied json which is one of object array bigint ubigint varchar boolean null if path is specified return the type of the element at the given path if path is a list the result will be list of types json_structure json return the structure of json throws an error if the structure is inconsistent incompatible types in an array examples create table example j json insert into example values family anatidae species duck goose swan null select json j from example -- family anatidae species duck goose swan null select json_valid j from example -- true select json_valid -- false select json_array_length duck goose swan null -- 4 select json_array_length j species from example -- 4 select json_array_length j species from example -- 4 select json_array_length j species from example -- 4 select json_array_length j species from example -- 4 select json_type j from example -- object select json_structure j from example -- family varchar species varchar select json_structure duck family anatidae -- invalid input error inconsistent json structure json extraction functions there are two extraction functions which have their respective operators the operators can only be used if the string is stored as the json logical type these functions supports the same two location notations as the previous functions function alias operator description --- --- --- json_extract json path json_extract_path - extract json from json at the given path if path is a list the result will be a list of json json_extract_string json path json_extract_path_text - extract varchar from json at the given path if path is a list the result will be a list of varchar examples create table example j json insert into example values family anatidae species duck goose swan null select json_extract j family from example -- anatidae select j- family from example -- anatidae select j- species 0 from example -- duck select j- species - 0 from example -- duck select j- species - 0 1 from example -- duck goose select json_extract_string j family from example -- anatidae select j- family from example -- anatidae select j- species 0 from example -- duck select j- species - 0 from example -- duck select j- species - 0 1 from example -- duck goose json creation functions the following functions are used to create json function description --- --- to_json any create json from a value of any type our list is converted to a json array and our struct and map are converted to a json object json_quote any alias for to_json array_to_json list alias for to_json that only accepts list row_to_json list alias for to_json that only accepts struct json_array any create a json array from any number of values json_object key value create a json object from any number of key value pairs json_merge_patch json json merge two json documents together read_json_objects filename read 1 or more newline-delimited json objects from a file read_ndjson_objects filename alias for read_json_objects examples select to_json duck -- duck select to_json 1 2 3 -- 1 2 3 select to_json duck 42 -- duck 42 select to_json map duck 42 -- duck 42 select json_array 42 duck null -- 42 duck null select json_object duck 42 -- duck 42 select json_merge_patch duck 42 goose 123 -- goose 123 duck 42 json aggregate functions there are three json aggregate functions function description --- --- json_group_array any return a json array with all values of any in the aggregation json_group_object key value return a json object with all key value pairs in the aggregation json_group_structure json return the combined json_structure of all json in the aggregation examples create table example k varchar v integer insert into example values duck 42 goose 7 select json_group_array v from example -- 42 7 select json_group_object k v from example -- duck 42 goose 7 drop table example create table example j json insert into example values family anatidae species duck goose coolness 42 42 family canidae species labrador bulldog hair true select json_group_structure j from example -- family varchar species varchar coolness double hair boolean transforming json in many cases it is inefficient to extract values from json one-by-one instead we can extract all values at once transforming json to the nested types list and struct function description --- --- json_transform json structure transform json according to the specified structure from_json json structure alias for json_transform json_transform_strict json structure same as json_transform but throws an error when type casting fails from_json_strict json structure alias for json_transform_strict the structure argument is json of the same form as returned by json_structure the structure argument can be modified to transform the json into the desired structure and types it is possible to extract fewer key value pairs than are present in the json and it is also possible to extract more missing keys become null examples create table example j json insert into example values family anatidae species duck goose coolness 42 42 family canidae species labrador bulldog hair true select json_transform j family varchar coolness double from example -- family anatidae coolness 42 420000 -- family canidae coolness null select json_transform j family tinyint coolness decimal 4 2 from example -- family null coolness 42 42 -- family null coolness null select json_transform_strict j family tinyint coolness double from example -- invalid input error failed to cast value anatidae",
- "category": "Extensions",
- "url": "../docs/extensions/json",
- "blurb": "The json extension is a loadable extension that implements SQL functions that are useful for reading values from..."
+ "title": "Importing Data",
+ "text": "the first step to using a database system is to insert data into that system duckdb provides several data ingestion methods that allow you to easily and efficiently fill up the database in this section we provide an overview of these methods so you can select which one is correct for you insert statements insert statements are the standard way of loading data into a database system they are suitable for quick prototyping but should be avoided for bulk loading as they have significant per-row overhead insert into people values 1 mark see here for a more detailed description of insert statements csv loading data can be efficiently loaded from csv files using the read_csv_auto function or the copy statement select from read_csv_auto test csv you can also load data from compressed e g compressed with gzip csv files for example select from read_csv_auto test csv gz see here for a detailed description of csv loading parquet loading parquet files can be efficiently loaded and queried using the read_parquet function select from read_parquet test parquet see here for a detailed description of parquet loading appender c and java in c and java the appender can be used as an alternative for bulk data loading this class can be used to efficiently add rows to the database system without needing to use sql c appender appender con people appender appendrow 1 mark appender close java con createappender main people appender beginrow appender append mark appender endrow appender close see here for a detailed description of the c appender",
+ "category": "Data",
+ "url": "../docs/data/overview",
+ "blurb": "The first step to using a database system is to insert data into that system. DuckDB provides several data ingestion..."
},
{
- "title": "Postgres Scanner",
- "text": "see here for usage",
- "category": "Extensions",
- "url": "../docs/extensions/postgres_scanner",
- "blurb": "See here for usage"
+ "title": "Connect",
+ "text": "connect or create a database to use duckdb you must first create a connection to a database the exact process varies by client most clients take a parameter pointing to a database file to read and write from the file extension may be anything e g db duckdb etc if the database file does not exist it will be created the special value memory can be used to create an in-memory database where no data is persisted to disk i e all data is lost when you exit the process see the api docs for client-specific details",
+ "category": "Docs",
+ "url": "../docs/connect",
+ "blurb": "Connect or Create a Database To use DuckDB, you must first create a connection to a database. The exact process varies..."
},
{
- "title": "Substrait",
- "text": "see repo for usage",
- "category": "Extensions",
- "url": "../docs/extensions/substrait",
- "blurb": "See repo for usage"
+ "title": "S3 Parquet Export",
+ "text": "how to write a parquet file directly to s3 to write a parquet file to s3 the httpfs extension is required this can be installed use the install sql command this only needs to be run once install httpfs to load the httpfs extension for usage use the load sql command load httpfs after loading the httpfs extension set up the credentials and s3 region to write data you may either use an access key and secret or a token set s3_region us-east-1 set s3_access_key_id aws access key id set s3_secret_access_key aws secret access key the alternative is to use a token set s3_region us-east-1 set s3_session_token aws session token after the httpfs extension is set up and the s3 credentials are correctly configured parquet files can be written to s3 using the following command copy table_name to s3 bucket file extension",
+ "category": "Import",
+ "url": "../docs/guides/import/s3_export",
+ "blurb": "How to write a Parquet file directly to S3 To write a Parquet file to S3, the HTTPFS extension is required. This can..."
},
{
- "title": "Sitemap",
- "text": "",
- "category": "Docs",
- "url": "../docs/index",
- "blurb": ""
+ "title": "S3 Parquet Import",
+ "text": "how to load a parquet file directly from s3 to load a parquet file from s3 the httpfs extension is required this can be installed use the install sql command this only needs to be run once install httpfs to load the httpfs extension for usage use the load sql command load httpfs after loading the httpfs extension set up the credentials and s3 region to read data firstly the region where the data resides needs to be configured set s3_region us-east-1 with the only the region set public s3 data can be queried to query private s3 data you need to either use an access key and secret set s3_access_key_id aws access key id set s3_secret_access_key aws secret access key or a session token set s3_session_token aws session token after the httpfs extension is set up and the s3 configuration is set correctly parquet files can be read from s3 using the following command select from read_parquet s3 bucket file",
+ "category": "Import",
+ "url": "../docs/guides/import/s3_import",
+ "blurb": "How to load a Parquet file directly from S3 To load a Parquet file from S3, the HTTPFS extension is required. This..."
},
{
- "title": "Guides",
- "text": "the guides section contains compact how-to guides that are focused on achieving a single goal for an in-depth reference see the documentation section sql data import export csv files how to load a csv file into a table how to export a table to a csv file parquet files how to load a parquet file into a table how to export a table to a parquet file how to run a query directly on a parquet file http s3 how to load a parquet file directly from http s how to load a parquet file directly from s3 sql meta queries how to list all tables how to view the schema of the result of a query how to quickly get a feel for a dataset using summarize how to view the query plan of a query how to profile a query python client how to install the python client how to execute sql queries how to easily query duckdb in jupyter notebooks pandas how to execute sql on a pandas dataframe how to create a table from a pandas dataframe how to export data to a pandas dataframe apache arrow how to execute sql on apache arrow how to create a duckdb table from apache arrow how to export data to apache arrow relational api how to query pandas dataframes with the relational api python library integrations how to use ibis to query duckdb with or without sql sql editors ide s how to set up the dbeaver sql ide data viewers how to use tad to view tabular data files and duckdb databases",
- "category": "Guides",
- "url": "../docs/guides/index",
- "blurb": "The guides section contains compact how-to guides that are focused on achieving a single goal. For an in-depth reference..."
+ "title": "CSV Export",
+ "text": "how to export a table to a csv file to export the data from a table to a csv file use the copy statement copy tbl to output csv header delimiter the result of queries can also be directly exported to a csv file copy select from tbl to output csv header delimiter for additional options see the copy statement documentation",
+ "category": "Import",
+ "url": "../docs/guides/import/csv_export",
+ "blurb": "How to export a table to a CSV file To export the data from a table to a CSV file, use the COPY statement. COPY tbl..."
+ },
+ {
+ "title": "HTTP Parquet Import",
+ "text": "how to load a parquet file directly from http s to load a parquet file over http s the httpfs extension is required this can be installed use the install sql command this only needs to be run once install httpfs to load the httpfs extension for usage use the load sql command load httpfs after the httpfs extension is set up parquet files can be read over http s using the following command select from read_parquet https domain path to file parquet",
+ "category": "Import",
+ "url": "../docs/guides/import/http_import",
+ "blurb": "How to load a Parquet file directly from HTTP(s) To load a Parquet file over http(s) , the HTTPFS extension is..."
+ },
+ {
+ "title": "Parquet Import",
+ "text": "how to run a query directly on a parquet file to run a query directly on a parquet file use the read_parquet function in the from clause of a query select from read_parquet input parquet the parquet file will be processed in parallel filters will be automatically pushed down into the parquet scan and only the relevant columns will be read automatically for more information see the blog post querying parquet with precision using duckdb",
+ "category": "Import",
+ "url": "../docs/guides/import/query_parquet",
+ "blurb": "How to run a query directly on a Parquet file To run a query directly on a Parquet file, use the read_parquet ..."
+ },
+ {
+ "title": "Parquet Export",
+ "text": "how to export a table to a parquet file to export the data from a table to a parquet file use the copy statement copy tbl to output parquet format parquet the result of queries can also be directly exported to a parquet file copy select from tbl to output parquet format parquet",
+ "category": "Import",
+ "url": "../docs/guides/import/parquet_export",
+ "blurb": "How to export a table to a Parquet file To export the data from a table to a Parquet file, use the COPY statement. ..."
+ },
+ {
+ "title": "CSV Import",
+ "text": "how to load a csv file into a table to read data from a csv file use the read_csv_auto function in the from clause of a query select from read_csv_auto input csv to create a new table using the result from a query use create table as from a select statement create table new_tbl as select from read_csv_auto input csv to load data into an existing table from a query use insert into from a select statement insert into tbl select from read_csv_auto input csv alternatively the copy statement can also be used to load data from a csv file into an existing table copy tbl from input csv for additional options see the csv loading reference and the copy statement documentation",
+ "category": "Import",
+ "url": "../docs/guides/import/csv_import",
+ "blurb": "How to load a CSV file into a table To read data from a CSV file, use the read_csv_auto function in the FROM clause..."
+ },
+ {
+ "title": "Parquet Import",
+ "text": "how to load a parquet file into a table to read data from a parquet file use the read_parquet function in the from clause of a query select from read_parquet input parquet to create a new table using the result from a query use create table as from a select statement create table new_tbl as select from read_parquet input parquet to load data into an existing table from a query use insert into from a select statement insert into tbl select from read_parquet input parquet alternatively the copy statement can also be used to load data from a parquet file into an existing table copy tbl from input parquet format parquet for additional options see the parquet loading reference",
+ "category": "Import",
+ "url": "../docs/guides/import/parquet_import",
+ "blurb": "How to load a Parquet file into a table To read data from a Parquet file, use the read_parquet function in the FROM ..."
+ },
+ {
+ "title": "Tad - A Tabular Data Viewer",
+ "text": "how to use tad to view tabular data files and duckdb databases tad is a fast free cross-platform tabular data viewer application powered by duckdb there are pre-built binary installers available for mac windows and linux and full source code is available on github tad is a low friction way to explore tabular data files csv and parquet as well as sqlite and duckdb database files a dirty little secret of the data world is that workflows often involve using commercial spreadsheet apps like excel to take a quick peek at the contents of a tabular data file like a csv before analysis work begins tad provides a pleasant and free alternative to this and thanks to duckdb is much faster than excel at opening csv files how much faster on a 2019 macbook pro opening metobjects csv a 230 mb csv with 450k rows takes around 33 seconds to open with the latest version of excel this same file opens in under 5 seconds in tad unlike commcercial spreadsheet applications tad also natively supports parquet files since duckdb operates on parquet files in place without an extra import step the speed for opening parquet files is pretty mind-blowing here is tad browsing a directory of multi-million-row parquet files from the tpc-h-sf10 benchmark data set tad-parquet data exploration with tad for data exploration the core of tad is a hierarchical pivot table that allows you to specify a combination of pivot filter aggregate sort column selection column ordering and basic column formatting operations directly in the ui tad delegates to a sql database duckdb for storage and analytics and generates sql queries to perform all analytic operations specified in the ui here is a quick view of using tad to pivot and sort a tabular data file tad-metobjects-pivoted launching tad there are three ways to launch tad application icon you can double click on the tad application icon to open tad use strong file open strong and the standard file open dialog to open a tabular data file in any of the supported file formats context menu or drag and drop the installation process registers tad as a viewer for the supported file format extensions on any code csv code or code parquet code file in the finder macos or explorer windows use strong open with strong and select strong tad strong to open the file in tad command line once tad is installed and available on your path simply type tad somefile csv to launch tad to explore the file you can also open code tad code files previously saved tad view configurations and all the other supported file types via the command line browsing database files of particular interest to duckdb users tad can serve as a lightweight browser for duckdb and sqlite database files just run tad mydatabase duckdb from the command line to get a browsable view of the tables in a duckdb database file this also works for sqlite databases files with a sqlite extension peeking under the bonnet if for some reason you want to see the gnarly details of the sql queries that tad is constructing and executing you can use the code -f code option on the command line to keep tad in the foreground and code --show-queries code to get tad to print the generated sql queries like so tad -f --show-queries somefile csv not for the faint-hearted downloading and installing tad you can download pre-built installers for mac linux and windows from the tad github releases page help bug reports feedback please send any questions feedback bug reports and feature requests to tad-feedback tadviewer com",
+ "category": "Data Viewers",
+ "url": "../docs/guides/data_viewers/tad",
+ "blurb": "How to use Tad to view tabular data files and DuckDb databases Tad is a fast, free cross-platform tabular data viewer..."
+ },
+ {
+ "title": "SQL on Pandas",
+ "text": "how to execute sql on a pandas dataframe pandas dataframes stored in local variables can be queried as if they are regular tables within duckdb import duckdb import pandas connect to an in-memory database con duckdb connect my_df pandas dataframe from_dict a 42 query the pandas dataframe my_df results con execute select from my_df df",
+ "category": "Python",
+ "url": "../docs/guides/python/sql_on_pandas",
+ "blurb": "How to execute SQL on a Pandas DataFrame Pandas DataFrames stored in local variables can be queried as if they are..."
},
{
"title": "Import From Pandas",
@@ -70,13 +119,6 @@
"url": "../docs/guides/python/import_pandas",
"blurb": "How to create a table from a Pandas DataFrame CREATE TABLE AS and INSERT INTO can be used to create a table from any..."
},
- {
- "title": "Export To Apache Arrow",
- "text": "how to export data to apache arrow all results of a query can be exported to an apache arrow table using the arrow function alternatively results can be returned as a recordbatchreader using the fetch_record_batch function and results can be read one batch at a time in addition relations built using duckdb s relational api can also be exported export to an arrow table import duckdb import pyarrow as pa connect to an in-memory database con duckdb connect my_arrow_table pa table from_pydict i 1 2 3 4 j one two three four query the apache arrow table my_arrow_table and return as an arrow table results con execute select from my_arrow_table arrow export as a recordbatchreader import duckdb import pyarrow as pa connect to an in-memory database con duckdb connect my_arrow_table pa table from_pydict i 1 2 3 4 j one two three four query the apache arrow table my_arrow_table and return as an arrow recordbatchreader chunk_size 1_000_000 results con execute select from my_arrow_table fetch_record_batch chunk_size loop through the results a stopiteration exception is thrown when the recordbatchreader is empty while true try process a single chunk here just printing as an example print results read_next_batch to_pandas except stopiteration print already fetched all batches break export from relational api arrow objects can also be exported from the relational api a relation can be converted to an arrow table using the arrow or to_arrow_table functions or a record batch using record_batch a result can be exported to an arrow table with arrow or the alias fetch_arrow_table or to a recordbatchreader using fetch_arrow_reader import duckdb connect to an in-memory database con duckdb connect con execute create table integers i integer con execute insert into integers values 0 1 2 3 4 5 6 7 8 9 null create a relation from the table and export the entire relation as arrow rel con table integers relation_as_arrow rel arrow or to_arrow_table or calculate a result using that relation and export that result to arrow res rel aggregate sum i execute result_as_arrow res arrow or fetch_arrow_table",
- "category": "Python",
- "url": "../docs/guides/python/export_arrow",
- "blurb": "How to export data to Apache Arrow All results of a query can be exported to an Apache Arrow Table using the arrow ..."
- },
{
"title": "Relational API and Pandas",
"text": "how to use the relational api to query pandas duckdb offers a relational api that can be used to chain together query operations these are lazily evaluated so that duckdb can optimize their execution these operators can act on pandas dataframes duckdb tables or views which can point to any underlying storage format that duckdb can read such as csv or parquet files etc here we show a simple example of reading from a pandas dataframe and returning a dataframe import duckdb import pandas connect to an in-memory database con duckdb connect input_df pandas dataframe from_dict i 1 2 3 4 j one two three four create a duckdb relation from a dataframe rel con df input_df chain together relational operators this is a lazy operation so the operations are not yet executed equivalent to select i j i 2 as two_i from input_df order by i desc limit 2 transformed_rel rel filter i 2 project i j i 2 as two_i order i desc limit 2 trigger execution by requesting df of the relation df could have been added to the end of the chain above - it was separated for clarity output_df transformed_rel df relational operators can also be used to group rows aggregate find distinct combinations of values join union and more they are also able to directly insert results into a duckdb table or write to a csv please see these additional examples and the available relational methods on the duckdbpyrelation class",
@@ -85,11 +127,11 @@
"blurb": "How to use the Relational API to query Pandas DuckDB offers a relational API that can be used to chain together query..."
},
{
- "title": "Jupyter Notebooks",
- "text": "duckdb in jupyter notebooks duckdb s python client can be used directly in jupyter notebooks with no additional configuration if desired however additional libraries can be used to simplify sql query development this guide will describe how to utilize those additional libraries see other guides in the python section for how to use duckdb and python together as a small note for maximum performance converting large output datasets to pandas dataframes using duckdb directly may be desirable however the difference is typically quite small this example workflow is also available as a google collab notebook library installation four additional libraries improve the duckdb experience in jupyter notebooks pandas clean table visualizations and compatibility with other analysis ipython-sql convert a jupyter code cell into a sql cell sqlalchemy used by ipython-sql to connect to databases duckdb_engine duckdb sqlalchemy driver used by sqlalchemy to connect to duckdb run these pip install commands from the command line if jupyter notebook is not yet installed otherwise see google collab link above for an in-notebook example pip install duckdb install jupyter notebook pip install notebook install supporting libraries pip install pandas conda install pandas pip install ipython-sql pip install sqlalchemy pip install duckdb-engine library import and configuration next open a jupyter notebook and import the relevant libraries import duckdb import pandas as pd import sqlalchemy no need to import duckdb_engine sqlalchemy will auto-detect the driver needed based on your connection string import ipython-sql jupyter extension to create sql cells load_ext sql set configrations on ipython-sql to directly output data to pandas and to simplify the output that is printed to the notebook config sqlmagic autopandas true config sqlmagic feedback false config sqlmagic displaycon false connect ipython-sql to duckdb using a sqlalchemy-style connection string you may either connect to an in memory duckdb or a file backed db sql duckdb memory sql duckdb path to file db querying duckdb single line sql queries can be run using sql at the start of a line query results will be displayed as a pandas df sql select off and flying as a_duckdb_column an entire jupyter cell can be used as a sql cell by placing sql at the start of the cell query results will be displayed as a pandas df sql select schema_name function_name from duckdb_functions order by all desc limit 5 to return query results into a pandas dataframe for future usage use as an assignment operator this can be used with both the sql and sql jupyter magics sql my_df select off and flying as a_duckdb_column querying pandas dataframes duckdb is able to find and query any dataframe stored as a variable in the jupyter notebook input_df pd dataframe from_dict i 1 2 3 j one two three the dataframe being queried can be specified just like any other table in the from clause sql output_df select sum i as total_i from input_df summary you now have the ability to alternate between sql and pandas in a simple and highly performant way dataframes can be read as tables in sql and sql results can be output into dataframes happy analyzing",
+ "title": "Execute SQL",
+ "text": "how to execute sql queries first connect to a database using the connect command by default an in-memory database will be opened import duckdb con duckdb connect after connecting sql queries can be executed using the execute command results con execute select 42 fetchall by default a list of python objects is returned use df if you would like the result to be returned as a python dataframe instead results con execute select 42 df",
"category": "Python",
- "url": "../docs/guides/python/jupyter",
- "blurb": "DuckDB in Jupyter Notebooks DuckDB's Python client can be used directly in Jupyter notebooks with no additional..."
+ "url": "../docs/guides/python/execute_sql",
+ "blurb": "How to execute SQL queries First connect to a database using the connect command. By default an in-memory database..."
},
{
"title": "Import From Apache Arrow",
@@ -98,13 +140,6 @@
"url": "../docs/guides/python/import_arrow",
"blurb": "How to create a table from Apache Arrow CREATE TABLE AS and INSERT INTO can be used to create a table from any..."
},
- {
- "title": "Execute SQL",
- "text": "how to execute sql queries first connect to a database using the connect command by default an in-memory database will be opened import duckdb con duckdb connect after connecting sql queries can be executed using the execute command results con execute select 42 fetchall by default a list of python objects is returned use df if you would like the result to be returned as a python dataframe instead results con execute select 42 df",
- "category": "Python",
- "url": "../docs/guides/python/execute_sql",
- "blurb": "How to execute SQL queries First connect to a database using the connect command. By default an in-memory database..."
- },
{
"title": "DuckDB with Ibis",
"text": "ibis is a python library that allows queries to be written in a pythonic relational style and then be compiled into sql ibis supports multiple database backends including duckdb by using duckdb s sqlalchemy driver ibis expressions can also be combined with sql statements installation to install only the duckdb backend for ibis use the commands below see the ibis duckdb installation instructions for a conda alternative note that duckdb support was added in ibis version 3 0 0 pip install ibis-framework duckdb duckdb sqlalchemy duckdb_engine and more are installed as dependencies querying duckdb with ibis the following example is loosely borrowed from the introduction to ibis tutorial which uses sqlite first we import ibis set it to interactive mode just for demo purposes - it is faster to not use this option and then connect to an in-memory duckdb instance we can then inspect the tables in our database import ibis ibis options interactive true use eager evaluation use only for demo purposes connection ibis duckdb connect memory use an in memory duckdb connection ibis duckdb connect path to my_db db use or create a physical duckdb at this path print connection list_tables output pragma_database_list duckdb_tables duckdb_views duckdb_indexes sqlite_master sqlite_schema sqlite_temp_master sqlite_temp_schema duckdb_constraints duckdb_columns duckdb_schemas duckdb_types we then create a handler to a specific table to be able to explore it further here we use a built in table called duckdb_types for simplicity the first thing we want to see is a list of columns duckdb_types_table connection table duckdb_types print duckdb_types_table columns output schema_name schema_oid type_oid type_name type_size type_category internal to access only certain columns use bracket syntax on the table handler we can also apply functions to transform the data for example to show only distinct values use the compile function to see the sql query that ibis generates print duckdb_types_table type_category type_size distinct print duckdb_types_table type_category type_size distinct compile type_category type_size --- --- boolean 1 numeric 1 numeric 2 numeric 4 numeric 8 datetime 4 datetime 8 string 16 nan 16 datetime 16 numeric 16 composite 16 composite 0 numeric nan select distinct t0 type_category t0 type_size from duckdb_types as t0 multiple methods can be chained together to build up more complex expressions this statement selects a subset of columns filters to rows containing a specific value in one column and sorts by another column the ibis-generated sql is shown below note that it uses a parameter as a part of the filter function print duckdb_types_table type_name type_category type_size filter duckdb_types_table type_category numeric sort_by type_size print duckdb_types_table type_name type_category type_size filter duckdb_types_table type_category numeric sort_by type_size compile type_name type_category type_size --- --- --- decimal numeric nan tinyint numeric 1 utinyint numeric 1 smallint numeric 2 usmallint numeric 2 integer numeric 4 float numeric 4 uinteger numeric 4 bigint numeric 8 double numeric 8 ubigint numeric 8 hugeint numeric 16 select t0 type_name t0 type_category t0 type_size from select t1 type_name as type_name t1 type_category as type_category t1 type_size as type_size from duckdb_types as t1 where t1 type_category cast as text as t0 order by t0 type_size combining sql and ibis expressions ibis can also be used to combine sql and relational operators sql can precede or follow ibis relational operations print duckdb_types_table sql select dense_rank over order by type_size as size_rank from duckdb_types group_by type_category aggregate avg_size_rank lambda t t size_rank mean print duckdb_types_table sql select dense_rank over order by type_size as size_rank from duckdb_types group_by type_category aggregate avg_size_rank lambda t t size_rank mean compile type_category avg_size_rank --- --- numeric 4 583333333333333 composite 3 6666666666666665 boolean 3 0 datetime 6 0 string 7 0 nan 7 0 with _ibis_view_11 as select dense_rank over order by type_size as size_rank from duckdb_types select t0 type_category avg t0 size_rank as avg_size_rank from _ibis_view_11 as t0 group by t0 type_category to learn more about ibis feel free to continue with the ibis introductory tutorial",
@@ -112,6 +147,13 @@
"url": "../docs/guides/python/ibis",
"blurb": "Ibis is a Python library that allows queries to be written in a pythonic relational style and then be compiled into SQL...."
},
+ {
+ "title": "Jupyter Notebooks",
+ "text": "duckdb in jupyter notebooks duckdb s python client can be used directly in jupyter notebooks with no additional configuration if desired however additional libraries can be used to simplify sql query development this guide will describe how to utilize those additional libraries see other guides in the python section for how to use duckdb and python together as a small note for maximum performance converting large output datasets to pandas dataframes using duckdb directly may be desirable however the difference is typically quite small this example workflow is also available as a google collab notebook library installation four additional libraries improve the duckdb experience in jupyter notebooks pandas clean table visualizations and compatibility with other analysis ipython-sql convert a jupyter code cell into a sql cell sqlalchemy used by ipython-sql to connect to databases duckdb_engine duckdb sqlalchemy driver used by sqlalchemy to connect to duckdb run these pip install commands from the command line if jupyter notebook is not yet installed otherwise see google collab link above for an in-notebook example pip install duckdb install jupyter notebook pip install notebook install supporting libraries pip install pandas conda install pandas pip install ipython-sql pip install sqlalchemy pip install duckdb-engine library import and configuration next open a jupyter notebook and import the relevant libraries import duckdb import pandas as pd import sqlalchemy no need to import duckdb_engine sqlalchemy will auto-detect the driver needed based on your connection string import ipython-sql jupyter extension to create sql cells load_ext sql set configrations on ipython-sql to directly output data to pandas and to simplify the output that is printed to the notebook config sqlmagic autopandas true config sqlmagic feedback false config sqlmagic displaycon false connect ipython-sql to duckdb using a sqlalchemy-style connection string you may either connect to an in memory duckdb or a file backed db sql duckdb memory sql duckdb path to file db querying duckdb single line sql queries can be run using sql at the start of a line query results will be displayed as a pandas df sql select off and flying as a_duckdb_column an entire jupyter cell can be used as a sql cell by placing sql at the start of the cell query results will be displayed as a pandas df sql select schema_name function_name from duckdb_functions order by all desc limit 5 to return query results into a pandas dataframe for future usage use as an assignment operator this can be used with both the sql and sql jupyter magics sql my_df select off and flying as a_duckdb_column querying pandas dataframes duckdb is able to find and query any dataframe stored as a variable in the jupyter notebook input_df pd dataframe from_dict i 1 2 3 j one two three the dataframe being queried can be specified just like any other table in the from clause sql output_df select sum i as total_i from input_df summary you now have the ability to alternate between sql and pandas in a simple and highly performant way dataframes can be read as tables in sql and sql results can be output into dataframes happy analyzing",
+ "category": "Python",
+ "url": "../docs/guides/python/jupyter",
+ "blurb": "DuckDB in Jupyter Notebooks DuckDB's Python client can be used directly in Jupyter notebooks with no additional..."
+ },
{
"title": "Install the Python Client",
"text": "installing the python client the latest release of the python client can be installed using pip pip install duckdb the pre-release python client can be installed using --pre pip install duckdb --upgrade --pre the python client can be installed from source from the tools pythonpkg directory in the duckdb github repository cd tools pythonpkg python setup py install",
@@ -120,11 +162,11 @@
"blurb": "Installing the Python Client The latest release of the Python client can be installed using pip . pip install duckdb..."
},
{
- "title": "SQL on Pandas",
- "text": "how to execute sql on a pandas dataframe pandas dataframes stored in local variables can be queried as if they are regular tables within duckdb import duckdb import pandas connect to an in-memory database con duckdb connect my_df pandas dataframe from_dict a 42 query the pandas dataframe my_df results con execute select from my_df df",
+ "title": "Export To Apache Arrow",
+ "text": "how to export data to apache arrow all results of a query can be exported to an apache arrow table using the arrow function alternatively results can be returned as a recordbatchreader using the fetch_record_batch function and results can be read one batch at a time in addition relations built using duckdb s relational api can also be exported export to an arrow table import duckdb import pyarrow as pa connect to an in-memory database con duckdb connect my_arrow_table pa table from_pydict i 1 2 3 4 j one two three four query the apache arrow table my_arrow_table and return as an arrow table results con execute select from my_arrow_table arrow export as a recordbatchreader import duckdb import pyarrow as pa connect to an in-memory database con duckdb connect my_arrow_table pa table from_pydict i 1 2 3 4 j one two three four query the apache arrow table my_arrow_table and return as an arrow recordbatchreader chunk_size 1_000_000 results con execute select from my_arrow_table fetch_record_batch chunk_size loop through the results a stopiteration exception is thrown when the recordbatchreader is empty while true try process a single chunk here just printing as an example print results read_next_batch to_pandas except stopiteration print already fetched all batches break export from relational api arrow objects can also be exported from the relational api a relation can be converted to an arrow table using the arrow or to_arrow_table functions or a record batch using record_batch a result can be exported to an arrow table with arrow or the alias fetch_arrow_table or to a recordbatchreader using fetch_arrow_reader import duckdb connect to an in-memory database con duckdb connect con execute create table integers i integer con execute insert into integers values 0 1 2 3 4 5 6 7 8 9 null create a relation from the table and export the entire relation as arrow rel con table integers relation_as_arrow rel arrow or to_arrow_table or calculate a result using that relation and export that result to arrow res rel aggregate sum i execute result_as_arrow res arrow or fetch_arrow_table",
"category": "Python",
- "url": "../docs/guides/python/sql_on_pandas",
- "blurb": "How to execute SQL on a Pandas DataFrame Pandas DataFrames stored in local variables can be queried as if they are..."
+ "url": "../docs/guides/python/export_arrow",
+ "blurb": "How to export data to Apache Arrow All results of a query can be exported to an Apache Arrow Table using the arrow ..."
},
{
"title": "Export To Pandas",
@@ -141,18 +183,18 @@
"blurb": "How to execute SQL on Apache Arrow DuckDB can query multiple different types of Apache Arrow objects. Apache Arrow..."
},
{
- "title": "Tad - A Tabular Data Viewer",
- "text": "how to use tad to view tabular data files and duckdb databases tad is a fast free cross-platform tabular data viewer application powered by duckdb there are pre-built binary installers available for mac windows and linux and full source code is available on github tad is a low friction way to explore tabular data files csv and parquet as well as sqlite and duckdb database files a dirty little secret of the data world is that workflows often involve using commercial spreadsheet apps like excel to take a quick peek at the contents of a tabular data file like a csv before analysis work begins tad provides a pleasant and free alternative to this and thanks to duckdb is much faster than excel at opening csv files how much faster on a 2019 macbook pro opening metobjects csv a 230 mb csv with 450k rows takes around 33 seconds to open with the latest version of excel this same file opens in under 5 seconds in tad unlike commcercial spreadsheet applications tad also natively supports parquet files since duckdb operates on parquet files in place without an extra import step the speed for opening parquet files is pretty mind-blowing here is tad browsing a directory of multi-million-row parquet files from the tpc-h-sf10 benchmark data set tad-parquet data exploration with tad for data exploration the core of tad is a hierarchical pivot table that allows you to specify a combination of pivot filter aggregate sort column selection column ordering and basic column formatting operations directly in the ui tad delegates to a sql database duckdb for storage and analytics and generates sql queries to perform all analytic operations specified in the ui here is a quick view of using tad to pivot and sort a tabular data file tad-metobjects-pivoted launching tad there are three ways to launch tad application icon you can double click on the tad application icon to open tad use strong file open strong and the standard file open dialog to open a tabular data file in any of the supported file formats context menu or drag and drop the installation process registers tad as a viewer for the supported file format extensions on any code csv code or code parquet code file in the finder macos or explorer windows use strong open with strong and select strong tad strong to open the file in tad command line once tad is installed and available on your path simply type tad somefile csv to launch tad to explore the file you can also open code tad code files previously saved tad view configurations and all the other supported file types via the command line browsing database files of particular interest to duckdb users tad can serve as a lightweight browser for duckdb and sqlite database files just run tad mydatabase duckdb from the command line to get a browsable view of the tables in a duckdb database file this also works for sqlite databases files with a sqlite extension peeking under the bonnet if for some reason you want to see the gnarly details of the sql queries that tad is constructing and executing you can use the code -f code option on the command line to keep tad in the foreground and code --show-queries code to get tad to print the generated sql queries like so tad -f --show-queries somefile csv not for the faint-hearted downloading and installing tad you can download pre-built installers for mac linux and windows from the tad github releases page help bug reports feedback please send any questions feedback bug reports and feature requests to tad-feedback tadviewer com",
- "category": "Data Viewers",
- "url": "../docs/guides/data_viewers/tad",
- "blurb": "How to use Tad to view tabular data files and DuckDb databases Tad is a fast, free cross-platform tabular data viewer..."
+ "title": "List Tables",
+ "text": "how to list all tables the show tables command can be used to obtain a list of all tables show tables to view the schema of an individual table use the describe command describe nation the describe command can also be used without a parameter to view all tables together with their columns and column types describe below is an example of describe without any parameters on the tpc-h dataset table_name column_names column_types temporary customer c_acctbal c_address c_comment c_custkey c_mktsegment c_name c_nationkey decimal 15 2 varchar varchar integer varchar varchar integer varchar false lineitem l_comment l_commitdate l_discount l_extendedprice l_linenumber l_linestat varchar date decimal 15 2 decimal 15 2 integer varchar integer intege false nation n_comment n_name n_nationkey n_regionkey varchar varchar integer integer false orders o_clerk o_comment o_custkey o_orderdate o_orderkey o_orderpriority o_ord varchar varchar integer date integer varchar varchar integer decimal 1 false part p_brand p_comment p_container p_mfgr p_name p_partkey p_retailprice p_s varchar varchar varchar varchar varchar integer decimal 15 2 integer false partsupp ps_availqty ps_comment ps_partkey ps_suppkey ps_supplycost integer varchar integer integer decimal 15 2 false region r_comment r_name r_regionkey varchar varchar integer false supplier s_acctbal s_address s_comment s_name s_nationkey s_phone s_suppkey decimal 15 2 varchar varchar varchar integer varchar integer false the sql-standard information_schema views are also defined duckdb also defines sqlite_master and many postgres system catalog tables for compatibility with sqlite and postgres respectively",
+ "category": "Meta",
+ "url": "../docs/guides/meta/list_tables",
+ "blurb": "How to list all tables The SHOW TABLES command can be used to obtain a list of all tables: SHOW TABLES; To view..."
},
{
- "title": "DBeaver SQL IDE",
- "text": "how to set up dbeaver sql ide for duckdb dbeaver is a powerful and popular desktop sql editor and integrated development environment ide it has both an open source and enterprise version it is useful for visually inspecting the available tables in duckdb and for quickly building complex queries duckdb s jdbc connector allows dbeaver to query duckdb files and by extension any other files that duckdb can access like parquet files install dbeaver using the download links and instructions found at their download page open dbeaver and create a new connection either click on the new database connector button or go to database new database connection in the menu bar img src images guides dbeaver_new_database_connection png alt dbeaver new database connection title dbeaver new database connection img src images guides dbeaver_new_database_connection_menu png alt dbeaver new database connection menu title dbeaver new database connection menu search for duckdb select it and click next img src images guides dbeaver_select_database_driver png alt dbeaver select database driver title dbeaver select database driver enter the path or browse to the duckdb database file you wish to query to use an in-memory duckdb useful primarily if just interested in querying parquet files or for testing enter memory as the path img src images guides dbeaver_connection_settings_path png alt dbeaver set path title dbeaver set path click test connection this will then prompt you to install the duckdb jdbc driver if you are not prompted see alternative driver installation instructions below img src images guides dbeaver_connection_settings_test_connection png alt dbeaver test connection title dbeaver test connection click download to download duckdb s jdbc driver from maven once download is complete click ok then click finish note if you are in a corporate environment or behind a firewall before clicking download click the download configuration link to configure your proxy settings img src images guides dbeaver_download_driver_files png alt dbeaver download driver files title dbeaver download driver files you should now see a database connection to your duckdb database in the left hand database navigator pane expand it to see the tables and views in your database right click on that connection and create a new sql script img src images guides dbeaver_new_sql_script png alt dbeaver new sql script title dbeaver new sql script write some sql and click the execute button img src images guides dbeaver_execute_query png alt dbeaver execute query title dbeaver execute query now you re ready to fly with duckdb and dbeaver img src images guides dbeaver_query_results png alt dbeaver query results title dbeaver query results alternative driver installation if not prompted to install the duckdb driver when testing your connection return to the connect to a database dialog and click edit driver settings img src images guides dbeaver_edit_driver_settings png alt dbeaver edit driver settings title dbeaver edit driver settings alternate you may also access the driver settings menu by returning to the main dbeaver window and clicking database driver manager in the menu bar then select duckdb then click edit img src images guides dbeaver_driver_manager png alt dbeaver driver manager title dbeaver driver manager img src images guides dbeaver_driver_manager_edit png alt dbeaver driver manager edit title dbeaver driver manager edit go to the libraries tab then click on the duckdb driver and click download update if you do not see the duckdb driver first click on reset to defaults img src images guides dbeaver_edit_driver_duckdb png alt dbeaver edit driver title dbeaver edit driver click download to download duckdb s jdbc driver from maven once download is complete click ok then return to the main dbeaver window and continue with step 7 above note if you are in a corporate environment or behind a firewall before clicking download click the download configuration link to configure your proxy settings img src images guides dbeaver_download_driver_files_from_driver_settings png alt dbeaver download driver files 2 title dbeaver download driver files 2",
- "category": "Sql Editors",
- "url": "../docs/guides/sql_editors/dbeaver",
- "blurb": "How to set up DBeaver SQL IDE for DuckDB DBeaver is a powerful and popular desktop sql editor and integrated..."
+ "title": "Summarize",
+ "text": "how to quickly get a feel for a dataset using summarize the summarize command can be used to easily compute a number of aggregates over a table or a query the summarize command launches a query that computes a number of aggregates over all columns including min max avg std and approx_count_distinct in order to summarize the contents of a table use summarize followed by the table name summarize tbl in order to summarize a query prepend summarize to a query summarize select from tbl below is an example of summarize on the lineitem table of tpc-h sf1 column_name column_type min max approx_unique avg std q25 q50 q75 count null_percentage l_orderkey integer 1 6000000 1486805 3000279 604204982 1732187 8734803426 1497471 3022276 4523225 6001215 0 0 l_partkey integer 1 200000 196125 100017 98932999402 57735 69082650517 50056 99973 150007 6001215 0 0 l_suppkey integer 1 10000 10010 5000 602606138924 2886 9619987306205 2499 5001 7498 6001215 0 0 l_linenumber integer 1 7 7 3 0005757167506912 1 7324314036519335 1 3 4 6001215 0 0 l_quantity integer 1 50 50 25 507967136654827 14 426262537016953 12 25 37 6001215 0 0 l_extendedprice decimal 15 2 901 00 104949 50 939196 38255 138484656854 23300 438710962204 18747 36719 55141 6001215 0 0 l_discount decimal 15 2 0 00 0 10 11 0 04999943011540163 0 031619855108125976 0 0 0 6001215 0 0 l_tax decimal 15 2 0 00 0 08 9 0 04001350893110812 0 02581655179884275 0 0 0 6001215 0 0 l_returnflag varchar a r 3 null null null null null 6001215 0 0 l_linestatus varchar f o 2 null null null null null 6001215 0 0 l_shipdate date 1992-01-02 1998-12-01 2554 null null null null null 6001215 0 0 l_commitdate date 1992-01-31 1998-10-31 2491 null null null null null 6001215 0 0 l_receiptdate date 1992-01-04 1998-12-31 2585 null null null null null 6001215 0 0 l_shipinstruct varchar collect cod take back return 4 null null null null null 6001215 0 0 l_shipmode varchar air truck 7 null null null null null 6001215 0 0 l_comment varchar tiresias zzle slyly final platelets sleep quickly 4587836 null null null null null 6001215 0 0",
+ "category": "Meta",
+ "url": "../docs/guides/meta/summarize",
+ "blurb": "How to quickly get a feel for a dataset using summarize The SUMMARIZE command can be used to easily compute a number..."
},
{
"title": "Describe",
@@ -168,20 +210,6 @@
"url": "../docs/guides/meta/explain_analyze",
"blurb": "In order to profile a query, prepend EXPLAIN ANALYZE to a query. EXPLAIN ANALYZE SELECT * FROM tbl; The query plan..."
},
- {
- "title": "Summarize",
- "text": "how to quickly get a feel for a dataset using summarize the summarize command can be used to easily compute a number of aggregates over a table or a query the summarize command launches a query that computes a number of aggregates over all columns including min max avg std and approx_count_distinct in order to summarize the contents of a table use summarize followed by the table name summarize tbl in order to summarize a query prepend summarize to a query summarize select from tbl below is an example of summarize on the lineitem table of tpc-h sf1 column_name column_type min max approx_unique avg std q25 q50 q75 count null_percentage l_orderkey integer 1 6000000 1486805 3000279 604204982 1732187 8734803426 1497471 3022276 4523225 6001215 0 0 l_partkey integer 1 200000 196125 100017 98932999402 57735 69082650517 50056 99973 150007 6001215 0 0 l_suppkey integer 1 10000 10010 5000 602606138924 2886 9619987306205 2499 5001 7498 6001215 0 0 l_linenumber integer 1 7 7 3 0005757167506912 1 7324314036519335 1 3 4 6001215 0 0 l_quantity integer 1 50 50 25 507967136654827 14 426262537016953 12 25 37 6001215 0 0 l_extendedprice decimal 15 2 901 00 104949 50 939196 38255 138484656854 23300 438710962204 18747 36719 55141 6001215 0 0 l_discount decimal 15 2 0 00 0 10 11 0 04999943011540163 0 031619855108125976 0 0 0 6001215 0 0 l_tax decimal 15 2 0 00 0 08 9 0 04001350893110812 0 02581655179884275 0 0 0 6001215 0 0 l_returnflag varchar a r 3 null null null null null 6001215 0 0 l_linestatus varchar f o 2 null null null null null 6001215 0 0 l_shipdate date 1992-01-02 1998-12-01 2554 null null null null null 6001215 0 0 l_commitdate date 1992-01-31 1998-10-31 2491 null null null null null 6001215 0 0 l_receiptdate date 1992-01-04 1998-12-31 2585 null null null null null 6001215 0 0 l_shipinstruct varchar collect cod take back return 4 null null null null null 6001215 0 0 l_shipmode varchar air truck 7 null null null null null 6001215 0 0 l_comment varchar tiresias zzle slyly final platelets sleep quickly 4587836 null null null null null 6001215 0 0",
- "category": "Meta",
- "url": "../docs/guides/meta/summarize",
- "blurb": "How to quickly get a feel for a dataset using summarize The SUMMARIZE command can be used to easily compute a number..."
- },
- {
- "title": "List Tables",
- "text": "how to list all tables the show tables command can be used to obtain a list of all tables show tables to view the schema of an individual table use the describe command describe nation the describe command can also be used without a parameter to view all tables together with their columns and column types describe below is an example of describe without any parameters on the tpc-h dataset table_name column_names column_types temporary customer c_acctbal c_address c_comment c_custkey c_mktsegment c_name c_nationkey decimal 15 2 varchar varchar integer varchar varchar integer varchar false lineitem l_comment l_commitdate l_discount l_extendedprice l_linenumber l_linestat varchar date decimal 15 2 decimal 15 2 integer varchar integer intege false nation n_comment n_name n_nationkey n_regionkey varchar varchar integer integer false orders o_clerk o_comment o_custkey o_orderdate o_orderkey o_orderpriority o_ord varchar varchar integer date integer varchar varchar integer decimal 1 false part p_brand p_comment p_container p_mfgr p_name p_partkey p_retailprice p_s varchar varchar varchar varchar varchar integer decimal 15 2 integer false partsupp ps_availqty ps_comment ps_partkey ps_suppkey ps_supplycost integer varchar integer integer decimal 15 2 false region r_comment r_name r_regionkey varchar varchar integer false supplier s_acctbal s_address s_comment s_name s_nationkey s_phone s_suppkey decimal 15 2 varchar varchar varchar integer varchar integer false the sql-standard information_schema views are also defined duckdb also defines sqlite_master and many postgres system catalog tables for compatibility with sqlite and postgres respectively",
- "category": "Meta",
- "url": "../docs/guides/meta/list_tables",
- "blurb": "How to list all tables The SHOW TABLES command can be used to obtain a list of all tables: SHOW TABLES; To view..."
- },
{
"title": "Explain",
"text": "how to view the query plan of a query in order to view the query plan of a query prepend explain to a query explain select from tbl by default only the final physical plan is shown in order to see the unoptimized and optimized logical plans change the explain_output setting set explain_output all below is an example of running explain on q1 of the tpc-h benchmark order_by lineitem l_returnflag asc lineitem l_linestatus asc hash_group_by 0 1 sum 2 sum 3 sum 4 sum 5 avg 6 avg 7 avg 8 count_star projection l_returnflag l_linestatus l_quantity l_extendedprice 4 4 1 00 l_tax l_quantity l_extendedprice l_discount projection l_returnflag l_linestatus l_quantity l_extendedprice l_extendedprice 1 00 - l_discount l_tax l_discount seq_scan lineitem l_shipdate l_returnflag l_linestatus l_quantity l_extendedprice l_discount l_tax filters l_shipdate 1998 -09-02 and l_shipdate null",
@@ -190,228 +218,214 @@
"blurb": "How to view the query plan of a query In order to view the query plan of a query, prepend EXPLAIN to a query. ..."
},
{
- "title": "Parquet Export",
- "text": "how to export a table to a parquet file to export the data from a table to a parquet file use the copy statement copy tbl to output parquet format parquet the result of queries can also be directly exported to a parquet file copy select from tbl to output parquet format parquet",
- "category": "Import",
- "url": "../docs/guides/import/parquet_export",
- "blurb": "How to export a table to a Parquet file To export the data from a table to a Parquet file, use the COPY statement. ..."
- },
- {
- "title": "CSV Export",
- "text": "how to export a table to a csv file to export the data from a table to a csv file use the copy statement copy tbl to output csv header delimiter the result of queries can also be directly exported to a csv file copy select from tbl to output csv header delimiter for additional options see the copy statement documentation",
- "category": "Import",
- "url": "../docs/guides/import/csv_export",
- "blurb": "How to export a table to a CSV file To export the data from a table to a CSV file, use the COPY statement. COPY tbl..."
+ "title": "Guides",
+ "text": "the guides section contains compact how-to guides that are focused on achieving a single goal for an in-depth reference see the documentation section sql data import export csv files how to load a csv file into a table how to export a table to a csv file parquet files how to load a parquet file into a table how to export a table to a parquet file how to run a query directly on a parquet file http s3 how to load a parquet file directly from http s how to load a parquet file directly from s3 sql meta queries how to list all tables how to view the schema of the result of a query how to quickly get a feel for a dataset using summarize how to view the query plan of a query how to profile a query python client how to install the python client how to execute sql queries how to easily query duckdb in jupyter notebooks pandas how to execute sql on a pandas dataframe how to create a table from a pandas dataframe how to export data to a pandas dataframe apache arrow how to execute sql on apache arrow how to create a duckdb table from apache arrow how to export data to apache arrow relational api how to query pandas dataframes with the relational api python library integrations how to use ibis to query duckdb with or without sql sql editors ide s how to set up the dbeaver sql ide data viewers how to use tad to view tabular data files and duckdb databases",
+ "category": "Guides",
+ "url": "../docs/guides/index",
+ "blurb": "The guides section contains compact how-to guides that are focused on achieving a single goal. For an in-depth reference..."
},
{
- "title": "CSV Import",
- "text": "how to load a csv file into a table to read data from a csv file use the read_csv_auto function in the from clause of a query select from read_csv_auto input csv to create a new table using the result from a query use create table as from a select statement create table new_tbl as select from read_csv_auto input csv to load data into an existing table from a query use insert into from a select statement insert into tbl select from read_csv_auto input csv alternatively the copy statement can also be used to load data from a csv file into an existing table copy tbl from input csv for additional options see the csv loading reference and the copy statement documentation",
- "category": "Import",
- "url": "../docs/guides/import/csv_import",
- "blurb": "How to load a CSV file into a table To read data from a CSV file, use the read_csv_auto function in the FROM clause..."
+ "title": "DBeaver SQL IDE",
+ "text": "how to set up dbeaver sql ide for duckdb dbeaver is a powerful and popular desktop sql editor and integrated development environment ide it has both an open source and enterprise version it is useful for visually inspecting the available tables in duckdb and for quickly building complex queries duckdb s jdbc connector allows dbeaver to query duckdb files and by extension any other files that duckdb can access like parquet files install dbeaver using the download links and instructions found at their download page open dbeaver and create a new connection either click on the new database connector button or go to database new database connection in the menu bar img src images guides dbeaver_new_database_connection png alt dbeaver new database connection title dbeaver new database connection img src images guides dbeaver_new_database_connection_menu png alt dbeaver new database connection menu title dbeaver new database connection menu search for duckdb select it and click next img src images guides dbeaver_select_database_driver png alt dbeaver select database driver title dbeaver select database driver enter the path or browse to the duckdb database file you wish to query to use an in-memory duckdb useful primarily if just interested in querying parquet files or for testing enter memory as the path img src images guides dbeaver_connection_settings_path png alt dbeaver set path title dbeaver set path click test connection this will then prompt you to install the duckdb jdbc driver if you are not prompted see alternative driver installation instructions below img src images guides dbeaver_connection_settings_test_connection png alt dbeaver test connection title dbeaver test connection click download to download duckdb s jdbc driver from maven once download is complete click ok then click finish note if you are in a corporate environment or behind a firewall before clicking download click the download configuration link to configure your proxy settings img src images guides dbeaver_download_driver_files png alt dbeaver download driver files title dbeaver download driver files you should now see a database connection to your duckdb database in the left hand database navigator pane expand it to see the tables and views in your database right click on that connection and create a new sql script img src images guides dbeaver_new_sql_script png alt dbeaver new sql script title dbeaver new sql script write some sql and click the execute button img src images guides dbeaver_execute_query png alt dbeaver execute query title dbeaver execute query now you re ready to fly with duckdb and dbeaver img src images guides dbeaver_query_results png alt dbeaver query results title dbeaver query results alternative driver installation if not prompted to install the duckdb driver when testing your connection return to the connect to a database dialog and click edit driver settings img src images guides dbeaver_edit_driver_settings png alt dbeaver edit driver settings title dbeaver edit driver settings alternate you may also access the driver settings menu by returning to the main dbeaver window and clicking database driver manager in the menu bar then select duckdb then click edit img src images guides dbeaver_driver_manager png alt dbeaver driver manager title dbeaver driver manager img src images guides dbeaver_driver_manager_edit png alt dbeaver driver manager edit title dbeaver driver manager edit go to the libraries tab then click on the duckdb driver and click download update if you do not see the duckdb driver first click on reset to defaults img src images guides dbeaver_edit_driver_duckdb png alt dbeaver edit driver title dbeaver edit driver click download to download duckdb s jdbc driver from maven once download is complete click ok then return to the main dbeaver window and continue with step 7 above note if you are in a corporate environment or behind a firewall before clicking download click the download configuration link to configure your proxy settings img src images guides dbeaver_download_driver_files_from_driver_settings png alt dbeaver download driver files 2 title dbeaver download driver files 2",
+ "category": "Sql Editors",
+ "url": "../docs/guides/sql_editors/dbeaver",
+ "blurb": "How to set up DBeaver SQL IDE for DuckDB DBeaver is a powerful and popular desktop sql editor and integrated..."
},
{
- "title": "HTTP Parquet Import",
- "text": "how to load a parquet file directly from http s to load a parquet file over http s the httpfs extension is required this can be installed use the install sql command this only needs to be run once install httpfs to load the httpfs extension for usage use the load sql command load httpfs after the httpfs extension is set up parquet files can be read over http s using the following command select from read_parquet https domain path to file parquet",
- "category": "Import",
- "url": "../docs/guides/import/http_import",
- "blurb": "How to load a Parquet file directly from HTTP(s) To load a Parquet file over http(s) , the HTTPFS extension is..."
+ "title": "Julia Package",
+ "text": "duckdb julia package the duckdb julia package provides a high-performance front-end for duckdb much like sqlite duckdb runs in-process within the julia client and provides a dbinterface front-end the package also supports multi-threaded execution it uses julia threads tasks for this purpose if you wish to run queries in parallel you must launch julia with multi-threading support by e g setting the julia_num_threads environment variable installation pkg add duckdb julia using duckdb basics create a new in-memory database con dbinterface connect duckdb db memory create a table dbinterface execute con create table integers i integer insert data using a prepared statement stmt dbinterface prepare con insert into integers values dbinterface execute stmt 42 query the database results dbinterface execute con select 42 a print results scanning dataframes the duckdb julia package also provides support for querying julia dataframes note that the dataframes are directly read by duckdb - they are not inserted or copied into the database itself if you wish to load data from a dataframe into a duckdb table you can run a create table as or insert into query using duckdb using dataframes create a new in-memory dabase con dbinterface connect duckdb db create a dataframe df dataframe a 1 2 3 b 42 84 42 register it as a view in the database duckdb register_data_frame con df my_df run a sql query over the dataframe results dbinterface execute con select from my_df print results original julia connector credits to kimmolinna for the original duckdb julia connector",
+ "category": "Api",
+ "url": "../docs/api/julia",
+ "blurb": "DuckDB Julia Package The DuckDB Julia package provides a high-performance front-end for DuckDB. Much like SQLite,..."
},
{
- "title": "S3 Parquet Import",
- "text": "how to load a parquet file directly from s3 to load a parquet file from s3 the httpfs extension is required this can be installed use the install sql command this only needs to be run once install httpfs to load the httpfs extension for usage use the load sql command load httpfs after loading the httpfs extension set up the credentials and s3 region to read data firstly the region where the data resides needs to be configured set s3_region us-east-1 with the only the region set public s3 data can be queried to query private s3 data you need to either use an access key and secret set s3_access_key_id aws access key id set s3_secret_access_key aws secret access key or a session token set s3_session_token aws session token after the httpfs extension is set up and the s3 configuration is set correctly parquet files can be read from s3 using the following command select from read_parquet s3 bucket file",
- "category": "Import",
- "url": "../docs/guides/import/s3_import",
- "blurb": "How to load a Parquet file directly from S3 To load a Parquet file from S3, the HTTPFS extension is required. This..."
+ "title": "NodeJS API",
+ "text": "modules typedefs a name module_duckdb a duckdb summary duckdb is an embeddable sql olap database management system duckdb connection run sql params callback code void code all sql params callback code void code arrowipcall sql params callback code void code arrowipcstream sql params callback each sql params callback code void code stream sql params register_udf name return_type fun code void code prepare sql params callback code statement code exec sql params callback code void code register_udf_bulk name return_type callback code void code unregister_udf name return_type callback code void code register_buffer name array force callback code void code unregister_buffer name callback code void code close callback code void code statement sql get run sql params callback code void code all sql params callback code void code arrowipcall sql params callback code void code each sql params callback code void code finalize sql params callback code void code stream sql params columns code array lt columninfo gt code queryresult nextchunk nextipcbuffer asynciterator database close callback code void code close_internal callback code void code wait callback code void code serialize callback code void code parallelize callback code void code connect path code connection code interrupt callback code void code prepare sql code statement code run sql params callback code void code scanarrowipc sql params callback code void code each sql params callback code void code all sql params callback code void code arrowipcall sql params callback code void code arrowipcstream sql params callback code void code exec sql params callback code void code register_udf name return_type fun code this code register_buffer name code this code unregister_buffer name code this code unregister_udf name code this code registerreplacementscan fun code this code get error code number code open_readonly code number code open_readwrite code number code open_create code number code open_fullmutex code number code open_sharedcache code number code open_privatecache code number code a name module_duckdb connection a duckdb connection kind inner class of code duckdb code connection run sql params callback code void code all sql params callback code void code arrowipcall sql params callback code void code arrowipcstream sql params callback each sql params callback code void code stream sql params register_udf name return_type fun code void code prepare sql params callback code statement code exec sql params callback code void code register_udf_bulk name return_type callback code void code unregister_udf name return_type callback code void code register_buffer name array force callback code void code unregister_buffer name callback code void code close callback code void code a name module_duckdb connection run a connection run sql params callback code void code run a sql statement and trigger a callback when done kind instance method of code connection code param type --- --- sql params code code callback a name module_duckdb connection all a connection all sql params callback code void code run a sql query and triggers the callback once for all result rows kind instance method of code connection code param type --- --- sql params code code callback a name module_duckdb connection arrowipcall a connection arrowipcall sql params callback code void code run a sql query and serialize the result into the apache arrow ipc format requires arrow extension to be loaded kind instance method of code connection code param type --- --- sql params code code callback a name module_duckdb connection arrowipcstream a connection arrowipcstream sql params callback run a sql query returns a ipcresultstreamiterator that allows streaming the result into the apache arrow ipc format requires arrow extension to be loaded kind instance method of code connection code returns promise ipcresultstreamiterator param type --- --- sql params code code callback a name module_duckdb connection each a connection each sql params callback code void code runs a sql query and triggers the callback for each result row kind instance method of code connection code param type --- --- sql params code code callback a name module_duckdb connection stream a connection stream sql params kind instance method of code connection code param type --- --- sql params code code a name module_duckdb connection register_udf a connection register _ udf name return_type fun code void code register a user defined function kind instance method of code connection code note this follows the wasm udfs somewhat but is simpler because we can pass data much more cleanly param --- name return_type fun a name module_duckdb connection prepare a connection prepare sql params callback code statement code prepare a sql query for execution kind instance method of code connection code param type --- --- sql params code code callback a name module_duckdb connection exec a connection exec sql params callback code void code execute a sql query kind instance method of code connection code param type --- --- sql params code code callback a name module_duckdb connection register_udf_bulk a connection register _ udf _ bulk name return_type callback code void code register a user defined function kind instance method of code connection code param --- name return_type callback a name module_duckdb connection unregister_udf a connection unregister _ udf name return_type callback code void code unregister a user defined function kind instance method of code connection code param --- name return_type callback a name module_duckdb connection register_buffer a connection register _ buffer name array force callback code void code register a buffer to be scanned using the apache arrow ipc scanner requires arrow extension to be loaded kind instance method of code connection code param --- name array force callback a name module_duckdb connection unregister_buffer a connection unregister _ buffer name callback code void code unregister the buffer kind instance method of code connection code param --- name callback a name module_duckdb connection close a connection close callback code void code closes connection kind instance method of code connection code param --- callback a name module_duckdb statement a duckdb statement kind inner class of code duckdb code statement sql get run sql params callback code void code all sql params callback code void code arrowipcall sql params callback code void code each sql params callback code void code finalize sql params callback code void code stream sql params columns code array lt columninfo gt code a name module_duckdb statement sql a statement sql kind instance property of code statement code returns sql contained in statement field a name module_duckdb statement get a statement get not implemented kind instance method of code statement code a name module_duckdb statement run a statement run sql params callback code void code kind instance method of code statement code param type --- --- sql params code code callback a name module_duckdb statement all a statement all sql params callback code void code kind instance method of code statement code param type --- --- sql params code code callback a name module_duckdb statement arrowipcall a statement arrowipcall sql params callback code void code kind instance method of code statement code param type --- --- sql params code code callback a name module_duckdb statement each a statement each sql params callback code void code kind instance method of code statement code param type --- --- sql params code code callback a name module_duckdb statement finalize a statement finalize sql params callback code void code kind instance method of code statement code param type --- --- sql params code code callback a name module_duckdb statement stream a statement stream sql params kind instance method of code statement code param type --- --- sql params code code a name module_duckdb statement columns a statement columns code array lt columninfo gt code kind instance method of code statement code returns code array lt columninfo gt code - - array of column names and types a name module_duckdb queryresult a duckdb queryresult kind inner class of code duckdb code queryresult nextchunk nextipcbuffer asynciterator a name module_duckdb queryresult nextchunk a queryresult nextchunk kind instance method of code queryresult code returns data chunk a name module_duckdb queryresult nextipcbuffer a queryresult nextipcbuffer function to fetch the next result blob of an arrow ipc stream in a zero-copy way requires arrow extension to be loaded kind instance method of code queryresult code returns data chunk a name module_duckdb queryresult asynciterator a queryresult asynciterator kind instance method of code queryresult code a name module_duckdb database a duckdb database main database interface kind inner property of code duckdb code param description --- --- path path to database file or memory for in-memory database access_mode access mode config the configuration object callback callback function database close callback code void code close_internal callback code void code wait callback code void code serialize callback code void code parallelize callback code void code connect path code connection code interrupt callback code void code prepare sql code statement code run sql params callback code void code scanarrowipc sql params callback code void code each sql params callback code void code all sql params callback code void code arrowipcall sql params callback code void code arrowipcstream sql params callback code void code exec sql params callback code void code register_udf name return_type fun code this code register_buffer name code this code unregister_buffer name code this code unregister_udf name code this code registerreplacementscan fun code this code get a name module_duckdb database close a database close callback code void code closes database instance kind instance method of code database code param --- callback a name module_duckdb database close_internal a database close _ internal callback code void code internal method do not use call connection close instead kind instance method of code database code param --- callback a name module_duckdb database wait a database wait callback code void code triggers callback when all scheduled database tasks have completed kind instance method of code database code param --- callback a name module_duckdb database serialize a database serialize callback code void code currently a no-op provided for sqlite compatibility kind instance method of code database code param --- callback a name module_duckdb database parallelize a database parallelize callback code void code currently a no-op provided for sqlite compatibility kind instance method of code database code param --- callback a name module_duckdb database connect a database connect path code connection code create a new database connection kind instance method of code database code param description --- --- path the database to connect to either a file path or memory a name module_duckdb database interrupt a database interrupt callback code void code supposedly interrupt queries but currently does not do anything kind instance method of code database code param --- callback a name module_duckdb database prepare a database prepare sql code statement code prepare a sql query for execution kind instance method of code database code param --- sql a name module_duckdb database run a database run sql params callback code void code convenience method for connection run using a built-in default connection kind instance method of code database code param type --- --- sql params code code callback a name module_duckdb database scanarrowipc a database scanarrowipc sql params callback code void code convenience method for connection scanarrowipc using a built-in default connection kind instance method of code database code param type --- --- sql params code code callback a name module_duckdb database each a database each sql params callback code void code kind instance method of code database code param type --- --- sql params code code callback a name module_duckdb database all a database all sql params callback code void code convenience method for connection apply using a built-in default connection kind instance method of code database code param type --- --- sql params code code callback a name module_duckdb database arrowipcall a database arrowipcall sql params callback code void code convenience method for connection arrowipcall using a built-in default connection kind instance method of code database code param type --- --- sql params code code callback a name module_duckdb database arrowipcstream a database arrowipcstream sql params callback code void code convenience method for connection arrowipcstream using a built-in default connection kind instance method of code database code param type --- --- sql params code code callback a name module_duckdb database exec a database exec sql params callback code void code kind instance method of code database code param type --- --- sql params code code callback a name module_duckdb database register_udf a database register _ udf name return_type fun code this code register a user defined function convenience method for connection register_udf kind instance method of code database code param --- name return_type fun a name module_duckdb database register_buffer a database register _ buffer name code this code register a buffer containing serialized data to be scanned from duckdb convenience method for connection unregister_buffer kind instance method of code database code param --- name a name module_duckdb database unregister_buffer a database unregister _ buffer name code this code unregister a buffer convenience method for connection unregister_buffer kind instance method of code database code param --- name a name module_duckdb database unregister_udf a database unregister _ udf name code this code unregister a udf convenience method for connection unregister_udf kind instance method of code database code param --- name a name module_duckdb database registerreplacementscan a database registerreplacementscan fun code this code register a table replace scan function kind instance method of code database code param description --- --- fun replacement scan function a name module_duckdb database get a database get not implemented kind instance method of code database code a name module_duckdb error a duckdb error code number code check that errno attribute equals this to check for a duckdb error kind inner constant of code duckdb code a name module_duckdb open_readonly a duckdb open _ readonly code number code open database in readonly mode kind inner constant of code duckdb code a name module_duckdb open_readwrite a duckdb open _ readwrite code number code currently ignored kind inner constant of code duckdb code a name module_duckdb open_create a duckdb open _ create code number code currently ignored kind inner constant of code duckdb code a name module_duckdb open_fullmutex a duckdb open _ fullmutex code number code currently ignored kind inner constant of code duckdb code a name module_duckdb open_sharedcache a duckdb open _ sharedcache code number code currently ignored kind inner constant of code duckdb code a name module_duckdb open_privatecache a duckdb open _ privatecache code number code currently ignored kind inner constant of code duckdb code a name columninfo a columninfo code object code kind global typedef properties name type description --- --- --- name code string code column name type code typeinfo code column type a name typeinfo a typeinfo code object code kind global typedef properties name type description --- --- --- id code string code type id alias code string code sql type alias sql_type code string code sql type name a name duckdberror a duckdberror code object code kind global typedef properties name type description --- --- --- errno code number code -1 for duckdb errors message code string code error message code code string code duckdb_nodejs_error for duckdb errors errortype code string code duckdb error type code eg http io catalog a name httperror a httperror code object code kind global typedef extends code duckdberror code properties name type description --- --- --- statuscode code number code http response status code reason code string code http response reason response code string code http response body headers code object code http headers",
+ "category": "Nodejs",
+ "url": "../docs/api/nodejs/reference",
+ "blurb": "Modules Typedefs duckdb Summary : DuckDB is an embeddable SQL OLAP Database..."
},
{
- "title": "Parquet Import",
- "text": "how to load a parquet file into a table to read data from a parquet file use the read_parquet function in the from clause of a query select from read_parquet input parquet to create a new table using the result from a query use create table as from a select statement create table new_tbl as select from read_parquet input parquet to load data into an existing table from a query use insert into from a select statement insert into tbl select from read_parquet input parquet alternatively the copy statement can also be used to load data from a parquet file into an existing table copy tbl from input parquet format parquet for additional options see the parquet loading reference",
- "category": "Import",
- "url": "../docs/guides/import/parquet_import",
- "blurb": "How to load a Parquet file into a table To read data from a Parquet file, use the read_parquet function in the FROM ..."
- },
- {
- "title": "Parquet Import",
- "text": "how to run a query directly on a parquet file to run a query directly on a parquet file use the read_parquet function in the from clause of a query select from read_parquet input parquet the parquet file will be processed in parallel filters will be automatically pushed down into the parquet scan and only the relevant columns will be read automatically for more information see the blog post querying parquet with precision using duckdb",
- "category": "Import",
- "url": "../docs/guides/import/query_parquet",
- "blurb": "How to run a query directly on a Parquet file To run a query directly on a Parquet file, use the read_parquet ..."
+ "title": "Node.js API",
+ "text": "duckdb node bindings this package provides a node js api for duckdb the sqlite for analytics the api for this client is somewhat compliant to the sqlite node js client for easier transition load the package and create a database object var duckdb require duckdb var db new duckdb database memory or a file name for a persistent db then you can run a query db all select 42 as fortytwo function err res if err throw err console log res 0 fortytwo other available methods are each where the callback is invoked for each row run to execute a single statement without results and exec which can execute several sql commands at once but also does not return results all those commands can work with prepared statements taking the values for the parameters as additional arguments for example like so db all select integer as fortytwo string as hello 42 hello world function err res if err throw err console log res 0 fortytwo console log res 0 hello however these are all shorthands for something much more elegant a database can have multiple connection s those are created using db connect var con db connect you can create multiple connections each with their own transaction context connection objects also contain shorthands to directly call run all and each with parameters and callbacks respectively for example con all select 42 as fortytwo function err res if err throw err console log res 0 fortytwo from connections you can create prepared statements and only that using con prepare var stmt con prepare select integer as fortytwo to execute this statement you can call for example all on the stmt object stmt all 42 function err res if err throw err console log res 0 fortytwo you can also execute the prepared statement multiple times this is for example useful to fill a table with data con run create table a i integer var stmt con prepare insert into a values for var i 0 i 10 i stmt run i stmt finalize con all select from a function err res if err throw err console log res prepare can also take a callback which gets the prepared statement as an argument var stmt con prepare select integer as fortytwo function err stmt stmt all 42 function err res if err throw err console log res 0 fortytwo",
+ "category": "Nodejs",
+ "url": "../docs/api/nodejs/overview",
+ "blurb": "DuckDB Node Bindings This package provides a node.js API for DuckDB , the SQLite for Analytics. The API for this..."
},
{
- "title": "S3 Parquet Export",
- "text": "how to write a parquet file directly to s3 to write a parquet file to s3 the httpfs extension is required this can be installed use the install sql command this only needs to be run once install httpfs to load the httpfs extension for usage use the load sql command load httpfs after loading the httpfs extension set up the credentials and s3 region to write data you may either use an access key and secret or a token set s3_region us-east-1 set s3_access_key_id aws access key id set s3_secret_access_key aws secret access key the alternative is to use a token set s3_region us-east-1 set s3_session_token aws session token after the httpfs extension is set up and the s3 credentials are correctly configured parquet files can be written to s3 using the following command copy table_name to s3 bucket file extension",
- "category": "Import",
- "url": "../docs/guides/import/s3_export",
- "blurb": "How to write a Parquet file directly to S3 To write a Parquet file to S3, the HTTPFS extension is required. This can..."
+ "title": "R API",
+ "text": "installation the duckdb r api can be installed using install packages please see the installation page for details basic api usage the standard duckdb r api implements the dbi interface for r if you are not familiar with dbi yet see here for an introduction startup shutdown to use duckdb you must first create a connection object that represents the database the connection object takes as parameter the database file to read and write from if the database file does not exist it will be created the file extension may be db duckdb or anything else the special value memory the default can be used to create an in-memory database note that for an in-memory database no data is persisted to disk i e all data is lost when you exit the r process if you would like to connect to an existing database in read-only mode set the read_only flag to true read-only mode is required if multiple r processes want to access the same database file at the same time library dbi to start an in-memory database con dbconnect duckdb duckdb dbdir memory to use a database file not shared between processes con dbconnect duckdb duckdb dbdir my-db duckdb read_only false to use a database file shared between processes con dbconnect duckdb duckdb dbdir my-db duckdb read_only true connections are closed implicitly when they go out of scope or if they are explicitly closed using dbdisconnect to shut down the database instance associated with the connection use dbdisconnect con shutdown true querying duckdb supports the standard dbi methods to send queries and retreive result sets dbexecute is meant for queries where no results are expected like create table or update etc and dbgetquery is meant to be used for queries that produce results e g select below an example create a table dbexecute con create table items item varchar value decimal 10 2 count integer insert two items into the table dbexecute con insert into items values jeans 20 0 1 hammer 42 2 2 retrieve the items again res dbgetquery con select from items print res item value count 1 jeans 20 0 1 2 hammer 42 2 2 duckdb also supports prepared statements in the r api with the dbexecute and dbgetquery methods here is an example prepared statement parameters are given as a list dbexecute con insert into items values list laptop 2000 1 if you want to reuse a prepared statement multiple times use dbsendstatement and dbbind stmt dbsendstatement con insert into items values dbbind stmt list iphone 300 2 dbbind stmt list android 3 5 1 dbclearresult stmt query the database using a prepared statement res dbgetquery con select item from items where value list 400 print res item 1 laptop do not use prepared statements to insert large amounts of data into duckdb see below for better options efficient transfer to write a r data frame into duckdb use the standard dbi function dbwritetable this creates a table in duckdb and populates it with the data frame contents for example dbwritetable con iris_table iris res dbgetquery con select from iris_table limit 1 print res sepal length sepal width petal length petal width species 1 5 1 3 5 1 4 0 2 setosa it is also possible to register a r data frame as a virtual table comparable to a sql view this does not actually transfer data into duckdb yet below is an example duckdb duckdb_register con iris_view iris res dbgetquery con select from iris_view limit 1 print res sepal length sepal width petal length petal width species 1 5 1 3 5 1 4 0 2 setosa duckdb keeps a reference to the r data frame after registration this prevents the data frame from being garbage-collected the reference is cleared when the connection is closed but can also be cleared manually using the duckdb duckdb_unregister method also refer to the data import documentation for more options of efficiently importing data dbplyr duckdb also plays well with the dbplyr dplyr packages for programmatic query construction from r here is an example library dbi library dplyr con - dbconnect duckdb duckdb duckdb duckdb_register con flights nycflights13 flights tbl con flights group_by dest summarise delay mean dep_time",
+ "category": "Api",
+ "url": "../docs/api/r",
+ "blurb": "Installation The DuckDB R API can be installed using install.packages . Please see the installation page for..."
},
{
- "title": "CSV Import/Export",
- "text": "copy moves data between duckdb tables and external comma separated value csv files csv import copy from imports data into duckdb from an external csv file into an existing table the data is appended to whatever data is in the table already the amount of columns inside the file must match the amount of columns in the table table_name and the contents of the columns must be convertible to the column types of the table in case this is not possible an error will be thrown if a list of columns is specified copy will only copy the data in the specified columns from the file if there are any columns in the table that are not in the column list copy from will insert the default values for those columns -- copy the contents of a comma-separated file test csv without a header into the table test copy test from test csv -- copy the contents of a comma-separated file with a header into the category table copy category from categories csv header -- copy the contents of lineitem tbl into the lineitem table where the contents are delimited by a pipe character copy lineitem from lineitem tbl delimiter -- read the contents of a comma-separated file names csv into the name column of the category table any other columns of this table are filled with their default value copy category name from names csv syntax csv export copy to exports data from duckdb to an external csv file it has mostly the same set of options as copy from however in the case of copy to the options specify how the csv file should be written to disk any csv file created by copy to can be copied back into the database by using copy from with the same set of options the copy to function can be called specifying either a table name or a query when a table name is specified the contents of the entire table will be written into the resulting csv file when a query is specified the query is executed and the result of the query is written to the resulting file -- copy the contents of the lineitem table to the file lineitem tbl where the columns are delimited by a pipe character including a header line copy lineitem to lineitem tbl delimiter header -- copy the l_orderkey column of the lineitem table to the file orderkey tbl copy lineitem l_orderkey to orderkey tbl delimiter -- copy the result of a query to the file query csv including a header with column names copy select 42 as a hello as b to query csv with header 1 delimiter syntax parameters name description --- --- table_name the name optionally schema-qualified of an existing table column_name an optional list of columns to be copied if no column list is specified all columns of the table will be copied filename the path and name of the input or output file boolean specifies whether the selected option should be turned on or off you can write true on or 1 to enable the option and false off or 0 to disable it the boolean value can also be omitted in which case true is assumed format if this option is used its value must be csv with any other format an error will be thrown delimiter specifies the string that separates columns within each row line of the file the default value is a comma null specifies the string that represents a null value the default is an empty string please note that in copy from both an unquoted empty string and a quoted empty string represent a null value if any other null string is specified again both its quoted and its unquoted appearance represent a null value copy to does not quote null values on output even if force_quote is true header specifies that the file contains a header line with the names of each column in the file thus copy from ignores the first line when importing data whereas on output copy to the first line of the file contains the column names of the exported columns quote specifies the quoting string to be used when a data value is quoted the default is double-quote escape specifies the string that should appear before a data character sequence that matches the quote value the default is the same as the quote value so that the quoting string is doubled if it appears in the data force_quote forces quoting to be used for all non-null values in each specified column null output is never quoted if is specified non-null values will be quoted in all columns this option is allowed only in copy to force_not_null do not match the specified columns values against the null string in the default case where the null string is empty this means that empty values will be read as zero-length strings rather than nulls this option is allowed only in copy from encoding if this option is used its value must be utf8 with any other encoding an error will be thrown notes it is recommended that the file name used in copy always be specified as an absolute path the values in each record are separated by the delimiter string if the value contains the delimiter string the quote string the null string a carriage return or line feed character then the whole value is prefixed and suffixed by the quote string and any occurrence within the value of a quote string or the escape string is preceded by the escape string you can also use force_quote to force quotes when outputting non-null values in specific columns the csv format has no standard way to distinguish a null value from an empty string you can use force_not_null to prevent null input comparisons for specific columns in csv format all characters are significant a quoted value surrounded by white space or any characters other than delimiter will include those characters this can cause errors if you import data from a system that pads csv lines with white space out to some fixed width if such a situation arises you might need to preprocess the csv file to remove the trailing white space before importing the data into duckdb",
- "category": "Docs",
- "url": "../docs/csv_import",
- "blurb": "COPY moves data between DuckDB tables and external Comma Separated Value (CSV) files. CSV Import COPY ... FROM ..."
+ "title": "C API - Configuration",
+ "text": "configuration options can be provided to change different settings of the database system note that many of these settings can be changed later on using pragma statements as well the configuration object should be created filled with values and passed to duckdb_open_ext example duckdb_database db duckdb_config config create the configuration object if duckdb_create_config config duckdberror handle error set some configuration options duckdb_set_config config access_mode read_write or read_only duckdb_set_config config threads 8 duckdb_set_config config max_memory 8gb duckdb_set_config config default_order desc open the database using the configuration if duckdb_open_ext null db config null duckdberror handle error cleanup the configuration object duckdb_destroy_config config run queries cleanup duckdb_close db api reference this will always succeed unless there is a malloc failure syntax the result configuration object returns duckdbsuccess on success or duckdberror on failure duckdb_config_count this returns the total amount of configuration options available for usage with duckdb_get_config_flag this should not be called in a loop as it internally loops over all the options syntax the amount of config options available duckdb_get_config_flag obtains a human-readable name and description of a specific configuration option this can be used to e g display configuration options this will succeed unless index is out of range i e duckdb_config_count the result name or description must not be freed syntax the index of the configuration option between 0 and duckdb_config_count out_name a name of the configuration flag out_description a description of the configuration flag returns duckdbsuccess on success or duckdberror on failure duckdb_set_config sets the specified option for the specified configuration the configuration option is indicated by name to obtain a list of config options see duckdb_get_config_flag in the source code configuration options are defined in config cpp this can fail if either the name is invalid or if the value provided for the option is invalid syntax the configuration object to set the option on name the name of the configuration flag to set option the value to set the configuration flag to returns duckdbsuccess on success or duckdberror on failure duckdb_destroy_config destroys the specified configuration option and de-allocates all memory allocated for the object syntax the configuration object to destroy",
+ "category": "C",
+ "url": "../docs/api/c/config",
+ "blurb": "Configuration options can be provided to change different settings of the database system. Note that many of these ..."
},
{
- "title": "Appender",
- "text": "the c appender can be used to load bulk data into a duckdb database the appender is tied to a connection and will use the transaction context of that connection when appending an appender always appends to a single table in the database file duckdb db connection con db create the table con query create table people id integer name varchar initialize the appender appender appender con people the appendrow function is the easiest way of appending data it uses recursive templates to allow you to put all the values of a single row within one function call as follows appender appendrow 1 mark rows can also be individually constructed using the beginrow endrow and append methods this is done internally by appendrow and hence has the same performance characteristics appender beginrow appender append int32_t 2 appender append string hannes appender endrow any values added to the appender are cached prior to being inserted into the database system for performance reasons that means that while appending the rows might not be immediately visible in the system the cache is automatically flushed when the appender goes out of scope or when appender close is called the cache can also be manually flushed using the appender flush method after either flush or close is called all the data has been written to the database system date time and timestamps while numbers and strings are rather self-explanatory dates times and timestamps require some explanation they can be directly appended using the methods provided by duckdb date duckdb time or duckdb timestamp they can also be appended using the internal duckdb value type however this adds some additional overheads and should be avoided if possible below is a short example con query create table dates d date t time ts timestamp appender appender con dates construct the values using the date time timestamp types - this is the most efficient appender appendrow date fromdate 1992 1 1 time fromtime 1 1 1 0 timestamp fromdatetime date fromdate 1992 1 1 time fromtime 1 1 1 0 construct duckdb value objects appender appendrow value date 1992 1 1 value time 1 1 1 0 value timestamp 1992 1 1 1 1 1 0",
- "category": "Data",
- "url": "../docs/data/appender",
- "blurb": "The C++ Appender can be used to load bulk data into a DuckDB database. The Appender is tied to a connection, and will use..."
+ "title": "C API - Complete API",
+ "text": "api reference open connect syntax path to the database file on disk or nullptr or memory to open an in-memory database out_database the result database object returns duckdbsuccess on success or duckdberror on failure duckdb_open_ext extended version of duckdb_open creates a new database or opens an existing database file stored at the given path syntax path to the database file on disk or nullptr or memory to open an in-memory database out_database the result database object config optional configuration used to start up the database system out_error if set and the function returns duckdberror this will contain the reason why the start-up failed note that the error must be freed using duckdb_free returns duckdbsuccess on success or duckdberror on failure duckdb_close closes the specified database and de-allocates all memory allocated for that database this should be called after you are done with any database allocated through duckdb_open note that failing to call duckdb_close in case of e g a program crash will not cause data corruption still it is recommended to always correctly close a database object after you are done with it syntax the database object to shut down duckdb_connect opens a connection to a database connections are required to query the database and store transactional state associated with the connection the instantiated connection should be closed using duckdb_disconnect syntax the database file to connect to out_connection the result connection object returns duckdbsuccess on success or duckdberror on failure duckdb_interrupt interrupt running query syntax the connection to interruot duckdb_query_progress get progress of the running query syntax the working connection returns -1 if no progress or a percentage of the progress duckdb_disconnect closes the specified connection and de-allocates all memory allocated for that connection syntax the connection to close duckdb_library_version returns the version of the linked duckdb with a version postfix for dev versions usually used for developing c extensions that must return this for a compatibility check syntax duckdb_create_config initializes an empty configuration object that can be used to provide start-up options for the duckdb instance through duckdb_open_ext this will always succeed unless there is a malloc failure syntax the result configuration object returns duckdbsuccess on success or duckdberror on failure duckdb_config_count this returns the total amount of configuration options available for usage with duckdb_get_config_flag this should not be called in a loop as it internally loops over all the options syntax the amount of config options available duckdb_get_config_flag obtains a human-readable name and description of a specific configuration option this can be used to e g display configuration options this will succeed unless index is out of range i e duckdb_config_count the result name or description must not be freed syntax the index of the configuration option between 0 and duckdb_config_count out_name a name of the configuration flag out_description a description of the configuration flag returns duckdbsuccess on success or duckdberror on failure duckdb_set_config sets the specified option for the specified configuration the configuration option is indicated by name to obtain a list of config options see duckdb_get_config_flag in the source code configuration options are defined in config cpp this can fail if either the name is invalid or if the value provided for the option is invalid syntax the configuration object to set the option on name the name of the configuration flag to set option the value to set the configuration flag to returns duckdbsuccess on success or duckdberror on failure duckdb_destroy_config destroys the specified configuration option and de-allocates all memory allocated for the object syntax the configuration object to destroy duckdb_query executes a sql query within a connection and stores the full materialized result in the out_result pointer if the query fails to execute duckdberror is returned and the error message can be retrieved by calling duckdb_result_error note that after running duckdb_query duckdb_destroy_result must be called on the result object even if the query fails otherwise the error stored within the result will not be freed correctly syntax the connection to perform the query in query the sql query to run out_result the query result returns duckdbsuccess on success or duckdberror on failure duckdb_destroy_result closes the result and de-allocates all memory allocated for that connection syntax the result to destroy duckdb_column_name returns the column name of the specified column the result should not need be freed the column names will automatically be destroyed when the result is destroyed returns null if the column is out of range syntax the result object to fetch the column name from col the column index returns the column name of the specified column duckdb_column_type returns the column type of the specified column returns duckdb_type_invalid if the column is out of range syntax the result object to fetch the column type from col the column index returns the column type of the specified column duckdb_column_logical_type returns the logical column type of the specified column the return type of this call should be destroyed with duckdb_destroy_logical_type returns null if the column is out of range syntax the result object to fetch the column type from col the column index returns the logical column type of the specified column duckdb_column_count returns the number of columns present in a the result object syntax the result object returns the number of columns present in the result object duckdb_row_count returns the number of rows present in a the result object syntax the result object returns the number of rows present in the result object duckdb_rows_changed returns the number of rows changed by the query stored in the result this is relevant only for insert update delete queries for other queries the rows_changed will be 0 syntax the result object returns the number of rows changed duckdb_column_data deprecated prefer using duckdb_result_get_chunk instead returns the data of a specific column of a result in columnar format the function returns a dense array which contains the result data the exact type stored in the array depends on the corresponding duckdb_type as provided by duckdb_column_type for the exact type by which the data should be accessed see the comments in the types section or the duckdb_type enum for example for a column of type duckdb_type_integer rows can be accessed in the following manner int32_t data int32_t duckdb_column_data result 0 printf data for row d d n row data row syntax the result object to fetch the column data from col the column index returns the column data of the specified column duckdb_nullmask_data deprecated prefer using duckdb_result_get_chunk instead returns the nullmask of a specific column of a result in columnar format the nullmask indicates for every row whether or not the corresponding row is null if a row is null the values present in the array provided by duckdb_column_data are undefined int32_t data int32_t duckdb_column_data result 0 bool nullmask duckdb_nullmask_data result 0 if nullmask row printf data for row d null n row else printf data for row d d n row data row syntax the result object to fetch the nullmask from col the column index returns the nullmask of the specified column duckdb_result_error returns the error message contained within the result the error is only set if duckdb_query returns duckdberror the result of this function must not be freed it will be cleaned up when duckdb_destroy_result is called syntax the result object to fetch the error from returns the error of the result duckdb_result_get_chunk fetches a data chunk from the duckdb_result this function should be called repeatedly until the result is exhausted the result must be destroyed with duckdb_destroy_data_chunk this function supersedes all duckdb_value functions as well as the duckdb_column_data and duckdb_nullmask_data functions it results in significantly better performance and should be preferred in newer code-bases if this function is used none of the other result functions can be used and vice versa i e this function cannot be mixed with the legacy result functions use duckdb_result_chunk_count to figure out how many chunks there are in the result syntax the result object to fetch the data chunk from chunk_index the chunk index to fetch from returns the resulting data chunk returns null if the chunk index is out of bounds duckdb_result_is_streaming checks if the type of the internal result is streamqueryresult syntax the result object to check returns whether or not the result object is of the type streamqueryresult duckdb_result_chunk_count returns the number of data chunks present in the result syntax the result object returns number of data chunks present in the result duckdb_value_boolean syntax the boolean value at the specified location or false if the value cannot be converted duckdb_value_int8 syntax the int8_t value at the specified location or 0 if the value cannot be converted duckdb_value_int16 syntax the int16_t value at the specified location or 0 if the value cannot be converted duckdb_value_int32 syntax the int32_t value at the specified location or 0 if the value cannot be converted duckdb_value_int64 syntax the int64_t value at the specified location or 0 if the value cannot be converted duckdb_value_hugeint syntax the duckdb_hugeint value at the specified location or 0 if the value cannot be converted duckdb_value_decimal syntax the duckdb_decimal value at the specified location or 0 if the value cannot be converted duckdb_value_uint8 syntax the uint8_t value at the specified location or 0 if the value cannot be converted duckdb_value_uint16 syntax the uint16_t value at the specified location or 0 if the value cannot be converted duckdb_value_uint32 syntax the uint32_t value at the specified location or 0 if the value cannot be converted duckdb_value_uint64 syntax the uint64_t value at the specified location or 0 if the value cannot be converted duckdb_value_float syntax the float value at the specified location or 0 if the value cannot be converted duckdb_value_double syntax the double value at the specified location or 0 if the value cannot be converted duckdb_value_date syntax the duckdb_date value at the specified location or 0 if the value cannot be converted duckdb_value_time syntax the duckdb_time value at the specified location or 0 if the value cannot be converted duckdb_value_timestamp syntax the duckdb_timestamp value at the specified location or 0 if the value cannot be converted duckdb_value_interval syntax the duckdb_interval value at the specified location or 0 if the value cannot be converted duckdb_value_varchar syntax use duckdb_value_string instead this function does not work correctly if the string contains null bytes returns the text value at the specified location as a null-terminated string or nullptr if the value cannot be converted the result must be freed with duckdb_free duckdb_value_varchar_internal syntax use duckdb_value_string_internal instead this function does not work correctly if the string contains null bytes returns the char value at the specified location only works on varchar columns and does not auto-cast if the column is not a varchar column this function will return null the result must not be freed duckdb_value_string_internal syntax use duckdb_value_string_internal instead this function does not work correctly if the string contains null bytes returns the char value at the specified location only works on varchar columns and does not auto-cast if the column is not a varchar column this function will return null the result must not be freed duckdb_value_blob syntax the duckdb_blob value at the specified location returns a blob with blob data set to nullptr if the value cannot be converted the resulting blob data must be freed with duckdb_free duckdb_value_is_null syntax returns true if the value at the specified index is null and false otherwise duckdb_malloc allocate size bytes of memory using the duckdb internal malloc function any memory allocated in this manner should be freed using duckdb_free syntax the number of bytes to allocate returns a pointer to the allocated memory region duckdb_free free a value returned from duckdb_malloc duckdb_value_varchar or duckdb_value_blob syntax the memory region to de-allocate duckdb_vector_size the internal vector size used by duckdb this is the amount of tuples that will fit into a data chunk created by duckdb_create_data_chunk syntax the vector size duckdb_string_is_inlined whether or not the duckdb_string_t value is inlined this means that the data of the string does not have a separate allocation syntax duckdb_from_date decompose a duckdb_date object into year month and date stored as duckdb_date_struct syntax the date object as obtained from a duckdb_type_date column returns the duckdb_date_struct with the decomposed elements duckdb_to_date re-compose a duckdb_date from year month and date duckdb_date_struct syntax the year month and date stored in a duckdb_date_struct returns the duckdb_date element duckdb_from_time decompose a duckdb_time object into hour minute second and microsecond stored as duckdb_time_struct syntax the time object as obtained from a duckdb_type_time column returns the duckdb_time_struct with the decomposed elements duckdb_to_time re-compose a duckdb_time from hour minute second and microsecond duckdb_time_struct syntax the hour minute second and microsecond in a duckdb_time_struct returns the duckdb_time element duckdb_from_timestamp decompose a duckdb_timestamp object into a duckdb_timestamp_struct syntax the ts object as obtained from a duckdb_type_timestamp column returns the duckdb_timestamp_struct with the decomposed elements duckdb_to_timestamp re-compose a duckdb_timestamp from a duckdb_timestamp_struct syntax the de-composed elements in a duckdb_timestamp_struct returns the duckdb_timestamp element duckdb_hugeint_to_double converts a duckdb_hugeint object as obtained from a duckdb_type_hugeint column into a double syntax the hugeint value returns the converted double element duckdb_double_to_hugeint converts a double value to a duckdb_hugeint object if the conversion fails because the double value is too big the result will be 0 syntax the double value returns the converted duckdb_hugeint element duckdb_double_to_decimal converts a double value to a duckdb_decimal object if the conversion fails because the double value is too big or the width scale are invalid the result will be 0 syntax the double value returns the converted duckdb_decimal element duckdb_decimal_to_double converts a duckdb_decimal object as obtained from a duckdb_type_decimal column into a double syntax the decimal value returns the converted double element duckdb_prepare create a prepared statement object from a query note that after calling duckdb_prepare the prepared statement should always be destroyed using duckdb_destroy_prepare even if the prepare fails if the prepare fails duckdb_prepare_error can be called to obtain the reason why the prepare failed syntax the connection object query the sql query to prepare out_prepared_statement the resulting prepared statement object returns duckdbsuccess on success or duckdberror on failure duckdb_destroy_prepare closes the prepared statement and de-allocates all memory allocated for the statement syntax the prepared statement to destroy duckdb_prepare_error returns the error message associated with the given prepared statement if the prepared statement has no error message this returns nullptr instead the error message should not be freed it will be de-allocated when duckdb_destroy_prepare is called syntax the prepared statement to obtain the error from returns the error message or nullptr if there is none duckdb_nparams returns the number of parameters that can be provided to the given prepared statement returns 0 if the query was not successfully prepared syntax the prepared statement to obtain the number of parameters for duckdb_param_type returns the parameter type for the parameter at the given index returns duckdb_type_invalid if the parameter index is out of range or the statement was not successfully prepared syntax the prepared statement param_idx the parameter index returns the parameter type duckdb_clear_bindings clear the params bind to the prepared statement syntax duckdb_bind_value binds a value to the prepared statement at the specified index syntax duckdb_bind_parameter_index retrieve the index of the parameter for the prepared statement identified by name syntax duckdb_bind_boolean binds a bool value to the prepared statement at the specified index syntax duckdb_bind_int8 binds an int8_t value to the prepared statement at the specified index syntax duckdb_bind_int16 binds an int16_t value to the prepared statement at the specified index syntax duckdb_bind_int32 binds an int32_t value to the prepared statement at the specified index syntax duckdb_bind_int64 binds an int64_t value to the prepared statement at the specified index syntax duckdb_bind_hugeint binds a duckdb_hugeint value to the prepared statement at the specified index syntax duckdb_bind_decimal binds a duckdb_decimal value to the prepared statement at the specified index syntax duckdb_bind_uint8 binds an uint8_t value to the prepared statement at the specified index syntax duckdb_bind_uint16 binds an uint16_t value to the prepared statement at the specified index syntax duckdb_bind_uint32 binds an uint32_t value to the prepared statement at the specified index syntax duckdb_bind_uint64 binds an uint64_t value to the prepared statement at the specified index syntax duckdb_bind_float binds a float value to the prepared statement at the specified index syntax duckdb_bind_double binds a double value to the prepared statement at the specified index syntax duckdb_bind_date binds a duckdb_date value to the prepared statement at the specified index syntax duckdb_bind_time binds a duckdb_time value to the prepared statement at the specified index syntax duckdb_bind_timestamp binds a duckdb_timestamp value to the prepared statement at the specified index syntax duckdb_bind_interval binds a duckdb_interval value to the prepared statement at the specified index syntax duckdb_bind_varchar binds a null-terminated varchar value to the prepared statement at the specified index syntax duckdb_bind_varchar_length binds a varchar value to the prepared statement at the specified index syntax duckdb_bind_blob binds a blob value to the prepared statement at the specified index syntax duckdb_bind_null binds a null value to the prepared statement at the specified index syntax duckdb_execute_prepared executes the prepared statement with the given bound parameters and returns a materialized query result this method can be called multiple times for each prepared statement and the parameters can be modified between calls to this function syntax the prepared statement to execute out_result the query result returns duckdbsuccess on success or duckdberror on failure duckdb_execute_prepared_arrow executes the prepared statement with the given bound parameters and returns an arrow query result syntax the prepared statement to execute out_result the query result returns duckdbsuccess on success or duckdberror on failure duckdb_arrow_scan scans the arrow stream and creates a view with the given name syntax the connection on which to execute the scan table_name name of the temporary view to create arrow arrow stream wrapper returns duckdbsuccess on success or duckdberror on failure duckdb_arrow_array_scan scans the arrow array and creates a view with the given name syntax the connection on which to execute the scan table_name name of the temporary view to create arrow_schema arrow schema wrapper arrow_array arrow array wrapper out_stream output array stream that wraps around the passed schema for releasing deleting once done returns duckdbsuccess on success or duckdberror on failure duckdb_extract_statements extract all statements from a query note that after calling duckdb_extract_statements the extracted statements should always be destroyed using duckdb_destroy_extracted even if no statements were extracted if the extract fails duckdb_extract_statements_error can be called to obtain the reason why the extract failed syntax the connection object query the sql query to extract out_extracted_statements the resulting extracted statements object returns the number of extracted statements or 0 on failure duckdb_prepare_extracted_statement prepare an extracted statement note that after calling duckdb_prepare_extracted_statement the prepared statement should always be destroyed using duckdb_destroy_prepare even if the prepare fails if the prepare fails duckdb_prepare_error can be called to obtain the reason why the prepare failed syntax the connection object extracted_statements the extracted statements object index the index of the extracted statement to prepare out_prepared_statement the resulting prepared statement object returns duckdbsuccess on success or duckdberror on failure duckdb_extract_statements_error returns the error message contained within the extracted statements the result of this function must not be freed it will be cleaned up when duckdb_destroy_extracted is called syntax the extracted statements to fetch the error from returns the error of the extracted statements duckdb_destroy_extracted de-allocates all memory allocated for the extracted statements syntax the extracted statements to destroy duckdb_pending_prepared executes the prepared statement with the given bound parameters and returns a pending result the pending result represents an intermediate structure for a query that is not yet fully executed the pending result can be used to incrementally execute a query returning control to the client between tasks note that after calling duckdb_pending_prepared the pending result should always be destroyed using duckdb_destroy_pending even if this function returns duckdberror syntax the prepared statement to execute out_result the pending query result returns duckdbsuccess on success or duckdberror on failure duckdb_pending_prepared_streaming executes the prepared statement with the given bound parameters and returns a pending result this pending result will create a streaming duckdb_result when executed the pending result represents an intermediate structure for a query that is not yet fully executed note that after calling duckdb_pending_prepared_streaming the pending result should always be destroyed using duckdb_destroy_pending even if this function returns duckdberror syntax the prepared statement to execute out_result the pending query result returns duckdbsuccess on success or duckdberror on failure duckdb_destroy_pending closes the pending result and de-allocates all memory allocated for the result syntax the pending result to destroy duckdb_pending_error returns the error message contained within the pending result the result of this function must not be freed it will be cleaned up when duckdb_destroy_pending is called syntax the pending result to fetch the error from returns the error of the pending result duckdb_pending_execute_task executes a single task within the query returning whether or not the query is ready if this returns duckdb_pending_result_ready the duckdb_execute_pending function can be called to obtain the result if this returns duckdb_pending_result_not_ready the duckdb_pending_execute_task function should be called again if this returns duckdb_pending_error an error occurred during execution the error message can be obtained by calling duckdb_pending_error on the pending_result syntax the pending result to execute a task within returns the state of the pending result after the execution duckdb_execute_pending fully execute a pending query result returning the final query result if duckdb_pending_execute_task has been called until duckdb_pending_result_ready was returned this will return fast otherwise all remaining tasks must be executed first syntax the pending result to execute out_result the result object returns duckdbsuccess on success or duckdberror on failure duckdb_pending_execution_is_finished returns whether a duckdb_pending_state is finished executing for example if pending_state is duckdb_pending_result_ready this function will return true syntax the pending state on which to decide whether to finish execution returns boolean indicating pending execution should be considered finished duckdb_destroy_value destroys the value and de-allocates all memory allocated for that type syntax the value to destroy duckdb_create_varchar creates a value from a null-terminated string syntax the null-terminated string returns the value this must be destroyed with duckdb_destroy_value duckdb_create_varchar_length creates a value from a string syntax the text length the length of the text returns the value this must be destroyed with duckdb_destroy_value duckdb_create_int64 creates a value from an int64 syntax the bigint value returns the value this must be destroyed with duckdb_destroy_value duckdb_get_varchar obtains a string representation of the given value the result must be destroyed with duckdb_free syntax the value returns the string value this must be destroyed with duckdb_free duckdb_get_int64 obtains an int64 of the given value syntax the value returns the int64 value or 0 if no conversion is possible duckdb_create_logical_type creates a duckdb_logical_type from a standard primitive type the resulting type should be destroyed with duckdb_destroy_logical_type this should not be used with duckdb_type_decimal syntax the primitive type to create returns the logical type duckdb_create_list_type creates a list type from its child type the resulting type should be destroyed with duckdb_destroy_logical_type syntax the child type of list type to create returns the logical type duckdb_create_map_type creates a map type from its key type and value type the resulting type should be destroyed with duckdb_destroy_logical_type syntax the key type and value type of map type to create returns the logical type duckdb_create_union_type creates a union type from the passed types array the resulting type should be destroyed with duckdb_destroy_logical_type syntax the array of types that the union should consist of type_amount the size of the types array returns the logical type duckdb_create_struct_type creates a struct type from the passed member name and type arrays the resulting type should be destroyed with duckdb_destroy_logical_type syntax the array of types that the struct should consist of member_names the array of names that the struct should consist of member_count the number of members that were specified for both arrays returns the logical type duckdb_create_decimal_type creates a duckdb_logical_type of type decimal with the specified width and scale the resulting type should be destroyed with duckdb_destroy_logical_type syntax the width of the decimal type scale the scale of the decimal type returns the logical type duckdb_get_type_id retrieves the type class of a duckdb_logical_type syntax the logical type object returns the type id duckdb_decimal_width retrieves the width of a decimal type syntax the logical type object returns the width of the decimal type duckdb_decimal_scale retrieves the scale of a decimal type syntax the logical type object returns the scale of the decimal type duckdb_decimal_internal_type retrieves the internal storage type of a decimal type syntax the logical type object returns the internal type of the decimal type duckdb_enum_internal_type retrieves the internal storage type of an enum type syntax the logical type object returns the internal type of the enum type duckdb_enum_dictionary_size retrieves the dictionary size of the enum type syntax the logical type object returns the dictionary size of the enum type duckdb_enum_dictionary_value retrieves the dictionary value at the specified position from the enum the result must be freed with duckdb_free syntax the logical type object index the index in the dictionary returns the string value of the enum type must be freed with duckdb_free duckdb_list_type_child_type retrieves the child type of the given list type the result must be freed with duckdb_destroy_logical_type syntax the logical type object returns the child type of the list type must be destroyed with duckdb_destroy_logical_type duckdb_map_type_key_type retrieves the key type of the given map type the result must be freed with duckdb_destroy_logical_type syntax the logical type object returns the key type of the map type must be destroyed with duckdb_destroy_logical_type duckdb_map_type_value_type retrieves the value type of the given map type the result must be freed with duckdb_destroy_logical_type syntax the logical type object returns the value type of the map type must be destroyed with duckdb_destroy_logical_type duckdb_struct_type_child_count returns the number of children of a struct type syntax the logical type object returns the number of children of a struct type duckdb_struct_type_child_name retrieves the name of the struct child the result must be freed with duckdb_free syntax the logical type object index the child index returns the name of the struct type must be freed with duckdb_free duckdb_struct_type_child_type retrieves the child type of the given struct type at the specified index the result must be freed with duckdb_destroy_logical_type syntax the logical type object index the child index returns the child type of the struct type must be destroyed with duckdb_destroy_logical_type duckdb_union_type_member_count returns the number of members that the union type has syntax the logical type union object returns the number of members of a union type duckdb_union_type_member_name retrieves the name of the union member the result must be freed with duckdb_free syntax the logical type object index the child index returns the name of the union member must be freed with duckdb_free duckdb_union_type_member_type retrieves the child type of the given union member at the specified index the result must be freed with duckdb_destroy_logical_type syntax the logical type object index the child index returns the child type of the union member must be destroyed with duckdb_destroy_logical_type duckdb_destroy_logical_type destroys the logical type and de-allocates all memory allocated for that type syntax the logical type to destroy duckdb_create_data_chunk creates an empty datachunk with the specified set of types syntax an array of types of the data chunk column_count the number of columns returns the data chunk duckdb_destroy_data_chunk destroys the data chunk and de-allocates all memory allocated for that chunk syntax the data chunk to destroy duckdb_data_chunk_reset resets a data chunk clearing the validity masks and setting the cardinality of the data chunk to 0 syntax the data chunk to reset duckdb_data_chunk_get_column_count retrieves the number of columns in a data chunk syntax the data chunk to get the data from returns the number of columns in the data chunk duckdb_data_chunk_get_vector retrieves the vector at the specified column index in the data chunk the pointer to the vector is valid for as long as the chunk is alive it does not need to be destroyed syntax the data chunk to get the data from returns the vector duckdb_data_chunk_get_size retrieves the current number of tuples in a data chunk syntax the data chunk to get the data from returns the number of tuples in the data chunk duckdb_data_chunk_set_size sets the current number of tuples in a data chunk syntax the data chunk to set the size in size the number of tuples in the data chunk duckdb_vector_get_column_type retrieves the column type of the specified vector the result must be destroyed with duckdb_destroy_logical_type syntax the vector get the data from returns the type of the vector duckdb_vector_get_data retrieves the data pointer of the vector the data pointer can be used to read or write values from the vector how to read or write values depends on the type of the vector syntax the vector to get the data from returns the data pointer duckdb_vector_get_validity retrieves the validity mask pointer of the specified vector if all values are valid this function might return null the validity mask is a bitset that signifies null-ness within the data chunk it is a series of uint64_t values where each uint64_t value contains validity for 64 tuples the bit is set to 1 if the value is valid i e not null or 0 if the value is invalid i e null validity of a specific value can be obtained like this idx_t entry_idx row_idx 64 idx_t idx_in_entry row_idx 64 bool is_valid validity_mask entry_idx 1 idx_in_entry alternatively the slower duckdb_validity_row_is_valid function can be used syntax the vector to get the data from returns the pointer to the validity mask or null if no validity mask is present duckdb_vector_ensure_validity_writable ensures the validity mask is writable by allocating it after this function is called duckdb_vector_get_validity will always return non-null this allows null values to be written to the vector regardless of whether a validity mask was present before syntax the vector to alter duckdb_vector_assign_string_element assigns a string element in the vector at the specified location syntax the vector to alter index the row position in the vector to assign the string to str the null-terminated string duckdb_vector_assign_string_element_len assigns a string element in the vector at the specified location syntax the vector to alter index the row position in the vector to assign the string to str the string str_len the length of the string in bytes duckdb_list_vector_get_child retrieves the child vector of a list vector the resulting vector is valid as long as the parent vector is valid syntax the vector returns the child vector duckdb_list_vector_get_size returns the size of the child vector of the list syntax the vector returns the size of the child list duckdb_list_vector_set_size sets the total size of the underlying child-vector of a list vector syntax the list vector size the size of the child list returns the duckdb state returns duckdberror if the vector is nullptr duckdb_list_vector_reserve sets the total capacity of the underlying child-vector of a list syntax the list vector required_capacity the total capacity to reserve return the duckdb state returns duckdberror if the vector is nullptr duckdb_struct_vector_get_child retrieves the child vector of a struct vector the resulting vector is valid as long as the parent vector is valid syntax the vector index the child index returns the child vector duckdb_validity_row_is_valid returns whether or not a row is valid i e not null in the given validity mask syntax the validity mask as obtained through duckdb_vector_get_validity row the row index returns true if the row is valid false otherwise duckdb_validity_set_row_validity in a validity mask sets a specific row to either valid or invalid note that duckdb_vector_ensure_validity_writable should be called before calling duckdb_vector_get_validity to ensure that there is a validity mask to write to syntax the validity mask as obtained through duckdb_vector_get_validity row the row index valid whether or not to set the row to valid or invalid duckdb_validity_set_row_invalid in a validity mask sets a specific row to invalid equivalent to duckdb_validity_set_row_validity with valid set to false syntax the validity mask row the row index duckdb_validity_set_row_valid in a validity mask sets a specific row to valid equivalent to duckdb_validity_set_row_validity with valid set to true syntax the validity mask row the row index duckdb_create_table_function creates a new empty table function the return value should be destroyed with duckdb_destroy_table_function syntax the table function object duckdb_destroy_table_function destroys the given table function object syntax the table function to destroy duckdb_table_function_set_name sets the name of the given table function syntax the table function name the name of the table function duckdb_table_function_add_parameter adds a parameter to the table function syntax the table function type the type of the parameter to add duckdb_table_function_add_named_parameter adds a named parameter to the table function syntax the table function name the name of the parameter type the type of the parameter to add duckdb_table_function_set_extra_info assigns extra information to the table function that can be fetched during binding etc syntax the table function extra_info the extra information destroy the callback that will be called to destroy the bind data if any duckdb_table_function_set_bind sets the bind function of the table function syntax the table function bind the bind function duckdb_table_function_set_init sets the init function of the table function syntax the table function init the init function duckdb_table_function_set_local_init sets the thread-local init function of the table function syntax the table function init the init function duckdb_table_function_set_function sets the main function of the table function syntax the table function function the function duckdb_table_function_supports_projection_pushdown sets whether or not the given table function supports projection pushdown if this is set to true the system will provide a list of all required columns in the init stage through the duckdb_init_get_column_count and duckdb_init_get_column_index functions if this is set to false the default the system will expect all columns to be projected syntax the table function pushdown true if the table function supports projection pushdown false otherwise duckdb_register_table_function register the table function object within the given connection the function requires at least a name a bind function an init function and a main function if the function is incomplete or a function with this name already exists duckdberror is returned syntax the connection to register it in function the function pointer returns whether or not the registration was successful duckdb_bind_get_extra_info retrieves the extra info of the function as set in duckdb_table_function_set_extra_info syntax the info object returns the extra info duckdb_bind_add_result_column adds a result column to the output of the table function syntax the info object name the name of the column type the logical type of the column duckdb_bind_get_parameter_count retrieves the number of regular non-named parameters to the function syntax the info object returns the number of parameters duckdb_bind_get_parameter retrieves the parameter at the given index the result must be destroyed with duckdb_destroy_value syntax the info object index the index of the parameter to get returns the value of the parameter must be destroyed with duckdb_destroy_value duckdb_bind_get_named_parameter retrieves a named parameter with the given name the result must be destroyed with duckdb_destroy_value syntax the info object name the name of the parameter returns the value of the parameter must be destroyed with duckdb_destroy_value duckdb_bind_set_bind_data sets the user-provided bind data in the bind object this object can be retrieved again during execution syntax the info object extra_data the bind data object destroy the callback that will be called to destroy the bind data if any duckdb_bind_set_cardinality sets the cardinality estimate for the table function used for optimization syntax the bind data object is_exact whether or not the cardinality estimate is exact or an approximation duckdb_bind_set_error report that an error has occurred while calling bind syntax the info object error the error message duckdb_init_get_extra_info retrieves the extra info of the function as set in duckdb_table_function_set_extra_info syntax the info object returns the extra info duckdb_init_get_bind_data gets the bind data set by duckdb_bind_set_bind_data during the bind note that the bind data should be considered as read-only for tracking state use the init data instead syntax the info object returns the bind data object duckdb_init_set_init_data sets the user-provided init data in the init object this object can be retrieved again during execution syntax the info object extra_data the init data object destroy the callback that will be called to destroy the init data if any duckdb_init_get_column_count returns the number of projected columns this function must be used if projection pushdown is enabled to figure out which columns to emit syntax the info object returns the number of projected columns duckdb_init_get_column_index returns the column index of the projected column at the specified position this function must be used if projection pushdown is enabled to figure out which columns to emit syntax the info object column_index the index at which to get the projected column index from 0 duckdb_init_get_column_count info returns the column index of the projected column duckdb_init_set_max_threads sets how many threads can process this table function in parallel default 1 syntax the info object max_threads the maximum amount of threads that can process this table function duckdb_init_set_error report that an error has occurred while calling init syntax the info object error the error message duckdb_function_get_extra_info retrieves the extra info of the function as set in duckdb_table_function_set_extra_info syntax the info object returns the extra info duckdb_function_get_bind_data gets the bind data set by duckdb_bind_set_bind_data during the bind note that the bind data should be considered as read-only for tracking state use the init data instead syntax the info object returns the bind data object duckdb_function_get_init_data gets the init data set by duckdb_init_set_init_data during the init syntax the info object returns the init data object duckdb_function_get_local_init_data gets the thread-local init data set by duckdb_init_set_init_data during the local_init syntax the info object returns the init data object duckdb_function_set_error report that an error has occurred while executing the function syntax the info object error the error message duckdb_add_replacement_scan add a replacement scan definition to the specified database syntax the database object to add the replacement scan to replacement the replacement scan callback extra_data extra data that is passed back into the specified callback delete_callback the delete callback to call on the extra data if any duckdb_replacement_scan_set_function_name sets the replacement function name to use if this function is called in the replacement callback the replacement scan is performed if it is not called the replacement callback is not performed syntax the info object function_name the function name to substitute duckdb_replacement_scan_add_parameter adds a parameter to the replacement scan function syntax the info object parameter the parameter to add duckdb_replacement_scan_set_error report that an error has occurred while executing the replacement scan syntax the info object error the error message duckdb_appender_create creates an appender object syntax the connection context to create the appender in schema the schema of the table to append to or nullptr for the default schema table the table name to append to out_appender the resulting appender object returns duckdbsuccess on success or duckdberror on failure duckdb_appender_error returns the error message associated with the given appender if the appender has no error message this returns nullptr instead the error message should not be freed it will be de-allocated when duckdb_appender_destroy is called syntax the appender to get the error from returns the error message or nullptr if there is none duckdb_appender_flush flush the appender to the table forcing the cache of the appender to be cleared and the data to be appended to the base table this should generally not be used unless you know what you are doing instead call duckdb_appender_destroy when you are done with the appender syntax the appender to flush returns duckdbsuccess on success or duckdberror on failure duckdb_appender_close close the appender flushing all intermediate state in the appender to the table and closing it for further appends this is generally not necessary call duckdb_appender_destroy instead syntax the appender to flush and close returns duckdbsuccess on success or duckdberror on failure duckdb_appender_destroy close the appender and destroy it flushing all intermediate state in the appender to the table and de-allocating all memory associated with the appender syntax the appender to flush close and destroy returns duckdbsuccess on success or duckdberror on failure duckdb_appender_begin_row a nop function provided for backwards compatibility reasons does nothing only duckdb_appender_end_row is required syntax duckdb_appender_end_row finish the current row of appends after end_row is called the next row can be appended syntax the appender returns duckdbsuccess on success or duckdberror on failure duckdb_append_bool append a bool value to the appender syntax duckdb_append_int8 append an int8_t value to the appender syntax duckdb_append_int16 append an int16_t value to the appender syntax duckdb_append_int32 append an int32_t value to the appender syntax duckdb_append_int64 append an int64_t value to the appender syntax duckdb_append_hugeint append a duckdb_hugeint value to the appender syntax duckdb_append_uint8 append a uint8_t value to the appender syntax duckdb_append_uint16 append a uint16_t value to the appender syntax duckdb_append_uint32 append a uint32_t value to the appender syntax duckdb_append_uint64 append a uint64_t value to the appender syntax duckdb_append_float append a float value to the appender syntax duckdb_append_double append a double value to the appender syntax duckdb_append_date append a duckdb_date value to the appender syntax duckdb_append_time append a duckdb_time value to the appender syntax duckdb_append_timestamp append a duckdb_timestamp value to the appender syntax duckdb_append_interval append a duckdb_interval value to the appender syntax duckdb_append_varchar append a varchar value to the appender syntax duckdb_append_varchar_length append a varchar value to the appender syntax duckdb_append_blob append a blob value to the appender syntax duckdb_append_null append a null value to the appender of any type syntax duckdb_append_data_chunk appends a pre-filled data chunk to the specified appender the types of the data chunk must exactly match the types of the table no casting is performed if the types do not match or the appender is in an invalid state duckdberror is returned if the append is successful duckdbsuccess is returned syntax the appender to append to chunk the data chunk to append returns the return state duckdb_query_arrow executes a sql query within a connection and stores the full materialized result in an arrow structure if the query fails to execute duckdberror is returned and the error message can be retrieved by calling duckdb_query_arrow_error note that after running duckdb_query_arrow duckdb_destroy_arrow must be called on the result object even if the query fails otherwise the error stored within the result will not be freed correctly syntax the connection to perform the query in query the sql query to run out_result the query result returns duckdbsuccess on success or duckdberror on failure duckdb_query_arrow_schema fetch the internal arrow schema from the arrow result syntax the result to fetch the schema from out_schema the output schema returns duckdbsuccess on success or duckdberror on failure duckdb_prepared_arrow_schema fetch the internal arrow schema from the prepared statement syntax the prepared statement to fetch the schema from out_schema the output schema returns duckdbsuccess on success or duckdberror on failure duckdb_query_arrow_array fetch an internal arrow array from the arrow result this function can be called multiple time to get next chunks which will free the previous out_array so consume the out_array before calling this function again syntax the result to fetch the array from out_array the output array returns duckdbsuccess on success or duckdberror on failure duckdb_arrow_column_count returns the number of columns present in a the arrow result object syntax the result object returns the number of columns present in the result object duckdb_arrow_row_count returns the number of rows present in a the arrow result object syntax the result object returns the number of rows present in the result object duckdb_arrow_rows_changed returns the number of rows changed by the query stored in the arrow result this is relevant only for insert update delete queries for other queries the rows_changed will be 0 syntax the result object returns the number of rows changed duckdb_query_arrow_error returns the error message contained within the result the error is only set if duckdb_query_arrow returns duckdberror the error message should not be freed it will be de-allocated when duckdb_destroy_arrow is called syntax the result object to fetch the nullmask from returns the error of the result duckdb_destroy_arrow closes the result and de-allocates all memory allocated for the arrow result syntax the result to destroy duckdb_execute_tasks execute duckdb tasks on this thread will return after max_tasks have been executed or if there are no more tasks present syntax the database object to execute tasks for max_tasks the maximum amount of tasks to execute duckdb_create_task_state creates a task state that can be used with duckdb_execute_tasks_state to execute tasks until duckdb_finish_execution is called on the state duckdb_destroy_state should be called on the result in order to free memory syntax the database object to create the task state for returns the task state that can be used with duckdb_execute_tasks_state duckdb_execute_tasks_state execute duckdb tasks on this thread the thread will keep on executing tasks forever until duckdb_finish_execution is called on the state multiple threads can share the same duckdb_task_state syntax the task state of the executor duckdb_execute_n_tasks_state execute duckdb tasks on this thread the thread will keep on executing tasks until either duckdb_finish_execution is called on the state max_tasks tasks have been executed or there are no more tasks to be executed multiple threads can share the same duckdb_task_state syntax the task state of the executor max_tasks the maximum amount of tasks to execute returns the amount of tasks that have actually been executed duckdb_finish_execution finish execution on a specific task syntax the task state to finish execution duckdb_task_state_is_finished check if the provided duckdb_task_state has finished execution syntax the task state to inspect returns whether or not duckdb_finish_execution has been called on the task state duckdb_destroy_task_state destroys the task state returned from duckdb_create_task_state note that this should not be called while there is an active duckdb_execute_tasks_state running on the task state syntax the task state to clean up duckdb_execution_is_finished returns true if execution of the current query is finished syntax the connection on which to check duckdb_stream_fetch_chunk fetches a data chunk from the streaming duckdb_result this function should be called repeatedly until the result is exhausted the result must be destroyed with duckdb_destroy_data_chunk this function can only be used on duckdb_results created with duckdb_pending_prepared_streaming if this function is used none of the other result functions can be used and vice versa i e this function cannot be mixed with the legacy result functions or the materialized result functions it is not known beforehand how many chunks will be returned by this result syntax the result object to fetch the data chunk from returns the resulting data chunk returns null if the result has an error",
+ "category": "C",
+ "url": "../docs/api/c/api",
+ "blurb": "API Reference Open/Connect Syntax Path to the database file on disk, or nullptr or :memory: to open an..."
},
{
- "title": "Parquet",
- "text": "parquet files are compressed columnar files that are efficient to load and process duckdb provides support for both reading and writing parquet files in an efficient manner as well as support for pushing filters and projections into the parquet file scans examples -- read a single parquet file select from test parquet -- figure out which columns types are in a parquet file describe select from test parquet -- create a table from a parquet file create table test as select from test parquet -- if the file does not end in parquet use the read_parquet function select from read_parquet test parq -- use list parameter to read 3 parquet files and treat them as a single table select from read_parquet file1 parquet file2 parquet file3 parquet -- read all files that match the glob pattern select from test parquet -- read all files that match the glob pattern and include a filename column that specifies which file each row came from select from read_parquet test parquet filename true -- use a list of globs to read all parquet files from 2 specific folders select from read_parquet folder1 parquet folder2 parquet -- query the metadata of a parquet file select from parquet_metadata test parquet -- query the schema of a parquet file select from parquet_schema test parquet -- write the results of a query to a parquet file copy select from tbl to result-snappy parquet format parquet -- export the table contents of the entire database as parquet export database target_directory format parquet single-file reads duckdb includes an efficient parquet reader in the form of the read_parquet function select from read_parquet test parquet if your file ends in parquet the read_parquet syntax is optional the system will automatically infer that you are reading a parquet file select from test parquet unlike csv files parquet files are structured and as such are unambiguous to read no parameters need to be passed to this function the read_parquet function will figure out the column names and column types present in the file and emit them multi-file reads and globs duckdb can also read a series of parquet files and treat them as if they were a single table note that this only works if the parquet files have the same schema you can specify which parquet files you want to read using a list parameter glob pattern matching syntax or a combination of both list parameter the read_parquet function can accept a list of filenames as the input parameter see the nested types documentation for more details on lists -- read 3 parquet files and treat them as a single table select from read_parquet file1 parquet file2 parquet file3 parquet glob syntax any file name input to the read_parquet function can either be an exact filename or use a glob syntax to read multple files that match a pattern wildcard description ------------ ----------------------------------------------------------- matches any number of any characters including none matches any single character abc matches one character given in the bracket a-z matches one character from the range given in the bracket here is an example that reads all the files that end with parquet located in the test folder -- read all files that match the glob pattern select from read_parquet test parquet list of globs the glob syntax and the list input parameter can be combined to scan files that meet one of multiple patterns -- read all parquet files from 2 specific folders select from read_parquet folder1 parquet folder2 parquet filenames the filename parameter can be passed into the read_parquet function to include an extra filename column that specifies for each row from which file it was read for example select from parquet_scan data parquet-testing glob filename true order by i i j filename 1 a data parquet-testing glob t1 parquet 2 b data parquet-testing glob t2 parquet partial reading duckdb supports projection pushdown into the parquet file itself that is to say when querying a parquet file only the columns required for the query are read this allows you to read only the part of the parquet file that you are interested in this will be done automatically by the system duckdb also supports filter pushdown into the parquet reader when you apply a filter to a column that is scanned from a parquet file the filter will be pushed down into the scan and can even be used to skip parts of the file using the built-in zonemaps note that this will depend on whether or not your parquet file contains zonemaps filter and projection pushdown provide significant performance benefits see our blog post on this for more information inserts and views you can also insert the data into a table or create a table from the parquet file directly this will load the data from the parquet file and insert it into the database -- insert the data from the parquet file in the table insert into people select from read_parquet test parquet -- create a table directly from a parquet file create table people as select from read_parquet test parquet if you wish to keep the data stored inside the parquet file but want to query the parquet file directly you can create a view over the read_parquet function you can then query the parquet file as if it were a built-in table -- create a view over the parquet file create view people as select from read_parquet test parquet -- query the parquet file select from people parquet metadata the parquet_metadata function can be used to query the metadata contained within a parquet file which reveals various internal details of the parquet file such as the statistics of the different columns this can be useful for figuring out what kind of skipping is possible in parquet files or even to obtain a quick overview of what the different columns contain select from parquet_metadata test parquet below is a table of the columns returned by parquet_metadata field type ------------------------- --------- file_name varchar row_group_id bigint row_group_num_rows bigint row_group_num_columns bigint row_group_bytes bigint column_id bigint file_offset bigint num_values bigint path_in_schema varchar type varchar stats_min varchar stats_max varchar stats_null_count bigint stats_distinct_count bigint stats_min_value varchar stats_max_value varchar compression varchar encodings varchar index_page_offset bigint dictionary_page_offset bigint data_page_offset bigint total_compressed_size bigint total_uncompressed_size bigint parquet schema the parquet_schema function can be used to query the internal schema contained within a parquet file note that this is the schema as it is contained within the metadata of the parquet file if you want to figure out the column names and types contained within a parquet file it is easier to use describe -- fetch the column names and column types describe select from test parquet -- fetch the internal schema of a parquet file select from parquet_schema test parquet below is a table of the columns returned by parquet_schema field type ----------------- --------- file_name varchar name varchar type varchar type_length varchar repetition_type varchar num_children bigint converted_type varchar scale bigint precision bigint field_id bigint logical_type varchar writing to parquet files duckdb also has support for writing to parquet files using the copy statement syntax you can specify which compression format should be used using the codec parameter options uncompressed snappy default zstd gzip -- write a query to a snappy compressed parquet file copy select from tbl to result-snappy parquet format parquet -- write tbl to a zstd compressed parquet file copy tbl to result-zstd parquet format parquet codec zstd -- write a csv file to an uncompressed parquet file copy test csv to result-uncompressed parquet format parquet codec uncompressed duckdb s export command can be used to export an entire database to a series of parquet files see the export statement documentation for more details -- export the table contents of the entire database as parquet export database target_directory format parquet",
- "category": "Data",
- "url": "../docs/data/parquet",
- "blurb": "Parquet files are compressed columnar files that are efficient to load and process. DuckDB provides support for both..."
+ "title": "C API - Appender",
+ "text": "appenders are the most efficient way of loading data into duckdb from within the c interface and are recommended for fast data loading the appender is much faster than using prepared statements or individual insert into statements appends are made in row-wise format for every column a duckdb_append_ type call should be made after which the row should be finished by calling duckdb_appender_end_row after all rows have been appended duckdb_appender_destroy should be used to finalize the appender and clean up the resulting memory note that duckdb_appender_destroy should always be called on the resulting appender even if the function returns duckdberror example duckdb_query con create table people id integer name varchar null duckdb_appender appender if duckdb_appender_create con null people appender duckdberror handle error append the first row 1 mark duckdb_append_int32 appender 1 duckdb_append_varchar appender mark duckdb_appender_end_row appender append the second row 2 hannes duckdb_append_int32 appender 2 duckdb_append_varchar appender hannes duckdb_appender_end_row appender finish appending and flush all the rows to the table duckdb_appender_destroy appender api reference syntax the connection context to create the appender in schema the schema of the table to append to or nullptr for the default schema table the table name to append to out_appender the resulting appender object returns duckdbsuccess on success or duckdberror on failure duckdb_appender_error returns the error message associated with the given appender if the appender has no error message this returns nullptr instead the error message should not be freed it will be de-allocated when duckdb_appender_destroy is called syntax the appender to get the error from returns the error message or nullptr if there is none duckdb_appender_flush flush the appender to the table forcing the cache of the appender to be cleared and the data to be appended to the base table this should generally not be used unless you know what you are doing instead call duckdb_appender_destroy when you are done with the appender syntax the appender to flush returns duckdbsuccess on success or duckdberror on failure duckdb_appender_close close the appender flushing all intermediate state in the appender to the table and closing it for further appends this is generally not necessary call duckdb_appender_destroy instead syntax the appender to flush and close returns duckdbsuccess on success or duckdberror on failure duckdb_appender_destroy close the appender and destroy it flushing all intermediate state in the appender to the table and de-allocating all memory associated with the appender syntax the appender to flush close and destroy returns duckdbsuccess on success or duckdberror on failure duckdb_appender_begin_row a nop function provided for backwards compatibility reasons does nothing only duckdb_appender_end_row is required syntax duckdb_appender_end_row finish the current row of appends after end_row is called the next row can be appended syntax the appender returns duckdbsuccess on success or duckdberror on failure duckdb_append_bool append a bool value to the appender syntax duckdb_append_int8 append an int8_t value to the appender syntax duckdb_append_int16 append an int16_t value to the appender syntax duckdb_append_int32 append an int32_t value to the appender syntax duckdb_append_int64 append an int64_t value to the appender syntax duckdb_append_hugeint append a duckdb_hugeint value to the appender syntax duckdb_append_uint8 append a uint8_t value to the appender syntax duckdb_append_uint16 append a uint16_t value to the appender syntax duckdb_append_uint32 append a uint32_t value to the appender syntax duckdb_append_uint64 append a uint64_t value to the appender syntax duckdb_append_float append a float value to the appender syntax duckdb_append_double append a double value to the appender syntax duckdb_append_date append a duckdb_date value to the appender syntax duckdb_append_time append a duckdb_time value to the appender syntax duckdb_append_timestamp append a duckdb_timestamp value to the appender syntax duckdb_append_interval append a duckdb_interval value to the appender syntax duckdb_append_varchar append a varchar value to the appender syntax duckdb_append_varchar_length append a varchar value to the appender syntax duckdb_append_blob append a blob value to the appender syntax duckdb_append_null append a null value to the appender of any type syntax duckdb_append_data_chunk appends a pre-filled data chunk to the specified appender the types of the data chunk must exactly match the types of the table no casting is performed if the types do not match or the appender is in an invalid state duckdberror is returned if the append is successful duckdbsuccess is returned syntax the appender to append to chunk the data chunk to append returns the return state",
+ "category": "C",
+ "url": "../docs/api/c/appender",
+ "blurb": "Appenders are the most efficient way of loading data into DuckDB from within the C interface, and are recommended for ..."
},
{
- "title": "CSV Loading",
- "text": "csv loading is a very common and yet surprisingly tricky task while csvs seem simple on the surface there are a lot of inconsistencies found within csv files that can make loading them a challenge csv files exist with different delimiters they can contain quoted values have an optional header row or even multiple or even be completely deformed the csv reader needs to cope with all of these different situations the duckdb csv reader can automatically infer which configuration flags to use by analyzing the csv file this will work correctly in most situations and should be the first option attempted in rare situations where the csv reader cannot figure out the correct configuration it is possible to manually configure the csv reader to correctly parse the csv file we use the following csv file in our examples keep in mind that you can also use compressed csv files in the following examples e g a gzipped file such as test csv gz will work just fine test csv flightdate uniquecarrier origincityname destcityname 1988-01-01 aa new york ny los angeles ca 1988-01-02 aa new york ny los angeles ca 1988-01-03 aa new york ny los angeles ca examples -- read a csv file from disk auto-infer options select from test csv -- read_csv with custom options select from read_csv_auto test csv delim header true columns flightdate date uniquecarrier varchar origincityname varchar destcityname varchar -- read a csv file into a table create table ontime flightdate date uniquecarrier varchar origincityname varchar destcityname varchar copy ontime from test csv auto_detect true -- alternatively create a table without specifying the schema manually create table ontime as select from test csv -- write the result of a query to a csv file copy select from ontime to test csv with header 1 delimiter parameters name description --- --- auto_detect option for csv parsing if true the parser will attempt to detect the input format and data types automatically delim sep quote escape and header parameters become optional read_csv_auto defaults to true for this parameter read_csv defaults to false columns a struct that specifies the column names and column types contained within the csv file e g col1 integer col2 varchar if auto_detect is enabled these will be inferred sep or delim specifies the string that separates columns within each row line of the file the default value is a comma nullstr specifies the string that represents a null value the default is an empty string header specifies that the file contains a header line with the names of each column in the file quote specifies the quoting string to be used when a data value is quoted the default is double-quote escape specifies the string that should appear before a data character sequence that matches the quote value the default is the same as the quote value so that the quoting string is doubled if it appears in the data dateformat specifies the date format to use when parsing dates see date format timestampformat specifies the date format to use when parsing timestamps see date format sample_size option to define number of sample rows for automatic csv type detection chunks of sample rows will be drawn from different locations of the input file set to -1 to scan the entire input file note only the first max 1024 rows will be used for dialect detection all_varchar option to skip type detection for csv parsing and assume all columns to be of type varchar normalize_names boolean value that specifies whether or not column names should be normalized removing any non-alphanumeric characters from them compression the compression type for the file by default this will be detected automatically from the file extension e g t csv gz will use gzip t csv will use none options are none gzip zstd filename boolean value that specifies whether or not an extra filename column should be included in the result skip the number of lines at the top of the file to skip read_csv_auto function the read_csv_auto is the simplest method of loading csv files it automatically attempts to figure out the correct configuration of the csv reader it also automatically deduces types of columns if the csv file has a header it will use the names found in that header to name the columns otherwise the columns will be named column0 column1 column2 select from read_csv_auto test csv flightdate uniquecarrier origincityname destcityname --------- ------------ ---------------- -------------- 1988-01-01 aa new york ny los angeles ca 1988-01-02 aa new york ny los angeles ca 1988-01-03 aa new york ny los angeles ca the path can either be a relative path relative to the current working directory or an absolute path we can use read_csv_auto to create a persistent table as well create table ontime as select from read_csv_auto test csv describe ontime field type null key default extra ------------- ------ --- --- ------ ---- flightdate date yes null null null uniquecarrier varchar yes null null null origincityname varchar yes null null null destcityname varchar yes null null null select from read_csv_auto test csv sample_size 20000 if we set delim sep quote escape or header explicitly we can bypass the automatic detection of this particular parameter select from read_csv_auto test csv header true note read_csv_auto is an alias for read_csv auto_detect true copy statement the copy statement can be used to load data from a csv file into a table this statement has the same syntax as the copy statement supported by postgresql for the copy statement we must first create a table with the correct schema to load the data into we then specify the csv file to load from plus any configuration options separately create table ontime flightdate date uniquecarrier varchar origincityname varchar destcityname varchar copy ontime from test csv delimiter header select from ontime flightdate uniquecarrier origincityname destcityname --------- ------------ ---------------- -------------- 1988-01-01 aa new york ny los angeles ca 1988-01-02 aa new york ny los angeles ca 1988-01-03 aa new york ny los angeles ca if we want to use the automatic format detection we can set auto_detect to true and omit the otherwise required configuration options create table ontime flightdate date uniquecarrier varchar origincityname varchar destcityname varchar copy ontime from test csv auto_detect true select from ontime more on the copy statement can be found here",
- "category": "Data",
- "url": "../docs/data/csv",
- "blurb": "CSV loading is a very common, and yet surprisingly tricky, task. While CSVs seem simple on the surface, there are a lot..."
+ "title": "C API - Data Chunks",
+ "text": "data chunks represent a horizontal slice of a table they hold a number of vectors that can each hold up to the vector_size rows the vector size can be obtained through the duckdb_vector_size function and is configurable but is usually set to 1024 data chunks and vectors are what duckdb uses natively to store and represent data for this reason the data chunk interface is the most efficient way of interfacing with duckdb be aware however that correctly interfacing with duckdb using the data chunk api does require knowledge of duckdb s internal vector format the primary manner of interfacing with data chunks is by obtaining the internal vectors of the data chunk using the duckdb_data_chunk_get_vector method and subsequently using the duckdb_vector_get_data and duckdb_vector_get_validity methods to read the internal data and the validity mask of the vector for composite types list and struct vectors duckdb_list_vector_get_child and duckdb_struct_vector_get_child should be used to read child vectors api reference syntax an array of types of the data chunk column_count the number of columns returns the data chunk duckdb_destroy_data_chunk destroys the data chunk and de-allocates all memory allocated for that chunk syntax the data chunk to destroy duckdb_data_chunk_reset resets a data chunk clearing the validity masks and setting the cardinality of the data chunk to 0 syntax the data chunk to reset duckdb_data_chunk_get_column_count retrieves the number of columns in a data chunk syntax the data chunk to get the data from returns the number of columns in the data chunk duckdb_data_chunk_get_vector retrieves the vector at the specified column index in the data chunk the pointer to the vector is valid for as long as the chunk is alive it does not need to be destroyed syntax the data chunk to get the data from returns the vector duckdb_data_chunk_get_size retrieves the current number of tuples in a data chunk syntax the data chunk to get the data from returns the number of tuples in the data chunk duckdb_data_chunk_set_size sets the current number of tuples in a data chunk syntax the data chunk to set the size in size the number of tuples in the data chunk duckdb_vector_get_column_type retrieves the column type of the specified vector the result must be destroyed with duckdb_destroy_logical_type syntax the vector get the data from returns the type of the vector duckdb_vector_get_data retrieves the data pointer of the vector the data pointer can be used to read or write values from the vector how to read or write values depends on the type of the vector syntax the vector to get the data from returns the data pointer duckdb_vector_get_validity retrieves the validity mask pointer of the specified vector if all values are valid this function might return null the validity mask is a bitset that signifies null-ness within the data chunk it is a series of uint64_t values where each uint64_t value contains validity for 64 tuples the bit is set to 1 if the value is valid i e not null or 0 if the value is invalid i e null validity of a specific value can be obtained like this idx_t entry_idx row_idx 64 idx_t idx_in_entry row_idx 64 bool is_valid validity_mask entry_idx 1 idx_in_entry alternatively the slower duckdb_validity_row_is_valid function can be used syntax the vector to get the data from returns the pointer to the validity mask or null if no validity mask is present duckdb_vector_ensure_validity_writable ensures the validity mask is writable by allocating it after this function is called duckdb_vector_get_validity will always return non-null this allows null values to be written to the vector regardless of whether a validity mask was present before syntax the vector to alter duckdb_vector_assign_string_element assigns a string element in the vector at the specified location syntax the vector to alter index the row position in the vector to assign the string to str the null-terminated string duckdb_vector_assign_string_element_len assigns a string element in the vector at the specified location syntax the vector to alter index the row position in the vector to assign the string to str the string str_len the length of the string in bytes duckdb_list_vector_get_child retrieves the child vector of a list vector the resulting vector is valid as long as the parent vector is valid syntax the vector returns the child vector duckdb_list_vector_get_size returns the size of the child vector of the list syntax the vector returns the size of the child list duckdb_list_vector_set_size sets the total size of the underlying child-vector of a list vector syntax the list vector size the size of the child list returns the duckdb state returns duckdberror if the vector is nullptr duckdb_list_vector_reserve sets the total capacity of the underlying child-vector of a list syntax the list vector required_capacity the total capacity to reserve return the duckdb state returns duckdberror if the vector is nullptr duckdb_struct_vector_get_child retrieves the child vector of a struct vector the resulting vector is valid as long as the parent vector is valid syntax the vector index the child index returns the child vector duckdb_validity_row_is_valid returns whether or not a row is valid i e not null in the given validity mask syntax the validity mask as obtained through duckdb_vector_get_validity row the row index returns true if the row is valid false otherwise duckdb_validity_set_row_validity in a validity mask sets a specific row to either valid or invalid note that duckdb_vector_ensure_validity_writable should be called before calling duckdb_vector_get_validity to ensure that there is a validity mask to write to syntax the validity mask as obtained through duckdb_vector_get_validity row the row index valid whether or not to set the row to valid or invalid duckdb_validity_set_row_invalid in a validity mask sets a specific row to invalid equivalent to duckdb_validity_set_row_validity with valid set to false syntax the validity mask row the row index duckdb_validity_set_row_valid in a validity mask sets a specific row to valid equivalent to duckdb_validity_set_row_validity with valid set to true syntax the validity mask row the row index",
+ "category": "C",
+ "url": "../docs/api/c/data_chunk",
+ "blurb": "Data chunks represent a horizontal slice of a table. They hold a number of vectors, that can each hold up to the ..."
},
{
- "title": "Importing Data",
- "text": "the first step to using a database system is to insert data into that system duckdb provides several data ingestion methods that allow you to easily and efficiently fill up the database in this section we provide an overview of these methods so you can select which one is correct for you insert statements insert statements are the standard way of loading data into a database system they are suitable for quick prototyping but should be avoided for bulk loading as they have significant per-row overhead insert into people values 1 mark see here for a more detailed description of insert statements csv loading data can be efficiently loaded from csv files using the read_csv_auto function or the copy statement select from read_csv_auto test csv you can also load data from compressed e g compressed with gzip csv files for example select from read_csv_auto test csv gz see here for a detailed description of csv loading parquet loading parquet files can be efficiently loaded and queried using the read_parquet function select from read_parquet test parquet see here for a detailed description of parquet loading appender c and java in c and java the appender can be used as an alternative for bulk data loading this class can be used to efficiently add rows to the database system without needing to use sql c appender appender con people appender appendrow 1 mark appender close java con createappender main people appender beginrow appender append mark appender endrow appender close see here for a detailed description of the c appender",
- "category": "Data",
- "url": "../docs/data/overview",
- "blurb": "The first step to using a database system is to insert data into that system. DuckDB provides several data ingestion..."
+ "title": "C API - Startup & Shutdown",
+ "text": "to use duckdb you must first initialize a duckdb_database handle using duckdb_open duckdb_open takes as parameter the database file to read and write from the special value null nullptr can be used to create an in-memory database note that for an in-memory database no data is persisted to disk i e all data is lost when you exit the process with the duckdb_database handle you can create one or many duckdb_connection using duckdb_connect while individual connections are thread-safe they will be locked during querying it is therefore recommended that each thread uses its own connection to allow for the best parallel performance all duckdb_connection s have to explicitly be disconnected with duckdb_disconnect and the duckdb_database has to be explicitly closed with duckdb_close to avoid memory and file handle leaking example duckdb_database db duckdb_connection con if duckdb_open null db duckdberror handle error if duckdb_connect db con duckdberror handle error run queries cleanup duckdb_disconnect con duckdb_close db api reference syntax path to the database file on disk or nullptr or memory to open an in-memory database out_database the result database object returns duckdbsuccess on success or duckdberror on failure duckdb_open_ext extended version of duckdb_open creates a new database or opens an existing database file stored at the given path syntax path to the database file on disk or nullptr or memory to open an in-memory database out_database the result database object config optional configuration used to start up the database system out_error if set and the function returns duckdberror this will contain the reason why the start-up failed note that the error must be freed using duckdb_free returns duckdbsuccess on success or duckdberror on failure duckdb_close closes the specified database and de-allocates all memory allocated for that database this should be called after you are done with any database allocated through duckdb_open note that failing to call duckdb_close in case of e g a program crash will not cause data corruption still it is recommended to always correctly close a database object after you are done with it syntax the database object to shut down duckdb_connect opens a connection to a database connections are required to query the database and store transactional state associated with the connection the instantiated connection should be closed using duckdb_disconnect syntax the database file to connect to out_connection the result connection object returns duckdbsuccess on success or duckdberror on failure duckdb_interrupt interrupt running query syntax the connection to interruot duckdb_query_progress get progress of the running query syntax the working connection returns -1 if no progress or a percentage of the progress duckdb_disconnect closes the specified connection and de-allocates all memory allocated for that connection syntax the connection to close duckdb_library_version returns the version of the linked duckdb with a version postfix for dev versions usually used for developing c extensions that must return this for a compatibility check syntax",
+ "category": "C",
+ "url": "../docs/api/c/connect",
+ "blurb": "To use DuckDB, you must first initialize a duckdb_database handle using duckdb_open() . duckdb_open() takes as..."
},
{
- "title": "Insert Statements",
- "text": "insert statements are the standard way of loading data into a relational database when using insert statements the values are supplied row-by-row while simple there is significant overhead involved in parsing and processing individual insert statements this makes them very inefficient for bulk insertion as a rule-of-thumb avoid insert statements when inserting more than a few rows i e avoid using insert statements as part of a loop if you must use insert statements to load data in a loop avoid executing the statements in auto-commit mode after every commit the database is required to sync the changes made to disk to ensure no data is lost in auto-commit mode every single statement will be wrapped in a separate transaction meaning fsync will be called for every statement this is typically unnecessary when bulk loading and will significantly slow down your program if you absolutely must use insert statements in a loop to load data wrap them in calls to begin transaction and commit syntax an example of using insert into to load data in a table is as follows create table people id integer name varchar insert into people values 1 mark 2 hannes a more detailed description together with syntax diagram can be found here",
- "category": "Data",
- "url": "../docs/data/insert",
- "blurb": "Insert statements are the standard way of loading data into a relational database. When using insert statements, the..."
+ "title": "C API - Table Functions",
+ "text": "the table function api can be used to define a table function that can then be called from within duckdb in the from clause of a query api reference the return value should be destroyed with duckdb_destroy_table_function syntax the table function object duckdb_destroy_table_function destroys the given table function object syntax the table function to destroy duckdb_table_function_set_name sets the name of the given table function syntax the table function name the name of the table function duckdb_table_function_add_parameter adds a parameter to the table function syntax the table function type the type of the parameter to add duckdb_table_function_add_named_parameter adds a named parameter to the table function syntax the table function name the name of the parameter type the type of the parameter to add duckdb_table_function_set_extra_info assigns extra information to the table function that can be fetched during binding etc syntax the table function extra_info the extra information destroy the callback that will be called to destroy the bind data if any duckdb_table_function_set_bind sets the bind function of the table function syntax the table function bind the bind function duckdb_table_function_set_init sets the init function of the table function syntax the table function init the init function duckdb_table_function_set_local_init sets the thread-local init function of the table function syntax the table function init the init function duckdb_table_function_set_function sets the main function of the table function syntax the table function function the function duckdb_table_function_supports_projection_pushdown sets whether or not the given table function supports projection pushdown if this is set to true the system will provide a list of all required columns in the init stage through the duckdb_init_get_column_count and duckdb_init_get_column_index functions if this is set to false the default the system will expect all columns to be projected syntax the table function pushdown true if the table function supports projection pushdown false otherwise duckdb_register_table_function register the table function object within the given connection the function requires at least a name a bind function an init function and a main function if the function is incomplete or a function with this name already exists duckdberror is returned syntax the connection to register it in function the function pointer returns whether or not the registration was successful duckdb_bind_get_extra_info retrieves the extra info of the function as set in duckdb_table_function_set_extra_info syntax the info object returns the extra info duckdb_bind_add_result_column adds a result column to the output of the table function syntax the info object name the name of the column type the logical type of the column duckdb_bind_get_parameter_count retrieves the number of regular non-named parameters to the function syntax the info object returns the number of parameters duckdb_bind_get_parameter retrieves the parameter at the given index the result must be destroyed with duckdb_destroy_value syntax the info object index the index of the parameter to get returns the value of the parameter must be destroyed with duckdb_destroy_value duckdb_bind_get_named_parameter retrieves a named parameter with the given name the result must be destroyed with duckdb_destroy_value syntax the info object name the name of the parameter returns the value of the parameter must be destroyed with duckdb_destroy_value duckdb_bind_set_bind_data sets the user-provided bind data in the bind object this object can be retrieved again during execution syntax the info object extra_data the bind data object destroy the callback that will be called to destroy the bind data if any duckdb_bind_set_cardinality sets the cardinality estimate for the table function used for optimization syntax the bind data object is_exact whether or not the cardinality estimate is exact or an approximation duckdb_bind_set_error report that an error has occurred while calling bind syntax the info object error the error message duckdb_init_get_extra_info retrieves the extra info of the function as set in duckdb_table_function_set_extra_info syntax the info object returns the extra info duckdb_init_get_bind_data gets the bind data set by duckdb_bind_set_bind_data during the bind note that the bind data should be considered as read-only for tracking state use the init data instead syntax the info object returns the bind data object duckdb_init_set_init_data sets the user-provided init data in the init object this object can be retrieved again during execution syntax the info object extra_data the init data object destroy the callback that will be called to destroy the init data if any duckdb_init_get_column_count returns the number of projected columns this function must be used if projection pushdown is enabled to figure out which columns to emit syntax the info object returns the number of projected columns duckdb_init_get_column_index returns the column index of the projected column at the specified position this function must be used if projection pushdown is enabled to figure out which columns to emit syntax the info object column_index the index at which to get the projected column index from 0 duckdb_init_get_column_count info returns the column index of the projected column duckdb_init_set_max_threads sets how many threads can process this table function in parallel default 1 syntax the info object max_threads the maximum amount of threads that can process this table function duckdb_init_set_error report that an error has occurred while calling init syntax the info object error the error message duckdb_function_get_extra_info retrieves the extra info of the function as set in duckdb_table_function_set_extra_info syntax the info object returns the extra info duckdb_function_get_bind_data gets the bind data set by duckdb_bind_set_bind_data during the bind note that the bind data should be considered as read-only for tracking state use the init data instead syntax the info object returns the bind data object duckdb_function_get_init_data gets the init data set by duckdb_init_set_init_data during the init syntax the info object returns the init data object duckdb_function_get_local_init_data gets the thread-local init data set by duckdb_init_set_init_data during the local_init syntax the info object returns the init data object duckdb_function_set_error report that an error has occurred while executing the function syntax the info object error the error message",
+ "category": "C",
+ "url": "../docs/api/c/table_functions",
+ "blurb": "The table function API can be used to define a table function that can then be called from within DuckDB in the FROM ..."
},
{
- "title": "Pragmas",
- "text": "the pragma statement is an sql extension adopted by duckdb from sqlite pragma statements can be issued in a similar manner to regular sql statements pragma commands may alter the internal state of the database engine and can influence the subsequent execution or behavior of the engine list of supported pragma statements below is a list of supported pragma statements database_list show_tables table_info show -- list all databases usually one pragma database_list -- list all tables pragma show_tables -- get info for a specific table pragma table_info table_name -- also show table structure but slightly different format for compatibility pragma show table_name table_info returns information about the columns of the table with name table_name the exact format of the table returned is given below cid integer -- cid of the column name varchar -- name fo the column type varchar -- type of the column notnull boolean -- if the column is marked as not null dflt_value varchar -- default value of the column or null if not specified pk boolean -- part of the primary key or not memory_limit threads -- set the memory limit pragma memory_limit 1gb -- set the amount of threads for parallel query execution pragma threads 4 database_size -- get the file and memory size of each database pragma database_size database_size returns information about the file and memory size of each table the column types of the returned results are given below database_size varchar -- total block count times the block size block_size bigint -- database block size total_blocks bigint -- total blocks in the database used_blocks bigint -- used blocks in the database free_blocks bigint -- free blocks in the database wal_size varchar -- write ahead log size memory_usage varchar -- memory used by the database buffer manager memory_limit varchar -- maximum memory allowed for the database collations default_collation -- list all available collations pragma collations -- set the default collation to one of the available ones pragma default_collation nocase default_null_order default_order -- set the ordering for nulls to be either nulls first or nulls last pragma default_null_order nulls last -- set the default result set ordering direction to ascending or descending pragma default_order descending version -- show duckdb version pragma version enable_progress_bar enable_profiling disable_profiling profiling_output -- show progress bar when running queries pragma enable_progress_bar -- enable profiling pragma enable_profiling -- enable profiling in a specified format pragma enable_profiling json query_tree query_tree_optimizer -- disable profiling pragma disable_profiling -- specify a file to save the profiling output to pragma profiling_output path to file json pragma profile_output path to file json enable the gathering and printing of profiling information after the execution of a query optionally the format of the resulting profiling information can be specified as either json query_tree or query_tree_optimizer the default format is query_tree which prints the physical operator tree together with the timings and cardinalities of each operator in the tree to the screen below is an example output of the profiling information for the simple query select 42 query profiling information select 42 total time 0 0001s projection 42 1 0 00s dummy_scan 1 0 00s the printing of profiling information can be disabled again using disable_profiling by default profiling information is printed to the console however if you prefer to write the profiling information to a file the pragma profiling_output can be used to write to a specified file note that the file contents will be overwritten for every new query that is issued hence the file will only contain the profiling information of the last query that is run disable_optimizer enable_optimizer -- disables the query optimizer pragma disable_optimizer -- enables the query optimizer pragma enable_optimizer log_query_path explain_output enable_verification disable_verification force_parallelism disable_force_parallelism -- set a path for query logging pragma log_query_path tmp duckdb_log -- disable query logging again pragma log_query_path -- either show all or only optimized plans in the explain output pragma explain_output optimized -- enable query verification for development pragma enable_verification -- disable query verification for development pragma disable_verification -- enable force parallel query processing for development pragma force_parallelism -- disable force parallel query processing for development pragma disable_force_parallelism -- force index joins where applicable pragma force_index_join these are pragma s mostly used for development and internal testing create_fts_index drop_fts_index only available when the fts extension is built documented here temp directory for spilling data to disk -- defaults to tmp pragma temp_directory path to temp tmp",
- "category": "SQL",
- "url": "../docs/sql/pragmas",
- "blurb": "The PRAGMA statement is an SQL extension adopted by DuckDB from SQLite. PRAGMA statements can be issued in a similar..."
+ "title": "C API - Prepared Statements",
+ "text": "a prepared statement is a parameterized query the query is prepared with question marks or dollar symbols 1 indicating the parameters of the query values can then be bound to these parameters after which the prepared statement can be executed using those parameters a single query can be prepared once and executed many times prepared statements are useful to easily supply parameters to functions while avoiding string concatenation sql injection attacks speeding up queries that will be executed many times with different parameters duckdb supports prepared statements in the c api with the duckdb_prepare method the duckdb_bind family of functions is used to supply values for subsequent execution of the prepared statement using duckdb_execute_prepared after we are done with the prepared statement it can be cleaned up using the duckdb_destroy_prepare method example duckdb_prepared_statement stmt duckdb_result result if duckdb_prepare con insert into integers values 1 2 stmt duckdberror handle error duckdb_bind_int32 stmt 1 42 the parameter index starts counting at 1 duckdb_bind_int32 stmt 2 43 null as second parameter means no result set is requested duckdb_execute_prepared stmt null duckdb_destroy_prepare stmt we can also query result sets using prepared statements if duckdb_prepare con select from integers where i stmt duckdberror handle error duckdb_bind_int32 stmt 1 42 duckdb_execute_prepared stmt result do something with result clean up duckdb_destroy_result result duckdb_destroy_prepare stmt after calling duckdb_prepare the prepared statement parameters can be inspected using duckdb_nparams and duckdb_param_type in case the prepare fails the error can be obtained through duckdb_prepare_error it is not required that the duckdb_bind family of functions matches the prepared statement parameter type exactly the values will be auto-cast to the required value as required for example calling duckdb_bind_int8 on a parameter type of duckdb_type_integer will work as expected do not use prepared statements to insert large amounts of data into duckdb instead it is recommended to use the appender api reference note that after calling duckdb_prepare the prepared statement should always be destroyed using duckdb_destroy_prepare even if the prepare fails if the prepare fails duckdb_prepare_error can be called to obtain the reason why the prepare failed syntax the connection object query the sql query to prepare out_prepared_statement the resulting prepared statement object returns duckdbsuccess on success or duckdberror on failure duckdb_destroy_prepare closes the prepared statement and de-allocates all memory allocated for the statement syntax the prepared statement to destroy duckdb_prepare_error returns the error message associated with the given prepared statement if the prepared statement has no error message this returns nullptr instead the error message should not be freed it will be de-allocated when duckdb_destroy_prepare is called syntax the prepared statement to obtain the error from returns the error message or nullptr if there is none duckdb_nparams returns the number of parameters that can be provided to the given prepared statement returns 0 if the query was not successfully prepared syntax the prepared statement to obtain the number of parameters for duckdb_param_type returns the parameter type for the parameter at the given index returns duckdb_type_invalid if the parameter index is out of range or the statement was not successfully prepared syntax the prepared statement param_idx the parameter index returns the parameter type duckdb_clear_bindings clear the params bind to the prepared statement syntax duckdb_bind_value binds a value to the prepared statement at the specified index syntax duckdb_bind_parameter_index retrieve the index of the parameter for the prepared statement identified by name syntax duckdb_bind_boolean binds a bool value to the prepared statement at the specified index syntax duckdb_bind_int8 binds an int8_t value to the prepared statement at the specified index syntax duckdb_bind_int16 binds an int16_t value to the prepared statement at the specified index syntax duckdb_bind_int32 binds an int32_t value to the prepared statement at the specified index syntax duckdb_bind_int64 binds an int64_t value to the prepared statement at the specified index syntax duckdb_bind_hugeint binds a duckdb_hugeint value to the prepared statement at the specified index syntax duckdb_bind_decimal binds a duckdb_decimal value to the prepared statement at the specified index syntax duckdb_bind_uint8 binds an uint8_t value to the prepared statement at the specified index syntax duckdb_bind_uint16 binds an uint16_t value to the prepared statement at the specified index syntax duckdb_bind_uint32 binds an uint32_t value to the prepared statement at the specified index syntax duckdb_bind_uint64 binds an uint64_t value to the prepared statement at the specified index syntax duckdb_bind_float binds a float value to the prepared statement at the specified index syntax duckdb_bind_double binds a double value to the prepared statement at the specified index syntax duckdb_bind_date binds a duckdb_date value to the prepared statement at the specified index syntax duckdb_bind_time binds a duckdb_time value to the prepared statement at the specified index syntax duckdb_bind_timestamp binds a duckdb_timestamp value to the prepared statement at the specified index syntax duckdb_bind_interval binds a duckdb_interval value to the prepared statement at the specified index syntax duckdb_bind_varchar binds a null-terminated varchar value to the prepared statement at the specified index syntax duckdb_bind_varchar_length binds a varchar value to the prepared statement at the specified index syntax duckdb_bind_blob binds a blob value to the prepared statement at the specified index syntax duckdb_bind_null binds a null value to the prepared statement at the specified index syntax duckdb_execute_prepared executes the prepared statement with the given bound parameters and returns a materialized query result this method can be called multiple times for each prepared statement and the parameters can be modified between calls to this function syntax the prepared statement to execute out_result the query result returns duckdbsuccess on success or duckdberror on failure duckdb_execute_prepared_arrow executes the prepared statement with the given bound parameters and returns an arrow query result syntax the prepared statement to execute out_result the query result returns duckdbsuccess on success or duckdberror on failure duckdb_arrow_scan scans the arrow stream and creates a view with the given name syntax the connection on which to execute the scan table_name name of the temporary view to create arrow arrow stream wrapper returns duckdbsuccess on success or duckdberror on failure duckdb_arrow_array_scan scans the arrow array and creates a view with the given name syntax the connection on which to execute the scan table_name name of the temporary view to create arrow_schema arrow schema wrapper arrow_array arrow array wrapper out_stream output array stream that wraps around the passed schema for releasing deleting once done returns duckdbsuccess on success or duckdberror on failure",
+ "category": "C",
+ "url": "../docs/api/c/prepared",
+ "blurb": "A prepared statement is a parameterized query. The query is prepared with question marks ( ? ) or dollar symbols ( $1 )..."
},
{
- "title": "Configuration",
- "text": "duckdb has a number of configuration options that can be used to change the behavior of the system the configuration options can be set using either the set statement or the pragma statement examples -- set the memory limit of the system to 10gb set memory_limit 10gb -- configure the system to use 1 thread set threads to 1 -- enable printing of a progress bar during long-running queries set enable_progress_bar true -- set the default null order to nulls last pragma default_null_order nulls_last -- show a list of all available settings select from duckdb_settings -- return the current value of a specific setting -- this example returns automatic select current_setting access_mode configuration reference below is a list of all available settings name description input_type default_value ------------------------------------------ --------------------------------------------------------------------------------------------------------------------------------------------------------- ------------ ----------------- calendar the current calendar varchar gregorian timezone the current time zone varchar system timezone access_mode access mode of the database automatic read_only or read_write varchar automatic allow_unsigned_extensions allow to load extensions with invalid or missing signatures boolean false binary_as_string in parquet files interpret binary data as a string boolean checkpoint_threshold wal_autocheckpoint the wal size threshold at which to automatically trigger a checkpoint e g 1gb varchar 16 7mb default_collation the collation setting used when none is specified varchar default_null_order null_order null ordering used when none is specified nulls_first or nulls_last varchar nulls_first default_order the order type used when none is specified asc or desc varchar asc enable_external_access allow the database to access external state through e g loading installing modules copy to from csv readers pandas replacement scans etc boolean true enable_object_cache whether or not object cache is used to cache e g parquet metadata boolean false enable_profiling enables profiling and sets the output format json query_tree query_tree_optimizer varchar null enable_progress_bar enables the progress bar printing progress to the terminal for long queries boolean false explain_output output of explain statements all optimized_only physical_only varchar physical_only external_threads the number of external threads that work on duckdb tasks bigint 0 file_search_path a comma separated list of directories to search for input files varchar home_directory sets the home directory used by the system varchar httpfs_timeout http timeout read write connection retry default 30000ms ubigint log_query_path specifies the path to which queries should be logged default empty string queries are not logged varchar null max_expression_depth the maximum expression depth limit in the parser warning increasing this setting and using very deep expressions might lead to stack overflow errors ubigint 1000 max_memory memory_limit the maximum memory of the system e g 1gb varchar 75 of ram perfect_ht_threshold threshold in bytes for when to use a perfect hash table default 12 bigint 12 preserve_identifier_case whether or not to preserve the identifier case instead of always lowercasing all non-quoted identifiers boolean true preserve_insertion_order whether or not to preserve insertion order if set to false the system is allowed to re-order any results that do not contain order by clauses boolean true profile_output profiling_output the file to which profile output should be saved or empty to print to the terminal varchar profiler_history_size sets the profiler history size bigint null profiling_mode the profiling mode standard or detailed varchar null progress_bar_time sets the time in milliseconds how long a query needs to take before we start printing a progress bar bigint 2000 s3_access_key_id s3 access key id varchar s3_endpoint s3 endpoint default s3 amazonaws com varchar s3_region s3 region varchar s3_secret_access_key s3 access key varchar s3_session_token s3 session token varchar s3_uploader_max_filesize s3 uploader max filesize between 50gb and 5tb default 800gb varchar s3_uploader_max_parts_per_file s3 uploader max parts per file between 1 and 10000 default 10000 ubigint s3_uploader_thread_limit s3 uploader global thread limit default 50 ubigint s3_url_style s3 url style vhost default or path varchar s3_use_ssl s3 use ssl default true boolean schema sets the default search schema equivalent to setting search_path to a single value varchar search_path sets the default search search path as a comma-separated list of values varchar temp_directory set the directory to which to write temp files varchar threads worker_threads the number of total threads used by the system bigint cores",
- "category": "SQL",
- "url": "../docs/sql/configuration",
- "blurb": "DuckDB has a number of configuration options that can be used to change the behavior of the system. The configuration..."
+ "title": "C API - Values",
+ "text": "the value class represents a single value of any type api reference syntax the value to destroy duckdb_create_varchar creates a value from a null-terminated string syntax the null-terminated string returns the value this must be destroyed with duckdb_destroy_value duckdb_create_varchar_length creates a value from a string syntax the text length the length of the text returns the value this must be destroyed with duckdb_destroy_value duckdb_create_int64 creates a value from an int64 syntax the bigint value returns the value this must be destroyed with duckdb_destroy_value duckdb_get_varchar obtains a string representation of the given value the result must be destroyed with duckdb_free syntax the value returns the string value this must be destroyed with duckdb_free duckdb_get_int64 obtains an int64 of the given value syntax the value returns the int64 value or 0 if no conversion is possible",
+ "category": "C",
+ "url": "../docs/api/c/value",
+ "blurb": "The value class represents a single value of any type. API Reference Syntax The value to destroy. ..."
},
{
- "title": "Aggregate Functions",
- "text": "aggregates are functions that combine multiple rows into a single value aggregates are different from scalar functions and window functions because they change the cardinality of the result as such aggregates can only be used in the select and having clauses of a sql query when the distinct clause is provided only distinct values are considered in the computation of the aggregate this is typically used in combination with the count aggregate to get the number of distinct elements but it can be used together with any aggregate function in the system when the order by clause is provided the values being aggregated are sorted before applying the function usually this is not important but there are some order-sensitive aggregates that can have indeterminate results e g first last list and string_agg these can be made deterministic by ordering the arguments for order-insensitive aggregates this clause is parsed but ignored general aggregate functions the table below shows the available general aggregate functions function description example alias es --- --- --- --- arg_max arg val calculates the arg value for a maximum val value arg_max a b argmax a b max_by a b arg_min arg val calculates the arg value for a minimum val value arg_min a b argmin a b min_by a b avg arg calculates the average value for all tuples in arg avg a - bit_and arg returns the bitwise and of all bits in a given expression bit_and a - bit_or arg returns the bitwise or of all bits in a given expression bit_or a - bit_xor arg returns the bitwise xor of all bits in a given expression bit_xor a - bool_and arg returns true if every input value is true otherwise false bool_and a - bool_or arg returns true if any input value is true otherwise false bool a - count arg calculates the number of tuples tuples in arg count a - favg arg calculates the average using a more accurate floating point summation kahan sum favg a - first arg returns the first value of a column first a arbitrary a fsum arg calculates the sum using a more accurate floating point summation kahan sum fsum a sumkahan kahan_sum histogram arg returns a list of struct s with the fields bucket and count histogram a - last arg returns the last value of a column last a - list arg returns a list containing all the values of a column list a array_agg max arg returns the maximum value present in arg max a - min arg returns the minumum value present in arg min a - product arg calculates the product of all tuples in arg product a - string_agg arg sep concatenates the column string values with a separator string_agg s group_concat sum arg calculates the sum value for all tuples in arg sum a - approximate aggregates the table below shows the available approximate aggregate functions function description example --- --- --- approx_count_distinct x gives the approximate count of distintinct elements using hyperloglog approx_count_distinct a approx_quantile x pos gives the approximate quantile using t-digest approx_quantile a 0 5 reservoir_quantile x quantile sample_size 8192 gives the approximate quantile using reservoir sampling the sample size is optional and uses 8192 as a default size reservoir_quantile a 0 5 1024 statistical aggregates the table below shows the available statistical aggregate functions function description formula alias --- --- --- --- corr y x returns the correlation coefficient for non-null pairs in a group covar_pop y x stddev_pop x stddev_pop y - covar_pop y x returns the population covariance of input values sum x y - sum x sum y count count - entropy x returns the log-2 entropy of count input-values - - kurtosis x returns the excess kurtosis of all input values - - mad x returns the median absolute deviation for the values within x null values are ignored temporal types return a positive interval median abs x-median x - median x returns the middle value of the set null values are ignored for even value counts quantitiative values are averaged and ordinal values return the lower value quantile_cont x 0 5 - mode x returns the most frequent value for the values within x null values are ignored - - quantile_cont x pos returns the intepolated quantile number between 0 and 1 if pos is a list of float s then the result is a list of the corresponding intepolated quantiles - - quantile_disc x pos returns the exact quantile number between 0 and 1 if pos is a list of float s then the result is a list of the corresponding exact quantiles - quantile regr_avgx y x returns the average of the independent variable for non-null pairs in a group where x is the independent variable and y is the dependent variable - - regr_avgy y x returns the average of the dependent variable for non-null pairs in a group where x is the independent variable and y is the dependent variable - - regr_count y x returns the number of non-null number pairs in a group sum x y - sum x sum y count count - regr_intercept y x returns the intercept of the univariate linear regression line for non-null pairs in a group avg y -regr_slope y x avg x - regr_r2 y x returns the coefficient of determination for non-null pairs in a group - - regr_slope y x returns the slope of the linear regression line for non-null pairs in a group covar_pop x y var_pop x - regr_sxx y x - regr_count y x var_pop x - regr_sxy y x returns the population covariance of input values regr_count y x covar_pop y x - regr_syy y x - regr_count y x var_pop y f - skewness x returns the skewness of all input values - - stddev_pop x returns the population standard deviation sqrt var_pop x - stddev_samp x returns the sample standard deviation sqrt var_samp x stddev x var_pop x returns the population variance - - var_samp x returns the sample variance of all input values sum x 2 - sum x 2 count x count x - 1 variance arg val ordered set aggregate functions the table below shows the available ordered set aggregate functions these functions are specified using the within group order by sort_expression syntax and they are converted to an equivalent aggregate function that takes the ordering expression as the first argument function equivalent --- --- mode within group order by sort_expression mode sort_expression percentile_cont fraction within group order by sort_expression quantile_cont sort_expression fraction percentile_cont fractions within group order by sort_expression quantile_cont sort_expression fractions percentile_disc fraction within group order by sort_expression quantile_disc sort_expression fraction percentile_disc fractions within group order by sort_expression quantile_disc sort_expression fractions",
- "category": "SQL",
- "url": "../docs/sql/aggregates",
- "blurb": "Aggregates are functions that combine multiple rows into a single value. Aggregates are different from scalar..."
+ "title": "C API - Types",
+ "text": "duckdb is a strongly typed database system as such every column has a single type specified this type is constant over the entire column that is to say a column that is labeled as an integer column will only contain integer values duckdb also supports columns of composite types for example it is possible to define an array of integers int it is also possible to define types as arbitrary structs row i integer j varchar for that reason native duckdb type objects are not mere enums but a class that can potentially be nested types in the c api are modeled using an enum duckdb_type and a complex class duckdb_logical_type for most primitive types e g integers or varchars the enum is sufficient for more complex types such as lists structs or decimals the logical type must be used typedef enum duckdb_type duckdb_type_invalid duckdb_type_boolean duckdb_type_tinyint duckdb_type_smallint duckdb_type_integer duckdb_type_bigint duckdb_type_utinyint duckdb_type_usmallint duckdb_type_uinteger duckdb_type_ubigint duckdb_type_float duckdb_type_double duckdb_type_timestamp duckdb_type_date duckdb_type_time duckdb_type_interval duckdb_type_hugeint duckdb_type_varchar duckdb_type_blob duckdb_type_decimal duckdb_type_timestamp_s duckdb_type_timestamp_ms duckdb_type_timestamp_ns duckdb_type_enum duckdb_type_list duckdb_type_struct duckdb_type_map duckdb_type_uuid duckdb_type_json duckdb_type the enum type of a column in the result can be obtained using the duckdb_column_type function the logical type of a column can be obtained using the duckdb_column_logical_type function duckdb_value the duckdb_value functions will auto-cast values as required for example it is no problem to use duckdb_value_double on a column of type duckdb_value_int32 the value will be auto-cast and returned as a double note that in certain cases the cast may fail for example this can happen if we request a duckdb_value_int8 and the value does not fit within an int8 value in this case a default value will be returned usually 0 or nullptr the same default value will also be returned if the corresponding value is null the duckdb_value_is_null function can be used to check if a specific value is null or not the exception to the auto-cast rule is the duckdb_value_varchar_internal function this function does not auto-cast and only works for varchar columns the reason this function exists is that the result does not need to be freed note that duckdb_value_varchar and duckdb_value_blob require the result to be de-allocated using duckdb_free duckdb_result_get_chunk the duckdb_result_get_chunk function can be used to read data chunks from a duckdb result set and is the most efficient way of reading data from a duckdb result using the c api it is also the only way of reading data of certain types from a duckdb result for example the duckdb_value functions do not support structural reading of composite types lists or structs or more complex types like enums and decimals for more information about data chunks see the documentation on data chunks api reference the result must be destroyed with duckdb_destroy_data_chunk this function supersedes all duckdb_value functions as well as the duckdb_column_data and duckdb_nullmask_data functions it results in significantly better performance and should be preferred in newer code-bases if this function is used none of the other result functions can be used and vice versa i e this function cannot be mixed with the legacy result functions use duckdb_result_chunk_count to figure out how many chunks there are in the result syntax the result object to fetch the data chunk from chunk_index the chunk index to fetch from returns the resulting data chunk returns null if the chunk index is out of bounds duckdb_result_is_streaming checks if the type of the internal result is streamqueryresult syntax the result object to check returns whether or not the result object is of the type streamqueryresult duckdb_result_chunk_count returns the number of data chunks present in the result syntax the result object returns number of data chunks present in the result duckdb_value_boolean syntax the boolean value at the specified location or false if the value cannot be converted duckdb_value_int8 syntax the int8_t value at the specified location or 0 if the value cannot be converted duckdb_value_int16 syntax the int16_t value at the specified location or 0 if the value cannot be converted duckdb_value_int32 syntax the int32_t value at the specified location or 0 if the value cannot be converted duckdb_value_int64 syntax the int64_t value at the specified location or 0 if the value cannot be converted duckdb_value_hugeint syntax the duckdb_hugeint value at the specified location or 0 if the value cannot be converted duckdb_value_decimal syntax the duckdb_decimal value at the specified location or 0 if the value cannot be converted duckdb_value_uint8 syntax the uint8_t value at the specified location or 0 if the value cannot be converted duckdb_value_uint16 syntax the uint16_t value at the specified location or 0 if the value cannot be converted duckdb_value_uint32 syntax the uint32_t value at the specified location or 0 if the value cannot be converted duckdb_value_uint64 syntax the uint64_t value at the specified location or 0 if the value cannot be converted duckdb_value_float syntax the float value at the specified location or 0 if the value cannot be converted duckdb_value_double syntax the double value at the specified location or 0 if the value cannot be converted duckdb_value_date syntax the duckdb_date value at the specified location or 0 if the value cannot be converted duckdb_value_time syntax the duckdb_time value at the specified location or 0 if the value cannot be converted duckdb_value_timestamp syntax the duckdb_timestamp value at the specified location or 0 if the value cannot be converted duckdb_value_interval syntax the duckdb_interval value at the specified location or 0 if the value cannot be converted duckdb_value_varchar syntax use duckdb_value_string instead this function does not work correctly if the string contains null bytes returns the text value at the specified location as a null-terminated string or nullptr if the value cannot be converted the result must be freed with duckdb_free duckdb_value_varchar_internal syntax use duckdb_value_string_internal instead this function does not work correctly if the string contains null bytes returns the char value at the specified location only works on varchar columns and does not auto-cast if the column is not a varchar column this function will return null the result must not be freed duckdb_value_string_internal syntax use duckdb_value_string_internal instead this function does not work correctly if the string contains null bytes returns the char value at the specified location only works on varchar columns and does not auto-cast if the column is not a varchar column this function will return null the result must not be freed duckdb_value_blob syntax the duckdb_blob value at the specified location returns a blob with blob data set to nullptr if the value cannot be converted the resulting blob data must be freed with duckdb_free duckdb_value_is_null syntax returns true if the value at the specified index is null and false otherwise duckdb_from_date decompose a duckdb_date object into year month and date stored as duckdb_date_struct syntax the date object as obtained from a duckdb_type_date column returns the duckdb_date_struct with the decomposed elements duckdb_to_date re-compose a duckdb_date from year month and date duckdb_date_struct syntax the year month and date stored in a duckdb_date_struct returns the duckdb_date element duckdb_from_time decompose a duckdb_time object into hour minute second and microsecond stored as duckdb_time_struct syntax the time object as obtained from a duckdb_type_time column returns the duckdb_time_struct with the decomposed elements duckdb_to_time re-compose a duckdb_time from hour minute second and microsecond duckdb_time_struct syntax the hour minute second and microsecond in a duckdb_time_struct returns the duckdb_time element duckdb_from_timestamp decompose a duckdb_timestamp object into a duckdb_timestamp_struct syntax the ts object as obtained from a duckdb_type_timestamp column returns the duckdb_timestamp_struct with the decomposed elements duckdb_to_timestamp re-compose a duckdb_timestamp from a duckdb_timestamp_struct syntax the de-composed elements in a duckdb_timestamp_struct returns the duckdb_timestamp element duckdb_hugeint_to_double converts a duckdb_hugeint object as obtained from a duckdb_type_hugeint column into a double syntax the hugeint value returns the converted double element duckdb_double_to_hugeint converts a double value to a duckdb_hugeint object if the conversion fails because the double value is too big the result will be 0 syntax the double value returns the converted duckdb_hugeint element duckdb_double_to_decimal converts a double value to a duckdb_decimal object if the conversion fails because the double value is too big or the width scale are invalid the result will be 0 syntax the double value returns the converted duckdb_decimal element duckdb_decimal_to_double converts a duckdb_decimal object as obtained from a duckdb_type_decimal column into a double syntax the decimal value returns the converted double element duckdb_create_logical_type creates a duckdb_logical_type from a standard primitive type the resulting type should be destroyed with duckdb_destroy_logical_type this should not be used with duckdb_type_decimal syntax the primitive type to create returns the logical type duckdb_create_list_type creates a list type from its child type the resulting type should be destroyed with duckdb_destroy_logical_type syntax the child type of list type to create returns the logical type duckdb_create_map_type creates a map type from its key type and value type the resulting type should be destroyed with duckdb_destroy_logical_type syntax the key type and value type of map type to create returns the logical type duckdb_create_union_type creates a union type from the passed types array the resulting type should be destroyed with duckdb_destroy_logical_type syntax the array of types that the union should consist of type_amount the size of the types array returns the logical type duckdb_create_struct_type creates a struct type from the passed member name and type arrays the resulting type should be destroyed with duckdb_destroy_logical_type syntax the array of types that the struct should consist of member_names the array of names that the struct should consist of member_count the number of members that were specified for both arrays returns the logical type duckdb_create_decimal_type creates a duckdb_logical_type of type decimal with the specified width and scale the resulting type should be destroyed with duckdb_destroy_logical_type syntax the width of the decimal type scale the scale of the decimal type returns the logical type duckdb_get_type_id retrieves the type class of a duckdb_logical_type syntax the logical type object returns the type id duckdb_decimal_width retrieves the width of a decimal type syntax the logical type object returns the width of the decimal type duckdb_decimal_scale retrieves the scale of a decimal type syntax the logical type object returns the scale of the decimal type duckdb_decimal_internal_type retrieves the internal storage type of a decimal type syntax the logical type object returns the internal type of the decimal type duckdb_enum_internal_type retrieves the internal storage type of an enum type syntax the logical type object returns the internal type of the enum type duckdb_enum_dictionary_size retrieves the dictionary size of the enum type syntax the logical type object returns the dictionary size of the enum type duckdb_enum_dictionary_value retrieves the dictionary value at the specified position from the enum the result must be freed with duckdb_free syntax the logical type object index the index in the dictionary returns the string value of the enum type must be freed with duckdb_free duckdb_list_type_child_type retrieves the child type of the given list type the result must be freed with duckdb_destroy_logical_type syntax the logical type object returns the child type of the list type must be destroyed with duckdb_destroy_logical_type duckdb_map_type_key_type retrieves the key type of the given map type the result must be freed with duckdb_destroy_logical_type syntax the logical type object returns the key type of the map type must be destroyed with duckdb_destroy_logical_type duckdb_map_type_value_type retrieves the value type of the given map type the result must be freed with duckdb_destroy_logical_type syntax the logical type object returns the value type of the map type must be destroyed with duckdb_destroy_logical_type duckdb_struct_type_child_count returns the number of children of a struct type syntax the logical type object returns the number of children of a struct type duckdb_struct_type_child_name retrieves the name of the struct child the result must be freed with duckdb_free syntax the logical type object index the child index returns the name of the struct type must be freed with duckdb_free duckdb_struct_type_child_type retrieves the child type of the given struct type at the specified index the result must be freed with duckdb_destroy_logical_type syntax the logical type object index the child index returns the child type of the struct type must be destroyed with duckdb_destroy_logical_type duckdb_union_type_member_count returns the number of members that the union type has syntax the logical type union object returns the number of members of a union type duckdb_union_type_member_name retrieves the name of the union member the result must be freed with duckdb_free syntax the logical type object index the child index returns the name of the union member must be freed with duckdb_free duckdb_union_type_member_type retrieves the child type of the given union member at the specified index the result must be freed with duckdb_destroy_logical_type syntax the logical type object index the child index returns the child type of the union member must be destroyed with duckdb_destroy_logical_type duckdb_destroy_logical_type destroys the logical type and de-allocates all memory allocated for that type syntax the logical type to destroy",
+ "category": "C",
+ "url": "../docs/api/c/types",
+ "blurb": "DuckDB is a strongly typed database system. As such, every column has a single type specified. This type is constant ..."
},
{
- "title": "List",
- "text": "list data type a list column can have values with different lengths but they must all have the same underlying type list s are typically used to store arrays of numbers but can contain any uniform data type including other list s and struct s list s are similar to postgres s array type duckdb uses the list terminology but some array functions are provided for postgres compatibility see the data types overview for a comparison between nested data types lists can be created using the list_value expr function or the equivalent bracket notation expr the expressions can be constants or arbitrary expressions creating lists -- list of integers select 1 2 3 -- list of strings with a null value select duck goose null heron -- list of lists with null values select duck goose heron null frog toad -- create a list with the list_value function select list_value 1 2 3 -- create a table with an integer list column and a varchar list column create table list_table int_list int varchar_list varchar retrieving from lists retrieving one or more values from a list can be accomplished using brackets and slicing notation or through list functions like list_extract multiple equivalent functions are provided as aliases for compatibility with systems that refer to lists as arrays for example the function array_slice -- retrieve an element from a list using brackets this returns c -- note that we wrap the list creation in parenthesis so that it happens first -- this is only needed in our basic examples here not when working with a list column -- for example this can t be parsed select a b c 1 select a b c 2 -- use a negative index to grab the nth element from the end of the list this returns c select a b c -1 -- any expression that evaluates to an integer can be used to retrieve a list value -- this includes using a column to determine which index to retrieve -- this returns c select a b c 1 1 -- the list_extract function may also be used in place of brackets for selecting individual elements -- this returns c select list_extract a b c 2 -- retrieve multiple list values using a bracketed slice syntax this returns b c select a b c 1 3 -- single sided slices are also supported here grab the first 2 elements this returns a b select a b c 2 -- use a negative index to grab the last 2 elements this returns b c select a b c -2 -- the list_slice function syntax is also supported this returns b c select list_slice a b c 1 3 ordering the ordering is defined positionally null values compare greater than all other values and are considered equal to each other null comparisons at the top level null nested values obey standard sql null comparison rules comparing a null nested value to a non- null nested value produces a null result comparing nested value members however uses the internal nested value rules for null s and a null nested value member will compare above a non- null nested value member functions see nested functions",
- "category": "Data Types",
- "url": "../docs/sql/data_types/list",
- "blurb": "List Data Type A LIST column can have values with different lengths, but they must all have the same underlying type...."
+ "title": "C API - Query",
+ "text": "the duckdb_query method allows sql queries to be run in duckdb from c this method takes two parameters a null-terminated sql query string and a duckdb_result result pointer the result pointer may be null if the application is not interested in the result set or if the query produces no result after the result is consumed the duckdb_destroy_result method should be used to clean up the result elements can be extracted from the duckdb_result object using a variety of methods the duckdb_column_count and duckdb_row_count methods can be used to extract the number of columns and the number of rows respectively duckdb_column_name and duckdb_column_type can be used to extract the names and types of individual columns example duckdb_state state duckdb_result result create a table state duckdb_query con create table integers i integer j integer null if state duckdberror handle error insert three rows into the table state duckdb_query con insert into integers values 3 4 5 6 7 null null if state duckdberror handle error query rows again state duckdb_query con select from integers result if state duckdberror handle error handle the result destroy the result after we are done with it duckdb_destroy_result result value extraction values can be extracted using either the duckdb_column_data duckdb_nullmask_data functions or using the duckdb_value convenience functions the duckdb_column_data duckdb_nullmask_data functions directly hand you a pointer to the result arrays in columnar format and can therefore be very fast the duckdb_value functions perform bounds- and type-checking and will automatically cast values to the desired type this makes them more convenient and easier to use at the expense of being slower see the types page for more information for optimal performance use duckdb_column_data and duckdb_nullmask_data to extract data from the query result the duckdb_value functions perform internal type-checking bounds-checking and casting which makes them slower duckdb_value below is an example that prints the above result to csv format using the duckdb_value_varchar function note that the function is generic we do not need to know about the types of the individual result columns print the above result to csv format using duckdb_value_varchar idx_t row_count duckdb_row_count result idx_t column_count duckdb_column_count result for idx_t row 0 row row_count row for idx_t col 0 col column_count col if col 0 printf auto str_val duckdb_value_varchar result col row printf s str_val duckdb_free str_val printf n duckdb_column_data below is an example that prints the above result to csv format using the duckdb_column_data function note that the function is not generic we do need to know exactly what the types of the result columns are int32_t i_data int32_t duckdb_column_data result 0 int32_t j_data int32_t duckdb_column_data result 1 bool i_mask duckdb_nullmask_data result 0 bool j_mask duckdb_nullmask_data result 1 idx_t row_count duckdb_row_count result for idx_t row 0 row row_count row if i_mask row printf null else printf d i_data row printf if j_mask row printf null else printf d j_data row printf n when using duckdb_column_data be careful that the type matches exactly what you expect it to be as the code directly accesses an internal array there is no type-checking accessing a duckdb_type_integer column as if it was a duckdb_type_bigint column will provide unpredictable results api reference note that after running duckdb_query duckdb_destroy_result must be called on the result object even if the query fails otherwise the error stored within the result will not be freed correctly syntax the connection to perform the query in query the sql query to run out_result the query result returns duckdbsuccess on success or duckdberror on failure duckdb_destroy_result closes the result and de-allocates all memory allocated for that connection syntax the result to destroy duckdb_column_name returns the column name of the specified column the result should not need be freed the column names will automatically be destroyed when the result is destroyed returns null if the column is out of range syntax the result object to fetch the column name from col the column index returns the column name of the specified column duckdb_column_type returns the column type of the specified column returns duckdb_type_invalid if the column is out of range syntax the result object to fetch the column type from col the column index returns the column type of the specified column duckdb_column_logical_type returns the logical column type of the specified column the return type of this call should be destroyed with duckdb_destroy_logical_type returns null if the column is out of range syntax the result object to fetch the column type from col the column index returns the logical column type of the specified column duckdb_column_count returns the number of columns present in a the result object syntax the result object returns the number of columns present in the result object duckdb_row_count returns the number of rows present in a the result object syntax the result object returns the number of rows present in the result object duckdb_rows_changed returns the number of rows changed by the query stored in the result this is relevant only for insert update delete queries for other queries the rows_changed will be 0 syntax the result object returns the number of rows changed duckdb_column_data deprecated prefer using duckdb_result_get_chunk instead returns the data of a specific column of a result in columnar format the function returns a dense array which contains the result data the exact type stored in the array depends on the corresponding duckdb_type as provided by duckdb_column_type for the exact type by which the data should be accessed see the comments in the types section or the duckdb_type enum for example for a column of type duckdb_type_integer rows can be accessed in the following manner int32_t data int32_t duckdb_column_data result 0 printf data for row d d n row data row syntax the result object to fetch the column data from col the column index returns the column data of the specified column duckdb_nullmask_data deprecated prefer using duckdb_result_get_chunk instead returns the nullmask of a specific column of a result in columnar format the nullmask indicates for every row whether or not the corresponding row is null if a row is null the values present in the array provided by duckdb_column_data are undefined int32_t data int32_t duckdb_column_data result 0 bool nullmask duckdb_nullmask_data result 0 if nullmask row printf data for row d null n row else printf data for row d d n row data row syntax the result object to fetch the nullmask from col the column index returns the nullmask of the specified column duckdb_result_error returns the error message contained within the result the error is only set if duckdb_query returns duckdberror the result of this function must not be freed it will be cleaned up when duckdb_destroy_result is called syntax the result object to fetch the error from returns the error of the result",
+ "category": "C",
+ "url": "../docs/api/c/query",
+ "blurb": "The duckdb_query method allows SQL queries to be run in DuckDB from C. This method takes two parameters, a..."
},
{
- "title": "Timestamp Type",
- "text": "timestamps represent points in absolute time usually called instants duckdb represents instants as the number of microseconds µs since 1970-01-01 00 00 00 00 name aliases description --- --- --- timestamp datetime time of day ignores time zone timestamp with time zone timestamptz time of day uses time zone a timestamp specifies a combination of date year month day and a time hour minute second millisecond timestamps can be created using the timestamp keyword where the data must be formatted according to the iso 8601 format yyyy-mm-dd hh mm ss zzzzzz -tt tt -- 11 30 am at 20 september 1992 gmt select timestamp 1992-09-20 11 30 00 -- 2 30 pm at 20 september 1992 gmt select timestamp 1992-09-20 14 30 00 special values there are also three special date values that can be used on input input string valid types description ------------- ---------------------------------- ----------------------------------------------- epoch timestamp timestamptz 1970-01-01 00 00 00 00 unix system time zero infinity timestamp timestamptz later than all other time stamps -infinity timestamp timestamptz earlier than all other time stamps the values infinity and -infinity are specially represented inside the system and will be displayed unchanged but epoch is simply a notational shorthand that will be converted to the time stamp value when read select -infinity timestamp epoch timestamp infinity timestamp negative epoch positive ---------- ------------------- --------- -infinity 1970-01-01 00 00 00 infinity functions see timestamp functions time zones the timestamptz type can be binned into calendar and clock bins using a suitable extension the built in icu extension implements all the binning and arithmetic functions using the international components for unicode time zone and calendar functions to set the time zone to use first load the icu extension the icu extension comes pre-bundled with several duckdb clients including python r jdbc and odbc so this step can be skipped in those cases require icu next use the set timezone command set timezone america los_angeles time binning operations for timestamptz will then be implemented using the given time zone a list of available time zones can be pulled from the pg_timezone_names table function select name abbrev utc_offset from pg_timezone_names order by name you can also find a reference table of available time zones here calendars the icu extension also supports non-gregorian calendars using the set calendar command note that the require icu step is only required if the duckdb client does not bundle the icu extension require icu set calendar japanese time binning operations for timestamptz will then be implemented using the given calendar in this example the era part will now report the japanese imperial era number a list of available calendars can be pulled from the icu_calendar_names table function select name from icu_calendar_names order by 1 settings the current value of the timezone and calendar settings are determined by icu when it starts up they can be looked from in the duckdb_settings table function select from duckdb_settings where name timezone -- america los_angeles select from duckdb_settings where name calendar -- gregorian",
- "category": "Data Types",
- "url": "../docs/sql/data_types/timestamp",
- "blurb": "A timestamp specifies a combination of a date (year, month, day) and a time (hour, minute, second, millisecond)."
+ "title": "C API - Replacement Scans",
+ "text": "the replacement scan api can be used to register a callback that is called when a table is read that does not exist in the catalog for example when a query such as select from my_table is executed and my_table does not exist the replacement scan callback will be called with my_table as parameter the replacement scan can then insert a table function with a specific parameter to replace the read of the table api reference syntax the database object to add the replacement scan to replacement the replacement scan callback extra_data extra data that is passed back into the specified callback delete_callback the delete callback to call on the extra data if any duckdb_replacement_scan_set_function_name sets the replacement function name to use if this function is called in the replacement callback the replacement scan is performed if it is not called the replacement callback is not performed syntax the info object function_name the function name to substitute duckdb_replacement_scan_add_parameter adds a parameter to the replacement scan function syntax the info object parameter the parameter to add duckdb_replacement_scan_set_error report that an error has occurred while executing the replacement scan syntax the info object error the error message",
+ "category": "C",
+ "url": "../docs/api/c/replacement_scans",
+ "blurb": "The replacement scan API can be used to register a callback that is called when a table is read that does not exist in..."
},
{
- "title": "Text Types",
- "text": "in duckdb strings can be stored in the varchar field name aliases description --- --- --- varchar char bpchar text string variable-length character string varchar n variable-length character string with maximum length n it is possible to supply a maximum length along with the type by initializing a type as varchar n where n is a positive integer note that specifying this length is not required and specifying this length will not improve performance or reduce storage space of the strings in the database specifying a maximum length is useful only for data integrity reasons not for performance reasons in fact the following sql statements are equivalent -- the following statements are equivalent create table strings val varchar 10 -- val has a maximum length of 10 characters create table strings val varchar check length val 10 -- val has a maximum length of 10 characters the varchar field allows storage of unicode characters internally the data is encoded as utf-8 functions see character functions and pattern matching",
- "category": "Data Types",
- "url": "../docs/sql/data_types/text",
- "blurb": "In DuckDB, strings can be stored in the VARCHAR field."
+ "title": "C API - Overview",
+ "text": "duckdb implements a custom c api modelled somewhat following the sqlite c api the api is contained in the duckdb h header continue to startup shutdown to get started or check out the full api overview we also provide a sqlite api wrapper which means that if your applications is programmed against the sqlite c api you can re-link to duckdb and it should continue working see the sqlite_api_wrapper folder in our source repository for more information installation the duckdb c api can be installed as part of the libduckdb packages please see the installation page for details pages in this section",
+ "category": "C",
+ "url": "../docs/api/c/overview",
+ "blurb": "DuckDB implements a custom C API modelled somewhat following the SQLite C API. The API is contained in the duckdb.h ..."
},
{
- "title": "Blob Type",
- "text": "name aliases description --- --- --- blob bytea variable-length binary data the blob b inary l arge ob ject type represents an arbitrary binary object stored in the database system the blob type can contain any type of binary data with no restrictions what the actual bytes represent is opaque to the database system -- create a blob value with a single byte 170 select xaa blob -- create a blob value with three bytes 170 171 172 select xaa xab xac blob -- create a blob value with two bytes 65 66 select ab blob blobs are typically used to store non-textual objects that the database does not provide explicit support for such as images while blobs can hold objects up to 4gb in size typically it is not recommended to store very large objects within the database system in many situations it is better to store the large file on the file system and store the path to the file in the database system in a varchar field functions see blob functions",
- "category": "Data Types",
- "url": "../docs/sql/data_types/blob",
- "blurb": "The blob (Binary Large OBject) type represents an arbitrary binary object stored in the database system."
+ "title": "Python Client API",
+ "text": "",
+ "category": "Reference",
+ "url": "../docs/api/python/reference/index",
+ "blurb": ""
},
{
- "title": "Interval Type",
- "text": "intervals represent a period of time this period can be measured in a variety of units for example years days or seconds see the date part functions docs for a list of available date parts for use with an interval name description --- --- interval period of time an interval can be generated directly or can be a result of a function for example calculating the difference between two timestamps intervals can be used to modify date timestamp or timestamp with time zone data types see the interval operators for details -- each date part can be either singular or plural -- in this example year or years can be used interchangeably -- 1 year select interval 1 year -- 1 year select interval 1 years -- the number used to specify an interval can optionally be wrapped in single quotes -- 28 days select interval 28 days -- the number and date part can optionally be wrapped entirely in single quotes -- 28 days select interval 28 days -- intervals can also be used to specify a time period rather than a date period -- 00 00 30 select interval 30 seconds -- intervals can also be produced as a result of a timestamp operator like subtraction -- these can include a date and time component on the interval -- 1 day 01 00 00 select 2022-01-02 01 00 00 timestamp - 2022-01-01 timestamp -- warning if a decimal value is specified it will be automatically truncated to an integer -- to use more precise values simply use a more granular date part -- in this example use 18 months instead of 1 5 years -- the statement below is equivalent to to_years cast 1 5 as integer -- 1 year select interval 1 5 years --warning this returns 1 year functions see interval functions",
- "category": "Data Types",
- "url": "../docs/sql/data_types/interval",
- "blurb": "An interval specifies a period of time measured in units of a specific date part like years, days, seconds, or others."
+ "title": "Python API",
+ "text": "installation the duckdb python api can be installed using pip pip install duckdb please see the installation page for details it is also possible to install duckdb using conda conda install python-duckdb -c conda-forge basic api usage the standard duckdb python api provides a sql interface compliant with the db-api 2 0 specification described by pep 249 similar to the sqlite python api startup shutdown to use the module you must first create a connection object that represents the database the connection object takes as parameter the database file to read and write from if the database file does not exist it will be created the file extension may be db duckdb or anything else the special value memory the default can be used to create an in-memory database note that for an in-memory database no data is persisted to disk i e all data is lost when you exit the python process if you would like to connect to an existing database in read-only mode you can set the read_only flag to true read-only mode is required if multiple python processes want to access the same database file at the same time import duckdb to start an in-memory database con duckdb connect database memory to use a database file not shared between processes con duckdb connect database my-db duckdb read_only false to use a database file shared between processes con duckdb connect database my-db duckdb read_only true if you want to create a second connection to an existing database you can use the cursor method this might be useful for example to allow parallel threads running queries independently a single connection is thread-safe but is locked for the duration of the queries effectively serializing database access in this case connections are closed implicitly when they go out of scope or if they are explicitly closed using close once the last connection to a database instance is closed the database instance is closed as well querying sql queries can be sent to duckdb using the execute method of connections once a query has been executed results can be retrieved using the fetchone and fetchall methods on the connection below is a short example create a table con execute create table items item varchar value decimal 10 2 count integer insert two items into the table con execute insert into items values jeans 20 0 1 hammer 42 2 2 retrieve the items again con execute select from items print con fetchall jeans 20 0 1 hammer 42 2 2 the description property of the connection object contains the column names as per the standard duckdb also supports prepared statements in the api with the execute and executemany methods in this case the parameters are passed as an additional parameter after a query that contains placeholders here is an example insert a row using prepared statements con execute insert into items values laptop 2000 1 insert several rows using prepared statements con executemany insert into items values chainsaw 500 10 iphone 300 2 query the database using a prepared statement con execute select item from items where value 400 print con fetchall laptop chainsaw do not use executemany to insert large amounts of data into duckdb see below for better options efficient transfer transferring large datasets to and from duckdb uses a separate api built around numpy and pandas or apache arrow this api works with entire columns of data instead of scalar values and is therefore far more efficient by default duckdb will automatically be able to query a pandas dataframe or arrow object that is stored in a python variable by name duckdb supports querying multiple types of apache arrow objects including tables datasets recordbatchreaders and scanners see the python guides for more examples duckdb also supports registering a dataframe or arrow object as a virtual table comparable to a sql view this is useful when querying a dataframe arrow object that is stored in another way as a class variable or a value in a dictionary below is a pandas example import pandas as pd test_df pd dataframe from_dict i 1 2 3 4 j one two three four con execute select from test_df con fetchall 1 one 2 two 3 three 4 four if your pandas dataframe is stored in another location here is an example of manually registering it import pandas as pd my_dictionary my_dictionary test_df pd dataframe from_dict i 1 2 3 4 j one two three four con register test_df_view my_dictionary test_df con execute select from test_df_view con fetchall 1 one 2 two 3 three 4 four pyarrow tables work in much the same way import pyarrow as pa arrow_table pa table from_pydict i 1 2 3 4 j one two three four con execute select from arrow_table con fetchall you can also create a persistent table in duckdb from the contents of the dataframe or the view con execute create table test_df_table as select from test_df duckdb keeps a reference to the pandas dataframe or arrow object after registration this prevents them from being garbage-collected by the python runtime the reference is cleared when the connection is closed but can also be cleared manually using the unregister method when retrieving the data from duckdb back into python the standard method of calling fetchall is inefficient as individual python objects need to be created for every value in the result set when retrieving a lot of data this can become very slow duckdb s python client provides multiple additional methods that can be used to efficiently retrieve data numpy fetchnumpy fetches the data as a dictionary of numpy arrays pandas df fetches the data as a pandas dataframe fetchdf is an alias of df fetch_df is an alias of df fetch_df_chunk vector_multiple fetches a portion of the results into a dataframe the number of rows returned in each chunk is the vector size 1024 by default vector_multiple 1 by default apache arrow arrow fetches the data as an arrow table fetch_arrow_table is an alias of arrow fetch_record_batch chunk_size returns an arrow record batch reader with chunk_size rows per batch below are some examples using this functionality see the python guides for more examples fetch as pandas dataframe df con execute select from items fetchdf print df item value count 0 jeans 20 0 1 1 hammer 42 2 2 2 laptop 2000 0 1 3 chainsaw 500 0 10 4 iphone 300 0 2 fetch as dictionary of numpy arrays arr con execute select from items fetchnumpy print arr item masked_array data jeans hammer laptop chainsaw iphone mask false false false false false fill_value dtype object value masked_array data 20 0 42 2 2000 0 500 0 300 0 mask false false false false false fill_value 1e 20 count masked_array data 1 2 1 10 2 mask false false false false false fill_value 999999 dtype int32 fetch as an arrow table converting to pandas afterwards just for pretty printing tbl con execute select from items fetch_arrow_table print tbl to_pandas item value count 0 jeans 20 00 1 1 hammer 42 20 2 2 laptop 2000 00 1 3 chainsaw 500 00 10 4 iphone 300 00 2 also refer to the data import documentation for more options of efficiently importing data relational api in addition to the sql api duckdb supports a programmatic method to construct queries see https github com duckdb duckdb blob master examples python duckdb-python py for an example",
+ "category": "Python",
+ "url": "../docs/api/python/overview",
+ "blurb": "Installation The DuckDB Python API can be installed using pip : pip install duckdb . Please see the installation..."
},
{
- "title": "Struct",
- "text": "struct data type conceptually a struct column contains an ordered list of other columns called entries the entries are referenced by name using strings this document refers to those entry names as keys each row in the struct column must have the same keys each key must have the same type of value for each row struct s are typically used to nest multiple columns into a single column and the nested column can be of any type including other struct s and list s struct s are similar to postgres s row type the key difference is that duckdb struct s require the same keys in each row of a struct column this allows duckdb to provide significantly improved performance by fully utilizing its vectorized execution engine and also enforces type consistency for improved correctness duckdb includes a row function as a special way to produce a struct but does not have a row data type see an example below and the nested functions docs for details see the data types overview for a comparison between nested data types structs can be created using the struct_pack name expr function or the equivalent array notation name expr notation the expressions can be constants or arbitrary expressions creating structs -- struct of integers select x 1 y 2 z 3 -- struct of strings with a null value select yes duck maybe goose huh null no heron -- struct with a different type for each key select key1 string key2 1 key3 12 345 -- struct using the struct_pack function -- note the lack of single quotes around the keys and the use of the operator select struct_pack key1 value1 key2 42 -- struct of structs with null values select birds yes duck maybe goose huh null no heron aliens null amphibians yes frog maybe salamander huh dragon no toad -- create a struct from columns and or expressions using the row function -- this returns x 1 v2 2 y a select row x x 1 y from select 1 as x a as y -- if using multiple expressions when creating a struct the row function is optional -- this also returns x 1 v2 2 y a select x x 1 y from select 1 as x a as y adding field s value s to structs -- add to a struct of integers select struct_insert a 1 b 2 c 3 d 4 retrieving from structs retrieving a value from a struct can be accomplished using dot notation bracket notation or through struct functions like struct_extract -- use dot notation to retrieve the value at a key s location this returns 1 -- the subquery generates a struct column a which we then query with a x select a x from select x 1 y 2 z 3 as a -- if key contains a space simply wrap it in double quotes this returns 1 -- note use double quotes not single quotes -- this is because this action is most similar to selecting a column from within the struct select a x space from select x space 1 y 2 z 3 as a -- bracket notation may also be used this returns 1 -- note use single quotes since the goal is to specify a certain string key -- only constant expressions may be used inside the brackets no columns select a x space from select x space 1 y 2 z 3 as a -- the struct_extract function is also equivalent this returns 1 select struct_extract x space 1 y 2 z 3 x space referring to structs with dot notation can be ambiguous with referring to schemas and tables in general duckdb looks for columns first then for struct keys within columns duckdb resolves references in these orders using the first match to occur no dots select part1 from tbl part1 is a column one dot select part1 part2 from tbl part1 is a table part2 is a column part1 is a column part2 is a property of that column two or more dots select part1 part2 part3 from tbl part1 is a schema part2 is a table part3 is a column part1 is a table part2 is a column part3 is a property of that column part1 is a column part2 is a property of that column part3 is a property of that column any extra parts e g part4 part5 etc are always treated as properties creating structs with the row function the row function can be used to automatically convert multiple columns to a single struct column the name of each input column is used as a key and the value of each column becomes the struct s value at that key when converting multiple expressions into a struct the row function name is optional - a set of parenthesis is all that is needed example data table named t1 my_column another_column --- --- 1 a 2 b row function example select row my_column another_column as my_struct_column my_column another_column as identical_struct_column from t1 example output my_struct_column identical_struct_column --- --- my_column 1 another_column a my_column 1 another_column a my_column 2 another_column b my_column 2 another_column b the row function or simplified parenthesis syntax may also be used with arbitrary expressions as input rather than column names in the case of an expression a key will be automatically generated in the format of vn where n is a number that refers to its parameter location in the row function ex v1 v2 etc this can be combined with column names as an input in the same call to the row function this example uses the same input table as above row function example with a column name a constant and an expression as input select row my_column 42 my_column 1 as my_struct_column my_column 42 my_column 1 as identical_struct_column from t1 example output my_struct_column identical_struct_column --- --- my_column 1 v2 42 v3 2 my_column 1 v2 42 v3 2 my_column 2 v2 42 v3 3 my_column 2 v2 42 v3 3 comparison operators nested types can be compared using all the comparison operators these comparisons can be used in logical expressions for both where and having clauses as well as for creating boolean values the ordering is defined positionally in the same way that words can be ordered in a dictionary null values compare greater than all other values and are considered equal to each other at the top level null nested values obey standard sql null comparison rules comparing a null nested value to a non- null nested value produces a null result comparing nested value members however uses the internal nested value rules for null s and a null nested value member will compare above a non- null nested value member functions see nested functions",
- "category": "Data Types",
- "url": "../docs/sql/data_types/struct",
- "blurb": "Struct Data Type Conceptually, a STRUCT column contains an ordered list of other columns called entries. The entries..."
+ "title": "C++ API",
+ "text": "installation the duckdb c api can be installed as part of the libduckdb packages please see the installation page for details basic api usage duckdb implements a custom c api this is built around the abstractions of a database instance duckdb class multiple connection s to the database instance and queryresult instances as the result of queries the header file for the c api is duckdb hpp the standard source distribution of libduckdb contains an amalgamation of the duckdb sources which combine all sources into two files duckdb hpp and duckdb cpp the duckdb hpp header is much larger in this case regardless of whether you are using the amalgamation or not just include duckdb hpp startup shutdown to use duckdb you must first initialize a duckdb instance using its constructor duckdb takes as parameter the database file to read and write from the special value nullptr can be used to create an in-memory database note that for an in-memory database no data is persisted to disk i e all data is lost when you exit the process the second parameter to the duckdb constructor is an optional dbconfig object in dbconfig you can set various database parameters for example the read write mode or memory limits the duckdb constructor may throw exceptions for example if the database file is not usable with the duckdb instance you can create one or many connection instances using the connection constructor while connections should be thread-safe they will be locked during querying it is therefore recommended that each thread uses its own connection if you are in a multithreaded environment duckdb db nullptr connection con db querying connections expose the query method to send a sql query string to duckdb from c query fully materializes the query result as a materializedqueryresult in memory before returning at which point the query result can be consumed there is also a streaming api for queries see further below create a table con query create table integers i integer j integer insert three rows into the table con query insert into integers values 3 4 5 6 7 null materializedqueryresult result con query select from integers if result- success cerr result- error the materializedqueryresult instance contains firstly two fields that indicate whether the query was successful query will not throw exceptions under normal circumstances instead invalid queries or other issues will lead to the success boolean field in the query result instance to be set to false in this case an error message may be available in error as a string if successful other fields are set the type of statement that was just executed e g statementtype insert_statement is contained in statement_type the high-level sql type types of the result set columns are in sql_types and the low-level data representation types are in types the names of the result columns are in the names string vector in case multiple result sets are returned for example because the result set contained multiple statements the result set can be chained using the next field todo duckdb also supports prepared statements in the c api with the prepare method this returns an instance of preparedstatement this instance can be used to execute the prepared statement with parameters below is an example std unique_ptr preparedstatement prepare con prepare select count from a where i 1 std unique_ptr queryresult result prepare- execute 12 do not use prepared statements to insert large amounts of data into duckdb see the data import documentation for better options streaming queries udf api the udf api is exposed in duckdb connection through the methods createscalarfunction and createvectorizedfunction and variants these methods created udfs into the temporary schema temp_schema of the owner connection that is the only one allowed to use and change them createscalarfunction the user can code an ordinary scalar function and invoke the createscalarfunction to register and afterward use the udf in a select statement for instance bool bigger_than_four int value return value 4 connection createscalarfunction bool int bigger_than_four bigger_than_four connection query select bigger_than_four i from values 3 5 tbl i - print the createscalarfunction methods automatically creates vectorized scalar udfs so they are as efficient as built-in functions we have two variants of this method interface as follows 1 template typename tr typename args nbsp nbsp nbsp void createscalarfunction string name tr udf_func args template parameters tr is the return type of the udf function args are the arguments up to 3 for the udf function this method only supports until ternary functions name is the name to register the udf function udf_func is a pointer to the udf function this method automatically discovers from the template typenames the corresponding sqltypes bool sqltype boolean int8_t sqltype tinyint int16_t sqltype smallint int32_t sqltype integer int64_t sqltype bigint float sqltype float double sqltype double string_t sqltype varchar in duckdb some primitive types e g int32_t are mapped to the same sqltype integer time and date then for disambiguation the users can use the following overloaded method 2 template typename tr typename args nbsp nbsp nbsp void createscalarfunction string name vector sqltype args sqltype ret_type tr udf_func args an example of use would be int32_t udf_date int32_t a return a con query create table dates d date con query insert into dates values 1992-01-01 con createscalarfunction int32_t int32_t udf_date sqltype date sqltype date udf_date con query select udf_date d from dates - print template parameters tr is the return type of the udf function args are the arguments up to 3 for the udf function this method only supports until ternary functions name is the name to register the udf function args are the sqltype arguments that the function uses which should match with the template args types ret_type is the sqltype of return of the function which should match with the template tr type udf_func is a pointer to the udf function this function checks the template types against the sqltypes passed as arguments and they must match as follow sqltypeid boolean bool sqltypeid tinyint int8_t sqltypeid smallint int16_t sqltypeid date sqltypeid time sqltypeid integer int32_t sqltypeid bigint sqltypeid timestamp int64_t sqltypeid float sqltypeid double sqltypeid decimal double sqltypeid varchar sqltypeid char sqltypeid blob string_t sqltypeid varbinary blob_t createvectorizedfunction the createvectorizedfunction methods register a vectorized udf such as this vectorized function copies the input values to the result vector template typename type static void udf_vectorized datachunk args expressionstate state vector result set the result vector type result vector_type vectortype flat_vector get a raw array from the result auto result_data flatvector getdata type result get the solely input vector auto input args data 0 now get an orrified vector vectordata vdata input orrify args size vdata get a raw array from the orrified input auto input_data type vdata data handling the data for idx_t i 0 i args size i auto idx vdata sel- get_index i if vdata nullmask idx continue result_data i input_data idx con query create table integers i integer con query insert into integers values 1 2 3 999 con createvectorizedfunction int int udf_vectorized_int udf_vectorized int con query select udf_vectorized_int i from integers - print the vectorized udf is a pointer of the type scalar_function_t typedef std function void datachunk args expressionstate expr vector result scalar_function_t args is a datachunk that holds a set of input vectors for the udf that all have the same length expr is an expressionstate that provides information to the query s expression state result is a vector to store the result values there are different vector types to handle in a vectorized udf constantvector dictionaryvector flatvector listvector stringvector structvector sequencevector the general api of the createvectorizedfunction method is as follows 1 template typename tr typename args nbsp nbsp nbsp void createvectorizedfunction string name scalar_function_t udf_func sqltype varargs sqltype invalid template parameters tr is the return type of the udf function args are the arguments up to 3 for the udf function name is the name to register the udf function udf_func is a vectorized udf function varargs the type of varargs to support or sqltypeid invalid default value if the function does not accept variable length arguments this method automatically discovers from the template typenames the corresponding sqltypes bool sqltype boolean int8_t sqltype tinyint int16_t sqltype smallint int32_t sqltype integer int64_t sqltype bigint float sqltype float double sqltype double string_t sqltype varchar 2 template typename tr typename args nbsp nbsp nbsp void createvectorizedfunction string name vector sqltype args sqltype ret_type scalar_function_t udf_func sqltype varargs sqltype invalid todo",
+ "category": "Api",
+ "url": "../docs/api/cpp",
+ "blurb": "Installation The DuckDB C++ API can be installed as part of the libduckdb packages. Please see the installation page..."
},
{
- "title": "Date Types",
- "text": "name aliases description ------- -------- -------------------------------- date calendar date year month day a date specifies a combination of year month and day duckdb follows the sql standard s lead by counting dates exclusively in the gregorian calendar even for years before that calendar was in use dates can be created using the date keyword where the data must be formatted according to the iso 8601 format yyyy-mm-dd -- 20 september 1992 select date 1992-09-20 special values there are also three special date values that can be used on input input string description ------------- ---------------------------------- epoch 1970-01-01 unix system day zero infinity later than all other dates -infinity earlier than all other dates the values infinity and -infinity are specially represented inside the system and will be displayed unchanged but epoch is simply a notational shorthand that will be converted to the date value when read select -infinity date epoch date infinity date negative epoch positive ---------- ----------- --------- -infinity 1970-01-01 infinity functions see date functions",
- "category": "Data Types",
- "url": "../docs/sql/data_types/date",
- "blurb": "A date specifies a combination of year, month and day."
+ "title": "ODBC API - Linux",
+ "text": "driver manager unixodbc a driver manager is required to manage communication between applications and the odbc driver we tested and support the unixodbc that is a complete odbc driver manager for linux users can install it from the command line debian so flavors sudo apt get install unixodbc fedora so flavors sudo yum install unixodbc or sudo dnf install unixodbc step 1 download odbc driver duckdb releases the odbc driver as asset for linux download it from a href https github com duckdb duckdb releases download v site currentduckdbversion duckdb_odbc-linux-amd64 zip odbc linux asset a that contains the following artifacts libduckdb_odbc so the duckdb driver compiled to ubuntu 16 04 unixodbc_setup sh a setup script to aid the configuration on linux step 2 extracting odbc artifacts run unzip to extract the files to a permanent directory mkdir duckdb_odbc unzip duckdb_odbc-linux-amd64 zip -d duckdb_odbc step 3 configuring with unixodbc the unixodbc_setup sh script aids the configuration of the duckdb odbc driver it is based on the unixodbc package that provides some commands to handle the odbc setup and test like odbcinst and isql in a terminal window change to the duckdb_odbc permanent directory and run the following commands with level options -u or -s either to configure duckdb odbc user-level odbc setup -u the -u option based on the user home directory to setup the odbc init files unixodbc_setup sh -u p s the default configuration consists of a database memory system-level odbc setup -s the -s changes the system level files that will be visible for all users because of that it requires root privileges sudo unixodbc_setup sh -s p s the default configuration consists of a database memory show usage --help the option --help shows the usage of unixodbc_setup sh that provides alternative options for a customer configuration like -db and -d unixodbc_setup sh --help usage unixodbc_setup sh level options example unixodbc_setup sh -u -db database_path -d driver_path libduckdb_odbc so level -s system-level using sudo to configure duckdb odbc at the system-level changing the files etc odbc inst ini -u user-level configuring the duckdb odbc at the user-level changing the files odbc inst ini options -db database_path the duckdb database file path the default is memory if not provided -d driver_path the driver file path i e the path for libduckdb_odbc so the default is using the base script directory step 4 optional configure the odbc driver the odbc setup on linux is based on files the well-known odbc ini and odbcinst ini these files can be placed at the system etc directory or at the user home directory home user shortcut as the dm prioritizes the user configuration files and then the system files the odbc ini file the odbc ini contains the dsns for the drivers which can have specific knobs an example of odbc ini with duckdb would be duckdb driver duckdb driver database memory duckdb between the brackets is a dsn for the duckdb driver it describes the driver s name and other configurations will be placed at the odbcinst ini database it describes the database name used by duckdb and it can also be a file path to a db in the system the odbcinst ini file the odbcinst ini contains general configurations for the odbc installed drivers in the system a driver section starts with the driver name between brackets and then it follows specific configuration knobs belonging to that driver an example of odbcinst ini with the duckdb driver would be odbc trace yes tracefile tmp odbctrace duckdb driver driver home user duckdb_odbc libduckdb_odbc so odbc it is the dm configuration section trace it enables the odbc trace file using the option yes tracefile the absolute system file path for the odbc trace file duckdb driver the section of the duckdb installed driver driver the absolute system file path of the duckdb driver",
+ "category": "Odbc",
+ "url": "../docs/api/odbc/linux",
+ "blurb": "Driver Manager: unixODBC A driver manager is required to manage communication between applications and the ODBC driver...."
},
{
- "title": "Map",
- "text": "map data type map s are similar to struct s in that they are an ordered list of entries where a key maps to a value however map s do not need to have the same keys present on each row and thus open additional use cases map s are useful when the schema is unknown beforehand and when adding or removing keys in subsequent rows their flexibility is a key differentiator map s must have a single type for all keys and a single type for all values keys and values can be any type and the type of the keys does not need to match the type of the values ex a map of int s to varchar s map s may also have duplicate keys this is possible and useful because maps are ordered map s are also more forgiving when extracting values as they return an empty list if a key is not found rather than throwing an error as structs do in contrast struct s must have string keys but each key may have a value of a different type struct s may not have duplicate keys see the data types overview for a comparison between nested data types to construct a map use the map function provide a list of keys as the first parameter and a list of values for the second creating maps -- a map with integer keys and varchar values this returns 1 a 5 e select map 1 5 a e -- a map with integer keys and numeric values this returns 1 42 001 5 -32 100 select map 1 5 42 001 -32 1 -- keys and or values can also be nested types -- this returns a b 1 1 2 2 c d 3 3 4 4 select map a b c d 1 1 2 2 3 3 4 4 -- create a table with a map column that has integer keys and double values create table map_table map_col map int double retrieving from maps map s use bracket notation for retrieving values this is due to the variety of types that can be used as a map s key selecting from a map also returns a list rather than an individual value -- use bracket notation to retrieve a list containing the value at a key s location this returns 42 -- note that the expression in bracket notation must match the type of the map s key select map 100 5 42 43 100 -- to retrieve the underlying value use list selection syntax to grab the 0th element -- this returns 42 select map 100 5 42 43 100 0 -- if the element is not in the map an empty list will be returned returns -- note that the expression in bracket notation must match the type of the map s key else an error is returned select map 100 5 42 43 123 -- the element_at function can also be used to retrieve a map value this returns 42 select element_at map 100 5 42 43 100 comparison operators nested types can be compared using all the comparison operators these comparisons can be used in logical expressions for both where and having clauses as well as for creating boolean values the ordering is defined positionally in the same way that words can be ordered in a dictionary null values compare greater than all other values and are considered equal to each other at the top level null nested values obey standard sql null comparison rules comparing a null nested value to a non- null nested value produces a null result comparing nested value members however uses the internal nested value rules for null s and a null nested value member will compare above a non- null nested value member functions see nested functions",
- "category": "Data Types",
- "url": "../docs/sql/data_types/map",
- "blurb": "Map Data Type MAP s are similar to STRUCT s in that they are an ordered list of entries where a key maps to a value...."
+ "title": "ODBC API - Windows",
+ "text": "windows odbc driver manager odbccp32 dll the microsoft windows requires an odbc driver manager to manage communication between applications and the odbc drivers the dm on windows is provided in a dll file odbccp32 dll and other files and tools for detailed information checkout out the common odbc component files step 1 download odbc driver duckdb releases the odbc driver as asset for windows download it from a href https github com duckdb duckdb releases download v site currentduckdbversion duckdb_odbc-windows-amd64 zip windows asset a that contains the following artifacts duckdb_odbc dll the duckdb driver compiled for windows duckdb_odbc_setup dll a setup dll used by the windows odbc data source administrator tool odbc_install exe a installation script to aid the configuration on windows step 2 extracting odbc artifacts unzip the file to a permanent directory e g duckdb_odbc an example with powershell and unzip command would be mkdir duckdb_odbc unzip duckdb_odbc-linux-amd64 zip -d duckdb_odbc step 3 odbc windows installer the odbc_install exe aids the configuration of the duckdb odbc driver on windows it depends on the odbccp32 dll that provides functions to configure the odbc registry entries inside the permanent directory e g duckdb_odbc double-click on the odbc_install exe windows administrator privileges is required in case of a non-administrator a user account control shall display step 4 configure the odbc driver the odbc_install exe adds a default dsn configuration into the odbc registries with a default database memory dsn windows setup after the installation it is possible to change the default dsn configuration or add a new one using the windows odbc data source administrator tool odbcad32 exe it also can be launched thought the windows start default duckdb dsn in the the windows odbc data source administrator tool at system dsn tab is placed the default installed dsn for duckdb windows odbc config tool changing duckdb dsn selecting the default dsn i e duckdb or add a new configuration the following setup window will display duckdb windows dsn setup for now it is possible to set the dsn and the database file path associated with that dsn more detailed windows setup the odbc setup on windows is based on registry keys see registry entries for odbc components the odbc entries can be placed at the current user registry key hkcu or the system registry key hklm we have tested and used the system entries based on hklm- software- odbc the odbc_install exe changes this entry that has two subkeys odbc ini and odbcinst ini the odbc ini is where users usually insert dsn registry entries for the drivers for example the dsn registry for duckdb would look like this hklm- software- odbc- odbc ini- duckdb the odbcinst ini contains one entry for each odbc driver and other keys predefined for windows odbc configuration",
+ "category": "Odbc",
+ "url": "../docs/api/odbc/windows",
+ "blurb": "Windows ODBC Driver Manager: Odbccp32.dll The Microsoft Windows requires an ODBC Driver Manager to manage communication..."
},
{
- "title": "Time Zones",
- "text": "time zone reference list an up-to-date version of this list can be pulled from the pg_timezone_names table function select name abbrev utc_offset from pg_timezone_names order by name name abbrev utc_offset ---------------------------------- ---------------------------------- ------------ act act 09 30 00 aet aet 10 00 00 agt agt -03 00 00 art art 02 00 00 ast ast -09 00 00 africa abidjan africa abidjan 00 00 00 africa accra africa accra 00 00 00 africa addis_ababa eat 03 00 00 africa algiers africa algiers 01 00 00 africa asmara eat 03 00 00 africa asmera eat 03 00 00 africa bamako africa bamako 00 00 00 africa bangui africa bangui 01 00 00 africa banjul africa banjul 00 00 00 africa bissau africa bissau 00 00 00 africa blantyre cat 02 00 00 africa brazzaville africa brazzaville 01 00 00 africa bujumbura cat 02 00 00 africa cairo art 02 00 00 africa casablanca africa casablanca 00 00 00 africa ceuta africa ceuta 01 00 00 africa conakry africa conakry 00 00 00 africa dakar africa dakar 00 00 00 africa dar_es_salaam eat 03 00 00 africa djibouti eat 03 00 00 africa douala africa douala 01 00 00 africa el_aaiun africa el_aaiun 00 00 00 africa freetown africa freetown 00 00 00 africa gaborone cat 02 00 00 africa harare cat 02 00 00 africa johannesburg africa johannesburg 02 00 00 africa juba africa juba 03 00 00 africa kampala eat 03 00 00 africa khartoum africa khartoum 02 00 00 africa kigali cat 02 00 00 africa kinshasa africa kinshasa 01 00 00 africa lagos africa lagos 01 00 00 africa libreville africa libreville 01 00 00 africa lome africa lome 00 00 00 africa luanda africa luanda 01 00 00 africa lubumbashi cat 02 00 00 africa lusaka cat 02 00 00 africa malabo africa malabo 01 00 00 africa maputo cat 02 00 00 africa maseru africa maseru 02 00 00 africa mbabane africa mbabane 02 00 00 africa mogadishu eat 03 00 00 africa monrovia africa monrovia 00 00 00 africa nairobi eat 03 00 00 africa ndjamena africa ndjamena 01 00 00 africa niamey africa niamey 01 00 00 africa nouakchott africa nouakchott 00 00 00 africa ouagadougou africa ouagadougou 00 00 00 africa porto-novo africa porto-novo 01 00 00 africa sao_tome africa sao_tome 00 00 00 africa timbuktu africa timbuktu 00 00 00 africa tripoli libya 02 00 00 africa tunis africa tunis 01 00 00 africa windhoek africa windhoek 02 00 00 america adak america adak -10 00 00 america anchorage ast -09 00 00 america anguilla america anguilla -04 00 00 america antigua america antigua -04 00 00 america araguaina america araguaina -03 00 00 america argentina buenos_aires agt -03 00 00 america argentina catamarca america argentina catamarca -03 00 00 america argentina comodrivadavia america argentina comodrivadavia -03 00 00 america argentina cordoba america argentina cordoba -03 00 00 america argentina jujuy america argentina jujuy -03 00 00 america argentina la_rioja america argentina la_rioja -03 00 00 america argentina mendoza america argentina mendoza -03 00 00 america argentina rio_gallegos america argentina rio_gallegos -03 00 00 america argentina salta america argentina salta -03 00 00 america argentina san_juan america argentina san_juan -03 00 00 america argentina san_luis america argentina san_luis -03 00 00 america argentina tucuman america argentina tucuman -03 00 00 america argentina ushuaia america argentina ushuaia -03 00 00 america aruba america aruba -04 00 00 america asuncion america asuncion -04 00 00 america atikokan america atikokan -05 00 00 america atka america atka -10 00 00 america bahia america bahia -03 00 00 america bahia_banderas america bahia_banderas -06 00 00 america barbados america barbados -04 00 00 america belem america belem -03 00 00 america belize america belize -06 00 00 america blanc-sablon america blanc-sablon -04 00 00 america boa_vista america boa_vista -04 00 00 america bogota america bogota -05 00 00 america boise america boise -07 00 00 america buenos_aires agt -03 00 00 america cambridge_bay america cambridge_bay -07 00 00 america campo_grande america campo_grande -04 00 00 america cancun america cancun -05 00 00 america caracas america caracas -04 00 00 america catamarca america catamarca -03 00 00 america cayenne america cayenne -03 00 00 america cayman america cayman -05 00 00 america chicago cst -06 00 00 america chihuahua america chihuahua -07 00 00 america coral_harbour america coral_harbour -05 00 00 america cordoba america cordoba -03 00 00 america costa_rica america costa_rica -06 00 00 america creston america creston -07 00 00 america cuiaba america cuiaba -04 00 00 america curacao america curacao -04 00 00 america danmarkshavn america danmarkshavn 00 00 00 america dawson america dawson -08 00 00 america dawson_creek america dawson_creek -07 00 00 america denver navajo -07 00 00 america detroit america detroit -05 00 00 america dominica america dominica -04 00 00 america edmonton america edmonton -07 00 00 america eirunepe america eirunepe -05 00 00 america el_salvador america el_salvador -06 00 00 america ensenada america ensenada -08 00 00 america fort_nelson america fort_nelson -07 00 00 america fort_wayne iet -05 00 00 america fortaleza america fortaleza -03 00 00 america glace_bay america glace_bay -04 00 00 america godthab america godthab -03 00 00 america goose_bay america goose_bay -04 00 00 america grand_turk america grand_turk -05 00 00 america grenada america grenada -04 00 00 america guadeloupe america guadeloupe -04 00 00 america guatemala america guatemala -06 00 00 america guayaquil america guayaquil -05 00 00 america guyana america guyana -04 00 00 america halifax america halifax -04 00 00 america havana cuba -05 00 00 america hermosillo america hermosillo -07 00 00 america indiana indianapolis iet -05 00 00 america indiana knox america indiana knox -06 00 00 america indiana marengo america indiana marengo -05 00 00 america indiana petersburg america indiana petersburg -05 00 00 america indiana tell_city america indiana tell_city -06 00 00 america indiana vevay america indiana vevay -05 00 00 america indiana vincennes america indiana vincennes -05 00 00 america indiana winamac america indiana winamac -05 00 00 america indianapolis iet -05 00 00 america inuvik america inuvik -07 00 00 america iqaluit america iqaluit -05 00 00 america jamaica jamaica -05 00 00 america jujuy america jujuy -03 00 00 america juneau america juneau -09 00 00 america kentucky louisville america kentucky louisville -05 00 00 america kentucky monticello america kentucky monticello -05 00 00 america knox_in america knox_in -06 00 00 america kralendijk america kralendijk -04 00 00 america la_paz america la_paz -04 00 00 america lima america lima -05 00 00 america los_angeles pst -08 00 00 america louisville america louisville -05 00 00 america lower_princes america lower_princes -04 00 00 america maceio america maceio -03 00 00 america managua america managua -06 00 00 america manaus america manaus -04 00 00 america marigot america marigot -04 00 00 america martinique america martinique -04 00 00 america matamoros america matamoros -06 00 00 america mazatlan america mazatlan -07 00 00 america mendoza america mendoza -03 00 00 america menominee america menominee -06 00 00 america merida america merida -06 00 00 america metlakatla america metlakatla -09 00 00 america mexico_city america mexico_city -06 00 00 america miquelon america miquelon -03 00 00 america moncton america moncton -04 00 00 america monterrey america monterrey -06 00 00 america montevideo america montevideo -03 00 00 america montreal america montreal -05 00 00 america montserrat america montserrat -04 00 00 america nassau america nassau -05 00 00 america new_york america new_york -05 00 00 america nipigon america nipigon -05 00 00 america nome america nome -09 00 00 america noronha america noronha -02 00 00 america north_dakota beulah america north_dakota beulah -06 00 00 america north_dakota center america north_dakota center -06 00 00 america north_dakota new_salem america north_dakota new_salem -06 00 00 america ojinaga america ojinaga -07 00 00 america panama america panama -05 00 00 america pangnirtung america pangnirtung -05 00 00 america paramaribo america paramaribo -03 00 00 america phoenix pnt -07 00 00 america port-au-prince america port-au-prince -05 00 00 america port_of_spain america port_of_spain -04 00 00 america porto_acre america porto_acre -05 00 00 america porto_velho america porto_velho -04 00 00 america puerto_rico prt -04 00 00 america punta_arenas america punta_arenas -03 00 00 america rainy_river america rainy_river -06 00 00 america rankin_inlet america rankin_inlet -06 00 00 america recife america recife -03 00 00 america regina america regina -06 00 00 america resolute america resolute -06 00 00 america rio_branco america rio_branco -05 00 00 america rosario america rosario -03 00 00 america santa_isabel america santa_isabel -08 00 00 america santarem america santarem -03 00 00 america santiago america santiago -04 00 00 america santo_domingo america santo_domingo -04 00 00 america sao_paulo bet -03 00 00 america scoresbysund america scoresbysund -01 00 00 america shiprock navajo -07 00 00 america sitka america sitka -09 00 00 america st_barthelemy america st_barthelemy -04 00 00 america st_johns cnt -03 30 00 america st_kitts america st_kitts -04 00 00 america st_lucia america st_lucia -04 00 00 america st_thomas america st_thomas -04 00 00 america st_vincent america st_vincent -04 00 00 america swift_current america swift_current -06 00 00 america tegucigalpa america tegucigalpa -06 00 00 america thule america thule -04 00 00 america thunder_bay america thunder_bay -05 00 00 america tijuana america tijuana -08 00 00 america toronto america toronto -05 00 00 america tortola america tortola -04 00 00 america vancouver america vancouver -08 00 00 america virgin america virgin -04 00 00 america whitehorse america whitehorse -08 00 00 america winnipeg america winnipeg -06 00 00 america yakutat america yakutat -09 00 00 america yellowknife america yellowknife -07 00 00 antarctica casey antarctica casey 08 00 00 antarctica davis antarctica davis 07 00 00 antarctica dumontdurville antarctica dumontdurville 10 00 00 antarctica macquarie antarctica macquarie 11 00 00 antarctica mawson antarctica mawson 05 00 00 antarctica mcmurdo nst 12 00 00 antarctica palmer antarctica palmer -03 00 00 antarctica rothera antarctica rothera -03 00 00 antarctica south_pole nst 12 00 00 antarctica syowa antarctica syowa 03 00 00 antarctica troll antarctica troll 00 00 00 antarctica vostok antarctica vostok 06 00 00 arctic longyearbyen arctic longyearbyen 01 00 00 asia aden asia aden 03 00 00 asia almaty asia almaty 06 00 00 asia amman asia amman 02 00 00 asia anadyr asia anadyr 12 00 00 asia aqtau asia aqtau 05 00 00 asia aqtobe asia aqtobe 05 00 00 asia ashgabat asia ashgabat 05 00 00 asia ashkhabad asia ashkhabad 05 00 00 asia atyrau asia atyrau 05 00 00 asia baghdad asia baghdad 03 00 00 asia bahrain asia bahrain 03 00 00 asia baku asia baku 04 00 00 asia bangkok asia bangkok 07 00 00 asia barnaul asia barnaul 07 00 00 asia beirut asia beirut 02 00 00 asia bishkek asia bishkek 06 00 00 asia brunei asia brunei 08 00 00 asia calcutta ist 05 30 00 asia chita asia chita 09 00 00 asia choibalsan asia choibalsan 08 00 00 asia chongqing ctt 08 00 00 asia chungking ctt 08 00 00 asia colombo asia colombo 05 30 00 asia dacca bst 06 00 00 asia damascus asia damascus 02 00 00 asia dhaka bst 06 00 00 asia dili asia dili 09 00 00 asia dubai asia dubai 04 00 00 asia dushanbe asia dushanbe 05 00 00 asia famagusta asia famagusta 02 00 00 asia gaza asia gaza 02 00 00 asia harbin ctt 08 00 00 asia hebron asia hebron 02 00 00 asia ho_chi_minh vst 07 00 00 asia hong_kong hongkong 08 00 00 asia hovd asia hovd 07 00 00 asia irkutsk asia irkutsk 08 00 00 asia istanbul turkey 03 00 00 asia jakarta asia jakarta 07 00 00 asia jayapura asia jayapura 09 00 00 asia jerusalem israel 02 00 00 asia kabul asia kabul 04 30 00 asia kamchatka asia kamchatka 12 00 00 asia karachi plt 05 00 00 asia kashgar asia kashgar 06 00 00 asia kathmandu asia kathmandu 05 45 00 asia katmandu asia katmandu 05 45 00 asia khandyga asia khandyga 09 00 00 asia kolkata ist 05 30 00 asia krasnoyarsk asia krasnoyarsk 07 00 00 asia kuala_lumpur asia kuala_lumpur 08 00 00 asia kuching asia kuching 08 00 00 asia kuwait asia kuwait 03 00 00 asia macao asia macao 08 00 00 asia macau asia macau 08 00 00 asia magadan asia magadan 11 00 00 asia makassar asia makassar 08 00 00 asia manila asia manila 08 00 00 asia muscat asia muscat 04 00 00 asia nicosia asia nicosia 02 00 00 asia novokuznetsk asia novokuznetsk 07 00 00 asia novosibirsk asia novosibirsk 07 00 00 asia omsk asia omsk 06 00 00 asia oral asia oral 05 00 00 asia phnom_penh asia phnom_penh 07 00 00 asia pontianak asia pontianak 07 00 00 asia pyongyang asia pyongyang 09 00 00 asia qatar asia qatar 03 00 00 asia qostanay asia qostanay 06 00 00 asia qyzylorda asia qyzylorda 05 00 00 asia rangoon asia rangoon 06 30 00 asia riyadh asia riyadh 03 00 00 asia saigon vst 07 00 00 asia sakhalin asia sakhalin 11 00 00 asia samarkand asia samarkand 05 00 00 asia seoul rok 09 00 00 asia shanghai ctt 08 00 00 asia singapore singapore 08 00 00 asia srednekolymsk asia srednekolymsk 11 00 00 asia taipei roc 08 00 00 asia tashkent asia tashkent 05 00 00 asia tbilisi asia tbilisi 04 00 00 asia tehran iran 03 30 00 asia tel_aviv israel 02 00 00 asia thimbu asia thimbu 06 00 00 asia thimphu asia thimphu 06 00 00 asia tokyo jst 09 00 00 asia tomsk asia tomsk 07 00 00 asia ujung_pandang asia ujung_pandang 08 00 00 asia ulaanbaatar asia ulaanbaatar 08 00 00 asia ulan_bator asia ulan_bator 08 00 00 asia urumqi asia urumqi 06 00 00 asia ust-nera asia ust-nera 10 00 00 asia vientiane asia vientiane 07 00 00 asia vladivostok asia vladivostok 10 00 00 asia yakutsk asia yakutsk 09 00 00 asia yangon asia yangon 06 30 00 asia yekaterinburg asia yekaterinburg 05 00 00 asia yerevan net 04 00 00 atlantic azores atlantic azores -01 00 00 atlantic bermuda atlantic bermuda -04 00 00 atlantic canary atlantic canary 00 00 00 atlantic cape_verde atlantic cape_verde -01 00 00 atlantic faeroe atlantic faeroe 00 00 00 atlantic faroe atlantic faroe 00 00 00 atlantic jan_mayen atlantic jan_mayen 01 00 00 atlantic madeira atlantic madeira 00 00 00 atlantic reykjavik iceland 00 00 00 atlantic south_georgia atlantic south_georgia -02 00 00 atlantic st_helena atlantic st_helena 00 00 00 atlantic stanley atlantic stanley -03 00 00 australia act aet 10 00 00 australia adelaide australia adelaide 09 30 00 australia brisbane australia brisbane 10 00 00 australia broken_hill australia broken_hill 09 30 00 australia canberra aet 10 00 00 australia currie australia currie 10 00 00 australia darwin act 09 30 00 australia eucla australia eucla 08 45 00 australia hobart australia hobart 10 00 00 australia lhi australia lhi 10 30 00 australia lindeman australia lindeman 10 00 00 australia lord_howe australia lord_howe 10 30 00 australia melbourne australia melbourne 10 00 00 australia nsw aet 10 00 00 australia north act 09 30 00 australia perth australia perth 08 00 00 australia queensland australia queensland 10 00 00 australia south australia south 09 30 00 australia sydney aet 10 00 00 australia tasmania australia tasmania 10 00 00 australia victoria australia victoria 10 00 00 australia west australia west 08 00 00 australia yancowinna australia yancowinna 09 30 00 bet bet -03 00 00 bst bst 06 00 00 brazil acre brazil acre -05 00 00 brazil denoronha brazil denoronha -02 00 00 brazil east bet -03 00 00 brazil west brazil west -04 00 00 cat cat 02 00 00 cet cet 01 00 00 cnt cnt -03 30 00 cst cst -06 00 00 cst6cdt cst6cdt -06 00 00 ctt ctt 08 00 00 canada atlantic canada atlantic -04 00 00 canada central canada central -06 00 00 canada east-saskatchewan canada east-saskatchewan -06 00 00 canada eastern canada eastern -05 00 00 canada mountain canada mountain -07 00 00 canada newfoundland cnt -03 30 00 canada pacific canada pacific -08 00 00 canada saskatchewan canada saskatchewan -06 00 00 canada yukon canada yukon -08 00 00 chile continental chile continental -04 00 00 chile easterisland chile easterisland -06 00 00 cuba cuba -05 00 00 eat eat 03 00 00 ect ect 01 00 00 eet eet 02 00 00 est est -05 00 00 est5edt est5edt -05 00 00 egypt art 02 00 00 eire eire 00 00 00 etc gmt gmt 00 00 00 etc gmt 0 gmt 00 00 00 etc gmt 1 etc gmt 1 -01 00 00 etc gmt 10 etc gmt 10 -10 00 00 etc gmt 11 etc gmt 11 -11 00 00 etc gmt 12 etc gmt 12 -12 00 00 etc gmt 2 etc gmt 2 -02 00 00 etc gmt 3 etc gmt 3 -03 00 00 etc gmt 4 etc gmt 4 -04 00 00 etc gmt 5 etc gmt 5 -05 00 00 etc gmt 6 etc gmt 6 -06 00 00 etc gmt 7 etc gmt 7 -07 00 00 etc gmt 8 etc gmt 8 -08 00 00 etc gmt 9 etc gmt 9 -09 00 00 etc gmt-0 gmt 00 00 00 etc gmt-1 etc gmt-1 01 00 00 etc gmt-10 etc gmt-10 10 00 00 etc gmt-11 etc gmt-11 11 00 00 etc gmt-12 etc gmt-12 12 00 00 etc gmt-13 etc gmt-13 13 00 00 etc gmt-14 etc gmt-14 14 00 00 etc gmt-2 etc gmt-2 02 00 00 etc gmt-3 etc gmt-3 03 00 00 etc gmt-4 etc gmt-4 04 00 00 etc gmt-5 etc gmt-5 05 00 00 etc gmt-6 etc gmt-6 06 00 00 etc gmt-7 etc gmt-7 07 00 00 etc gmt-8 etc gmt-8 08 00 00 etc gmt-9 etc gmt-9 09 00 00 etc gmt0 gmt 00 00 00 etc greenwich gmt 00 00 00 etc uct uct 00 00 00 etc utc uct 00 00 00 etc universal uct 00 00 00 etc zulu uct 00 00 00 europe amsterdam europe amsterdam 01 00 00 europe andorra europe andorra 01 00 00 europe astrakhan europe astrakhan 04 00 00 europe athens europe athens 02 00 00 europe belfast gb 00 00 00 europe belgrade europe belgrade 01 00 00 europe berlin europe berlin 01 00 00 europe bratislava europe bratislava 01 00 00 europe brussels europe brussels 01 00 00 europe bucharest europe bucharest 02 00 00 europe budapest europe budapest 01 00 00 europe busingen europe busingen 01 00 00 europe chisinau europe chisinau 02 00 00 europe copenhagen europe copenhagen 01 00 00 europe dublin eire 00 00 00 europe gibraltar europe gibraltar 01 00 00 europe guernsey gb 00 00 00 europe helsinki europe helsinki 02 00 00 europe isle_of_man gb 00 00 00 europe istanbul turkey 03 00 00 europe jersey gb 00 00 00 europe kaliningrad europe kaliningrad 02 00 00 europe kiev europe kiev 02 00 00 europe kirov europe kirov 03 00 00 europe lisbon portugal 00 00 00 europe ljubljana europe ljubljana 01 00 00 europe london gb 00 00 00 europe luxembourg europe luxembourg 01 00 00 europe madrid europe madrid 01 00 00 europe malta europe malta 01 00 00 europe mariehamn europe mariehamn 02 00 00 europe minsk europe minsk 03 00 00 europe monaco europe monaco 01 00 00 europe moscow w-su 03 00 00 europe nicosia europe nicosia 02 00 00 europe oslo europe oslo 01 00 00 europe paris ect 01 00 00 europe podgorica europe podgorica 01 00 00 europe prague europe prague 01 00 00 europe riga europe riga 02 00 00 europe rome europe rome 01 00 00 europe samara europe samara 04 00 00 europe san_marino europe san_marino 01 00 00 europe sarajevo europe sarajevo 01 00 00 europe saratov europe saratov 04 00 00 europe simferopol europe simferopol 03 00 00 europe skopje europe skopje 01 00 00 europe sofia europe sofia 02 00 00 europe stockholm europe stockholm 01 00 00 europe tallinn europe tallinn 02 00 00 europe tirane europe tirane 01 00 00 europe tiraspol europe tiraspol 02 00 00 europe ulyanovsk europe ulyanovsk 04 00 00 europe uzhgorod europe uzhgorod 02 00 00 europe vaduz europe vaduz 01 00 00 europe vatican europe vatican 01 00 00 europe vienna europe vienna 01 00 00 europe vilnius europe vilnius 02 00 00 europe volgograd europe volgograd 04 00 00 europe warsaw poland 01 00 00 europe zagreb europe zagreb 01 00 00 europe zaporozhye europe zaporozhye 02 00 00 europe zurich europe zurich 01 00 00 factory factory 00 00 00 gb gb 00 00 00 gb-eire gb 00 00 00 gmt gmt 00 00 00 gmt 0 gmt 00 00 00 gmt-0 gmt 00 00 00 gmt0 gmt 00 00 00 greenwich gmt 00 00 00 hst hst -10 00 00 hongkong hongkong 08 00 00 iet iet -05 00 00 ist ist 05 30 00 iceland iceland 00 00 00 indian antananarivo eat 03 00 00 indian chagos indian chagos 06 00 00 indian christmas indian christmas 07 00 00 indian cocos indian cocos 06 30 00 indian comoro eat 03 00 00 indian kerguelen indian kerguelen 05 00 00 indian mahe indian mahe 04 00 00 indian maldives indian maldives 05 00 00 indian mauritius indian mauritius 04 00 00 indian mayotte eat 03 00 00 indian reunion indian reunion 04 00 00 iran iran 03 30 00 israel israel 02 00 00 jst jst 09 00 00 jamaica jamaica -05 00 00 japan jst 09 00 00 kwajalein kwajalein 12 00 00 libya libya 02 00 00 met met 01 00 00 mit mit 13 00 00 mst mst -07 00 00 mst7mdt mst7mdt -07 00 00 mexico bajanorte mexico bajanorte -08 00 00 mexico bajasur mexico bajasur -07 00 00 mexico general mexico general -06 00 00 net net 04 00 00 nst nst 12 00 00 nz nst 12 00 00 nz-chat nz-chat 12 45 00 navajo navajo -07 00 00 plt plt 05 00 00 pnt pnt -07 00 00 prc ctt 08 00 00 prt prt -04 00 00 pst pst -08 00 00 pst8pdt pst8pdt -08 00 00 pacific apia mit 13 00 00 pacific auckland nst 12 00 00 pacific bougainville pacific bougainville 11 00 00 pacific chatham nz-chat 12 45 00 pacific chuuk pacific chuuk 10 00 00 pacific easter pacific easter -06 00 00 pacific efate pacific efate 11 00 00 pacific enderbury pacific enderbury 13 00 00 pacific fakaofo pacific fakaofo 13 00 00 pacific fiji pacific fiji 12 00 00 pacific funafuti pacific funafuti 12 00 00 pacific galapagos pacific galapagos -06 00 00 pacific gambier pacific gambier -09 00 00 pacific guadalcanal sst 11 00 00 pacific guam pacific guam 10 00 00 pacific honolulu pacific honolulu -10 00 00 pacific johnston pacific johnston -10 00 00 pacific kiritimati pacific kiritimati 14 00 00 pacific kosrae pacific kosrae 11 00 00 pacific kwajalein kwajalein 12 00 00 pacific majuro pacific majuro 12 00 00 pacific marquesas pacific marquesas -09 30 00 pacific midway pacific midway -11 00 00 pacific nauru pacific nauru 12 00 00 pacific niue pacific niue -11 00 00 pacific norfolk pacific norfolk 11 00 00 pacific noumea pacific noumea 11 00 00 pacific pago_pago pacific pago_pago -11 00 00 pacific palau pacific palau 09 00 00 pacific pitcairn pacific pitcairn -08 00 00 pacific pohnpei pacific pohnpei 11 00 00 pacific ponape pacific ponape 11 00 00 pacific port_moresby pacific port_moresby 10 00 00 pacific rarotonga pacific rarotonga -10 00 00 pacific saipan pacific saipan 10 00 00 pacific samoa pacific samoa -11 00 00 pacific tahiti pacific tahiti -10 00 00 pacific tarawa pacific tarawa 12 00 00 pacific tongatapu pacific tongatapu 13 00 00 pacific truk pacific truk 10 00 00 pacific wake pacific wake 12 00 00 pacific wallis pacific wallis 12 00 00 pacific yap pacific yap 10 00 00 poland poland 01 00 00 portugal portugal 00 00 00 roc roc 08 00 00 rok rok 09 00 00 sst sst 11 00 00 singapore singapore 08 00 00 systemv ast4 systemv ast4 -04 00 00 systemv ast4adt systemv ast4adt -04 00 00 systemv cst6 systemv cst6 -06 00 00 systemv cst6cdt systemv cst6cdt -06 00 00 systemv est5 systemv est5 -05 00 00 systemv est5edt systemv est5edt -05 00 00 systemv hst10 systemv hst10 -10 00 00 systemv mst7 systemv mst7 -07 00 00 systemv mst7mdt systemv mst7mdt -07 00 00 systemv pst8 systemv pst8 -08 00 00 systemv pst8pdt systemv pst8pdt -08 00 00 systemv yst9 systemv yst9 -09 00 00 systemv yst9ydt systemv yst9ydt -09 00 00 turkey turkey 03 00 00 uct uct 00 00 00 us alaska ast -09 00 00 us aleutian us aleutian -10 00 00 us arizona pnt -07 00 00 us central cst -06 00 00 us east-indiana iet -05 00 00 us eastern us eastern -05 00 00 us hawaii us hawaii -10 00 00 us indiana-starke us indiana-starke -06 00 00 us michigan us michigan -05 00 00 us mountain navajo -07 00 00 us pacific pst -08 00 00 us pacific-new pst -08 00 00 us samoa us samoa -11 00 00 utc uct 00 00 00 universal uct 00 00 00 vst vst 07 00 00 w-su w-su 03 00 00 wet wet 00 00 00 zulu uct 00 00 00",
- "category": "Data Types",
- "url": "../docs/sql/data_types/timezones",
- "blurb": "A reference list for Time Zones."
+ "title": "ODBC API - Overview",
+ "text": "odbc api the odbc open database connectivity is a c-style api that provides access to different flavors of database management systems dbmss the odbc api consists of the driver manager dm and the odbc drivers the dm is part of the system library e g unixodbc which manages the communications between the user applications and the odbc drivers typically applications are linked against the dm which uses data source name dsn to look up the correct odbc driver the odbc driver is a dbms implementation of the odbc api which handles all the internals of that dbms the dm maps user application calls of odbc functions to the correct odbc driver that performs the specified function and returns the proper values duckdb odbc driver duckdb supports the odbc version 3 0 according to the core interface conformance we release the odbc driver as assets for linux and windows users can download them from the latest release of duckdb operating system pages in this section",
+ "category": "Odbc",
+ "url": "../docs/api/odbc/overview",
+ "blurb": "ODBC API The ODBC (Open Database Connectivity) is a C-style API that provides access to different flavors of Database..."
},
{
- "title": "Data Types",
- "text": "general-purpose data types the table below shows all the built-in general-purpose data types the alternatives listed in the aliases column can be used to refer to these types as well however note that the aliases are not part of the sql standard and hence might not be accepted by other database engines name aliases description --- --- --- bigint int8 long signed eight-byte integer boolean bool logical logical boolean true false blob bytea binary varbinary variable-length binary data date calendar date year month day double float8 numeric decimal double precision floating-point number 8 bytes decimal s p fixed-precision floating point number with the given scale and precision hugeint signed sixteen-byte integer integer int4 int signed signed four-byte integer interval date time delta real float4 float single precision floating-point number 4 bytes smallint int2 short signed two-byte integer time time of day no time zone timestamp datetime combination of time and date timestamp with time zone timestamptz combination of time and date that uses the current time zone tinyint int1 signed one-byte integer ubigint unsigned eight-byte integer uinteger unsigned four-byte integer usmallint unsigned two-byte integer utinyint unsigned one-byte integer uuid uuid data type varchar char bpchar text string variable-length character string nested composite types duckdb supports three nested data types list struct and map each supports different use cases and has a different structure name description rules when used in a column build from values define in ddl create --- --- --- --- --- list an ordered sequence of data values of the same type each row must have the same data type within each list but can have any number of elements 1 2 3 int struct a dictionary of multiple named values where each key is a string but the value can be a different type for each key each row must have the same keys i 42 j a struct i int j varchar map a dictionary of multiple named values each key having the same type and each value having the same type keys and values can be any type and can be different types from one another rows may have different keys map 1 2 a b map int varchar nesting list s struct s and map s can be arbitrarily nested to any depth so long as the type rules are observed -- struct with lists select birds duck goose heron aliens null amphibians frog toad -- struct with list of maps select test map 1 5 42 1 45 map 1 5 42 1 45 links to detailed documentation",
- "category": "Data Types",
- "url": "../docs/sql/data_types/overview",
- "blurb": "The table below shows all the built-in general-purpose data types."
+ "title": "Java JDBC API",
+ "text": "installation the duckdb java jdbc api can be installed from maven central please see the installation page for details basic api usage duckdb s jdbc api implements the main parts of the standard java database connectivity jdbc api version 4 0 describing jdbc is beyond the scope of this page see the official documentation for details below we focus on the duckdb-specific parts startup shutdown in jdbc database connections are created through the standard java sql drivermanager class the driver should auto-register in the drivermanager if that does not work for some reason you can enforce registration like so class forname org duckdb duckdbdriver to create a duckdb connection call drivermanager with the jdbc duckdb jdbc url prefix like so connection conn drivermanager getconnection jdbc duckdb when using the jdbc duckdb url alone an in-memory database is created note that for an in-memory database no data is persisted to disk i e all data is lost when you exit the java program if you would like to access or create a persistent database append its file name after the path for example if your database is stored in tmp my_database use the jdbc url jdbc duckdb tmp my_database to create a connection to it it is possible to open a duckdb database file in read-only mode this is for example useful if multiple java processes want to read the same database file at the same time to open an existing database file in read-only mode set the connection property duckdb read_only like so properties ro_prop new properties ro_prop setproperty duckdb read_only true connection conn_ro drivermanager getconnection jdbc duckdb tmp my_database ro_prop the jdbc drivermanager api is a relatively poor fit for embedded database management systems such as duckdb if you would like to create multiple connections to the same database it would be somewhat logical to just create additional connections with the same url this is however only supported for read-only connections if you would like to create multiple read-write connections to the same database file or the same in-memory database instance you can use the custom duplicate method like so connection conn2 duckdbconnection conn duplicate querying duckdb supports the standard jdbc methods to send queries and retreive result sets first a statement object has to be created from the connection this object can then be used to send queries using execute and executequery execute is meant for queries where no results are expected like create table or update etc and executequery is meant to be used for queries that produce results e g select below two examples see also the jdbc statement and resultset documentations create a table statement stmt conn createstatement stmt execute create table items item varchar value decimal 10 2 count integer insert two items into the table stmt execute insert into items values jeans 20 0 1 hammer 42 2 2 resultset rs stmt executequery select from items while rs next system out println rs getstring 1 system out println rs getint 3 rs close jeans 1 hammer 2 duckdb also supports prepared statements as per the jdbc api preparedstatement p_stmt conn preparestatement insert into test values p_stmt setstring 1 chainsaw p_stmt setdouble 2 500 0 p_stmt setint 3 42 p_stmt execute more calls to execute possible p_stmt close do not use prepared statements to insert large amounts of data into duckdb see the data import documentation for better options",
+ "category": "Api",
+ "url": "../docs/api/java",
+ "blurb": "Installation The DuckDB Java JDBC API can be installed from Maven Central . Please see the installation page for..."
},
{
- "title": "Enum Types",
- "text": "name description --- --- enum dictionary encoding representing all possible string values of a column enums the enum type represents a dictionary data structure with all possible unique values of a column for example a column storing the days of the week can be an enum holding all possible days enums are particularly interesting for string columns with high cardinality this is because the column only stores a numerical reference to the string in the enum dictionary resulting in immense savings in disk storage and faster query performance enum definition enum types are created using the command create type enum_name as enum value_1 value_2 for example -- creates new user defined type mood as an enum create type mood as enum sad ok happy -- this will fail since the mood type already exists create type mood as enum sad ok happy anxious -- this will fail since enums cannot hold null values create type breed as enum maltese null -- this will fail since enum values must be unique create type breed as enum maltese maltese enum usage after an enum has been created it can be used anywhere a standard built-in type is used for example we can create a table with a column that references the enum -- creates a table person with attributes name string type and current_mood mood type create table person name text current_mood mood -- inserts tuples in the person table insert into person values pedro happy mark null pagliacci sad mr mackey ok -- this will fail since the mood type does not have a quackity-quack value insert into person values hannes quackity-quack -- the string sad is cast to the type mood returning a numerical reference value -- this makes the comparison a numerical comparison instead of a string comparison select from person where current_mood sad ---- pagliacci enum vs strings duckdb enums are automatically cast to varchar types whenever necessary this characteristic allows for enum columns to be used in any varchar function in addition it also allows for comparisons between different enum columns or an enum and a varchar column for example -- regexp_matches is a function that takes a varchar hence current_mood is cast to varchar select regexp_matches current_mood a from person ---- true false true false create type new_mood as enum happy anxious create table person_2 name text current_mood mood future_mood new_mood past_mood varchar -- since current_mood and future_mood are constructed on different enums -- duckdb will cast both enums to strings and perform a string comparison select from person_2 where current_mood future_mood -- since current_mood is an enum -- duckdb will cast the current_mood enum to varchar and perform a string comparison select from person_2 where current_mood past_mood enum removal enum types are stored in the catalog and a catalog dependency is added to each table that uses them it is possible to drop an enum from the catalog using the following command drop type enum_name note that any dependent must be removed before dropping the enum or the enum must be dropped with the additional cascade parameter for example -- this will fail since person has a catalog dependency to the mood type drop type mood drop table person drop table person_2 -- this successfully removes the mood type -- another option would be to drop type mood cascade drops the type and its dependents drop type mood",
- "category": "Data Types",
- "url": "../docs/sql/data_types/enum",
- "blurb": "The ENUM type represents a dictionary data structure with all possible unique values of a column."
+ "title": "Rust API",
+ "text": "installation the duckdb rust api can be installed from crate io please see the docs rs for details basic api usage duckdb-rs is an ergonomic wrapper based on the duckdb c api please refer to the readme for details startup shutdown to use duckdb you must first initialize a connection handle using connection open connection open takes as parameter the database file to read and write from if the database file does not exist it will be created the file extension may be db duckdb or anything else you can also use connection open_in_memory to create an in-memory database note that for an in-memory database no data is persisted to disk i e all data is lost when you exit the process use duckdb params connection result let conn connection open_in_memory you can conn close the connection manually or just leave it out of scope we had implement the drop trait which will automatically close the underlining db connection for you querying sql queries can be sent to duckdb using the execute method of connections or we can also prepare the statement and then query on that conn execute insert into person name data values params me name me data let mut stmt conn prepare select id name data from person let person_iter stmt query_map row ok person id row get 0 name row get 1 data row get 2 for person in person_iter println found person person unwrap",
+ "category": "Api",
+ "url": "../docs/api/rust",
+ "blurb": "Installation The DuckDB Rust API can be installed from crate.io . Please see the docs.rs for details. Basic API..."
},
{
- "title": "NULL Values",
- "text": "null values are special values that are used to represent missing data in sql columns of any type can contain null values logically a null value can be seen as the value of this field is unknown -- insert a null value into a table create table integers i integer insert into integers values null null values have special semantics in many parts of the query as well as in many functions any comparison with a null value returns null including null null you can use is not distinct from to perform an equality comparison where null values compare equal to each other use is not null to check if a value is null select null null -- returns null select null is not distinct from null -- returns true select null is null -- returns true null and functions a function that has input argument as null usually returns null select cos null -- null coalesce is an exception to this coalesce takes any number of arguments and returns for each row the first argument that is not null if all arguments are null coalesce also returns null select coalesce null null 1 -- 1 select coalesce 10 20 -- 10 select coalesce null null -- null null and conjunctions null values have special semantics in and or conjunctions for the ternary logic truth tables see the boolean type documentation null and aggregate functions null values are ignored in most aggregate functions aggregate functions that do not ignore null values include first last list and array_agg to exclude null values from those aggregate functions the filter clause can be used create table integers i integer insert into integers values 1 10 null select min i from integers -- 1 select max i from integers -- 10",
- "category": "Data Types",
- "url": "../docs/sql/data_types/nulls",
- "blurb": "The NULL value represents a missing value."
+ "title": "WebAssembly API",
+ "text": "getting started with duckdb-wasm a great starting point is to read the duckdb-wasm launch blog post another great resource is the github repository for details see the full duckdb-wasm api documentation",
+ "category": "Api",
+ "url": "../docs/api/wasm",
+ "blurb": "Getting Started with DuckDB-WASM A great starting point is to read the DuckDB-WASM launch blog post ! Another great..."
},
{
- "title": "Boolean Type",
- "text": "name aliases description --- --- --- boolean bool logical boolean true false the boolean type represents a statement of truth true or false in sql the boolean field can also have a third state unknown which is represented by the sql null value -- select the three possible values of a boolean column select true false null boolean boolean values can be explicitly created using the literals true and false however they are most often created as a result of comparisons or conjunctions for example the comparison i 10 results in a boolean value boolean values can be used in the where and having clauses of a sql statement to filter out tuples from the result in this case tuples for which the predicate evaluates to true will pass the filter and tuples for which the predicate evaluates to false or null will be filtered out consider the following example -- create a table with the value 5 15 and null create table integers i integer insert into integers values 5 15 null -- select all entries where i 10 select from integers where i 10 -- in this case 5 and null are filtered out -- 5 10 false -- null 10 null -- the result is 15 conjunctions the and or conjunctions can be used to combine boolean values below is a truth table for the and conjunction i e x and y x x and true x and false x and null ------- ------- ------- ------- true true false null false false false false null null false null below is a truth table for the or conjunction i e x or y x x or true x or false x or null ------- ------ ------- ------ true true true true false true false null null true null null expressions see logical operators and comparison operators",
- "category": "Data Types",
- "url": "../docs/sql/data_types/boolean",
- "blurb": "The BOOLEAN type represents a statement of truth (true or false)."
+ "title": "Scala JDBC API",
+ "text": "installation the duckdb java jdbc api can be used in scala and can be installed from maven central please see the installation page for details basic api usage scala uses duckdb s jdbc api implements the main parts of the standard java database connectivity jdbc api version 4 0 describing jdbc is beyond the scope of this page see the official documentation for details below we focus on the duckdb-specific parts startup shutdown in scala database connections are created through the standard java sql drivermanager class the driver should auto-register in the drivermanager if that does not work for some reason you can enforce registration like so class forname org duckdb duckdbdriver to create a duckdb connection call drivermanager with the jdbc duckdb jdbc url prefix like so val conn drivermanager getconnection jdbc duckdb when using the jdbc duckdb url alone an in-memory database is created note that for an in-memory database no data is persisted to disk i e all data is lost when you exit the java program if you would like to access or create a persistent database append its file name after the path for example if your database is stored in tmp my_database use the jdbc url jdbc duckdb tmp my_database to create a connection to it it is possible to open a duckdb database file in read-only mode this is for example useful if multiple java processes want to read the same database file at the same time to open an existing database file in read-only mode set the connection property duckdb read_only like so val ro_prop new properties ro_prop setproperty duckdb read_only true val conn_ro drivermanager getconnection jdbc duckdb tmp my_database ro_prop the jdbc drivermanager api is a relatively poor fit for embedded database management systems such as duckdb if you would like to create multiple connections to the same database it would be somewhat logical to just create additional connections with the same url this is however only supported for read-only connections if you would like to create multiple read-write connections to the same database file or the same in-memory database instance you can use the custom duplicate method like so val conn2 duckdbconnection conn duplicate querying duckdb supports the standard jdbc methods to send queries and retreive result sets first a statement object has to be created from the connection this object can then be used to send queries using execute and executequery execute is meant for queries where no results are expected like create table or update etc and executequery is meant to be used for queries that produce results e g select below two examples see also the jdbc statement and resultset documentations create a table val stmt conn createstatement stmt execute create table items item varchar value decimal 10 2 count integer insert two items into the table stmt execute insert into items values jeans 20 0 1 hammer 42 2 2 val rs stmt executequery select from items while rs next system out println rs getstring 1 system out println rs getint 3 rs close jeans 1 hammer 2 duckdb also supports prepared statements as per the jdbc api val p_stmt conn preparestatement insert into test values p_stmt setstring 1 chainsaw p_stmt setdouble 2 500 0 p_stmt setint 3 42 p_stmt execute more calls to execute possible p_stmt close do not use prepared statements to insert large amounts of data into duckdb see the data import documentation for better options",
+ "category": "Api",
+ "url": "../docs/api/scala",
+ "blurb": "Installation The DuckDB Java JDBC API can be used in Scala and can be installed from Maven Central . Please see the ..."
},
{
- "title": "Numeric Types",
- "text": "integer types the types tinyint smallint integer bigint and hugeint store whole numbers that is numbers without fractional components of various ranges attempts to store values outside of the allowed range will result in an error the types utinyint usmallint uinteger ubigint store whole unsigned numbers attempts to store negative numbers or values outside of the allowed range will result in an error name aliases min max --- --- --- --- tinyint int1 -128 127 smallint int2 short -32768 32767 integer int4 int signed -2147483648 2147483647 bigint int8 long -9223372036854775808 9223372036854775807 hugeint -170141183460469231731687303715884105727 170141183460469231731687303715884105727 utinyint - 0 255 usmallint - 0 65535 uinteger - 0 4294967295 ubigint - 0 18446744073709551615 the type integer is the common choice as it offers the best balance between range storage size and performance the smallint type is generally only used if disk space is at a premium the bigint and hugeint types are designed to be used when the range of the integer type is insufficient -170141183460469231731687303715884105728 -1 127 is not representable by the internal structure fixed-point decimals the data type decimal width scale represents an exact fixed-point decimal value when creating a value of type decimal the width and scale can be specified to define which size of decimal values can be held in the field the width field determines how many digits can be held and the scale determines the amount of digits after the decimal point for example the type decimal 3 2 can fit the value 1 23 but cannot fit the value 12 3 or the value 1 234 the default width and scale is decimal 18 3 if none are specified internally decimals are represented as integers depending on their specified width width internal size bytes --- --- --- 1-4 int16 2 5-9 int32 4 10-18 int64 8 19-38 int128 16 performance can be impacted by using too large decimals when not required in particular decimal values with a width above 19 are very slow as arithmetic involving the int128 type is much more expensive than operations involving the int32 or int64 types it is therefore recommended to stick with a width of 18 or below unless there is a good reason for why this is insufficient floating-point types the data types real and double precision are inexact variable-precision numeric types in practice these types are usually implementations of ieee standard 754 for binary floating-point arithmetic single and double precision respectively to the extent that the underlying processor operating system and compiler support it name aliases description --- --- --- real float4 single precision floating-point number 4 bytes double float8 double precision floating-point number 8 bytes inexact means that some values cannot be converted exactly to the internal format and are stored as approximations so that storing and retrieving a value might show slight discrepancies managing these errors and how they propagate through calculations is the subject of an entire branch of mathematics and computer science and will not be discussed here except for the following points if you require exact storage and calculations such as for monetary amounts use the numeric type instead if you want to do complicated calculations with these types for anything important especially if you rely on certain behavior in boundary cases infinity underflow you should evaluate the implementation carefully comparing two floating-point values for equality might not always work as expected on most platforms the real type has a range of at least 1e-37 to 1e 37 with a precision of at least 6 decimal digits the double type typically has a range of around 1e-307 to 1e 308 with a precision of at least 15 digits values that are too large or too small will cause an error rounding might take place if the precision of an input number is too high numbers too close to zero that are not representable as distinct from zero will cause an underflow error in addition to ordinary numeric values the floating-point types have several special values infinity -infinity nan these represent the ieee 754 special values infinity negative infinity and not-a-number respectively on a machine whose floating-point arithmetic does not follow ieee 754 these values will probably not work as expected when writing these values as constants in an sql command you must put quotes around them for example update table set x -infinity on input these strings are recognized in a case-insensitive manner functions see numeric functions and operators",
- "category": "Data Types",
- "url": "../docs/sql/data_types/numeric",
- "blurb": "Numeric types are used to store numbers, and come in different shapes and sizes."
+ "title": "CLI API",
+ "text": "installation the duckdb cli command line interface is a single dependency free executable it is precompiled for windows mac and linux please see the installation page under the cli tab or download the version for your environment from the duckdb github releases page in the assets section for pre-release versions you may compile from source or download the executable file that is produced from github actions linux mac windows the duckdb cli is based on the sqlite command line shell so cli-client-specific functionality is similar to what is described in the sqlite documentation although duckdb s sql syntax follows postgresql conventions getting started once the cli executable has been downloaded unzip it and save it to any directory navigate to that directory in a terminal and enter the command duckdb to run the executable if in a powershell or posix shell environment use the command duckdb instead to see additional command line options to use when starting the cli use the command duckdb --help by default the cli will open a temporary in-memory database to open or create a persistent database simply include a path as a command line argument like duckdb path to my_database duckdb this path can point to an existing database or to a file that does not yet exist and duckdb will open or create a database at that location as needed the file may have any arbitrary extension but db or duckdb are two common choices you will see a prompt like the below with a d on the final line v0 3 4 662041e2b enter help for usage hints connected to a transient in-memory database use open filename to reopen on a persistent database d once the cli has been opened enter a sql statement followed by a semicolon then hit enter and it will be executed results will be displayed in a table in the terminal if a semicolon is omitted hitting enter will allow for multi-line sql statements to be entered d select quack as my_column my_column ----------- quack d select nicely formatted quack as my_column excited quacking as another_column my_column another_column ------------------------ ------------------ nicely formatted quack excited quacking the cli supports all of duckdb s rich sql syntax including select create and alter statements etc to exit the cli press ctrl-d if your platform supports it otherwise press ctrl-c if using a persistent database it will automatically checkpoint save the latest edits to disk and close this will remove the wal file the write-ahead-log and consolidate all of your data into the single file database special commands dot commands in addition to sql syntax special dot commands may be entered that are specific to the cli client to use one of these commands begin the line with a period immediately followed by the name of the command you wish to execute additional arguments to the command are entered space separated after the command if an argument must contain a space either single or double quotes may be used to wrap that parameter dot commands must be entered on a single line and no whitespace may occur before the period no semicolon is required at the end of the line to see available commands use the help command d help auth on off show authorizer callbacks backup db file backup db default main to file bail on off stop after hitting an error default off binary on off turn binary output on or off default off cd directory change the working directory to directory changes on off show number of rows changed by sql check glob fail if output since testcase does not match clone newdb clone data into newdb from the existing database databases list names and files of attached databases dbconfig op val list or change sqlite3_db_config options dbinfo db show status information about the database dump table render database content as sql echo on off turn command echo on or off eqp on off full enable or disable automatic explain query plan excel display the output of next command in spreadsheet exit code exit this program with return-code code expert experimental suggest indexes for queries explain on off auto change the explain formatting mode default auto filectrl cmd run various sqlite3_file_control operations fullschema --indent show schema and the content of sqlite_stat tables headers on off turn display of headers on or off help -all pattern show help text for pattern import file table import data from file into table imposter index table create imposter table table on index index indexes table show names of indexes limit limit val display or change the value of an sqlite_limit lint options report potential schema issues log file off turn logging on or off file can be stderr stdout mode mode table set output mode nullvalue string use string in place of null values once options file output for the next sql command only to file open options file close existing database and reopen file output file send output to file or stdout if file is omitted parameter cmd manage sql parameter bindings print string print literal string progress n invoke progress handler after every n opcodes prompt main continue replace the standard prompts quit exit this program read file read input from file restore db file restore content of db default main from file save file write in-memory database into file scanstats on off turn sqlite3_stmt_scanstatus metrics on or off schema pattern show the create statements matching pattern selftest options run tests defined in the selftest table separator col row change the column and row separators sha3sum compute a sha3 hash of database content shell cmd args run cmd args in a system shell show show the current values for various settings stats on off show stats or turn stats on or off system cmd args run cmd args in a system shell tables table list names of tables matching like pattern table testcase name begin redirecting output to testcase-out txt testctrl cmd run various sqlite3_test_control operations timeout ms try opening locked tables for ms milliseconds timer on off turn sql timer on or off trace options output each sql statement as it is run vfsinfo aux information about the top-level vfs vfslist list all available vfses vfsname aux print the name of the vfs stack width num1 num2 set minimum column widths for columnar output note that the above list of methods is extensive and duckdb supports only a subset of the commands that are displayed please file a github issue if a command that is central to your workflow is not yet supported as an example of passing an argument to a dot command the help text may be filtered by passing in a text string as the second argument d help sh sha3sum compute a sha3 hash of database content shell cmd args run cmd args in a system shell show show the current values for various settings output formats the mode command may be used to change the appearance of the tables returned in the terminal output in addition to customizing the appearance these modes have additional benefits this can be useful for presenting duckdb output elsewhere by redirecting the terminal output to a file for example see writing results to a file section below using the insert mode will build a series of sql statements that can be used to insert the data at a later point the markdown mode is particularly useful for building documentation ascii box csv column html insert json jsonlines line list markdown quote table tabs tcl trash d mode markdown d select quacking intensifies as incoming_ducks incoming_ducks ---------------------- quacking intensifies the output appearance can also be adjusted with the separator command if using an export mode that relies on a separator csv or tabs for example the separator will be reset when the mode is changed for example mode csv will set the separator to a comma using separator will then convert the output to be pipe separated d mode csv d select 1 as col_1 2 as col_2 union all select 10 as col1 20 as col_2 col_1 col_2 1 2 10 20 d separator d select 1 as col_1 2 as col_2 union all select 10 as col1 20 as col_2 col_1 col_2 1 2 10 20 prepared statements the duckdb cli supports executing prepared statements in addition to normal select statements to create a prepared statement use the prepare statement d prepare s1 as select from my_table where my_column 1 or my_column 2 to run the prepared statement with parameters use the execute statement d execute s1 42 101 querying the database schema all duckdb clients support querying the database schema with sql but the cli has additional dot commands that can make it easier to understand the contents of a database the tables command will return a list of tables in the database it has an optional argument that will filter the results according to a like pattern d create table swimmers as select duck as animal d create table fliers as select duck as animal d create table walkers as select duck as animal d tables fliers swimmers walkers for example to filter to only tables that contain an l use the like pattern l d tables l fliers walkers the schema command will show all of the sql statements used to define the schema of the database d schema create table fliers animal varchar create table swimmers animal varchar create table walkers animal varchar opening database files in addition to connecting to a database when opening the cli a new database connection can be made by using the open command if no additional parameters are supplied a new in-memory database connection is created this database will not be persisted when the cli connection is closed d open the open command optionally accepts several options but the final parameter can be used to indicate a path to a persistent database or where one should be created the special string memory can also be used to open a temporary in-memory database d open persistent duckdb one important option accepted by open is the --readonly flag this disallows any editing of the database to open in read only mode the database must already exist this also means that a new in-memory database can t be opened in read only mode since in-memory databases are created upon connection d open --readonly preexisting duckdb writing results to a file by default the duckdb cli sends results to the terminal s standard output however this can be modified using either the output or once commands pass in the desired output file location as a parameter the once command will only output the next set of results and then revert to standard out but output will redirect all subsequent output to that file location note that each result will overwrite the entire file at that destination to revert back to standard output enter output with no file parameter in this example the output format is changed to markdown the destination is identified as a markdown file and then duckdb will write the output of the sql statement to that file output is then reverted to standard output using output with no parameter d mode markdown d output my_results md d select taking flight as output_column d output d select back to the terminal as displayed_column the file my_results md will then contain output_column --------------- taking flight the terminal will then display displayed_column ---------------------- back to the terminal a common output format is csv or comma separated values duckdb supports sql syntax to export data as csv or parquet but the cli-specific commands may be used to write a csv instead if desired d mode csv d once my_output_file csv d select 1 as col_1 2 as col_2 union all select 10 as col1 20 as col_2 the file my_output_file csv will then contain col_1 col_2 1 2 10 20 by passing special options flags to the once command query results can also be sent to a temporary file and automatically opened in the user s default program use either the -e flag for a text file opened in the default text editor or the -x flag for a csv file opened in the default spreadsheet editor this is useful for more detailed inspection of query results especially if there is a relatively large result set the excel command is equivalent to once -x d once -e d select quack as hello the results then open in the default text file editor of the system for example import data from csv duckdb supports sql syntax to directly query or import csv files but the cli-specific commands may be used to import a csv instead if desired the import command takes two arguments and also supports several options the first argument is the path to the csv file and the second is the name of the duckdb table to create since duckdb requires stricter typing than sqlite upon which the duckdb cli is based the destination table must be created before using the import command to automatically detect the schema and create a table from a csv see the read_csv_auto examples in the import docs in this example a csv file is generated by changing to csv mode and setting an output file location d mode csv d output import_example csv d select 1 as col_1 2 as col_2 union all select 10 as col1 20 as col_2 now that the csv has been written a table can be created with the desired schema and the csv can be imported the output is reset to the terminal to avoid continuing to edit the output file specified above the --skip n option is used to ignore the first row of data since it is a header row and the table has already been created with the correct column names d mode csv d output d create table test_table col_1 int col_2 int d import import_example csv test_table --skip 1 note that the import command utilizes the current mode and separator settings when identifying the structure of the data to import the --csv option can be used to override that behavior d import import_example csv test_table --skip 1 --csv reading sql from a file the duckdb cli can read both sql commands and dot commands from an external file intead of the terminal using the read command this allows for a number of commands to be run in sequence and allows command sequences to be saved and reused the read command requires only one argument the path to the file containing the sql and or commands to execute after running the commands in the file control will revert back to the terminal output from the execution of that file is governed by the same output and once commands that have been discussed previously this allows the output to be displayed back to the terminal as in the first example below or out to another file as in the second example in this example the file select_example sql is located in the same directory as duckdb exe and contains the following sql statement select from generate_series 5 to execute it from the cli the read command is used d read select_example sql the output below is returned to the terminal by default but can be adjusted using the output or once commands generate_series ----------------- 0 1 2 3 4 5 multiple commands including both sql and dot commands can also be run in a single read command in this example the file write_markdown_to_file sql is located in the same directory as duckdb exe and contains the following commands mode markdown output series md select from generate_series 5 to execute it from the cli the read command is used as before d read write_markdown_to_file sql in this case no output is returned to the terminal instead the file series md is created or replaced if it already existed with the markdown-formatted results shown here generate_series ----------------- 0 1 2 3 4 5 loading extensions the cli does not use the sqlite shell s load command instead directly execute duckdb s sql install and load commands as you would other sql statements see the extension docs for details d install fts d load fts",
+ "category": "Api",
+ "url": "../docs/api/cli",
+ "blurb": "Installation The DuckDB CLI (Command Line Interface) is a single, dependency free executable. It is precompiled for..."
},
{
- "title": "Window Functions",
- "text": "window functions can only be used in the select clause to share over specifications between functions use the statement s window clause and use the over window-name syntax general-purpose window functions the table below shows the available general window functions function return type description example --- --- --- --- row_number bigint the number of the current row within the partition counting from 1 row_number rank bigint the rank of the current row with gaps same as row_number of its first peer rank dense_rank bigint the rank of the current row without gaps this function counts peer groups dense_rank percent_rank double the relative rank of the current row rank - 1 total partition rows - 1 percent_rank cume_dist double the cumulative distribution number of partition rows preceding or peer with current row total partition rows cume_dist ntile num_buckets integer bigint an integer ranging from 1 to the argument value dividing the partition as equally as possible ntile 4 lag expr any offset integer default any same type as expr returns expr evaluated at the row that is offset rows before the current row within the partition if there is no such row instead return default which must be of the same type as expr both offset and default are evaluated with respect to the current row if omitted offset defaults to 1 and default to null lag column 3 0 lead expr any offset integer default any same type as expr returns expr evaluated at the row that is offset rows after the current row within the partition if there is no such row instead return default which must be of the same type as expr both offset and default are evaluated with respect to the current row if omitted offset defaults to 1 and default to null lead column 3 0 first_value expr any same type as expr returns expr evaluated at the row that is the first row of the window frame first_value column last_value expr any same type as expr returns expr evaluated at the row that is the last row of the window frame last_value column nth_value expr any nth integer same type as expr returns expr evaluated at the nth row of the window frame counting from 1 null if no such row nth_value column 2 aggregate window functions all aggregate functions can be used in a windowing context evaluation windowing works by breaking a relation up into independent partitions ordering those partitions and then computing a new column for each row as a function the nearby values some window functions depend only on the partition boundary and the ordering but a few including all the aggregates also use a frame frames are specified as a number of rows on either side preceding or following of the current row the distance can either be specified as a number of rows or a range of values using the partition s ordering value and a distance the full syntax is shown in the diagram at the top of the page and this diagram visually illustrates computation environment partition and ordering partitioning breaks the relation up into independent unrelated pieces partitioning is optional and if none is specified then the entire relation is treated as a single partition window functions cannot access values outside of the partition containing the row they are being evaluated at ordering is also optional but without it the results are not well-defined each partition is ordered using the same ordering clause here is a table of power generation data after partitioning by plant and ordering by date it will have this layout plant date mwh --- --- --- boston 2019-01-02 564337 boston 2019-01-03 507405 boston 2019-01-04 528523 boston 2019-01-05 469538 boston 2019-01-06 474163 boston 2019-01-07 507213 boston 2019-01-08 613040 boston 2019-01-09 582588 boston 2019-01-10 499506 boston 2019-01-11 482014 boston 2019-01-12 486134 boston 2019-01-13 531518 worcester 2019-01-02 118860 worcester 2019-01-03 101977 worcester 2019-01-04 106054 worcester 2019-01-05 92182 worcester 2019-01-06 94492 worcester 2019-01-07 99932 worcester 2019-01-08 118854 worcester 2019-01-09 113506 worcester 2019-01-10 96644 worcester 2019-01-11 93806 worcester 2019-01-12 98963 worcester 2019-01-13 107170 in what follows we shall use this table or small sections of it to illustrate various pieces of window function evaluation the simplest window function is row_number this function just computes the 1-based row number within the partition using the query select plant date row_number over as row from history order by 1 2 the result will be plant date row --- --- --- boston 2019-01-02 1 boston 2019-01-03 2 boston 2019-01-04 3 worcester 2019-01-02 1 worcester 2019-01-03 2 worcester 2019-01-04 3 note that even though the function is computed with an order by clause the result does not have to be sorted so the select also needs to be explicitly sorted if that is desired framing framing specifies a set of rows relative to each row where the function is evaluated the distance from the current row is given as an expression either preceding or following the current row this distance can either be specified as an integral number of rows or as a range delta expression from the value of the ordering expression for a range specification there must be only one ordering expression and it has to support addition and subtraction i e numbers or interval s the default values for frames are from unbounded preceding to current row it is invalid for a frame to start after it ends row framing here is a simple row frame query using an aggregate function select points sum points over rows between 1 preceding and 1 following we from results this query computes the sum of each point and the points on either side of it notice that at the edge of the partition there are only two values added together this is because frames are cropped to the edge of the partition range framing returning to the power data suppose the data is noisy we might want to compute a 7 day moving average for each plant to smooth out the noise to do this we can use this window query select plant date avg mwh over partition by plant order by date asc range between interval 3 days preceding and interval 3 days following as mwh 7-day moving average from generation history order by 1 2 this query partitions the data by plant to keep the different power plants data separate orders each plant s partition by date to put the energy measurements next to each other and uses a range frame of three days on either side of each day for the avg to handle any missing days this is the result plant date mwh 7-day br moving average --- --- --- boston 2019-01-02 517450 75 boston 2019-01-03 508793 20 boston 2019-01-04 508529 83 boston 2019-01-13 499793 00 worcester 2019-01-02 104768 25 worcester 2019-01-03 102713 00 worcester 2019-01-04 102249 50 window clauses multiple different over clauses can be specified in the same select and each will be computed separately often however we want to use the same layout for multiple window functions the window clause can be used to define a named window that can be shared between multiple window functions select plant date min mwh over seven as mwh 7-day moving minimum avg mwh over seven as mwh 7-day moving average max mwh over seven as mwh 7-day moving maximum from generation history window seven as partition by plant order by date asc range between interval 3 days preceding and interval 3 days following order by 1 2 the three window functions will also share the data layout which will improve performance box and whisker queries all aggregates can be used as windowing functions including the complex statistical functions these function implementations have been optimised for windowing and we can use the window syntax to write queries that generate the data for moving box-and-whisker plots select plant date min mwh over seven as mwh 7-day moving minimum quantile_cont mwh 0 25 0 5 0 75 over seven as mwh 7-day moving iqr max mwh over seven as mwh 7-day moving maximum from generation history window seven as partition by plant order by date asc range between interval 3 days preceding and interval 3 days following order by 1 2",
- "category": "SQL",
- "url": "../docs/sql/window_functions",
- "blurb": "Window functions can only be used in the SELECT clause. To share OVER specifications between functions, use the..."
+ "title": "Client APIs Overview",
+ "text": "there are various client apis for duckdb duckdb s native api is c with official wrappers available for c python r java node js webassembly wasm odbc api julia and a command line interface cli there are also contributed third-party duckdb wrappers for ruby go c and rust pages in this section",
+ "category": "Api",
+ "url": "../docs/api/overview",
+ "blurb": "There are various client APIs for DuckDB. DuckDB's native API is C++ , with official wrappers available for C , ..."
},
{
"title": "Indexes",
@@ -421,158 +435,151 @@
"blurb": "Index types DuckDB currently uses two index types: A min-max index is automatically created for columns of all ..."
},
{
- "title": "Samples",
- "text": "samples are used to randomly select a subset of a dataset examples -- select a sample of 5 rows from tbl using reservoir sampling select from tbl using sample 5 -- select a sample of 10 of the table using system sampling cluster sampling select from tbl using sample 10 -- select a sample of 10 of the table using bernoulli sampling select from tbl using sample 10 percent bernoulli -- select a sample of 50 rows of the table using reservoir sampling with a fixed seed 100 select from tbl using sample reservoir 50 rows repeatable 100 -- select a sample of 20 of the table using system sampling with a fixed seed 377 select from tbl using sample 10 system 377 -- select a sample of 10 of tbl before the join with tbl2 select from tbl tablesample reservoir 20 tbl2 where tbl i tbl2 i -- select a sample of 10 of tbl after the join with tbl2 select from tbl tbl2 where tbl i tbl2 i using sample reservoir 20 syntax samples allow you to randomly extract a subset of a dataset samples are useful for exploring a dataset faster as often you might not be interested in the exact answers to queries but only in rough indications of what the data looks like and what is in the data samples allow you to get approximate answers to queries faster as they reduce the amount of data that needs to pass through the query engine duckdb supports three different types of sampling methods reservoir bernoulli and system by default duckdb uses reservoir sampling when an exact number of rows is sampled and system sampling when a percentage is specified the sampling methods are described in detail below samples require a sample size which is an indication of how many elements will be sampled from the total population samples can either be given as a percentage 10 or as a fixed number of rows 10 rows all three sampling methods support sampling over a percentage but only reservoir sampling supports sampling a fixed number of rows samples are probablistic that is to say samples can be different between runs unless the seed is specifically specified specifying the seed only guarantees that the sample is the same if multi-threading is not enabled i e pragma threads 1 in the case of multiple threads running over a sample samples are not necessarily consistent even with a fixed seed reservoir reservoir sampling is a stream sampling technique that selects a random sample by keeping a reservoir of size equal to the sample size and randomly replacing elements as more elements come in reservoir sampling allows us to specify exactly how many elements we want in the resulting sample by selecting the size of the reservoir as a result reservoir sampling always outputs the same amount of elements unlike system and bernoulli sampling reservoir sampling is only recommended for small sample sizes and is not recommended for use with percentages that is because reservoir sampling needs to materialize the entire sample and randomly replace tuples within the materialized sample the larger the sample size the higher the performance hit incurred by this process reservoir sampling also incurs an additional performance penalty when multi-processing is used since the reservoir is to be shared amongst the different threads to ensure unbiased sampling this is not a big problem when the reservoir is very small but becomes costly when the sample is large bernoulli bernoulli sampling can only be used when a sampling percentage is specified it is rather straightforward every tuple in the underlying table is included with a chance equal to the specified percentage as a result bernoulli sampling can return a different number of tuples even if the same percentage is specified the amount of rows will generally be more or less equal to the specified percentage of the table but there will be some variance because bernoulli sampling is completely independent there is no shared state there is no penalty for using bernoulli sampling together with multiple threads system system sampling is a variant of bernoulli sampling with one crucial difference every vector is included with a chance equal to the sampling percentage this is a form of cluster sampling system sampling is more efficient than bernoulli sampling as no per-tuple selections have to be performed there is almost no extra overhead for using system sampling whereas bernoulli sampling can add additional cost as it has to perform random number generation for every single tuple system sampling is not suitable for smaller data sets as the granularity of the sampling is on the order of 1000 tuples that means that if system sampling is used for small data sets e g 100 rows either all the data will be filtered out or all the data will be included table samples the tablesample and using sample clauses are identical in terms of syntax and effect with one important difference tablesamples sample directly from the table for which they are specified whereas the sample clause samples after the entire from clause has been resolved this is relevant when there are joins present in the query plan the tablesample clause is essentially equivalent to creating a subquery with the using sample clause i e the following two queries are identical -- sample 20 of tbl before the join select from tbl tablesample reservoir 20 tbl2 where tbl i tbl2 i -- sample 20 of tbl before the join select from select from tbl using sample reservoir 20 tbl tbl2 where tbl i tbl2 i -- sample 20 after the join i e sample 20 of the join result select from tbl tbl2 where tbl i tbl2 i using sample reservoir 20",
- "category": "SQL",
- "url": "../docs/sql/samples",
- "blurb": "Samples are used to randomly select a subset of a dataset. Examples -- select a sample of 5 rows from tbl using..."
- },
- {
- "title": "SQL Introduction",
- "text": "here we provide an overview of how to perform simple operations in sql this tutorial is only intended to give you an introduction and is in no way a complete tutorial on sql this tutorial is adapted from the postgresql tutorial in the examples that follow we assume that you have installed the duckdb command line interface cli shell see here for information on how to install the cli if you build from the source tree you can launch the cli from the build directory build release duckdb launching the shell should give you the following prompt duckdb 5fb6fe57ab enter help for usage hints connected to a transient in-memory database use open filename to reopen on a persistent database d by launching the database like this an in-memory database is launched that means that no data is persisted on disk to persist data on disk you should also pass a database path to the shell the database will then be stored at that path and can be reloaded from disk later concepts duckdb is a relational database management system rdbms that means it is a system for managing data stored in relations a relation is essentially a mathematical term for a table each table is a named collection of rows each row of a given table has the same set of named columns and each column is of a specific data type tables themselves are stored inside schemas and a collection of schemas constitutes the entire database that you can access creating a new table you can create a new table by specifying the table name along with all column names and their types create table weather city varchar temp_lo integer -- minimum temperature on a day temp_hi integer -- maximum temperature on a day prcp real date date you can enter this into the shell with the line breaks the command is not terminated until the semicolon white space i e spaces tabs and newlines can be used freely in sql commands that means you can type the command aligned differently than above or even all on one line two dash characters -- introduce comments whatever follows them is ignored up to the end of the line sql is case insensitive about key words and identifiers except when identifiers are double-quoted to preserve the case not done above in the sql command we first specify the type of command that we want to perform create table after that follows the parameters for the command first the table name weather is given then the column names and column types follow city varchar specifies that the table has a column called city that is of type varchar varchar specifies a data type that can store text of arbitrary length the temperature fields are stored in an integer type a type that stores integer numbers i e whole numbers without a decimal point real columns store single precision floating-point numbers i e numbers with a decimal point date stores a date i e year month day combination date only stores the specific day not a time associated with that day duckdb supports the standard sql types integer smallint real double decimal char n varchar n date time and timestamp the second example will store cities and their associated geographical location create table cities name varchar lat decimal lon decimal finally it should be mentioned that if you don t need a table any longer or want to recreate it differently you can remove it using the following command drop table tablename populating a table with rows the insert statement is used to populate a table with rows insert into weather values san francisco 46 50 0 25 1994-11-27 constants that are not numeric values e g text and dates must be surrounded by single quotes as in the example input dates for the date type must be formatted as yyyy-mm-dd we can insert into the cities table in the same manner insert into cities values san francisco -194 0 53 0 the syntax used so far requires you to remember the order of the columns an alternative syntax allows you to list the columns explicitly insert into weather city temp_lo temp_hi prcp date values san francisco 43 57 0 0 1994-11-29 you can list the columns in a different order if you wish or even omit some columns e g if the prcp is unknown insert into weather date city temp_hi temp_lo values 1994-11-29 hayward 54 37 many developers consider explicitly listing the columns better style than relying on the order implicitly please enter all the commands shown above so you have some data to work with in the following sections you could also have used copy to load large amounts of data from csv files this is usually faster because the copy command is optimized for this application while allowing less flexibility than insert an example would be copy weather from home user weather csv where the file name for the source file must be available on the machine running the process there are many other ways of loading data into duckdb see the corresponding documentation section for more information querying a table to retrieve data from a table the table is queried a sql select statement is used to do this the statement is divided into a select list the part that lists the columns to be returned a table list the part that lists the tables from which to retrieve the data and an optional qualification the part that specifies any restrictions for example to retrieve all the rows of table weather type select from weather here is a shorthand for all columns so the same result would be had with select city temp_lo temp_hi prcp date from weather the output should be city temp_lo temp_hi prcp date --------------- --------- ---------- ------ ------------ san francisco 46 50 0 25 1994-11-27 san francisco 43 57 0 0 1994-11-29 hayward 37 54 1994-11-29 3 rows you can write expressions not just simple column references in the select list for example you can do select city temp_hi temp_lo 2 as temp_avg date from weather this should give city temp_avg date --------------- ---------- ------------ san francisco 48 1994-11-27 san francisco 50 1994-11-29 hayward 45 1994-11-29 3 rows notice how the as clause is used to relabel the output column the as clause is optional a query can be qualified by adding a where clause that specifies which rows are wanted the where clause contains a boolean truth value expression and only rows for which the boolean expression is true are returned the usual boolean operators and or and not are allowed in the qualification for example the following retrieves the weather of san francisco on rainy days select from weather where city san francisco and prcp 0 0 result city temp_lo temp_hi prcp date --------------- --------- --------- ------ ------------ san francisco 46 50 0 25 1994-11-27 you can request that the results of a query be returned in sorted order select from weather order by city city temp_lo temp_hi prcp date --------------- --------- --------- ------ ------------ hayward 37 54 1994-11-29 san francisco 43 57 0 0 1994-11-29 san francisco 46 50 0 25 1994-11-27 in this example the sort order isn t fully specified and so you might get the san francisco rows in either order but you d always get the results shown above if you do select from weather order by city temp_lo you can request that duplicate rows be removed from the result of a query select distinct city from weather city --------------- hayward san francisco 2 rows here again the result row ordering might vary you can ensure consistent results by using distinct and order by together select distinct city from weather order by city joins between tables thus far our queries have only accessed one table at a time queries can access multiple tables at once or access the same table in such a way that multiple rows of the table are being processed at the same time a query that accesses multiple rows of the same or different tables at one time is called a join query as an example say you wish to list all the weather records together with the location of the associated city to do that we need to compare the city column of each row of the weather table with the name column of all rows in the cities table and select the pairs of rows where these values match this would be accomplished by the following query select from weather cities where city name city temp_lo temp_hi prcp date name lon lat --------------- --------- --------- ------ ------------ --------------- ----- ---- san francisco 46 50 0 25 1994-11-27 san francisco -194 53 san francisco 43 57 0 1994-11-29 san francisco -194 53 2 rows observe two things about the result set there is no result row for the city of hayward this is because there is no matching entry in the cities table for hayward so the join ignores the unmatched rows in the weather table we will see shortly how this can be fixed there are two columns containing the city name this is correct because the lists of columns from the weather and cities tables are concatenated in practice this is undesirable though so you will probably want to list the output columns explicitly rather than using select city temp_lo temp_hi prcp date lon lat from weather cities where city name since the columns all had different names the parser automatically found which table they belong to if there were duplicate column names in the two tables you d need to qualify the column names to show which one you meant as in select weather city weather temp_lo weather temp_hi weather prcp weather date cities lon cities lat from weather cities where cities name weather city it is widely considered good style to qualify all column names in a join query so that the query won t fail if a duplicate column name is later added to one of the tables join queries of the kind seen thus far can also be written in this alternative form select from weather inner join cities on weather city cities name this syntax is not as commonly used as the one above but we show it here to help you understand the following topics now we will figure out how we can get the hayward records back in what we want the query to do is to scan the weather table and for each row to find the matching cities row s if no matching row is found we want some empty values to be substituted for the cities table s columns this kind of query is called an outer join the joins we have seen so far are inner joins the command looks like this select from weather left outer join cities on weather city cities name city temp_lo temp_hi prcp date name lon lat --------------- --------- --------- ------ ------------ --------------- ----- ---- san francisco 46 50 0 25 1994-11-27 san francisco -194 53 san francisco 43 57 0 1994-11-29 san francisco -194 53 hayward 37 54 1994-11-29 3 rows this query is called a left outer join because the table mentioned on the left of the join operator will have each of its rows in the output at least once whereas the table on the right will only have those rows output that match some row of the left table when outputting a left-table row for which there is no right-table match empty null values are substituted for the right-table columns aggregate functions like most other relational database products duckdb supports aggregate functions an aggregate function computes a single result from multiple input rows for example there are aggregates to compute the count sum avg average max maximum and min minimum over a set of rows as an example we can find the highest low-temperature reading anywhere with select max temp_lo from weather max ----- 46 1 row if we wanted to know what city or cities that reading occurred in we might try select city from weather where temp_lo max temp_lo -- wrong but this will not work since the aggregate max cannot be used in the where clause this restriction exists because the where clause determines which rows will be included in the aggregate calculation so obviously it has to be evaluated before aggregate functions are computed however as is often the case the query can be restated to accomplish the desired result here by using a subquery select city from weather where temp_lo select max temp_lo from weather city --------------- san francisco 1 row this is ok because the subquery is an independent computation that computes its own aggregate separately from what is happening in the outer query aggregates are also very useful in combination with group by clauses for example we can get the maximum low temperature observed in each city with select city max temp_lo from weather group by city city max --------------- ----- hayward 37 san francisco 46 2 rows which gives us one output row per city each aggregate result is computed over the table rows matching that city we can filter these grouped rows using having select city max temp_lo from weather group by city having max temp_lo 40 city max --------- ----- hayward 37 1 row which gives us the same results for only the cities that have all temp_lo values below 40 finally if we only care about cities whose names begin with s we can use the like operator select city max temp_lo from weather where city like s -- 1 group by city having max temp_lo 40 more information about the like operator can be found here it is important to understand the interaction between aggregates and sql s where and having clauses the fundamental difference between where and having is this where selects input rows before groups and aggregates are computed thus it controls which rows go into the aggregate computation whereas having selects group rows after groups and aggregates are computed thus the where clause must not contain aggregate functions it makes no sense to try to use an aggregate to determine which rows will be inputs to the aggregates on the other hand the having clause always contains aggregate functions in the previous example we can apply the city name restriction in where since it needs no aggregate this is more efficient than adding the restriction to having because we avoid doing the grouping and aggregate calculations for all rows that fail the where check updates you can update existing rows using the update command suppose you discover the temperature readings are all off by 2 degrees after november 28 you can correct the data as follows update weather set temp_hi temp_hi - 2 temp_lo temp_lo - 2 where date 1994-11-28 look at the new state of the data select from weather city temp_lo temp_hi prcp date --------------- --------- --------- ------ ------------ san francisco 46 50 0 25 1994-11-27 san francisco 41 55 0 1994-11-29 hayward 35 52 1994-11-29 3 rows deletions rows can be removed from a table using the delete command suppose you are no longer interested in the weather of hayward then you can do the following to delete those rows from the table delete from weather where city hayward all weather records belonging to hayward are removed select from weather city temp_lo temp_hi prcp date --------------- --------- --------- ------ ------------ san francisco 46 50 0 25 1994-11-27 san francisco 41 55 0 1994-11-29 2 rows one should be wary of statements of the form delete from tablename without a qualification delete will remove all rows from the given table leaving it empty the system will not request confirmation before doing this",
+ "title": "Aggregate Functions",
+ "text": "aggregates are functions that combine multiple rows into a single value aggregates are different from scalar functions and window functions because they change the cardinality of the result as such aggregates can only be used in the select and having clauses of a sql query when the distinct clause is provided only distinct values are considered in the computation of the aggregate this is typically used in combination with the count aggregate to get the number of distinct elements but it can be used together with any aggregate function in the system when the order by clause is provided the values being aggregated are sorted before applying the function usually this is not important but there are some order-sensitive aggregates that can have indeterminate results e g first last list and string_agg these can be made deterministic by ordering the arguments for order-insensitive aggregates this clause is parsed but ignored general aggregate functions the table below shows the available general aggregate functions function description example alias es --- --- --- --- arg_max arg val calculates the arg value for a maximum val value arg_max a b argmax a b max_by a b arg_min arg val calculates the arg value for a minimum val value arg_min a b argmin a b min_by a b avg arg calculates the average value for all tuples in arg avg a - bit_and arg returns the bitwise and of all bits in a given expression bit_and a - bit_or arg returns the bitwise or of all bits in a given expression bit_or a - bit_xor arg returns the bitwise xor of all bits in a given expression bit_xor a - bool_and arg returns true if every input value is true otherwise false bool_and a - bool_or arg returns true if any input value is true otherwise false bool a - count arg calculates the number of tuples tuples in arg count a - favg arg calculates the average using a more accurate floating point summation kahan sum favg a - first arg returns the first value of a column first a arbitrary a fsum arg calculates the sum using a more accurate floating point summation kahan sum fsum a sumkahan kahan_sum histogram arg returns a list of struct s with the fields bucket and count histogram a - last arg returns the last value of a column last a - list arg returns a list containing all the values of a column list a array_agg max arg returns the maximum value present in arg max a - min arg returns the minumum value present in arg min a - product arg calculates the product of all tuples in arg product a - string_agg arg sep concatenates the column string values with a separator string_agg s group_concat sum arg calculates the sum value for all tuples in arg sum a - approximate aggregates the table below shows the available approximate aggregate functions function description example --- --- --- approx_count_distinct x gives the approximate count of distintinct elements using hyperloglog approx_count_distinct a approx_quantile x pos gives the approximate quantile using t-digest approx_quantile a 0 5 reservoir_quantile x quantile sample_size 8192 gives the approximate quantile using reservoir sampling the sample size is optional and uses 8192 as a default size reservoir_quantile a 0 5 1024 statistical aggregates the table below shows the available statistical aggregate functions function description formula alias --- --- --- --- corr y x returns the correlation coefficient for non-null pairs in a group covar_pop y x stddev_pop x stddev_pop y - covar_pop y x returns the population covariance of input values sum x y - sum x sum y count count - entropy x returns the log-2 entropy of count input-values - - kurtosis x returns the excess kurtosis of all input values - - mad x returns the median absolute deviation for the values within x null values are ignored temporal types return a positive interval median abs x-median x - median x returns the middle value of the set null values are ignored for even value counts quantitiative values are averaged and ordinal values return the lower value quantile_cont x 0 5 - mode x returns the most frequent value for the values within x null values are ignored - - quantile_cont x pos returns the intepolated quantile number between 0 and 1 if pos is a list of float s then the result is a list of the corresponding intepolated quantiles - - quantile_disc x pos returns the exact quantile number between 0 and 1 if pos is a list of float s then the result is a list of the corresponding exact quantiles - quantile regr_avgx y x returns the average of the independent variable for non-null pairs in a group where x is the independent variable and y is the dependent variable - - regr_avgy y x returns the average of the dependent variable for non-null pairs in a group where x is the independent variable and y is the dependent variable - - regr_count y x returns the number of non-null number pairs in a group sum x y - sum x sum y count count - regr_intercept y x returns the intercept of the univariate linear regression line for non-null pairs in a group avg y -regr_slope y x avg x - regr_r2 y x returns the coefficient of determination for non-null pairs in a group - - regr_slope y x returns the slope of the linear regression line for non-null pairs in a group covar_pop x y var_pop x - regr_sxx y x - regr_count y x var_pop x - regr_sxy y x returns the population covariance of input values regr_count y x covar_pop y x - regr_syy y x - regr_count y x var_pop y f - skewness x returns the skewness of all input values - - stddev_pop x returns the population standard deviation sqrt var_pop x - stddev_samp x returns the sample standard deviation sqrt var_samp x stddev x var_pop x returns the population variance - - var_samp x returns the sample variance of all input values sum x 2 - sum x 2 count x count x - 1 variance arg val ordered set aggregate functions the table below shows the available ordered set aggregate functions these functions are specified using the within group order by sort_expression syntax and they are converted to an equivalent aggregate function that takes the ordering expression as the first argument function equivalent --- --- mode within group order by sort_expression mode sort_expression percentile_cont fraction within group order by sort_expression quantile_cont sort_expression fraction percentile_cont fractions within group order by sort_expression quantile_cont sort_expression fractions percentile_disc fraction within group order by sort_expression quantile_disc sort_expression fraction percentile_disc fractions within group order by sort_expression quantile_disc sort_expression fractions",
"category": "SQL",
- "url": "../docs/sql/introduction",
- "blurb": "Here we provide an overview of how to perform simple operations in SQL. This tutorial is only intended to give you an..."
+ "url": "../docs/sql/aggregates",
+ "blurb": "Aggregates are functions that combine multiple rows into a single value. Aggregates are different from scalar..."
},
{
- "title": "Timestamp Functions",
- "text": "this section describes functions and operators for examining and manipulating timestamp values timestamp operators the table below shows the available mathematical operators for timestamp types operator description example result --- --- --- --- addition of an interval timestamp 1992-03-22 01 02 03 interval 5 day 1992-03-27 01 02 03 - subtraction of timestamp s timestamp 1992-03-27 - timestamp 1992-03-22 5 days - subtraction of an interval timestamp 1992-03-27 01 02 03 - interval 5 day 1992-03-22 01 02 03 adding to or subtracting from infinite values produces the same infinite value timestamp functions the table below shows the available scalar functions for timestamp types function description example result --- --- --- --- age timestamp timestamp subtract arguments resulting in the time difference between the two timestamps age timestamp 2001-04-10 timestamp 1992-09-20 8 years 6 months 20 days age timestamp subtract from current_date age timestamp 1992-09-20 29 years 1 month 27 days 12 39 00 844 century timestamp extracts the century of a timestamp century timestamp 1992-03-22 20 current_timestamp current date and time start of current transaction date_diff part startdate enddate the number of partition boundaries between the timestamps date_diff hour timestamp 1992-09-30 23 59 59 timestamp 1992-10-01 01 58 00 2 date_part part timestamp get subfield equivalent to extract date_part minute timestamp 1992-09-20 20 38 40 38 date_sub part startdate enddate the number of complete partitions between the timestamps date_sub hour timestamp 1992-09-30 23 59 59 timestamp 1992-10-01 01 58 00 1 date_trunc part timestamp truncate to specified precision date_trunc hour timestamp 1992-09-20 20 38 40 1992-09-20 20 00 00 dayname timestamp the english name of the weekday dayname timestamp 1992-03-22 sunday epoch_ms ms converts ms since epoch to a timestamp epoch_ms 701222400000 1992-03-22 00 00 00 extract field from timestamp get subfield from a timestamp extract hour from timestamp 1992-09-20 20 38 48 20 greatest timestamp timestamp the later of two timestamps greatest timestamp 1992-09-20 20 38 48 timestamp 1992-03-22 01 02 03 1234 1992-09-20 20 38 48 isfinite timestamp returns true if the timestamp is finite false otherwise isfinite timestamp 1992-03-07 true isinf timestamp returns true if the timestamp is infinite false otherwise isinf timestamp -infinity true last_day timestamp the last day of the month last_day timestamp 1992-03-22 01 02 03 1234 1992-03-31 least timestamp timestamp the earlier of two timestamps least timestamp 1992-09-20 20 38 48 timestamp 1992-03-22 01 02 03 1234 1992-03-22 01 02 03 1234 make_timestamp bigint bigint bigint bigint bigint double the timestamp for the given parts make_timestamp 1992 9 20 13 34 27 123456 1992-09-20 13 34 27 123456 monthname timestamp the english name of the month monthname timestamp 1992-09-20 september now current date and time start of current transaction strftime timestamp format converts timestamp to string according to the format string strftime timestamp 1992-01-01 20 38 40 a -d b y - i m s p wed 1 january 1992 - 08 38 40 pm strptime text format converts string to timestamp according to the format string strptime wed 1 january 1992 - 08 38 40 pm a -d b y - i m s p 1992-01-01 20 38 40 to_timestamp sec converts sec since epoch to a timestamp to_timestamp 701222400 1992-03-22 00 00 00 there are also dedicated extraction functions to get the subfields functions applied to infinite dates will either return the same infinite dates e g greatest or null e g date_part depending on what makes sense in general if the function needs to examine the parts of the infinite date the result will be null timestamp table functions the table below shows the available table functions for timestamp types function description example --- --- --- --- generate_series timestamp timestamp interval generate a table of timestamps in the closed range stepping by the interval generate_series timestamp 2001-04-10 timestamp 2001-04-11 interval 30 minute range timestamp timestamp interval generate a table of timestamps in the half open range stepping by the interval range timestamp 2001-04-10 timestamp 2001-04-11 interval 30 minute infinite values are not allowed as table function bounds",
- "category": "Functions",
- "url": "../docs/sql/functions/timestamp",
- "blurb": "This section describes functions and operators for examining and manipulating TIMESTAMP values. Timestamp Operators ..."
+ "title": "WITH Clause",
+ "text": "the with clause allows you to specify common table expressions ctes regular non-recursive common-table-expressions are essentially views that are limited in scope to a particular query ctes can reference each-other and can be nested basic cte examples -- create a cte called cte and use it in the main query with cte as select 42 as x select from cte x 42 -- create two ctes where the second cte references the first cte with cte as select 42 as i cte2 as select i 100 as x from cte select from cte2 x 4200 recursive cte examples tree traversal with recursive can be used to traverse trees for example take a hierarchy of tags create table tag id int name varchar subclassof int insert into tag values 1 u2 5 2 blur 5 3 oasis 5 4 2pac 6 5 rock 7 6 rap 7 7 music 9 8 movies 9 9 art null the following query returns the path from the node oasis to the root of the tree art with recursive tag_hierarchy id source path as select id name name as path from tag where subclassof is null union all select tag id tag name list_prepend tag name tag_hierarchy path from tag tag_hierarchy where tag subclassof tag_hierarchy id select path from tag_hierarchy where source oasis path oasis rock music art graph traversal the with recursive clause can be used to express graph traversal on arbitrary graphs however if the graph has cycles the query must perform cycle detection to prevent infinite loops one way to achieve this is to store the path of a traversal in a list and before extending the path with a new edge check whether its endpoint has been visited before see the example later take the following directed graph from the ldbc graphalytics benchmark create table edge node1id int node2id int insert into edge values 1 3 1 5 2 4 2 5 2 10 3 1 3 5 3 8 3 10 5 3 5 4 5 8 6 3 6 4 7 4 8 1 9 4 note that the graph contains directed cycles e g between nodes 1 2 and 5 enumerate all paths from a node the following query returns all paths starting in node 1 with recursive paths startnode endnode path as select -- define the path as the first edge of the traversal node1id as startnode node2id as endnode node1id node2id as path from edge where startnode 1 union all select -- concatenate new edge to the path paths startnode as startnode node2id as endnode array_append path node2id as path from paths join edge on paths endnode node1id -- prevent adding a repeated node to the path -- this ensures that no cycles occur where node2id all paths path select startnode endnode path from paths order by length path path startnode endnode path 1 3 1 3 1 5 1 5 1 5 1 3 5 1 8 1 3 8 1 10 1 3 10 1 3 1 5 3 1 4 1 5 4 1 8 1 5 8 1 4 1 3 5 4 1 8 1 3 5 8 1 8 1 5 3 8 1 10 1 5 3 10 note that the result of this query is not restricted to shortest paths e g for node 5 the results include paths 1 5 and 1 3 5 enumerate unweighted shortest paths from a node in most cases enumerating all paths is not practical or feasible instead only the unweighted shortest paths are of interest to find these the second half of the with recursive query should be adjusted such that it only includes a node if it has not yet been visited this is implemented by using a subquery that checks if any of the previous paths includes the node with recursive paths startnode endnode path as select -- define the path as the first edge of the traversal node1id as startnode node2id as endnode node1id node2id as path from edge where startnode 1 union all select -- concatenate new edge to the path paths startnode as startnode node2id as endnode array_append path node2id as path from paths join edge on paths endnode node1id -- prevent adding a node that was visited previously by any path -- this ensures that 1 no cycles occur and 2 only nodes that -- were not visited by previous shorter paths are added to a path where not exists select 1 from paths previous_paths where list_contains previous_paths path node2id select startnode endnode path from paths order by length path path startnode endnode path 1 3 1 3 1 5 1 5 1 8 1 3 8 1 10 1 3 10 1 4 1 5 4 1 8 1 5 8 enumerate unweighted shortest paths between two nodes with recursive can also be used to find all unweighted shortest paths between two nodes to ensure that the recursive query is stopped as soon as we reach the end node we use a window function which checks whether the end node is among the newly added nodes the following query returns all unweighted shortest paths between nodes 1 start node and 8 end node with recursive paths startnode endnode path endreached as select -- define the path as the first edge of the traversal node1id as startnode node2id as endnode node1id node2id as path node2id 8 as endreached from edge where startnode 1 union all select -- concatenate new edge to the path paths startnode as startnode node2id as endnode array_append path node2id as path max case when node2id 8 then 1 else 0 end over rows between unbounded preceding and unbounded following as endreached from paths join edge on paths endnode node1id where not exists select 1 from paths previous_paths where list_contains previous_paths path node2id and paths endreached 0 select startnode endnode path from paths where endnode 8 order by length path path startnode endnode path 1 8 1 3 8 1 8 1 5 8 common table expressions",
+ "category": "Query Syntax",
+ "url": "../docs/sql/query_syntax/with",
+ "blurb": "The WITH clause allows you to specify common table expressions (CTEs). Regular (non-recursive) common-table-expressions..."
},
{
- "title": "Blob Functions",
- "text": "this section describes functions and operators for examining and manipulating blob values function description example result --- --- --- --- blob blob blob concatenation xaa blob xbb blob xaabb decode blob convert blob to varchar fails if blob is not valid utf-8 decode xc3 xbc blob ü encode string convert varchar to blob converts utf-8 characters into literal encoding encode my_string_with_ü my_string_with_ xc3 xbc octet_length blob number of bytes in blob octet_length xaabb blob 2",
- "category": "Functions",
- "url": "../docs/sql/functions/blob",
- "blurb": "This section describes functions and operators for examining and manipulating blob values. | Function | Description |..."
+ "title": "GROUPING SETS",
+ "text": "grouping sets rollup and cube can be used in the group by clause to perform a grouping over multiple dimensions within the same query examples -- compute the average income along the provided four different dimensions -- signifies the empty set i e computing an ungrouped aggregate select city street_name avg income from addresses group by grouping sets city street_name city street_name -- compute the average income along the same dimensions select city street_name avg income from addresses group by cube city street_name -- compute the average income along the dimensions city street_name city and select city street_name avg income from addresses group by rollup city street_name description grouping sets perform the same aggregate across different group by clauses in a single query create table students course varchar type varchar insert into students course type values cs bachelor cs bachelor cs phd math masters cs null cs null math null select course type count from students group by grouping sets course type course type course type count_star cs bachelor 2 cs phd 1 math masters 1 cs null 2 math null 1 cs null 5 math null 2 null bachelor 2 null phd 1 null masters 1 null null 3 null null 7 in the above query we group across four different sets course type course type and the empty group the result contains null for a group which is not in the grouping set for the result i e the above query is equivalent to the following union statement -- group by course type select course type count from students group by course type union all -- group by type select null as course type count from students group by type union all -- group by course select course null as type count from students group by course union all -- group by nothing select null as course null as type count from students cube and rollup are syntactic sugar to easily produce commonly used grouping sets the rollup clause will produce all sub-groups of a grouping set e g rollup country city zip produces the grouping sets country city zip country city country this can be useful for producing different levels of detail of a group by clause this produces n 1 grouping sets where n is the amount of terms in the rollup clause cube produces grouping sets for all combinations of the inputs e g cube country city zip will produce country city zip country city country zip city zip country city zip this produces 2 n grouping sets grouping alias grouping_id is a special aggregate function that can be used in combination with grouping sets the grouping function takes as parameters a group and returns 0 if the group is included in the grouping for that row or 1 otherwise this is primarily useful because the grouping columns by which we do not aggregate return null which is ambiguous with groups that are actually the value null the grouping or grouping_id function can be used to distinguish these two cases syntax",
+ "category": "Query Syntax",
+ "url": "../docs/sql/query_syntax/grouping_sets",
+ "blurb": "GROUPING SETS , ROLLUP and CUBE can be used in the GROUP BY clause to perform a grouping over multiple dimensions..."
},
{
- "title": "Interval Functions",
- "text": "this section describes functions and operators for examining and manipulating interval values interval operators the table below shows the available mathematical operators for interval types operator description example result --- --- --- --- addition of an interval interval 1 hour interval 5 hour interval 6 hour addition to a date date 1992-03-22 interval 5 day 1992-03-27 addition to a timestamp timestamp 1992-03-22 01 02 03 interval 5 day 1992-03-27 01 02 03 addition to a time time 01 02 03 interval 5 hour 06 02 03 - subtraction of an interval interval 5 hour - interval 1 hour interval 4 hour - subtraction from a date date 1992-03-27 - interval 5 day 1992-03-22 - subtraction from a timestamp timestamp 1992-03-27 01 02 03 - interval 5 day 1992-03-22 01 02 03 - subtraction from a time time 06 02 03 - interval 5 hour 01 02 03 interval functions the table below shows the available scalar functions for interval types function description example result --- --- --- --- date_part part interval get subfield equivalent to extract date_part year interval 14 months 1 extract part from interval get subfield from a date extract month from interval 14 months 2 to_years integer construct a year interval to_years 5 interval 5 year to_months integer construct a month interval to_months 5 interval 5 month to_days integer construct a day interval to_days 5 interval 5 day to_hours integer construct a hour interval to_hours 5 interval 5 hour to_minutes integer construct a minute interval to_minutes 5 interval 5 minute to_seconds integer construct a second interval to_seconds 5 interval 5 second to_milliseconds integer construct a millisecond interval to_milliseconds 5 interval 5 millisecond to_microseconds integer construct a microsecond interval to_microseconds 5 interval 5 microsecond only the documented date parts are defined for intervals",
- "category": "Functions",
- "url": "../docs/sql/functions/interval",
- "blurb": "This section describes functions and operators for examining and manipulating INTERVAL values. Interval Operators ..."
+ "title": "FILTER Clause",
+ "text": "the filter clause may optionally follow an aggregate function in a select statement this will filter the rows of data that are fed into the aggregate function in the same way that a where clause filters rows but localized to the specific aggregate function filter s are not currently able to be used when the aggregate function is in a windowing context there are multiple types of situations where this is useful including when evaluating multiple aggregates with different filters and when creating a pivoted view of a dataset filter provides a cleaner syntax for pivoting data when compared with the more traditional case when approach discussed below some aggregate functions also do not filter out null values so using a filter clause will return valid results when at times the case when approach will not this occurs with the functions first and last which are desirable in a non-aggregating pivot operation where the goal is to simply re-orient the data into columns rather than re-aggregate it filter also improves null handling when using the list and array_agg functions as the case when approach will include null values in the list result while the filter clause will remove them examples -- compare total row count to -- the number of rows where i 5 -- the number of rows where i is odd select count as total_rows count filter where i 5 as lte_five count filter where i 2 1 as odds from generate_series 1 10 tbl i total_rows lte_five odds --- --- --- 10 5 5 -- different aggregate functions may be used and multiple where expressions are also permitted -- the sum of i for rows where i 5 -- the median of i where i is odd select sum i filter where i 5 as lte_five_sum median i filter where i 2 1 as odds_median median i filter where i 2 1 and i 5 as odds_lte_five_median from generate_series 1 10 tbl i lte_five_sum odds_median odds_lte_five_median --- --- --- 15 5 0 3 0 the filter clause can also be used to pivot data from rows into columns this is a static pivot as columns must be defined prior to runtime in sql however this kind of statement can be dynamically generated in a host programming language to leverage duckdb s sql engine for rapid larger than memory pivoting --first generate an example dataset create temp table stacked_data as select i case when i rows 0 25 then 2022 when i rows 0 5 then 2023 when i rows 0 75 then 2024 when i rows 0 875 then 2025 else null end as year from select i count over as rows from generate_series 1 100000000 tbl i tbl -- pivot the data out by year move each year out to a separate column select count i filter where year 2022 as 2022 count i filter where year 2023 as 2023 count i filter where year 2024 as 2024 count i filter where year 2025 as 2025 count i filter where year is null as nulls from stacked_data --this syntax produces the same results as the the filter clauses above select count case when year 2022 then i end as 2022 count case when year 2023 then i end as 2023 count case when year 2024 then i end as 2024 count case when year 2025 then i end as 2025 count case when year is null then i end as nulls from stacked_data 2022 2023 2024 2025 nulls --- --- --- --- --- 25000000 25000000 25000000 12500000 12500000 however the case when approach will not work as expected when using an aggregate function that does not ignore null values the first function falls into this category so filter is preferred in this case -- pivot the data out by year move each year out to a separate column select first i filter where year 2022 as 2022 first i filter where year 2023 as 2023 first i filter where year 2024 as 2024 first i filter where year 2025 as 2025 first i filter where year is null as nulls from stacked_data 2022 2023 2024 2025 nulls --- --- --- --- --- 1474561 25804801 50749441 76431361 87500001 --this will produce null values whenever the first evaluation of the case when clause returns a null select first case when year 2022 then i end as 2022 first case when year 2023 then i end as 2023 first case when year 2024 then i end as 2024 first case when year 2025 then i end as 2025 first case when year is null then i end as nulls from stacked_data 2022 2023 2024 2025 nulls --- --- --- --- --- 1228801 null null null null aggregate function syntax including filter clause",
+ "category": "Query Syntax",
+ "url": "../docs/sql/query_syntax/filter",
+ "blurb": "The FILTER clause may optionally follow an aggregate function in a SELECT statement. This will filter the rows of..."
},
{
- "title": "Date Functions",
- "text": "this section describes functions and operators for examining and manipulating date values date operators the table below shows the available mathematical operators for date types operator description example result --- --- --- --- addition of days integers date 1992-03-22 5 1992-03-27 addition of an interval date 1992-03-22 interval 5 day 1992-03-27 - subtraction of date s date 1992-03-27 - date 1992-03-22 5 - subtraction of an interval date 1992-03-27 - interval 5 day 1992-03-22 adding to or subtracting from infinite values produces the same infinite value date functions the table below shows the available functions for date types dates can also be manipulated with the timestamp functions through type promotion function description example result --- --- --- --- current_date current date at start of current transaction date_diff part startdate enddate the number of partition boundaries between the dates date_diff month date 1992-09-15 date 1992-11-14 2 datediff part startdate enddate alias of date_diff the number of partition boundaries between the dates datediff month date 1992-09-15 date 1992-11-14 2 date_part part date get the subfield equivalent to extract date_part year date 1992-09-20 1992 datepart part date alias of date_part get the subfield equivalent to extract datepart year date 1992-09-20 1992 date_sub part startdate enddate the number of complete partitions between the dates date_sub month date 1992-09-15 date 1992-11-14 1 datesub part startdate enddate alias of date_sub the number of complete partitions between the dates datesub month date 1992-09-15 date 1992-11-14 1 date_trunc part date truncate to specified precision date_trunc month date 1992-03-07 1992-03-01 datetrunc part date alias of date_trunc truncate to specified precision datetrunc month date 1992-03-07 1992-03-01 dayname date the english name of the weekday dayname date 1992-09-20 sunday isfinite date returns true if the date is finite false otherwise isfinite date 1992-03-07 true isinf date returns true if the date is infinite false otherwise isinf date -infinity true extract part from date get subfield from a date extract year from date 1992-09-20 1992 greatest date date the later of two dates greatest date 1992-09-20 date 1992-03-07 1992-09-20 last_day date the last day of the corresponding month in the date last_day date 1992-09-20 1992-09-30 least date date the earlier of two dates least date 1992-09-20 date 1992-03-07 1992-03-07 make_date bigint bigint bigint the date for the given parts make_date 1992 9 20 1992-09-20 monthname date the english name of the month monthname date 1992-09-20 september strftime date format converts a date to a string according to the format string strftime date 1992-01-01 a -d b y wed 1 january 1992 there are also dedicated extraction functions to get the subfields functions applied to infinite dates will either return the same infinite dates e g greatest or null e g date_part depending on what makes sense in general if the function needs to examine the parts of the infinite date the result will be null",
- "category": "Functions",
- "url": "../docs/sql/functions/date",
- "blurb": "This section describes functions and operators for examining and manipulating date values. Date Operators The table..."
+ "title": "SELECT Clause",
+ "text": "the select clause specifies the list of columns that will be returned by the query while it appears first in the clause logically the expressions here are executed only at the end the select clause can contain arbitrary expressions that transform the output as well as aggregates and window functions examples -- select all columns from the table called table_name select from table_name -- select all unique cities from the addresses table select distinct city from addresses -- return the total number of rows in the addresses table select count from addresses -- select all columns except the city column from the addresses table select exclude city from addresses -- select all columns from the addresses table but replace city with lower city select replace lower city as city from addresses syntax",
+ "category": "Query Syntax",
+ "url": "../docs/sql/query_syntax/select",
+ "blurb": "The SELECT clause specifies the list of columns that will be returned by the query."
},
{
- "title": "Pattern Matching",
- "text": "pattern matching there are four separate approaches to pattern matching provided by duckdb the traditional sql like operator the more recent similar to operator added in sql 1999 a glob operator and posix-style regular expressions like the like expression returns true if the string matches the supplied pattern as expected the not like expression returns false if like returns true and vice versa an equivalent expression is not string like pattern if pattern does not contain percent signs or underscores then the pattern only represents the string itself in that case like acts like the equals operator an underscore _ in pattern stands for matches any single character a percent sign matches any sequence of zero or more characters like pattern matching always covers the entire string therefore if it s desired to match a sequence anywhere within a string the pattern must start and end with a percent sign some examples abc like abc -- true abc like a -- true abc like _b_ -- true abc like c -- false abc like c -- false abc like c -- true abc not like c -- false the keyword ilike can be used instead of like to make the match case-insensitive according to the active locale abc ilike c -- true abc not ilike c -- false to search within a string for a character that is a wildcard or _ the pattern must use an escape clause and an escape character to indicate the wildcard should be treated as a literal character instead of a wildcard see an example below additionally the function like_escape has the same functionality as a like expression with an escape clause but using function syntax see the text functions docs for details --search for strings with a then a literal percent sign then c a c like a c escape -- true azc like a c escape -- false --case insensitive ilike with escape a c ilike a c escape --true there are also alternative characters that can be used as keywords in place of like expressions these enhance postgres compatibility like-style postgres-style --- --- like not like ilike not ilike similar to the similar to operator returns true or false depending on whether its pattern matches the given string it is similar to like except that it interprets the pattern using a regular expression like like the similar to operator succeeds only if its pattern matches the entire string this is unlike common regular expression behavior where the pattern can match any part of the string a regular expression is a character sequence that is an abbreviated definition of a set of strings a regular set a string is said to match a regular expression if it is a member of the regular set described by the regular expression as with like pattern characters match string characters exactly unless they are special characters in the regular expression language but regular expressions use different special characters than like does some examples abc similar to abc -- true abc similar to a -- false abc similar to b d -- true abc similar to b c -- false abc not similar to abc -- false there are also alternative characters that can be used as keywords in place of similar to expressions these follow posix syntax similar to-style posix-style --- --- similar to not similar to glob the glob operator returns true or false if the string matches the glob pattern the glob operator is most commonly used when searching for filenames that follow a specific pattern for example a specific file extension use the question mark wildcard to match any single character and use the asterisk to match zero or more characters in addition use bracket syntax to match any single character contained within the brackets or within the character range specified by the brackets an exclamation mark may be used inside the first bracket to search for a character that is not contained within the brackets to learn more visit the glob programming wikipedia page some examples best txt glob txt -- true best txt glob txt -- true best txt glob txt -- false best txt glob abc est txt -- true best txt glob a-z est txt -- true -- the bracket syntax is case sensitive best txt glob a-z est txt -- false best txt glob a-za-z est txt -- true -- the applies to all characters within the brackets best txt glob a-za-z est txt -- false -- to negate a glob operator negate the entire expression -- not glob is not valid syntax not best txt glob txt -- false three tildes may also be used in place of the glob keyword glob-style symbolic-style --- --- glob regular expressions function description --- --- regexp_matches string pattern returns true if string contains the regexp pattern false otherwise regexp_replace string pattern replacement if string contains the regexp pattern replaces the matching part with replacement regexp_extract string pattern idx if string contains the regexp pattern returns the capturing group specified by optional parameter idx the regexp_matches function is similar to the similar to operator however it does not require the entire string to match instead regexp_matches returns true if the string merely contains the pattern unless the special tokens and are used to anchor the regular expression to the start and end of the string below are some examples regexp_matches abc abc -- true regexp_matches abc abc -- true regexp_matches abc a -- true regexp_matches abc a -- false regexp_matches abc b d -- true regexp_matches abc b c -- true regexp_matches abc b c -- false the regexp_replace function can be used to replace the part of a string that matches the regexp pattern with a replacement string the notation d where d is a number indicating the group can be used to refer to groups captured in the regular expression in the replacement string below are some examples regexp_replace abc b c x -- axc regexp_replace abc b c 1 1 1 1 -- abbbbc regexp_replace abc c 1e -- abe regexp_replace abc a b 2 1 -- bac the regexp_extract function is used to extract a part of a string that matches the regexp pattern a specific capturing group within the pattern can be extracted using the idx parameter if idx is not specified it defaults to 0 extracting the first match with the whole pattern regexp_extract abc b -- abc regexp_extract abc b 0 -- abc regexp_extract abc b 1 -- empty regexp_extract abc a-z b 1 -- a regexp_extract abc a-z b 2 -- b",
- "category": "Functions",
- "url": "../docs/sql/functions/patternmatching",
- "blurb": "Pattern Matching There are four separate approaches to pattern matching provided by DuckDB: the traditional SQL LIKE ..."
+ "title": "ORDER BY Clause",
+ "text": "order by is an output modifier logically it is applied at the very end of the query the order by clause sorts the rows on the sorting criteria in either ascending or descending order in addition every order clause can specify whether null values should be moved to the beginning or to the end by default if no modifiers are provided duckdb sorts asc nulls first i e the values are sorted in ascending order and null values are placed first this is identical to the default sort order of sqlite postgresql by default sorts in asc nulls last order the default sort order can be changed using the following pragma statements -- change the default null sorting order to either nulls first and nulls last pragma default_null_order nulls last -- change the default sorting order to either desc or asc pragma default_order desc text is sorted using the binary comparison collation by default which means values are sorted on their binary utf8 values while this works well for ascii text e g for english language data the sorting order can be incorrect for other languages for this purpose duckdb provides collations for more information on collations see the collation page examples -- select the addresses ordered by city name using the default null order and default order select from addresses order by city -- select the addresses ordered by city name in descending order with nulls at the end select from addresses order by city desc nulls last -- order by city and then by zip code both using the default orderings select from addresses order by city zip -- order by city using german collation rules select from addresses order by city collate de syntax",
+ "category": "Query Syntax",
+ "url": "../docs/sql/query_syntax/orderby",
+ "blurb": "ORDER BY is an output modifier. Logically it is applied at the very end of the query. The ORDER BY clause sorts the..."
},
{
- "title": "Text Functions",
- "text": "this section describes functions and operators for examining and manipulating string values denotes a space character function description example result --- --- --- --- string string string concatenation duck db duckdb string index alias for array_extract duckdb 4 k string begin end alias for array_slice missing arguments are interprete as null s duckdb 4 duck array_extract list index extract a single character using a 1-based index array_extract duckdb 2 u array_slice list begin end extract a string using slice conventions null s are interpreted as the bounds of the string negative values are accepted array_slice duckdb 5 null db ascii string returns an integer that represents the unicode code point of the first character of the string ascii ω 937 base64 blob convert a blob to a base64 encoded string alias of to_base64 base64 a blob qq bit_length string number of bits in a string bit_length abc 24 concat string concatenate many strings together concat hello world hello world concat_ws separator string concatenate strings together separated by the specified separator concat_ws banana apple melon banana apple melon contains string search_string return true if search_string is found within string contains abc a true format format parameters formats a string using fmt syntax format benchmark took seconds csv 42 benchmark csv took 42 seconds from_base64 string convert a base64 encoded string to a character string from_base64 qq a instr string search_string return location of first occurrence of search_string in string counting from 1 returns 0 if no match found instr test test es 2 lcase string alias of lower convert string to lower case lcase hello hello left string count extract the left-most count characters left hello 2 he length string number of characters in string length hello 5 string like target returns true if the string matches the like specifier see pattern matching hello like lo true like_escape string like_specifier escape_character returns true if the string matches the like_specifier see pattern matching escape_character is used to search for wildcard characters in the string like_escape a c a c true list_element string index an alias for array_extract list_element duckdb 2 u list_extract string index an alias for array_extract list_extract duckdb 2 u lower string convert string to lower case lower hello hello lpad string count character pads the string with the character from the left until it has count characters lpad hello 10 hello ltrim string removes any spaces from the left side of the string ltrim test test ltrim string characters removes any occurrences of any of the characters from the left side of the string ltrim test test md5 value returns the md5 hash of the value md5 123 202cb962ac59075b964b07152d234b70 nfc_normalize string convert string to unicode nfc normalized string useful for comparisons and ordering if text data is mixed between nfc normalized and not nfc_normalize arde ch arde ch not_like_escape string like_specifier escape_character returns false if the string matches the like_specifier see pattern matching escape_character is used to search for wildcard characters in the string like_escape a c a c true ord string return ascii character code of the leftmost character in a string ord ü 252 position search_string in string return location of first occurrence of search_string in string counting from 1 returns 0 if no match found position b in abc 2 prefix string search_string return true if string starts with search_string prefix abc ab true printf format parameters formats a string using printf syntax printf benchmark s took d seconds csv 42 benchmark csv took 42 seconds regexp_full_match string regex returns true if the entire string matches the regex see pattern matching regexp_full_match anabanana an false regexp_matches string regex returns true if a part of string matches the regex see pattern matching regexp_matches anabanana an true regexp_replace string regex replacement modifiers replaces the first occurrence of regex with the replacement use g modifier to replace all occurrences instead see pattern matching select regexp_replace hello lo - he-lo regexp_split_to_array string regex alias of string_split_regex splits the string along the regex regexp_split_to_array hello world 42 hello world 42 repeat string count repeats the string count number of times repeat a 5 aaaaa replace string source target replaces any occurrences of the source with target in string replace hello l - he--o reverse string reverses the string reverse hello olleh right string count extract the right-most count characters right hello 3 llo rpad string count character pads the string with the character from the right until it has count characters rpad hello 10 hello rtrim string removes any spaces from the right side of the string rtrim test test rtrim string characters removes any occurrences of any of the characters from the right side of the string rtrim test test string similar to regex returns true if the string matches the regex identical to regexp_full_match see pattern matching hello similar to l false strlen string number of bytes in string length 1 strpos string search_string alias of instr return location of first occurrence of search_string in string counting from 1 returns 0 if no match found strpos test test es 2 strip_accents string strips accents from string strip_accents mühleisen muhleisen str_split string separator alias of string_split splits the string along the separator str_split hello world hello world str_split_regex string regex alias of string_split_regex splits the string along the regex str_split_regex hello world 42 hello world 42 string_split string separator splits the string along the separator string_split hello world hello world string_split_regex string regex splits the string along the regex string_split_regex hello world 42 hello world 42 string_to_array string separator alias of string_split splits the string along the separator string_to_array hello world hello world substr string start length alias of substring extract substring of length characters starting from character start note that a start value of 1 refers to the first character of the string substr hello 2 2 el substring string start length extract substring of length characters starting from character start note that a start value of 1 refers to the first character of the string substring hello 2 2 el suffix string search_string return true if string ends with search_string suffix abc bc true strpos string characters alias of instr return location of first occurrence of characters in string counting from 1 returns 0 if no match found strpos test test es 2 to_base64 blob convert a blob to a base64 encoded string alias of base64 to_base64 a blob qq trim string removes any spaces from either side of the string trim test test trim string characters removes any occurrences of any of the characters from either side of the string trim test test ucase string alias of upper convert string to upper case ucase hello hello unicode string returns the unicode code of the first character of the string unicode ü 252 upper string convert string to upper case upper hello hello text similarity functions these functions are used to measure the similarity of two strings using various metrics function description example result --- --- --- --- editdist3 string string alias of levenshtein for sqlite compatibility the minimum number of single-character edits insertions deletions or substitutions required to change one string to the other different case is considered different editdist3 duck db 3 hamming string string the number of positions with different characters for 2 strings of equal length different case is considered different hamming duck luck 1 jaccard string string the jaccard similarity between two strings different case is considered different returns a number between 0 and 1 jaccard duck luck 0 6 jaro_similarity string string the jaro similarity between two strings different case is considered different returns a number between 0 and 1 jaro_similarity duck duckdb 0 88 jaro_winkler_similarity string string the jaro-winkler similarity between two strings different case is considered different returns a number between 0 and 1 jaro_winkler_similarity duck duckdb 0 93 levenshtein string string the minimum number of single-character edits insertions deletions or substitutions required to change one string to the other different case is considered different levenshtein duck db 3 mismatches string string the number of positions with different characters for 2 strings of equal length different case is considered different mismatches duck luck 1",
- "category": "Functions",
- "url": "../docs/sql/functions/char",
- "blurb": "This section describes functions and operators for examining and manipulating string values. ␣ denotes a space..."
+ "title": "VALUES Clause",
+ "text": "the values clause is used to specify a fixed number of rows the values clause can be used as a stand-alone statement as part of the from clause or as input to an insert into statement examples -- generate two rows and directly return them values amsterdam 1 london 2 -- generate two rows as part of a from clause and rename the columns select from values amsterdam 1 london 2 cities name id -- generate two rows and insert them into a table insert into cities values amsterdam 1 london 2 -- create a table directly from a values clause create table cities as select from values amsterdam 1 london 2 cities name id syntax",
+ "category": "Query Syntax",
+ "url": "../docs/sql/query_syntax/values",
+ "blurb": "The VALUES clause is used to specify a fixed number of rows. The VALUES clause can be used as a stand-alone..."
},
{
- "title": "Functions",
- "text": "functions are query functions duckdb_functions table function shows the list of functions currently built into the system d select distinct on function_name function_name function_type return_type parameters parameter_types from duckdb_functions where function_type scalar limit 10 function_name function_type return_type parameters parameter_types log10 scalar double col0 double mod scalar tinyint col0 col1 tinyint tinyint date_diff scalar bigint col0 col1 col2 varchar date date writefile scalar varchar regexp_replace scalar varchar col0 col1 col2 col3 varchar varchar varchar varchar age scalar interval col0 timestamp age scalar interval col0 col1 timestamp timestamp datediff scalar bigint col0 col1 col2 varchar date date map scalar map year scalar bigint col0 timestamp with time zone currently the description and parameter names of functions are still missing more",
- "category": "Functions",
- "url": "../docs/sql/functions/overview",
- "blurb": "Functions are ... Query functions duckdb_functions table function shows the list of functions currently built into..."
+ "title": "FROM Clause",
+ "text": "the from clause specifies the source of the data on which the remainder of the query should operate logically the from clause is where the query starts execution the from clause can contain a single table a combination of multiple tables that are joined together or another select query inside a subquery node examples -- select all columns from the table called table_name select from table_name -- select all columns from the table called table_name in the schema schema_name select from schema_name table_name -- select the column i from the table function range where the first column of the range function is renamed to i select t i from range 100 as t i -- select all columns from the csv file called test csv select from test csv -- select all columns from a subquery select from select from table_name -- join two tables together select from table_name join other_table on table_name key other_table key -- select a 10 sample from a table select from table_name tablesample 10 -- select a sample of 10 rows from a table select from table_name tablesample 10 rows syntax",
+ "category": "Query Syntax",
+ "url": "../docs/sql/query_syntax/from",
+ "blurb": "The FROM clause can contain a single table, a combination of multiple tables that are joined together, or another SELECT..."
},
{
- "title": "Date Format",
- "text": "the strftime and strptime functions can be used to convert between dates timestamps and strings this is often required when parsing csv files displaying output to the user or transferring information between programs because there are many possible date representations these functions accept a format string that describes how the date or timestamp should be structured strftime examples strftime timestamp format converts timestamps or dates to strings according to the specified pattern select strftime date 1992-03-02 d m y -- 02 03 1992 select strftime timestamp 1992-03-02 20 32 45 a -d b y - i m s p -- monday 2 march 1992 - 08 32 45 pm strptime examples strptime string format converts strings to timestamps according to the specified pattern select strptime 02 03 1992 d m y -- 1992-03-02 00 00 00 select strptime monday 2 march 1992 - 08 32 45 pm a -d b y - i m s p -- 1992-03-02 20 32 45 csv parsing the date formats can also be specified during csv parsing either in the copy statement or in the read_csv function this can be done by either specifying a dateformat or a timestampformat or both dateformat will be used for converting dates and timestampformat will be used for converting timestamps below are some examples for how to use this -- in copy statement copy dates from test csv dateformat d m y timestampformat a -d b y - i m s p -- in read_csv function select from read_csv test csv dateformat m d y format specifiers below is a full list of all available format specifiers specifier description example --- --- --- --- a abbreviated weekday name sun mon a full weekday name sunday monday w weekday as a decimal number 0 1 6 d day of the month as a zero-padded decimal 01 02 31 -d day of the month as a decimal number 1 2 30 b abbreviated month name jan feb dec b full month name january february m month as a zero-padded decimal number 01 02 12 -m month as a decimal number 1 2 12 y year without century as a zero-padded decimal number 00 01 99 -y year without century as a decimal number 0 1 99 y year with century as a decimal number 2013 2019 etc h hour 24-hour clock as a zero-padded decimal number 00 01 23 -h hour 24-hour clock as a decimal number 0 1 23 i hour 12-hour clock as a zero-padded decimal number 01 02 12 -i hour 12-hour clock as a decimal number 1 2 12 p locale s am or pm am pm m minute as a zero-padded decimal number 00 01 59 -m minute as a decimal number 0 1 59 s second as a zero-padded decimal number 00 01 59 -s second as a decimal number 0 1 59 g millisecond as a decimal number zero-padded on the left 000 - 999 f microsecond as a decimal number zero-padded on the left 000000 - 999999 z utc offset in the form hhmm or -hhmm z time zone name j day of the year as a zero-padded decimal number 001 002 366 -j day of the year as a decimal number 1 2 366 u week number of the year sunday as the first day of the week 00 01 53 w week number of the year monday as the first day of the week 00 01 53 c iso date and time representation 1992-03-02 10 30 20 x iso date representation 1992-03-02 x iso time representation 10 30 20 a literal character",
- "category": "Functions",
- "url": "../docs/sql/functions/dateformat",
- "blurb": "The strftime and strptime functions can be used to convert between dates/timestamps and strings. This is often..."
+ "title": "UNNEST",
+ "text": "the unnest function is used to unnest a list by one level the function can be used as a regular scalar function but only in the select clause unnest is a special function in the sense that it changes the cardinality of the result the result of the unnest function is one tuple per entry in the list when unnest is combined with regular scalar expressions those expressions are repeated for every entry in the list when multiple lists are unnested in the same select clause the lists are unnested side-by-side if one list is longer than the other the shorter list will be padded with null values an empty list and a null list will both unnest to zero elements untyped and types null arguments will both return zero rows examples -- unnest a scalar list generating 3 rows 1 2 3 select unnest 1 2 3 -- unnest a scalar list generating 3 rows 1 10 2 11 3 null select unnest 1 2 3 unnest 10 11 -- unnest a scalar list generating 3 rows 1 10 2 10 3 10 select unnest 1 2 3 10 -- unnest a list column generated from a subquery select unnest l 10 from values 1 2 3 4 5 tbl l -- empty result select unnest -- zero rows untyped null select unnest null -- zero rows typed null select unnest null int",
+ "category": "Query Syntax",
+ "url": "../docs/sql/query_syntax/unnest",
+ "blurb": "The UNNEST function is used to unnest a list by one level. The function can be used as a regular scalar function, but..."
},
{
- "title": "Utility Functions",
- "text": "utility functions the functions below are difficult to categorize into specific function types and are broadly useful function description example result --- --- --- --- alias column return the name of the column alias column1 column1 coalesce expr return the first expression that evaluates to a non- null value accepts 1 or more parameters each expression can be a column literal value function result or many others coalesce null null default_string default_string current_schema return the name of the currently active schema default is main current_schema main current_schemas boolean return list of schemas pass a parameter of true to include implicit schemas current_schemas true temp main pg_catalog current_setting setting_name return the current value of the configuration setting current_setting access_mode automatic currval sequence_name return the current value of the sequence note that nextval must be called at least once prior to calling currval currval my_sequence_name 1 gen_random_uuid alias of uuid return a random uuid similar to this eeccb8c5-9943-b2bb-bb5e-222f4e14b687 gen_random_uuid various icu_sort_key string collator surrogate key used to sort special characters according to the specific locale collator parameter is optional valid only when icu extension is installed icu_sort_key ö de 460145960106 md5 string return an md5 one-way hash of the string md5 123 202cb962ac59075b964b07152d234b70 nextval sequence_name return the following value of the sequence nextval my_sequence_name 2 pg_typeof expression returns the lower case name of the data type of the result of the expression for postgres compatibility pg_typeof abc varchar stats expression returns a string with statistics about the expression expression can be a column constant or sql expression stats 5 min 5 max 5 has null false txid_current returns the current transaction s id a bigint it will assign a new one if the current transaction does not have one already txid_current various typeof expression returns the name of the data type of the result of the expression typeof abc varchar uuid return a random uuid similar to this eeccb8c5-9943-b2bb-bb5e-222f4e14b687 uuid various version return the currently active version of duckdb in this format v0 3 2 version various",
- "category": "Functions",
- "url": "../docs/sql/functions/utility",
- "blurb": "Utility Functions The functions below are difficult to categorize into specific function types and are broadly useful. ..."
+ "title": "HAVING Clause",
+ "text": "the having clause can be used after the group by clause to provide filter criteria after the grouping has been completed in terms of syntax the having clause is identical to the where clause but while the where clause occurs before the grouping the having clause occurs after the grouping examples -- count the number of entries in the addresses table that belong to each different city -- filtering out cities with a count below 50 select city count from addresses group by city having count 50 -- compute the average income per city per street_name -- filtering out cities with an average income bigger than twice the median income select city street_name avg income from addresses group by city street_name having avg income 2 median income syntax",
+ "category": "Query Syntax",
+ "url": "../docs/sql/query_syntax/having",
+ "blurb": "The HAVING clause can be used after the GROUP BY clause to provide filter criteria after the grouping has been..."
},
{
- "title": "Date Parts",
- "text": "the date_part and date_diff and date_trunc functions can be used to manipulate the fields of temporal types the fields are specified as strings that contain the part name of the field part specifiers below is a full list of all available date part specifiers the examples are the corresponding parts of the timestamp 2021-08-03 11 59 44 123456 usable as date part specifiers and in intervals specifier description synonyms example --- --- --- --- year gregorian year y years 2021 month gregorian month mon months mons 8 day gregorian day days d dayofmonth 3 decade gregorian decade decades 202 century gregorian century centuries 21 millennium gregorian millennium millenia millenium 3 quarter quarter of the year 1-4 quarters 3 microseconds sub-minute microseconds microsecond 44123456 milliseconds sub-minute milliseconds millisecond ms msec msecs 44123 second seconds seconds s 44 minute minutes minutes m 59 hour hours hours h 11 usable in date part specifiers only specifier description synonyms example --- --- --- --- epoch seconds since 1970-01-01 1627991984 dayofweek day of the week sunday 0 saturday 6 weekday dow 2 isodow iso day of the week monday 1 sunday 7 2 isoyear iso year number starts on monday of week containing jan 4th 2021 week week number weeks w 31 yearweek iso year and week number in yyyyww format 202131 dayofyear day of the year 1-365 366 doy 215 era gregorian era ce ad bce bc 1 timezone time zone offset in seconds 0 timezone_hour time zone offset hour portion 0 timezone_minute time zone offset minute portion 0 note that the time zone parts are all zero unless a time zone plugin such as icu has been installed to support timestamp with time zone part functions there are dedicated extraction functions to get certain subfields function description example result --- --- --- --- year date year year date 1992-02-15 1992 month date month month date 1992-02-15 2 day date day day date 1992-02-15 15 dayofmonth date day synonym dayofmonth date 1992-02-15 15 decade date decade year 10 decade date 1992-02-15 199 century date century century date 1992-02-15 20 millennium date millennium millennium date 1992-02-15 2 quarter date quarter quarter date 1992-02-15 1 microsecond date sub-minute microseconds microsecond timestamp 2021-08-03 11 59 44 123456 44123456 millisecond date sub-minute milliseconds millisecond timestamp 2021-08-03 11 59 44 123456 44123 second date seconds second timestamp 2021-08-03 11 59 44 123456 44 minute date minutes minute timestamp 2021-08-03 11 59 44 123456 59 hour date hours hour timestamp 2021-08-03 11 59 44 123456 11 epoch date seconds since 1970-01-01 epoch date 1992-02-15 698112000 dayofweek date numeric weekday sunday 0 saturday 6 dayofweek date 1992-02-15 6 weekday date numeric weekday synonym sunday 0 saturday 6 weekday date 1992-02-15 6 isodow date numeric iso weekday monday 1 sunday 7 isodow date 1992-02-15 6 isoyear date iso year number starts on monday of week containing jan 4th isoyear date 2022-01-01 2021 week date iso week week date 1992-02-15 7 weekofyear date iso week synonym weekofyear date 1992-02-15 7 yearweek date bigint of combined iso year number and 2-digit version of iso week number yearweek date 1992-02-15 199207 dayofyear date numeric iso weekday monday 1 sunday 7 isodow date 1992-02-15 46 quarter date quarter quarter date 1992-02-15 1 era date calendar era era date 0044-03-15 bc 0 timezone date time zone offset in minutes timezone date 1992-02-15 0 timezone_hour date time zone offset hour portion timezone_hour date 1992-02-15 0 timezone_minute date time zone offset minutes portion timezone_minute date 1992-02-15 0",
- "category": "Functions",
- "url": "../docs/sql/functions/datepart",
- "blurb": "The date_part and date_diff and date_trunc functions can be used to manipulate the fields of temporal types. The..."
+ "title": "QUALIFY Clause",
+ "text": "the qualify clause is used to filter the results of window functions this filtering of results is similar to how a having clause filters the results of aggregate functions applied based on the group by clause the qualify clause avoids the need for a subquery or with clause to perform this filtering much like having avoids a subquery an example using a with clause instead of qualify is included below the qualify examples note that this is filtering based on window functions not necessarily based on the window clause the window clause is optional and can be used to simplify the creation of multiple window function expressions the position of where to specify a qualify clause is following the window clause in a select statement window does not need to be specified and before the order by examples each of the following examples produce the same output located below -- filter based on a window function defined in the qualify clause select schema_name function_name -- in this example the function_rank column in the select clause is for reference row_number over partition by schema_name order by function_name as function_rank from duckdb_functions qualify row_number over partition by schema_name order by function_name 3 -- filter based on a window function defined in the select clause select schema_name function_name row_number over partition by schema_name order by function_name as function_rank from duckdb_functions qualify function_rank 3 -- filter based on a window function defined in the qualify clause but using the window clause select schema_name function_name -- in this example the function_rank column in the select clause is for reference row_number over my_window as function_rank from duckdb_functions window my_window as partition by schema_name order by function_name qualify row_number over my_window 3 -- filter based on a window function defined in the select clause but using the window clause select schema_name function_name row_number over my_window as function_rank from duckdb_functions window my_window as partition by schema_name order by function_name qualify function_rank 3 -- equivalent query based on a with clause without qualify clause with ranked_functions as select schema_name function_name row_number over partition by schema_name order by function_name as function_rank from duckdb_functions select from ranked_functions where function_rank 3 schema_name function_name function_rank --- --- --- main __postfix 1 main 2 pg_catalog col_description 1 pg_catalog format_pg_type 2 syntax",
+ "category": "Query Syntax",
+ "url": "../docs/sql/query_syntax/qualify",
+ "blurb": "The QUALIFY clause is used to filter the results of WINDOW functions."
},
{
- "title": "Time Functions",
- "text": "this section describes functions and operators for examining and manipulating time values time operators the table below shows the available mathematical operators for time types operator description example result --- --- --- --- addition of an interval time 01 02 03 interval 5 hour 06 02 03 - subtraction of an interval time 06 02 03 - interval 5 hour 01 02 03 time functions the table below shows the available scalar functions for time types function description example result --- --- --- --- current_time current time start of current transaction date_diff part starttime endtime the number of partition boundaries between the times date_diff hour time 01 02 03 time 06 01 03 5 date_part part time get subfield equivalent to extract date_part minute time 14 21 13 21 date_sub part starttime endtime the number of complete partitions between the times date_sub hour time 01 02 03 time 06 01 03 4 extract part from time get subfield from a time extract hour from time 14 21 13 14 make_time bigint bigint double the time for the given parts make_time 13 34 27 123456 13 34 27 123456 the only date parts that are defined for times are epoch hours minutes seconds milliseconds and microseconds",
- "category": "Functions",
- "url": "../docs/sql/functions/time",
- "blurb": "This section describes functions and operators for examining and manipulating TIME values. Time Operators The table..."
+ "title": "SAMPLE Clause",
+ "text": "the sample clause allows you to run the query on a sample from the base table this can significantly speed up processing of queries at the expense of accuracy in the result samples can also be used to quickly see a snapshot of the data when exploring a data set the sample clause is applied right after anything in the from clause i e after any joins but before the where clause or any aggregates see the sample page for more information examples -- select a sample of 1 of the addresses table using default system sampling select from addresses using sample 1 -- select a sample of 1 of the addresses table using bernoulli sampling select from addresses using sample 1 bernoulli -- select a sample of 10 rows from the subquery select from select from addresses using sample 10 rows syntax",
+ "category": "Query Syntax",
+ "url": "../docs/sql/query_syntax/sample",
+ "blurb": "The SAMPLE clause allows you to run the query on a sample from the base table. This can significantly speed up..."
},
{
- "title": "Nested Functions",
- "text": "this section describes functions and operators for examining and manipulating nested values there are three nested data types lists structs and maps list functions in the descriptions l is the three element list 4 5 6 function description example result -------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ --------------------------------------- ------------- list index bracket notation serves as an alias for list_extract l 3 6 list_extract list index extract the index th 1-based value from the list list_extract l 3 6 list_element list index alias for list_extract list_element l 3 6 array_extract list index alias for list_extract array_extract l 3 6 list begin end bracket notation with colon is an alias for list_slice missing arguments are interpreted as null s l 2 3 5 6 list_slice list begin end extract a sublist using slice conventions null s are interpreted as the bounds of the list negative values are accepted list_slice l 2 null 5 6 array_slice list begin end alias for list_slice array_slice l 2 null 5 6 array_pop_front list returns the list without the first element array_pop_front l 5 6 array_pop_back list returns the list without the last element array_pop_back l 4 5 list_value any create a list containing the argument values list_value 4 5 6 4 5 6 list_pack any alias for list_value list_pack 4 5 6 4 5 6 len list return the length of the list len 1 2 3 3 array_length list alias for len array_length 1 2 3 3 unnest list unnests a list by one level note that this is a special function that alters the cardinality of the result see the unnest page for more details unnest 1 2 3 1 2 3 list_concat list1 list2 concatenates two lists list_concat 2 3 4 5 6 2 3 4 5 6 list_cat list1 list2 alias for list_concat list_cat 2 3 4 5 6 2 3 4 5 6 array_concat list1 list2 alias for list_concat array_concat 2 3 4 5 6 2 3 4 5 6 array_cat list1 list2 alias for list_concat array_cat 2 3 4 5 6 2 3 4 5 6 list_prepend element list prepends element to list list_prepend 3 4 5 6 3 4 5 6 array_prepend element list alias for list_prepend array_prepend 3 4 5 6 3 4 5 6 array_push_front list element alias for list_prepend array_push_front l 3 3 4 5 6 list_append list element appends element to list list_append 2 3 4 2 3 4 array_append list element alias for list_append array_append 2 3 4 2 3 4 array_push_back list element alias for list_append array_push_back l 7 4 5 6 7 list_contains list element returns true if the list contains the element list_contains 1 2 null 1 true list_has list element alias for list_contains list_has 1 2 null 1 true array_contains list element alias for list_contains array_contains 1 2 null 1 true array_has list element alias for list_contains array_has 1 2 null 1 true list_position list element returns the index of the element if the list contains the element list_contains 1 2 null 2 2 list_indexof list element alias for list_position list_indexof 1 2 null 2 2 array_position list element alias for list_position array_position 1 2 null 2 2 array_indexof list element alias for list_position array_indexof 1 2 null 2 2 list_aggregate list name executes the aggregate function name on the elements of list see the list aggregates section for more details list_aggregate 1 2 null min 1 list_aggr list name alias for list_aggregate list_aggr 1 2 null min 1 array_aggregate list name alias for list_aggregate array_aggregate 1 2 null min 1 array_aggr list name alias for list_aggregate array_aggr 1 2 null min 1 list_sort list sorts the elements of the list see the sorting lists section for more details about the sorting order and the null sorting order list_sort 3 6 1 2 1 2 3 6 array_sort list alias for list_sort array_sort 3 6 1 2 1 2 3 6 list_reverse_sort list sorts the elements of the list in reverse order see the sorting lists section for more details about the null sorting order list_reverse_sort 3 6 1 2 6 3 2 1 array_reverse_sort list alias for list_reverse_sort array_reverse_sort 3 6 1 2 6 3 2 1 list_transform list lambda returns a list that is the result of applying the lambda function to each element of the input list see the lambda functions section for more details list_transform l x - x 1 5 6 7 array_transform list lambda alias for list_transform array_transform l x - x 1 5 6 7 list_apply list lambda alias for list_transform list_apply l x - x 1 5 6 7 array_apply list lambda alias for list_transform array_apply l x - x 1 5 6 7 list_filter list lambda constructs a list from those elements of the input list for which the lambda function returns true see the lambda functions section for more details list_filter l x - x 4 5 6 array_filter list lambda alias for list_filter array_filter l x - x 4 5 6 struct functions function description example result --- --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- --- --- struct entry dot notation serves as an alias for struct_extract i 3 s string s string struct entry bracket notation serves as an alias for struct_extract i 3 s string s string row any create a struct containing the argument values if the values are column references the entry name will be the column name otherwise it will be the string vn where n is the 1-based position of the argument row i i 4 i 4 i 3 v2 3 v3 0 struct_extract struct entry extract the named entry from the struct struct_extract s i 4 struct_pack name any create a struct containing the argument values the entry name will be the bound variable name struct_pack i 4 s string i 3 s string struct_insert struct name any add field s value s to an existing struct with the argument values the entry name s will be the bound variable name s struct_insert a 1 b 2 a 1 b 2 map functions function description example result --- --- --- --- map entry alias for element_at map 100 5 a b 100 a element_at map key return a list containing the value for a given key or an empty list if the key is not contained in the map the type of the key provided in the second parameter must match the type of the map s keys else an error is returned element_at map 100 5 42 43 100 42 map_extract map key alias of element_at return a list containing the value for a given key or an empty list if the key is not contained in the map the type of the key provided in the second parameter must match the type of the map s keys else an error is returned map_extract map 100 5 42 43 100 42 cardinality map return the size of the map or the number of entries in the map cardinality map 4 2 a b 2 map returns an empty map map range functions the functions range and generate_series create a list of values in the range between start and stop the start parameter is inclusive for the range function the stop parameter is exclusive while for generate_series it is inclusive based on the number of arguments the following variants exist range start stop step range start stop range stop generate_series start stop step generate_series start stop generate_series stop the default value of start is 0 and the default value of step is 1 select range 5 -- 0 1 2 3 4 select range 2 5 -- 2 3 4 select range 2 5 3 -- 2 select generate_series 5 -- 0 1 2 3 4 5 select generate_series 2 5 -- 2 3 4 5 select generate_series 2 5 3 -- 2 5 date ranges are also supported select from range date 1992-01-01 date 1992-03-01 interval 1 month range 1992-01-01 00 00 00 1992-02-01 00 00 00 list aggregates the function list_aggregate allows the execution of arbitrary existing aggregate functions on the elements of a list its first argument is the list column its second argument is the aggregate function name e g min histogram or sum select list_aggregate 1 2 -4 null min -- -4 select list_aggregate 2 4 8 42 sum -- 56 select list_aggregate 1 2 null 2 10 3 last -- 2 10 3 the following is a list of existing rewrites rewrites simplify the use of the list aggregate function by only taking the list column as their argument list_avg list_var_samp list_var_pop list_stddev_pop list_stddev_samp list_sem list_approx_count_distinct list_bit_xor list_bit_or list_bit_and list_bool_and list_bool_or list_count list_entropy list_last list_first list_kurtosis list_min list_max list_product list_skewness list_sum list_string_agg list_mode list_median list_mad and list_histogram select list_min 1 2 -4 null -- -4 select list_sum 2 4 8 42 -- 56 select list_last 1 2 null 2 10 3 -- 2 10 3 sorting lists the function list_sort sorts the elements of a list either in ascending or descending order in addition it allows to provide whether null values should be moved to the beginning or to the end of the list by default if no modifiers are provided duckdb sorts asc nulls first i e the values are sorted in ascending order and null values are placed first this is identical to the default sort order of sqlite the default sort order can be changed using these pragma statements list_sort leaves it open to the user whether they want to use the default sort order or a custom order list_sort takes up to two additional optional parameters the second parameter provides the sort order and can be either asc or desc the third parameter provides the null sort order and can be either nulls first or nulls last -- default sort order and default null sort order select list_sort 1 3 null 5 null -5 ---- null null -5 1 3 5 -- only providing the sort order select list_sort 1 3 null 2 asc ---- null 1 2 3 -- providing the sort order and the null sort order select list_sort 1 3 null 2 desc nulls first ---- null 3 2 1 list_reverse_sort has an optional second parameter providing the null sort order it can be either nulls first or nulls last -- default null sort order select list_sort 1 3 null 5 null -5 ---- null null -5 1 3 5 -- providing the null sort order select list_reverse_sort 1 3 null 2 nulls last ---- 3 2 1 null lambda functions parameter1 parameter2 - expression if the lambda function has only one parameter then the brackets can be omitted the parameters can have any names param - param 1 duck - contains concat duck db duck x y - x y transform list_transform list lambda returns a list that is the result of applying the lambda function to each element of the input list the lambda function must have exactly one left-hand side parameter the return type of the lambda function defines the type of the list elements -- incrementing each list element by one select list_transform 1 2 null 3 x - x 1 ---- 2 3 null 4 -- transforming strings select list_transform duck a b duck - concat duck db ---- duckdb adb bdb -- combining lambda functions with other functions select list_transform 5 null 6 x - coalesce x 0 1 ---- 6 1 7 filter list_filter list lambda constructs a list from those elements of the input list for which the lambda function returns true the lambda function must have exactly one left-hand side parameter and its return type must be of type boolean -- filter out negative values select list_filter 5 -6 null 7 x - x 0 ---- 5 7 -- divisible by 2 and 5 select list_filter list_filter 2 4 3 1 20 10 3 30 x - x 2 0 y - y 5 0 ---- 20 10 30 -- in combination with range to construct lists select list_filter 1 2 3 4 x - x 1 from range 4 ---- 1 2 3 4 2 3 4 3 4 4 lambda functions can be arbitrarily nested -- nested lambda functions to get all squares of even list elements select list_transform list_filter 0 1 2 3 4 5 x - x 2 0 y - y y ---- 0 4 16 generate_subscripts the generate_subscript arr dim function generates indexes along the dim th dimension of array arr select generate_subscripts 4 5 6 1 as i i 1 2 3 related functions there are also aggregate functions list and histogram that produces lists and lists of structs unnest is used to unnest a list by one level",
- "category": "Functions",
- "url": "../docs/sql/functions/nested",
- "blurb": "This section describes functions and operators for examining and manipulating nested values. There are three nested data..."
+ "title": "WHERE Clause",
+ "text": "the where clause specifies any filters to apply to the data this allows you to select only a subset of the data in which you are interested logically the where clause is applied immediately after the from clause examples -- select all rows that have id equal to 3 select from table_name where id 3 -- select all rows that match the given case-insensitive like expression select from table_name where name ilike mark -- select all rows that match the given composite expression select from table_name where id 3 or id 7 syntax",
+ "category": "Query Syntax",
+ "url": "../docs/sql/query_syntax/where",
+ "blurb": "The WHERE clause specifies any filters to apply to the data. This allows you to select only a subset of the data in..."
},
{
- "title": "Numeric Functions",
- "text": "numeric operators the table below shows the available mathematical operators for numeric types operator description example result --- --- --- --- addition 2 3 5 - subtraction 2 - 3 -1 multiplication 2 3 6 division 4 2 2 modulo remainder 5 4 1 exponent 3 4 81 exponent alias for 3 4 81 bitwise and 91 15 11 bitwise or 32 3 35 bitwise shift left 1 4 16 bitwise shift right 8 2 2 factorial of x computes the product of the current integer and all integers below it 4 24 the modulo bitwise and factorial operators work only on integral data types whereas the others are available for all numeric data types numeric functions the table below shows the available mathematical functions function description example result --- --- --- --- abs x absolute value abs -17 4 17 4 acos x computes the arccosine of x acos 0 5 1 0471975511965976 asin x computes the arcsine of x asin 0 5 0 5235987755982989 atan x computes the arctangent of x atan 0 5 0 4636476090008061 atan2 x y computes the arctangent x y atan2 0 5 0 5 0 7853981633974483 bit_count x returns the number of bits that are set bit_count 31 5 cbrt x returns the cube root of the number cbrt 8 2 ceil x rounds the number up ceil 17 4 18 ceiling x rounds the number up alias of ceil ceiling 17 4 18 chr x returns a character which is corresponding the the ascii code value or unicode code point chr 65 a cos x computes the cosine of x cos 90 -0 4480736161291701 cot x computes the cotangent of x cot 0 5 1 830487721712452 degrees x converts radians to degrees degrees pi 180 even x round to next even number by rounding away from zero even 2 9 4 factorial x see operator computes the product of the current integer and all integers below it factorial 4 24 floor x rounds the number down floor 17 4 17 gamma x interpolation of x-1 factorial so decimal inputs are allowed gamma 5 5 52 34277778455352 greatest x1 x2 selects the largest value greatest 3 2 4 4 4 isfinite x returns true if the floating point value is finite false otherwise isfinite 5 5 true isinf x returns true if the floating point value is infinite false otherwise isinf infinity true isnan x returns true if the floating point value is not a number false otherwise isnan nan true least x1 x2 selects the smallest value least 3 2 4 4 2 lgamma x computes the log of the gamma function lgamma 2 0 ln x computes the natural logarithm of x ln 2 0 693 log x computes the 10-log of x log 100 2 log2 x computes the 2-log of x log2 8 3 log10 x alias of log computes the 10-log of x log10 1000 3 nextafter x y return the next floating point value after x in the direction of y nextafter 1 float 2 float 1 0000001 pi returns the value of pi pi 3 141592653589793 pow x y computes x to the power of y pow 2 3 8 power x y alias of pow computes x to the power of y power 2 3 8 radians x converts degrees to radians radians 90 1 5707963267948966 random returns a random number between 0 and 1 random various round v numeric s int round to s decimal places round 42 4332 2 42 43 setseed x sets the seed to be used for the random function setseed 0 42 sin x computes the sin of x sin 90 0 8939966636005579 sign x returns the sign of x as -1 0 or 1 sign -349 -1 sqrt x returns the square root of the number sqrt 9 3 xor x bitwise xor xor 17 5 20 tan x computes the tangent of x tan 90 -1 995200412208242 absolute value parentheses optional if operating on a column -2 2",
- "category": "Functions",
- "url": "../docs/sql/functions/numeric",
- "blurb": "Numeric Operators The table below shows the available mathematical operators for numeric types. | Operator |..."
+ "title": "GROUP BY Clause",
+ "text": "the group by clause specifies which grouping columns should be used to perform any aggregations in the select clause if the group by clause is specified the query is always an aggregate query even if no aggregations are present in the select clause when a group by clause is specified all tuples that have matching data in the grouping columns i e all tuples that belong to the same group will be combined the values of the grouping columns themselves are unchanged and any other columns can be combined using an aggregate function such as count sum avg etc normally the group by clause groups along a single dimension using the grouping sets cube or rollup clauses it is possible to group along multiple dimensions see the grouping sets page for more information examples -- count the number of entries in the addresses table that belong to each different city select city count from addresses group by city -- compute the average income per city per street_name select city street_name avg income from addresses group by city street_name syntax",
+ "category": "Query Syntax",
+ "url": "../docs/sql/query_syntax/groupby",
+ "blurb": "The GROUP BY clause specifies which grouping columns should be used to perform any aggregations in the SELECT clause...."
},
{
- "title": "Create Table",
- "text": "the create table statement creates a table in the catalog examples -- create a table with two integer columns i and j create table t1 i integer j integer -- create a table with a primary key create table t1 id integer primary key j varchar -- create a table with a composite primary key create table t1 id integer j varchar primary key id j -- create a table with various different types and constraints create table t1 i integer not null decimalnr double check decimalnr 10 date date unique time timestamp -- create a table from the result of a query create table t1 as select 42 as i 84 as j -- create a table from a csv file using auto-detect i e automatically detecting column names and types create table t1 as select from read_csv_auto path file csv temporary tables temporary tables can be created using a create temp table statement see diagram below temporary tables are session scoped similar to postgres for example meaning that only the specific connection that created them can access them and once the connection to duckdb is closed they will be automatically dropped temporary tables reside in memory rather than on disk even when connecting to a persistent duckdb but if the temp_directory configuration is set when connecting or with a set command data will be spilled to disk if memory becomes constrained -- create a temporary table from a csv file using auto-detect i e automatically detecting column names and types create temp table t1 as select from read_csv_auto path file csv -- allow temporary tables to off-load excess memory to disk set temp_directory path to directory create or replace the create or replace syntax allows a new table to be created or for an existing table to be overwritten by the new table this is shorthand for dropping the existing table and then creating the new one -- create a table with two integer columns i and j even if t1 already exists create or replace table t1 i integer j integer if not exists the if not exists syntax will only proceed with the creation of the table if it does not already exist if the table already exists no action will be taken and the existing table will remain in the database -- create a table with two integer columns i and j only if t1 does not exist yet create table if not exists t1 i integer j integer generated columns the type generated always as expr virtual stored syntax will create a generated column the data in this kind of column is generated from its expression which can reference other regular or generated columns of the table since they are produced by calculations these columns can not be inserted into directly duckdb can infer the type of the generated column based on the expression s return type this allows you to leave out the type when declaring a generated column it is possible to explicitly set a type but insertions into the referenced columns might fail if the type can not be cast to the type of the generated column generated columns come in two varieties virtual and stored the data of virtual generated columns is not stored on disk instead it is computed from the expression every time the column is referenced through a select statement the data of stored generated columns is stored on disk and is computed every time the data of their dependencies change through an insert update drop statement currently only the virtual kind is supported and it is also the default option if the last field is left blank -- the simplest syntax for a generated column -- the type is derived from the expression and the variant defaults to virtual create table t1 x float two_x as 2 x -- fully specifying the same generated column for completeness create table t1 x float two_x float generated always as 2 x virtual syntax",
- "category": "Statements",
- "url": "../docs/sql/statements/create_table",
- "blurb": "The CREATE TABLE statement creates a table in the catalog. Examples -- create a table with two integer columns (i..."
+ "title": "WINDOW Clause",
+ "text": "the window clause allows you to specify named windows that can be used within window functions these are useful when you have multiple window functions as they allow you to avoid repeating the same window clause syntax",
+ "category": "Query Syntax",
+ "url": "../docs/sql/query_syntax/window",
+ "blurb": "The WINDOW clause allows you to specify named windows that can be used within window functions. These are useful when..."
},
{
- "title": "Drop Statement",
- "text": "the drop statement removes a catalog entry added previously with the create command examples -- delete the table with the name tbl drop table tbl -- drop the view with the name v1 do not throw an error if the view does not exist drop view if exists v1 syntax the optional if exists clause suppresses the error that would normally result if the table does not exist by default or if the restrict clause is provided the entry will not be dropped if there are any other objects that depend on it if the cascade clause is provided then all the objects that are dependent on the object will be dropped as well create schema myschema create table myschema t1 i integer -- error cannot drop myschema because the table myschema t1 depends on it drop schema myschema -- cascade drops both myschema and myschema 1 drop schema myschema cascade",
- "category": "Statements",
- "url": "../docs/sql/statements/drop",
- "blurb": "The DROP statement removes a catalog entry added previously with the CREATE command. Examples -- delete the table..."
+ "title": "LIMIT Clause",
+ "text": "limit is an output modifier logically it is applied at the very end of the query the limit clause restricts the amount of rows fetched the offset clause indicates at which position to start reading the values i e the first offset values are ignored note that while limit can be used without an order by clause the results might not be deterministic without the order by clause this can still be useful however for example when you want to inspect a quick snapshot of the data examples -- select the first 5 rows from the addresses table select from addresses limit 5 -- select the 5 rows from the addresses table starting at position 5 i e ignoring the first 5 rows select from addresses limit 5 offset 5 -- select the top 5 cities with the highest population select city count as population from addresses group by city order by population desc limit 5 syntax",
+ "category": "Query Syntax",
+ "url": "../docs/sql/query_syntax/limit",
+ "blurb": "LIMIT is an output modifier. Logically it is applied at the very end of the query. The LIMIT clause restricts the..."
},
{
- "title": "Create Macro",
- "text": "the create macro statement creates a scalar macro in the catalog examples -- create a macro that adds two expressions a and b create macro add a b as a b -- create a macro for a case expression create macro ifelse a b c as case when a then b else c end -- create a macro that does a subquery create macro one as select 1 -- create a macro with a common table expression -- parameter names get priority over column names disambiguate using the table name create macro plus_one a as with cte as select 1 as a select cte a a from cte -- macro s are schema-dependent and have an alias function create function main myavg x as sum x count x -- create a macro with default constant parameters create macro add_default a b 5 as a b -- create a macro arr_append with a functionality equivalent to array_append create macro arr_append l e as list_concat l list_value e syntax macros allow you to create shortcuts for combinations of expressions -- failure cannot find column b create macro add a as a b -- this works create macro add a b as a b -- error cannot bind varchar integer select add hello 3 -- success select add 1 2 -- 3 macro s can have default parameters -- b is a default parameter create macro add_default a b 5 as a b -- the following will result in 42 select add_default 37 -- error add_default only has one positional parameter select add_default 40 2 -- success default parameters are used by assigning them like so select add_default 40 b 2 -- error default parameters must come after positional parameters select add_default b 2 40 -- the order of default parameters does not matter create macro triple_add a b 5 c 10 as a b c -- success select triple_add 40 c 1 b 1 -- 42 when macro s are used they are expanded i e replaced with the original expression and the parameters within the expanded expression are replaced with the supplied arguments step by step -- the add macro we defined above is used in a query select add 40 2 -- internally add is replaced with its definition of a b select a b -- then the parameters are replaced by the supplied arguments select 40 2 -- 42",
- "category": "Statements",
- "url": "../docs/sql/statements/create_macro",
- "blurb": "The CREATE MACRO statement creates a scalar macro in the catalog. Examples -- create a macro that adds two..."
+ "title": "Window Functions",
+ "text": "window functions can only be used in the select clause to share over specifications between functions use the statement s window clause and use the over window-name syntax general-purpose window functions the table below shows the available general window functions function return type description example --- --- --- --- row_number bigint the number of the current row within the partition counting from 1 row_number rank bigint the rank of the current row with gaps same as row_number of its first peer rank dense_rank bigint the rank of the current row without gaps this function counts peer groups dense_rank percent_rank double the relative rank of the current row rank - 1 total partition rows - 1 percent_rank cume_dist double the cumulative distribution number of partition rows preceding or peer with current row total partition rows cume_dist ntile num_buckets integer bigint an integer ranging from 1 to the argument value dividing the partition as equally as possible ntile 4 lag expr any offset integer default any same type as expr returns expr evaluated at the row that is offset rows before the current row within the partition if there is no such row instead return default which must be of the same type as expr both offset and default are evaluated with respect to the current row if omitted offset defaults to 1 and default to null lag column 3 0 lead expr any offset integer default any same type as expr returns expr evaluated at the row that is offset rows after the current row within the partition if there is no such row instead return default which must be of the same type as expr both offset and default are evaluated with respect to the current row if omitted offset defaults to 1 and default to null lead column 3 0 first_value expr any same type as expr returns expr evaluated at the row that is the first row of the window frame first_value column last_value expr any same type as expr returns expr evaluated at the row that is the last row of the window frame last_value column nth_value expr any nth integer same type as expr returns expr evaluated at the nth row of the window frame counting from 1 null if no such row nth_value column 2 aggregate window functions all aggregate functions can be used in a windowing context evaluation windowing works by breaking a relation up into independent partitions ordering those partitions and then computing a new column for each row as a function of the nearby values some window functions depend only on the partition boundary and the ordering but a few including all the aggregates also use a frame frames are specified as a number of rows on either side preceding or following of the current row the distance can either be specified as a number of rows or a range of values using the partition s ordering value and a distance the full syntax is shown in the diagram at the top of the page and this diagram visually illustrates computation environment partition and ordering partitioning breaks the relation up into independent unrelated pieces partitioning is optional and if none is specified then the entire relation is treated as a single partition window functions cannot access values outside of the partition containing the row they are being evaluated at ordering is also optional but without it the results are not well-defined each partition is ordered using the same ordering clause here is a table of power generation data after partitioning by plant and ordering by date it will have this layout plant date mwh --- --- --- boston 2019-01-02 564337 boston 2019-01-03 507405 boston 2019-01-04 528523 boston 2019-01-05 469538 boston 2019-01-06 474163 boston 2019-01-07 507213 boston 2019-01-08 613040 boston 2019-01-09 582588 boston 2019-01-10 499506 boston 2019-01-11 482014 boston 2019-01-12 486134 boston 2019-01-13 531518 worcester 2019-01-02 118860 worcester 2019-01-03 101977 worcester 2019-01-04 106054 worcester 2019-01-05 92182 worcester 2019-01-06 94492 worcester 2019-01-07 99932 worcester 2019-01-08 118854 worcester 2019-01-09 113506 worcester 2019-01-10 96644 worcester 2019-01-11 93806 worcester 2019-01-12 98963 worcester 2019-01-13 107170 in what follows we shall use this table or small sections of it to illustrate various pieces of window function evaluation the simplest window function is row_number this function just computes the 1-based row number within the partition using the query select plant date row_number over as row from history order by 1 2 the result will be plant date row --- --- --- boston 2019-01-02 1 boston 2019-01-03 2 boston 2019-01-04 3 worcester 2019-01-02 1 worcester 2019-01-03 2 worcester 2019-01-04 3 note that even though the function is computed with an order by clause the result does not have to be sorted so the select also needs to be explicitly sorted if that is desired framing framing specifies a set of rows relative to each row where the function is evaluated the distance from the current row is given as an expression either preceding or following the current row this distance can either be specified as an integral number of rows or as a range delta expression from the value of the ordering expression for a range specification there must be only one ordering expression and it has to support addition and subtraction i e numbers or interval s the default values for frames are from unbounded preceding to current row it is invalid for a frame to start after it ends row framing here is a simple row frame query using an aggregate function select points sum points over rows between 1 preceding and 1 following we from results this query computes the sum of each point and the points on either side of it notice that at the edge of the partition there are only two values added together this is because frames are cropped to the edge of the partition range framing returning to the power data suppose the data is noisy we might want to compute a 7 day moving average for each plant to smooth out the noise to do this we can use this window query select plant date avg mwh over partition by plant order by date asc range between interval 3 days preceding and interval 3 days following as mwh 7-day moving average from generation history order by 1 2 this query partitions the data by plant to keep the different power plants data separate orders each plant s partition by date to put the energy measurements next to each other and uses a range frame of three days on either side of each day for the avg to handle any missing days this is the result plant date mwh 7-day br moving average --- --- --- boston 2019-01-02 517450 75 boston 2019-01-03 508793 20 boston 2019-01-04 508529 83 boston 2019-01-13 499793 00 worcester 2019-01-02 104768 25 worcester 2019-01-03 102713 00 worcester 2019-01-04 102249 50 window clauses multiple different over clauses can be specified in the same select and each will be computed separately often however we want to use the same layout for multiple window functions the window clause can be used to define a named window that can be shared between multiple window functions select plant date min mwh over seven as mwh 7-day moving minimum avg mwh over seven as mwh 7-day moving average max mwh over seven as mwh 7-day moving maximum from generation history window seven as partition by plant order by date asc range between interval 3 days preceding and interval 3 days following order by 1 2 the three window functions will also share the data layout which will improve performance box and whisker queries all aggregates can be used as windowing functions including the complex statistical functions these function implementations have been optimised for windowing and we can use the window syntax to write queries that generate the data for moving box-and-whisker plots select plant date min mwh over seven as mwh 7-day moving minimum quantile_cont mwh 0 25 0 5 0 75 over seven as mwh 7-day moving iqr max mwh over seven as mwh 7-day moving maximum from generation history window seven as partition by plant order by date asc range between interval 3 days preceding and interval 3 days following order by 1 2",
+ "category": "SQL",
+ "url": "../docs/sql/window_functions",
+ "blurb": "Window functions can only be used in the SELECT clause. To share OVER specifications between functions, use the..."
},
{
- "title": "Alter Table",
- "text": "the alter table statement changes the schema of an existing table in the catalog examples -- add a new column with name k to the table integers it will be filled with the default value null alter table integers add column k integer -- add a new column with name l to the table integers it will be filled with the default value 10 alter table integers add column l integer default 10 -- drop the column k from the table integers alter table integers drop k -- change the type of the column i to the type varchar using a standard cast alter table integers alter i type varchar -- change the type of the column i to the type varchar using the specified expression to convert the data for each row alter table integers alter i set data type varchar using concat i _ j -- set the default value of a column alter table integers alter column i set default 10 -- drop the default value of a column alter table integers alter column i drop default -- rename a table alter table integers rename to integers_old -- rename a column of a table alter table integers rename i to j syntax alter table changes the schema of an existing table all the changes made by alter table fully respect the transactional semantics - that is - they will not be visible to other transactions until committed and can be fully reverted through a rollback rename table -- rename a table alter table integers rename to integers_old the rename to clause renames an entire table changing its name in the schema note that any views that rely on the table are not automatically updated rename column -- rename a column of a table alter table integers rename i to j alter table integers rename column j to k the rename column clause renames a single column within a table any constraints that rely on this name e g check constraints are automatically updated however note that any views that rely on this column name are not automatically updated add column -- add a new column with name k to the table integers it will be filled with the default value null alter table integers add column k integer -- add a new column with name l to the table integers it will be filled with the default value 10 alter table integers add column l integer default 10 the add column clause can be used to add a new column of a specified type to a table the new column will be filled with the specified default value or null if none is specified drop column -- drop the column k from the table integers alter table integers drop k the drop column clause can be used to remove a column from a table note that columns can only be removed if they do not have any indexes that rely on them this includes any indexes created as part of a primary key or unique constraint columns that are part of multi-column check constraints cannot be dropped either alter type -- change the type of the column i to the type varchar using a standard cast alter table integers alter i type varchar -- change the type of the column i to the type varchar using the specified expression to convert the data for each row alter table integers alter i set data type varchar using concat i _ j the set data type clause changes the type of a column in a table any data present in the column is converted according to the provided expression in the using clause or if the using clause is absent cast to the new data type note that columns can only have their type changed if they do not have any indexes that rely on them and are not part of any check constraints set drop default -- set the default value of a column alter table integers alter column i set default 10 -- drop the default value of a column alter table integers alter column i drop default the set drop default clause modifies the default value of an existing column note that this does not modify any existing data in the column dropping the default is equivalent to setting the default value to null",
- "category": "Statements",
- "url": "../docs/sql/statements/alter_table",
- "blurb": "The ALTER TABLE statement changes the schema of an existing table in the catalog. Examples -- add a new column with..."
+ "title": "Pragmas",
+ "text": "the pragma statement is an sql extension adopted by duckdb from sqlite pragma statements can be issued in a similar manner to regular sql statements pragma commands may alter the internal state of the database engine and can influence the subsequent execution or behavior of the engine list of supported pragma statements below is a list of supported pragma statements database_list show_tables table_info show -- list all databases usually one pragma database_list -- list all tables pragma show_tables -- get info for a specific table pragma table_info table_name -- also show table structure but slightly different format for compatibility pragma show table_name table_info returns information about the columns of the table with name table_name the exact format of the table returned is given below cid integer -- cid of the column name varchar -- name fo the column type varchar -- type of the column notnull boolean -- if the column is marked as not null dflt_value varchar -- default value of the column or null if not specified pk boolean -- part of the primary key or not memory_limit threads -- set the memory limit pragma memory_limit 1gb -- set the amount of threads for parallel query execution pragma threads 4 database_size -- get the file and memory size of each database pragma database_size database_size returns information about the file and memory size of each table the column types of the returned results are given below database_size varchar -- total block count times the block size block_size bigint -- database block size total_blocks bigint -- total blocks in the database used_blocks bigint -- used blocks in the database free_blocks bigint -- free blocks in the database wal_size varchar -- write ahead log size memory_usage varchar -- memory used by the database buffer manager memory_limit varchar -- maximum memory allowed for the database collations default_collation -- list all available collations pragma collations -- set the default collation to one of the available ones pragma default_collation nocase default_null_order default_order -- set the ordering for nulls to be either nulls first or nulls last pragma default_null_order nulls last -- set the default result set ordering direction to ascending or descending pragma default_order descending version -- show duckdb version pragma version enable_progress_bar enable_profiling disable_profiling profiling_output -- show progress bar when running queries pragma enable_progress_bar -- enable profiling pragma enable_profiling -- enable profiling in a specified format pragma enable_profiling json query_tree query_tree_optimizer -- disable profiling pragma disable_profiling -- specify a file to save the profiling output to pragma profiling_output path to file json pragma profile_output path to file json enable the gathering and printing of profiling information after the execution of a query optionally the format of the resulting profiling information can be specified as either json query_tree or query_tree_optimizer the default format is query_tree which prints the physical operator tree together with the timings and cardinalities of each operator in the tree to the screen below is an example output of the profiling information for the simple query select 42 query profiling information select 42 total time 0 0001s projection 42 1 0 00s dummy_scan 1 0 00s the printing of profiling information can be disabled again using disable_profiling by default profiling information is printed to the console however if you prefer to write the profiling information to a file the pragma profiling_output can be used to write to a specified file note that the file contents will be overwritten for every new query that is issued hence the file will only contain the profiling information of the last query that is run disable_optimizer enable_optimizer -- disables the query optimizer pragma disable_optimizer -- enables the query optimizer pragma enable_optimizer log_query_path explain_output enable_verification disable_verification force_parallelism disable_force_parallelism -- set a path for query logging pragma log_query_path tmp duckdb_log -- disable query logging again pragma log_query_path -- either show all or only optimized plans in the explain output pragma explain_output optimized -- enable query verification for development pragma enable_verification -- disable query verification for development pragma disable_verification -- enable force parallel query processing for development pragma force_parallelism -- disable force parallel query processing for development pragma disable_force_parallelism -- force index joins where applicable pragma force_index_join these are pragma s mostly used for development and internal testing create_fts_index drop_fts_index only available when the fts extension is built documented here temp directory for spilling data to disk -- defaults to tmp pragma temp_directory path to temp tmp",
+ "category": "SQL",
+ "url": "../docs/sql/pragmas",
+ "blurb": "The PRAGMA statement is an SQL extension adopted by DuckDB from SQLite. PRAGMA statements can be issued in a similar..."
},
{
- "title": "Export & Import Database",
- "text": "the export database command allows you to export the contents of the database to a specific directory the import database command allows you to then read the contents again examples -- export the database to the target directory export database target_directory -- export the table contents with the given options export database target_directory format csv delimiter -- export the table contents as parquet export database target_directory format parquet --reload the database again import database target_directory syntax the export database command exports the full contents of the database - including schema information tables views and sequences - to a specific directory that can then be loaded again the created directory will be structured as follows target_directory schema sql target_directory load sql target_directory t_1 csv target_directory t_n csv the schema sql file contains the schema statements that are found in the database it contains any create schema create table create view and create sequence commands that are necessary to re-construct the database the load sql file contains a set of copy statements that can be used to read the data from the csv files again the file contains a single copy statement for every table found in the schema the database can be reloaded by using the import database command again or manually by running schema sql followed by load sql to re-load the data",
- "category": "Statements",
- "url": "../docs/sql/statements/export",
- "blurb": "The EXPORT DATABASE command allows you to export the contents of the database to a specific directory. The IMPORT..."
+ "title": "Configuration",
+ "text": "duckdb has a number of configuration options that can be used to change the behavior of the system the configuration options can be set using either the set statement or the pragma statement examples -- set the memory limit of the system to 10gb set memory_limit 10gb -- configure the system to use 1 thread set threads to 1 -- enable printing of a progress bar during long-running queries set enable_progress_bar true -- set the default null order to nulls last pragma default_null_order nulls_last -- show a list of all available settings select from duckdb_settings -- return the current value of a specific setting -- this example returns automatic select current_setting access_mode configuration reference below is a list of all available settings name description input_type default_value ------------------------------------------ --------------------------------------------------------------------------------------------------------------------------------------------------------- ------------ ------------------ calendar the current calendar varchar gregorian timezone the current time zone varchar system timezone access_mode access mode of the database automatic read_only or read_write varchar automatic allocator_flush_threshold peak allocation threshold at which to flush the allocator after completing a task varchar 134 2mb allow_unsigned_extensions allow to load extensions with invalid or missing signatures boolean false arrow_large_buffer_size if arrow buffers for strings blobs uuids and bits should be exported using large buffers boolean false binary_as_string in parquet files interpret binary data as a string boolean checkpoint_threshold wal_autocheckpoint the wal size threshold at which to automatically trigger a checkpoint e g 1gb varchar 16 7mb custom_extension_repository overrides the custom endpoint for remote extension installation varchar default_collation the collation setting used when none is specified varchar default_null_order null_order null ordering used when none is specified nulls_first or nulls_last varchar nulls_last default_order the order type used when none is specified asc or desc varchar asc disabled_filesystems disable specific file systems preventing access e g localfilesystem varchar enable_external_access allow the database to access external state through e g loading installing modules copy to from csv readers pandas replacement scans etc boolean true enable_fsst_vectors allow scans on fsst compressed segments to emit compressed vectors to utilize late decompression boolean false enable_http_metadata_cache whether or not the global http metadata is used to cache http metadata boolean false enable_object_cache whether or not object cache is used to cache e g parquet metadata boolean false enable_profiling enables profiling and sets the output format json query_tree query_tree_optimizer varchar null enable_progress_bar enables the progress bar printing progress to the terminal for long queries boolean false enable_progress_bar_print controls the printing of the progress bar when enable_progress_bar is true boolean true experimental_parallel_csv whether or not to use the experimental parallel csv reader boolean null explain_output output of explain statements all optimized_only physical_only varchar physical_only extension_directory set the directory to store extensions in varchar external_threads the number of external threads that work on duckdb tasks bigint 0 file_search_path a comma separated list of directories to search for input files varchar force_download forces upfront download of file boolean 0 home_directory sets the home directory used by the system varchar http_retries http retries on i o error default 3 ubigint 3 http_retry_backoff backoff factor for exponentially increasing retry wait time default 4 float 4 http_retry_wait_ms time between retries default 100ms ubigint 100 http_timeout http timeout read write connection retry default 30000ms ubigint 30000 immediate_transaction_mode whether transactions should be started lazily when needed or immediately when begin transaction is called boolean false integer_division whether or not the operator defaults to integer division or to floating point division boolean 0 lock_configuration whether or not the configuration can be altered boolean false log_query_path specifies the path to which queries should be logged default empty string queries are not logged varchar null max_expression_depth the maximum expression depth limit in the parser warning increasing this setting and using very deep expressions might lead to stack overflow errors ubigint 1000 max_memory memory_limit the maximum memory of the system e g 1gb varchar 75 of ram ordered_aggregate_threshold the number of rows to accumulate before sorting used for tuning ubigint 262144 password the password to use ignored for legacy compatibility varchar null perfect_ht_threshold threshold in bytes for when to use a perfect hash table default 12 bigint 12 pivot_filter_threshold the threshold to switch from using filtered aggregates to list with a dedicated pivot operator bigint 10 pivot_limit the maximum number of pivot columns in a pivot statement default 100000 bigint 100000 prefer_range_joins force use of range joins with mixed predicates boolean false preserve_identifier_case whether or not to preserve the identifier case instead of always lowercasing all non-quoted identifiers boolean true preserve_insertion_order whether or not to preserve insertion order if set to false the system is allowed to re-order any results that do not contain order by clauses boolean true profile_output profiling_output the file to which profile output should be saved or empty to print to the terminal varchar profiler_history_size sets the profiler history size bigint null profiling_mode the profiling mode standard or detailed varchar null progress_bar_time sets the time in milliseconds how long a query needs to take before we start printing a progress bar bigint 2000 s3_access_key_id s3 access key id varchar s3_endpoint s3 endpoint default s3 amazonaws com varchar s3 amazonaws com s3_region s3 region varchar s3_secret_access_key s3 access key varchar s3_session_token s3 session token varchar s3_uploader_max_filesize s3 uploader max filesize between 50gb and 5tb default 800gb varchar 800gb s3_uploader_max_parts_per_file s3 uploader max parts per file between 1 and 10000 default 10000 ubigint 10000 s3_uploader_thread_limit s3 uploader global thread limit default 50 ubigint 50 s3_url_compatibility_mode disable globs and query parameters on s3 urls boolean 0 s3_url_style s3 url style vhost default or path varchar vhost s3_use_ssl s3 use ssl default true boolean 1 schema sets the default search schema equivalent to setting search_path to a single value varchar main search_path sets the default search search path as a comma-separated list of values varchar temp_directory set the directory to which to write temp files varchar threads worker_threads the number of total threads used by the system bigint cores username user the username to use ignored for legacy compatibility varchar null",
+ "category": "SQL",
+ "url": "../docs/sql/configuration",
+ "blurb": "DuckDB has a number of configuration options that can be used to change the behavior of the system. The configuration..."
},
{
- "title": "Statements Overview",
- "text": "more",
+ "title": "Copy",
+ "text": "copy moves data between duckdb tables and external comma separated value csv files csv import copy from imports data into duckdb from an external csv file into an existing table the data is appended to whatever data is in the table already the amount of columns inside the file must match the amount of columns in the table table_name and the contents of the columns must be convertible to the column types of the table in case this is not possible an error will be thrown if a list of columns is specified copy will only copy the data in the specified columns from the file if there are any columns in the table that are not in the column list copy from will insert the default values for those columns -- copy the contents of a comma-separated file test csv without a header into the table test copy test from test csv -- copy the contents of a comma-separated file with a header into the category table copy category from categories csv header -- copy the contents of lineitem tbl into the lineitem table where the contents are delimited by a pipe character copy lineitem from lineitem tbl delimiter -- copy the contents of lineitem tbl into the lineitem table where the delimiter quote character and presence of a header are automatically detected copy lineitem from lineitem tbl auto_detect true -- read the contents of a comma-separated file names csv into the name column of the category table any other columns of this table are filled with their default value copy category name from names csv syntax csv export copy to exports data from duckdb to an external csv file it has mostly the same set of options as copy from however in the case of copy to the options specify how the csv file should be written to disk any csv file created by copy to can be copied back into the database by using copy from with the same set of options the copy to function can be called specifying either a table name or a query when a table name is specified the contents of the entire table will be written into the resulting csv file when a query is specified the query is executed and the result of the query is written to the resulting file -- copy the contents of the lineitem table to the file lineitem tbl where the columns are delimited by a pipe character including a header line copy lineitem to lineitem tbl delimiter header -- copy the l_orderkey column of the lineitem table to the file orderkey tbl copy lineitem l_orderkey to orderkey tbl delimiter -- copy the result of a query to the file query csv including a header with column names copy select 42 as a hello as b to query csv with header 1 delimiter syntax parameters name description --- --- table_name the name optionally schema-qualified of an existing table column_name an optional list of columns to be copied if no column list is specified all columns of the table will be copied filename the path and name of the input or output file boolean specifies whether the selected option should be turned on or off you can write true on or 1 to enable the option and false off or 0 to disable it the boolean value can also be omitted in which case true is assumed format specifies the copy function to use this defaults to csv but other options can be available e g parquet delimiter specifies the string that separates columns within each row line of the file the default value is a comma null specifies the string that represents a null value the default is an empty string please note that in copy from both an unquoted empty string and a quoted empty string represent a null value if any other null string is specified again both its quoted and its unquoted appearance represent a null value copy to does not quote null values on output even if force_quote is true header specifies that the file contains a header line with the names of each column in the file thus copy from ignores the first line when importing data whereas on output copy to the first line of the file contains the column names of the exported columns quote specifies the quoting string to be used when a data value is quoted the default is double-quote escape specifies the string that should appear before a data character sequence that matches the quote value the default is the same as the quote value so that the quoting string is doubled if it appears in the data dateformat specifies the date format to use when parsing dates see date format timestampformat specifies the date format to use when parsing timestamps see date format force_quote forces quoting to be used for all non-null values in each specified column null output is never quoted if is specified non-null values will be quoted in all columns this option is allowed only in copy to force_not_null do not match the specified columns values against the null string in the default case where the null string is empty this means that empty values will be read as zero-length strings rather than nulls this option is allowed only in copy from encoding if this option is used its value must be utf8 with any other encoding an error will be thrown auto_detect option for csv parsing if true the parser will attempt to detect the input format and data types automatically delim sep quote escape and header parameters become optional sample_size option to define number of sample rows for automatic csv type detection chunks of sample rows will be drawn from different locations of the input file set to -1 to scan the entire input file only the first max 1024 rows will be used for dialect detection all_varchar option to skip type detection for csv parsing and assume all columns to be of type varchar it is recommended that the file name used in copy always be specified as an absolute path the values in each record are separated by the delimiter string if the value contains the delimiter string the quote string the null string a carriage return or line feed character then the whole value is prefixed and suffixed by the quote string and any occurrence within the value of a quote string or the escape string is preceded by the escape string you can also use force_quote to force quotes when outputting non-null values in specific columns the csv format has no standard way to distinguish a null value from an empty string you can use force_not_null to prevent null input comparisons for specific columns in csv format all characters are significant a quoted value surrounded by white space or any characters other than delimiter will include those characters this can cause errors if you import data from a system that pads csv lines with white space out to some fixed width if such a situation arises you might need to preprocess the csv file to remove the trailing white space before importing the data into duckdb",
"category": "Statements",
- "url": "../docs/sql/statements/overview",
- "blurb": ""
+ "url": "../docs/sql/statements/copy",
+ "blurb": "COPY moves data between DuckDB tables and external Comma Separated Value (CSV) files. CSV Import COPY ... FROM ..."
},
{
- "title": "Create Schema",
- "text": "the create schema statement creates a schema in the catalog the default schema is main examples -- create a schema create schema s1 -- create a schema if it does not exist yet create schema if not exists s2 -- create table in the schemas create table s1 t id integer primary key other_id integer create table s2 t id integer primary key j varchar -- compute a join between tables from two schemas select from s1 t s1t s2 t s2t where s1t other_id s2t id syntax",
+ "title": "Create View",
+ "text": "the create view statement defines a new view in the catalog examples -- create a simple view create view v1 as select from tbl -- create a view or replace it if a view with that name already exists create or replace view v1 as select 42 -- create a view and replace the column names create view v1 a as select 42 syntax create view defines a view of a query the view is not physically materialized instead the query is run every time the view is referenced in a query create or replace view is similar but if a view of the same name already exists it is replaced if a schema name is given then the view is created in the specified schema otherwise it is created in the current schema temporary views exist in a special schema so a schema name cannot be given when creating a temporary view the name of the view must be distinct from the name of any other view or table in the same schema",
"category": "Statements",
- "url": "../docs/sql/statements/create_schema",
- "blurb": "The CREATE SCHEMA statement creates a schema in the catalog. The default schema is main . Examples -- create a..."
+ "url": "../docs/sql/statements/create_view",
+ "blurb": "The CREATE VIEW statement defines a new view in the catalog. Examples -- create a simple view CREATE VIEW v1 AS..."
},
{
"title": "Insert Statement",
@@ -582,25 +589,32 @@
"blurb": "The INSERT statement inserts new data into a table. Examples -- insert the values (1), (2), (3) into tbl INSERT INTO..."
},
{
- "title": "Delete Statement",
- "text": "the delete statement removes rows from the table identified by the table-name examples -- remove the rows matching the condition i 2 from the database delete from tbl where i 2 -- delete all rows in the table tbl delete from tbl syntax the delete statement removes rows from the table identified by the table-name if the where clause is not present all records in the table are deleted if a where clause is supplied then only those rows for which the where clause results in true are deleted rows for which the expression is false or null are retained the using clause allows deleting based on the content of other tables or subqueries",
+ "title": "Export & Import Database",
+ "text": "the export database command allows you to export the contents of the database to a specific directory the import database command allows you to then read the contents again examples -- export the database to the target directory export database target_directory -- export the table contents with the given options export database target_directory format csv delimiter -- export the table contents as parquet export database target_directory format parquet --reload the database again import database target_directory syntax the export database command exports the full contents of the database - including schema information tables views and sequences - to a specific directory that can then be loaded again the created directory will be structured as follows target_directory schema sql target_directory load sql target_directory t_1 csv target_directory t_n csv the schema sql file contains the schema statements that are found in the database it contains any create schema create table create view and create sequence commands that are necessary to re-construct the database the load sql file contains a set of copy statements that can be used to read the data from the csv files again the file contains a single copy statement for every table found in the schema the database can be reloaded by using the import database command again or manually by running schema sql followed by load sql to re-load the data",
"category": "Statements",
- "url": "../docs/sql/statements/delete",
- "blurb": "The DELETE statement removes rows from the table identified by the table-name. Examples -- remove the rows matching..."
+ "url": "../docs/sql/statements/export",
+ "blurb": "The EXPORT DATABASE command allows you to export the contents of the database to a specific directory. The IMPORT..."
},
{
- "title": "Select Statement",
- "text": "the select statement retrieves rows from the database examples -- select all columns from the table tbl select from tbl -- select the rows from tbl select j from tbl where i 3 -- perform an aggregate grouped by the column i select i sum j from tbl group by i -- select only the top 3 rows from the tbl select from tbl order by i desc limit 3 -- join two tables together using the using clause select from t1 join t2 using a b syntax the select statement retrieves rows from the database the canonical order of a select statement is as follows with less common clauses being indented select select_list from tables using sample sample_expr where condition group by groups having group_filter window window_expr qualify qualify_filter order by order_expr limit n optionally the select statement can be prefixed with a with clause as the select statement is so complex we have split up the syntax diagrams into several parts the full syntax diagram can be found at the bottom of the page select clause the select clause specifies the list of columns that will be returned by the query while it appears first in the clause logically the expressions here are executed only at the end the select clause can contain arbitrary expressions that transform the output as well as aggregates and window functions from clause the from clause specifies the source of the data on which the remainder of the query should operate logically the from clause is where the query starts execution the from clause can contain a single table a combination of multiple tables that are joined together or another select query inside a subquery node sample clause the sample clause allows you to run the query on a sample from the base table this can significantly speed up processing of queries at the expense of accuracy in the result samples can also be used to quickly see a snapshot of the data when exploring a data set the sample clause is applied right after anything in the from clause i e after any joins but before the where clause or any aggregates see the sample page for more information where clause the where clause specifies any filters to apply to the data this allows you to select only a subset of the data in which you are interested logically the where clause is applied immediately after the from clause group by having clause the group by clause specifies which grouping columns should be used to perform any aggregations in the select clause if the group by clause is specified the query is always an aggregate query even if no aggregations are present in the select clause window clause the window clause allows you to specify named windows that can be used within window functions these are useful when you have multiple window functions as they allow you to avoid repeating the same window clause qualify clause the qualify clause is used to filter the result of window functions order by limit clause order by and limit are output modifiers logically they are applied at the very end of the query the limit clause restricts the amount of rows fetched and the order by clause sorts the rows on the sorting criteria in either ascending or descending order values list a values list is a set of values that is supplied instead of a select statement row ids for each table the rowid pseudocolumn returns the row identifiers based on the physical storage d create table t id int content string d insert into t values 42 hello 43 world d select rowid id content from t rowid id content 0 42 hello 1 43 world in the current storage these identifiers are contiguous unsigned integers 0 1 if no rows were deleted deletions introduce gaps in the rowids which may be reclaimed later therefore it is strongly recommended not to use rowids as identifiers the rowid values are stable within a transaction if there is a user-defined column named rowid it shadows the rowid pseudocolumn common table expressions full syntax diagram below is the full syntax diagram of the select statement",
+ "title": "Create Schema",
+ "text": "the create schema statement creates a schema in the catalog the default schema is main examples -- create a schema create schema s1 -- create a schema if it does not exist yet create schema if not exists s2 -- create table in the schemas create table s1 t id integer primary key other_id integer create table s2 t id integer primary key j varchar -- compute a join between tables from two schemas select from s1 t s1t s2 t s2t where s1t other_id s2t id syntax",
"category": "Statements",
- "url": "../docs/sql/statements/select",
- "blurb": "The SELECT statement retrieves rows from the database."
+ "url": "../docs/sql/statements/create_schema",
+ "blurb": "The CREATE SCHEMA statement creates a schema in the catalog. The default schema is main . Examples -- create a..."
},
{
- "title": "Copy",
- "text": "copy moves data between duckdb tables and external comma separated value csv files csv import copy from imports data into duckdb from an external csv file into an existing table the data is appended to whatever data is in the table already the amount of columns inside the file must match the amount of columns in the table table_name and the contents of the columns must be convertible to the column types of the table in case this is not possible an error will be thrown if a list of columns is specified copy will only copy the data in the specified columns from the file if there are any columns in the table that are not in the column list copy from will insert the default values for those columns -- copy the contents of a comma-separated file test csv without a header into the table test copy test from test csv -- copy the contents of a comma-separated file with a header into the category table copy category from categories csv header -- copy the contents of lineitem tbl into the lineitem table where the contents are delimited by a pipe character copy lineitem from lineitem tbl delimiter -- copy the contents of lineitem tbl into the lineitem table where the delimiter quote character and presence of a header are automatically detected copy lineitem from lineitem tbl auto_detect true -- read the contents of a comma-separated file names csv into the name column of the category table any other columns of this table are filled with their default value copy category name from names csv syntax csv export copy to exports data from duckdb to an external csv file it has mostly the same set of options as copy from however in the case of copy to the options specify how the csv file should be written to disk any csv file created by copy to can be copied back into the database by using copy from with the same set of options the copy to function can be called specifying either a table name or a query when a table name is specified the contents of the entire table will be written into the resulting csv file when a query is specified the query is executed and the result of the query is written to the resulting file -- copy the contents of the lineitem table to the file lineitem tbl where the columns are delimited by a pipe character including a header line copy lineitem to lineitem tbl delimiter header -- copy the l_orderkey column of the lineitem table to the file orderkey tbl copy lineitem l_orderkey to orderkey tbl delimiter -- copy the result of a query to the file query csv including a header with column names copy select 42 as a hello as b to query csv with header 1 delimiter syntax parameters name description --- --- table_name the name optionally schema-qualified of an existing table column_name an optional list of columns to be copied if no column list is specified all columns of the table will be copied filename the path and name of the input or output file boolean specifies whether the selected option should be turned on or off you can write true on or 1 to enable the option and false off or 0 to disable it the boolean value can also be omitted in which case true is assumed format specifies the copy function to use this defaults to csv but other options can be available e g parquet delimiter specifies the string that separates columns within each row line of the file the default value is a comma null specifies the string that represents a null value the default is an empty string please note that in copy from both an unquoted empty string and a quoted empty string represent a null value if any other null string is specified again both its quoted and its unquoted appearance represent a null value copy to does not quote null values on output even if force_quote is true header specifies that the file contains a header line with the names of each column in the file thus copy from ignores the first line when importing data whereas on output copy to the first line of the file contains the column names of the exported columns quote specifies the quoting string to be used when a data value is quoted the default is double-quote escape specifies the string that should appear before a data character sequence that matches the quote value the default is the same as the quote value so that the quoting string is doubled if it appears in the data dateformat specifies the date format to use when parsing dates see date format timestampformat specifies the date format to use when parsing timestamps see date format force_quote forces quoting to be used for all non-null values in each specified column null output is never quoted if is specified non-null values will be quoted in all columns this option is allowed only in copy to force_not_null do not match the specified columns values against the null string in the default case where the null string is empty this means that empty values will be read as zero-length strings rather than nulls this option is allowed only in copy from encoding if this option is used its value must be utf8 with any other encoding an error will be thrown auto_detect option for csv parsing if true the parser will attempt to detect the input format and data types automatically delim sep quote escape and header parameters become optional sample_size option to define number of sample rows for automatic csv type detection chunks of sample rows will be drawn from different locations of the input file set to -1 to scan the entire input file only the first max 1024 rows will be used for dialect detection all_varchar option to skip type detection for csv parsing and assume all columns to be of type varchar it is recommended that the file name used in copy always be specified as an absolute path the values in each record are separated by the delimiter string if the value contains the delimiter string the quote string the null string a carriage return or line feed character then the whole value is prefixed and suffixed by the quote string and any occurrence within the value of a quote string or the escape string is preceded by the escape string you can also use force_quote to force quotes when outputting non-null values in specific columns the csv format has no standard way to distinguish a null value from an empty string you can use force_not_null to prevent null input comparisons for specific columns in csv format all characters are significant a quoted value surrounded by white space or any characters other than delimiter will include those characters this can cause errors if you import data from a system that pads csv lines with white space out to some fixed width if such a situation arises you might need to preprocess the csv file to remove the trailing white space before importing the data into duckdb",
+ "title": "Drop Statement",
+ "text": "the drop statement removes a catalog entry added previously with the create command examples -- delete the table with the name tbl drop table tbl -- drop the view with the name v1 do not throw an error if the view does not exist drop view if exists v1 syntax the optional if exists clause suppresses the error that would normally result if the table does not exist by default or if the restrict clause is provided the entry will not be dropped if there are any other objects that depend on it if the cascade clause is provided then all the objects that are dependent on the object will be dropped as well create schema myschema create table myschema t1 i integer -- error cannot drop myschema because the table myschema t1 depends on it drop schema myschema -- cascade drops both myschema and myschema 1 drop schema myschema cascade",
"category": "Statements",
- "url": "../docs/sql/statements/copy",
- "blurb": "COPY moves data between DuckDB tables and external Comma Separated Value (CSV) files. CSV Import COPY ... FROM ..."
+ "url": "../docs/sql/statements/drop",
+ "blurb": "The DROP statement removes a catalog entry added previously with the CREATE command. Examples -- delete the table..."
+ },
+ {
+ "title": "Alter Table",
+ "text": "the alter table statement changes the schema of an existing table in the catalog examples -- add a new column with name k to the table integers it will be filled with the default value null alter table integers add column k integer -- add a new column with name l to the table integers it will be filled with the default value 10 alter table integers add column l integer default 10 -- drop the column k from the table integers alter table integers drop k -- change the type of the column i to the type varchar using a standard cast alter table integers alter i type varchar -- change the type of the column i to the type varchar using the specified expression to convert the data for each row alter table integers alter i set data type varchar using concat i _ j -- set the default value of a column alter table integers alter column i set default 10 -- drop the default value of a column alter table integers alter column i drop default -- rename a table alter table integers rename to integers_old -- rename a column of a table alter table integers rename i to j syntax alter table changes the schema of an existing table all the changes made by alter table fully respect the transactional semantics - that is - they will not be visible to other transactions until committed and can be fully reverted through a rollback rename table -- rename a table alter table integers rename to integers_old the rename to clause renames an entire table changing its name in the schema note that any views that rely on the table are not automatically updated rename column -- rename a column of a table alter table integers rename i to j alter table integers rename column j to k the rename column clause renames a single column within a table any constraints that rely on this name e g check constraints are automatically updated however note that any views that rely on this column name are not automatically updated add column -- add a new column with name k to the table integers it will be filled with the default value null alter table integers add column k integer -- add a new column with name l to the table integers it will be filled with the default value 10 alter table integers add column l integer default 10 the add column clause can be used to add a new column of a specified type to a table the new column will be filled with the specified default value or null if none is specified drop column -- drop the column k from the table integers alter table integers drop k the drop column clause can be used to remove a column from a table note that columns can only be removed if they do not have any indexes that rely on them this includes any indexes created as part of a primary key or unique constraint columns that are part of multi-column check constraints cannot be dropped either alter type -- change the type of the column i to the type varchar using a standard cast alter table integers alter i type varchar -- change the type of the column i to the type varchar using the specified expression to convert the data for each row alter table integers alter i set data type varchar using concat i _ j the set data type clause changes the type of a column in a table any data present in the column is converted according to the provided expression in the using clause or if the using clause is absent cast to the new data type note that columns can only have their type changed if they do not have any indexes that rely on them and are not part of any check constraints set drop default -- set the default value of a column alter table integers alter column i set default 10 -- drop the default value of a column alter table integers alter column i drop default the set drop default clause modifies the default value of an existing column note that this does not modify any existing data in the column dropping the default is equivalent to setting the default value to null",
+ "category": "Statements",
+ "url": "../docs/sql/statements/alter_table",
+ "blurb": "The ALTER TABLE statement changes the schema of an existing table in the catalog. Examples -- add a new column with..."
},
{
"title": "Create Sequence",
@@ -617,123 +631,88 @@
"blurb": "The UPDATE statement modifies the values of rows in a table. Examples -- for every row where i is NULL, set the..."
},
{
- "title": "Create View",
- "text": "the create view statement defines a new view in the catalog examples -- create a simple view create view v1 as select from tbl -- create a view or replace it if a view with that name already exists create or replace view v1 as select 42 -- create a view and replace the column names create view v1 a as select 42 syntax create view defines a view of a query the view is not physically materialized instead the query is run every time the view is referenced in a query create or replace view is similar but if a view of the same name already exists it is replaced if a schema name is given then the view is created in the specified schema otherwise it is created in the current schema temporary views exist in a special schema so a schema name cannot be given when creating a temporary view the name of the view must be distinct from the name of any other view or table in the same schema",
+ "title": "Select Statement",
+ "text": "the select statement retrieves rows from the database examples -- select all columns from the table tbl select from tbl -- select the rows from tbl select j from tbl where i 3 -- perform an aggregate grouped by the column i select i sum j from tbl group by i -- select only the top 3 rows from the tbl select from tbl order by i desc limit 3 -- join two tables together using the using clause select from t1 join t2 using a b syntax the select statement retrieves rows from the database the canonical order of a select statement is as follows with less common clauses being indented select select_list from tables using sample sample_expr where condition group by groups having group_filter window window_expr qualify qualify_filter order by order_expr limit n optionally the select statement can be prefixed with a with clause as the select statement is so complex we have split up the syntax diagrams into several parts the full syntax diagram can be found at the bottom of the page select clause the select clause specifies the list of columns that will be returned by the query while it appears first in the clause logically the expressions here are executed only at the end the select clause can contain arbitrary expressions that transform the output as well as aggregates and window functions from clause the from clause specifies the source of the data on which the remainder of the query should operate logically the from clause is where the query starts execution the from clause can contain a single table a combination of multiple tables that are joined together or another select query inside a subquery node sample clause the sample clause allows you to run the query on a sample from the base table this can significantly speed up processing of queries at the expense of accuracy in the result samples can also be used to quickly see a snapshot of the data when exploring a data set the sample clause is applied right after anything in the from clause i e after any joins but before the where clause or any aggregates see the sample page for more information where clause the where clause specifies any filters to apply to the data this allows you to select only a subset of the data in which you are interested logically the where clause is applied immediately after the from clause group by having clause the group by clause specifies which grouping columns should be used to perform any aggregations in the select clause if the group by clause is specified the query is always an aggregate query even if no aggregations are present in the select clause window clause the window clause allows you to specify named windows that can be used within window functions these are useful when you have multiple window functions as they allow you to avoid repeating the same window clause qualify clause the qualify clause is used to filter the result of window functions order by limit clause order by and limit are output modifiers logically they are applied at the very end of the query the limit clause restricts the amount of rows fetched and the order by clause sorts the rows on the sorting criteria in either ascending or descending order values list a values list is a set of values that is supplied instead of a select statement row ids for each table the rowid pseudocolumn returns the row identifiers based on the physical storage d create table t id int content string d insert into t values 42 hello 43 world d select rowid id content from t rowid id content 0 42 hello 1 43 world in the current storage these identifiers are contiguous unsigned integers 0 1 if no rows were deleted deletions introduce gaps in the rowids which may be reclaimed later therefore it is strongly recommended not to use rowids as identifiers the rowid values are stable within a transaction if there is a user-defined column named rowid it shadows the rowid pseudocolumn common table expressions full syntax diagram below is the full syntax diagram of the select statement",
"category": "Statements",
- "url": "../docs/sql/statements/create_view",
- "blurb": "The CREATE VIEW statement defines a new view in the catalog. Examples -- create a simple view CREATE VIEW v1 AS..."
+ "url": "../docs/sql/statements/select",
+ "blurb": "The SELECT statement retrieves rows from the database."
},
{
- "title": "FILTER Clause",
- "text": "the filter clause may optionally follow an aggregate function in a select statement this will filter the rows of data that are fed into the aggregate function in the same way that a where clause filters rows but localized to the specific aggregate function filter s are not currently able to be used when the aggregate function is in a windowing context there are multiple types of situations where this is useful including when evaluating multiple aggregates with different filters and when creating a pivoted view of a dataset filter provides a cleaner syntax for pivoting data when compared with the more traditional case when approach discussed below some aggregate functions also do not filter out null values so using a filter clause will return valid results when at times the case when approach will not this occurs with the functions first and last which are desirable in a non-aggregating pivot operation where the goal is to simply re-orient the data into columns rather than re-aggregate it filter also improves null handling when using the list and array_agg functions as the case when approach will include null values in the list result while the filter clause will remove them examples -- compare total row count to -- the number of rows where i 5 -- the number of rows where i is odd select count as total_rows count filter where i 5 as lte_five count filter where i 2 1 as odds from generate_series 1 10 tbl i total_rows lte_five odds --- --- --- 10 5 5 -- different aggregate functions may be used and multiple where expressions are also permitted -- the sum of i for rows where i 5 -- the median of i where i is odd select sum i filter where i 5 as lte_five_sum median i filter where i 2 1 as odds_median median i filter where i 2 1 and i 5 as odds_lte_five_median from generate_series 1 10 tbl i lte_five_sum odds_median odds_lte_five_median --- --- --- 15 5 0 3 0 the filter clause can also be used to pivot data from rows into columns this is a static pivot as columns must be defined prior to runtime in sql however this kind of statement can be dynamically generated in a host programming language to leverage duckdb s sql engine for rapid larger than memory pivoting --first generate an example dataset create temp table stacked_data as select i case when i rows 0 25 then 2022 when i rows 0 5 then 2023 when i rows 0 75 then 2024 when i rows 0 875 then 2025 else null end as year from select i count over as rows from generate_series 1 100000000 tbl i tbl -- pivot the data out by year move each year out to a separate column select count i filter where year 2022 as 2022 count i filter where year 2023 as 2023 count i filter where year 2024 as 2024 count i filter where year 2025 as 2025 count i filter where year is null as nulls from stacked_data --this syntax produces the same results as the the filter clauses above select count case when year 2022 then i end as 2022 count case when year 2023 then i end as 2023 count case when year 2024 then i end as 2024 count case when year 2025 then i end as 2025 count case when year is null then i end as nulls from stacked_data 2022 2023 2024 2025 nulls --- --- --- --- --- 25000000 25000000 25000000 12500000 12500000 however the case when approach will not work as expected when using an aggregate function that does not ignore null values the first function falls into this category so filter is preferred in this case -- pivot the data out by year move each year out to a separate column select first i filter where year 2022 as 2022 first i filter where year 2023 as 2023 first i filter where year 2024 as 2024 first i filter where year 2025 as 2025 first i filter where year is null as nulls from stacked_data 2022 2023 2024 2025 nulls --- --- --- --- --- 1474561 25804801 50749441 76431361 87500001 --this will produce null values whenever the first evaluation of the case when clause returns a null select first case when year 2022 then i end as 2022 first case when year 2023 then i end as 2023 first case when year 2024 then i end as 2024 first case when year 2025 then i end as 2025 first case when year is null then i end as nulls from stacked_data 2022 2023 2024 2025 nulls --- --- --- --- --- 1228801 null null null null aggregate function syntax including filter clause",
- "category": "Query Syntax",
- "url": "../docs/sql/query_syntax/filter",
- "blurb": "The FILTER clause may optionally follow an aggregate function in a SELECT statement. This will filter the rows of..."
+ "title": "Create Table",
+ "text": "the create table statement creates a table in the catalog examples -- create a table with two integer columns i and j create table t1 i integer j integer -- create a table with a primary key create table t1 id integer primary key j varchar -- create a table with a composite primary key create table t1 id integer j varchar primary key id j -- create a table with various different types and constraints create table t1 i integer not null decimalnr double check decimalnr 10 date date unique time timestamp -- create a table from the result of a query create table t1 as select 42 as i 84 as j -- create a table from a csv file using auto-detect i e automatically detecting column names and types create table t1 as select from read_csv_auto path file csv temporary tables temporary tables can be created using a create temp table statement see diagram below temporary tables are session scoped similar to postgres for example meaning that only the specific connection that created them can access them and once the connection to duckdb is closed they will be automatically dropped temporary tables reside in memory rather than on disk even when connecting to a persistent duckdb but if the temp_directory configuration is set when connecting or with a set command data will be spilled to disk if memory becomes constrained -- create a temporary table from a csv file using auto-detect i e automatically detecting column names and types create temp table t1 as select from read_csv_auto path file csv -- allow temporary tables to off-load excess memory to disk set temp_directory path to directory create or replace the create or replace syntax allows a new table to be created or for an existing table to be overwritten by the new table this is shorthand for dropping the existing table and then creating the new one -- create a table with two integer columns i and j even if t1 already exists create or replace table t1 i integer j integer if not exists the if not exists syntax will only proceed with the creation of the table if it does not already exist if the table already exists no action will be taken and the existing table will remain in the database -- create a table with two integer columns i and j only if t1 does not exist yet create table if not exists t1 i integer j integer generated columns the type generated always as expr virtual stored syntax will create a generated column the data in this kind of column is generated from its expression which can reference other regular or generated columns of the table since they are produced by calculations these columns can not be inserted into directly duckdb can infer the type of the generated column based on the expression s return type this allows you to leave out the type when declaring a generated column it is possible to explicitly set a type but insertions into the referenced columns might fail if the type can not be cast to the type of the generated column generated columns come in two varieties virtual and stored the data of virtual generated columns is not stored on disk instead it is computed from the expression every time the column is referenced through a select statement the data of stored generated columns is stored on disk and is computed every time the data of their dependencies change through an insert update drop statement currently only the virtual kind is supported and it is also the default option if the last field is left blank -- the simplest syntax for a generated column -- the type is derived from the expression and the variant defaults to virtual create table t1 x float two_x as 2 x -- fully specifying the same generated column for completeness create table t1 x float two_x float generated always as 2 x virtual syntax",
+ "category": "Statements",
+ "url": "../docs/sql/statements/create_table",
+ "blurb": "The CREATE TABLE statement creates a table in the catalog. Examples -- create a table with two integer columns (i..."
},
{
- "title": "WHERE Clause",
- "text": "the where clause specifies any filters to apply to the data this allows you to select only a subset of the data in which you are interested logically the where clause is applied immediately after the from clause examples -- select all rows that have id equal to 3 select from table_name where id 3 -- select all rows that match the given case-insensitive like expression select from table_name where name ilike mark -- select all rows that match the given composite expression select from table_name where id 3 or id 7 syntax",
- "category": "Query Syntax",
- "url": "../docs/sql/query_syntax/where",
- "blurb": "The WHERE clause specifies any filters to apply to the data. This allows you to select only a subset of the data in..."
- },
- {
- "title": "QUALIFY Clause",
- "text": "the qualify clause is used to filter the results of window functions this filtering of results is similar to how a having clause filters the results of aggregate functions applied based on the group by clause the qualify clause avoids the need for a subquery or with clause to perform this filtering much like having avoids a subquery an example using a with clause instead of qualify is included below the qualify examples note that this is filtering based on window functions not necessarily based on the window clause the window clause is optional and can be used to simplify the creation of multiple window function expressions the position of where to specify a qualify clause is following the window clause in a select statement window does not need to be specified and before the order by examples each of the following examples produce the same output located below -- filter based on a window function defined in the qualify clause select schema_name function_name -- in this example the function_rank column in the select clause is for reference row_number over partition by schema_name order by function_name as function_rank from duckdb_functions qualify row_number over partition by schema_name order by function_name 3 -- filter based on a window function defined in the select clause select schema_name function_name row_number over partition by schema_name order by function_name as function_rank from duckdb_functions qualify function_rank 3 -- filter based on a window function defined in the qualify clause but using the window clause select schema_name function_name -- in this example the function_rank column in the select clause is for reference row_number over my_window as function_rank from duckdb_functions window my_window as partition by schema_name order by function_name qualify row_number over my_window 3 -- filter based on a window function defined in the select clause but using the window clause select schema_name function_name row_number over my_window as function_rank from duckdb_functions window my_window as partition by schema_name order by function_name qualify function_rank 3 -- equivalent query based on a with clause without qualify clause with ranked_functions as select schema_name function_name row_number over partition by schema_name order by function_name as function_rank from duckdb_functions select from ranked_functions where function_rank 3 schema_name function_name function_rank --- --- --- main __postfix 1 main 2 pg_catalog col_description 1 pg_catalog format_pg_type 2 syntax",
- "category": "Query Syntax",
- "url": "../docs/sql/query_syntax/qualify",
- "blurb": "The QUALIFY clause is used to filter the results of WINDOW functions."
- },
- {
- "title": "GROUP BY Clause",
- "text": "the group by clause specifies which grouping columns should be used to perform any aggregations in the select clause if the group by clause is specified the query is always an aggregate query even if no aggregations are present in the select clause when a group by clause is specified all tuples that have matching data in the grouping columns i e all tuples that belong to the same group will be combined the values of the grouping columns themselves are unchanged and any other columns can be combined using an aggregate function such as count sum avg etc normally the group by clause groups along a single dimension using the grouping sets cube or rollup clauses it is possible to group along multiple dimensions see the grouping sets page for more information examples -- count the number of entries in the addresses table that belong to each different city select city count from addresses group by city -- compute the average income per city per street_name select city street_name avg income from addresses group by city street_name syntax",
- "category": "Query Syntax",
- "url": "../docs/sql/query_syntax/groupby",
- "blurb": "The GROUP BY clause specifies which grouping columns should be used to perform any aggregations in the SELECT clause...."
- },
- {
- "title": "WINDOW Clause",
- "text": "the window clause allows you to specify named windows that can be used within window functions these are useful when you have multiple window functions as they allow you to avoid repeating the same window clause syntax",
- "category": "Query Syntax",
- "url": "../docs/sql/query_syntax/window",
- "blurb": "The WINDOW clause allows you to specify named windows that can be used within window functions. These are useful when..."
- },
- {
- "title": "SELECT Clause",
- "text": "the select clause specifies the list of columns that will be returned by the query while it appears first in the clause logically the expressions here are executed only at the end the select clause can contain arbitrary expressions that transform the output as well as aggregates and window functions examples -- select all columns from the table called table_name select from table_name -- select all unique cities from the addresses table select distinct city from addresses -- return the total number of rows in the addresses table select count from addresses -- select all columns except the city column from the addresses table select exclude city from addresses -- select all columns from the addresses table but replace city with lower city select replace lower city as city from addresses syntax",
- "category": "Query Syntax",
- "url": "../docs/sql/query_syntax/select",
- "blurb": "The SELECT clause specifies the list of columns that will be returned by the query."
- },
- {
- "title": "WITH Clause",
- "text": "the with clause allows you to specify common table expressions ctes regular non-recursive common-table-expressions are essentially views that are limited in scope to a particular query ctes can reference each-other and can be nested basic cte examples -- create a cte called cte and use it in the main query with cte as select 42 as x select from cte x 42 -- create two ctes where the second cte references the first cte with cte as select 42 as i cte2 as select i 100 as x from cte select from cte2 x 4200 recursive cte examples tree traversal with recursive can be used to traverse trees for example take a hierarchy of tags create table tag id int name varchar subclassof int insert into tag values 1 u2 5 2 blur 5 3 oasis 5 4 2pac 6 5 rock 7 6 rap 7 7 music 9 8 movies 9 9 art null the following query returns the path from the node oasis to the root of the tree art with recursive tag_hierarchy id source path as select id name name as path from tag where subclassof is null union all select tag id tag name list_prepend tag name tag_hierarchy path from tag tag_hierarchy where tag subclassof tag_hierarchy id select path from tag_hierarchy where source oasis path oasis rock music art graph traversal the with recursive clause can be used to express graph traversal on arbitrary graphs however if the graph has cycles the query must perform cycle detection to prevent infinite loops one way to achieve this is to store the path of a traversal in a list and before extending the path with a new edge check whether its endpoint has been visited before see the example later take the following directed graph from the ldbc graphalytics benchmark create table edge node1id int node2id int insert into edge values 1 3 1 5 2 4 2 5 2 10 3 1 3 5 3 8 3 10 5 3 5 4 5 8 6 3 6 4 7 4 8 1 9 4 note that the graph contains directed cycles e g between nodes 1 2 and 5 enumerate all paths from a node the following query returns all paths starting in node 1 with recursive paths startnode endnode path as select -- define the path as the first edge of the traversal node1id as startnode node2id as endnode node1id node2id as path from edge where startnode 1 union all select -- concatenate new edge to the path paths startnode as startnode node2id as endnode array_append path node2id as path from paths join edge on paths endnode node1id -- prevent adding a repeated node to the path -- this ensures that no cycles occur where node2id all paths path select startnode endnode path from paths order by length path path startnode endnode path 1 3 1 3 1 5 1 5 1 5 1 3 5 1 8 1 3 8 1 10 1 3 10 1 3 1 5 3 1 4 1 5 4 1 8 1 5 8 1 4 1 3 5 4 1 8 1 3 5 8 1 8 1 5 3 8 1 10 1 5 3 10 note that the result of this query is not restricted to shortest paths e g for node 5 the results include paths 1 5 and 1 3 5 enumerate unweighted shortest paths from a node in most cases enumerating all paths is not practical or feasible instead only the unweighted shortest paths are of interest to find these the second half of the with recursive query should be adjusted such that it only includes a node if it has not yet been visited this is implemented by using a subquery that checks if any of the previous paths includes the node with recursive paths startnode endnode path as select -- define the path as the first edge of the traversal node1id as startnode node2id as endnode node1id node2id as path from edge where startnode 1 union all select -- concatenate new edge to the path paths startnode as startnode node2id as endnode array_append path node2id as path from paths join edge on paths endnode node1id -- prevent adding a node that was visited previously by any path -- this ensures that 1 no cycles occur and 2 only nodes that -- were not visited by previous shorter paths are added to a path where not exists select 1 from paths previous_paths where list_contains previous_paths path node2id select startnode endnode path from paths order by length path path startnode endnode path 1 3 1 3 1 5 1 5 1 8 1 3 8 1 10 1 3 10 1 4 1 5 4 1 8 1 5 8 enumerate unweighted shortest paths between two nodes with recursive can also be used to find all unweighted shortest paths between two nodes to ensure that the recursive query is stopped as soon as we reach the end node we use a window function which checks whether the end node is among the newly added nodes the following query returns all unweighted shortest paths between nodes 1 start node and 8 end node with recursive paths startnode endnode path endreached as select -- define the path as the first edge of the traversal node1id as startnode node2id as endnode node1id node2id as path node2id 8 as endreached from edge where startnode 1 union all select -- concatenate new edge to the path paths startnode as startnode node2id as endnode array_append path node2id as path max case when node2id 8 then 1 else 0 end over rows between unbounded preceding and unbounded following as endreached from paths join edge on paths endnode node1id where not exists select 1 from paths previous_paths where list_contains previous_paths path node2id and paths endreached 0 select startnode endnode path from paths where endnode 8 order by length path path startnode endnode path 1 8 1 3 8 1 8 1 5 8 common table expressions",
- "category": "Query Syntax",
- "url": "../docs/sql/query_syntax/with",
- "blurb": "The WITH clause allows you to specify common table expressions (CTEs). Regular (non-recursive) common-table-expressions..."
+ "title": "Create Macro",
+ "text": "the create macro statement creates a scalar macro in the catalog examples -- create a macro that adds two expressions a and b create macro add a b as a b -- create a macro for a case expression create macro ifelse a b c as case when a then b else c end -- create a macro that does a subquery create macro one as select 1 -- create a macro with a common table expression -- parameter names get priority over column names disambiguate using the table name create macro plus_one a as with cte as select 1 as a select cte a a from cte -- macro s are schema-dependent and have an alias function create function main myavg x as sum x count x -- create a macro with default constant parameters create macro add_default a b 5 as a b -- create a macro arr_append with a functionality equivalent to array_append create macro arr_append l e as list_concat l list_value e syntax macros allow you to create shortcuts for combinations of expressions -- failure cannot find column b create macro add a as a b -- this works create macro add a b as a b -- error cannot bind varchar integer select add hello 3 -- success select add 1 2 -- 3 macro s can have default parameters -- b is a default parameter create macro add_default a b 5 as a b -- the following will result in 42 select add_default 37 -- error add_default only has one positional parameter select add_default 40 2 -- success default parameters are used by assigning them like so select add_default 40 b 2 -- error default parameters must come after positional parameters select add_default b 2 40 -- the order of default parameters does not matter create macro triple_add a b 5 c 10 as a b c -- success select triple_add 40 c 1 b 1 -- 42 when macro s are used they are expanded i e replaced with the original expression and the parameters within the expanded expression are replaced with the supplied arguments step by step -- the add macro we defined above is used in a query select add 40 2 -- internally add is replaced with its definition of a b select a b -- then the parameters are replaced by the supplied arguments select 40 2 -- 42",
+ "category": "Statements",
+ "url": "../docs/sql/statements/create_macro",
+ "blurb": "The CREATE MACRO statement creates a scalar macro in the catalog. Examples -- create a macro that adds two..."
},
{
- "title": "GROUPING SETS",
- "text": "grouping sets rollup and cube can be used in the group by clause to perform a grouping over multiple dimensions within the same query examples -- compute the average income along the provided four different dimensions -- signifies the empty set i e computing an ungrouped aggregate select city street_name avg income from addresses group by grouping sets city street_name city street_name -- compute the average income along the same dimensions select city street_name avg income from addresses group by cube city street_name -- compute the average income along the dimensions city street_name city and select city street_name avg income from addresses group by rollup city street_name description grouping sets perform the same aggregate across different group by clauses in a single query create table students course varchar type varchar insert into students course type values cs bachelor cs bachelor cs phd math masters cs null cs null math null select course type count from students group by grouping sets course type course type course type count_star cs bachelor 2 cs phd 1 math masters 1 cs null 2 math null 1 cs null 5 math null 2 null bachelor 2 null phd 1 null masters 1 null null 3 null null 7 in the above query we group across four different sets course type course type and the empty group the result contains null for a group which is not in the grouping set for the result i e the above query is equivalent to the following union statement -- group by course type select course type count from students group by course type union all -- group by type select null as course type count from students group by type union all -- group by course select course null as type count from students group by course union all -- group by nothing select null as course null as type count from students cube and rollup are syntactic sugar to easily produce commonly used grouping sets the rollup clause will produce all sub-groups of a grouping set e g rollup country city zip produces the grouping sets country city zip country city country this can be useful for producing different levels of detail of a group by clause this produces n 1 grouping sets where n is the amount of terms in the rollup clause cube produces grouping sets for all combinations of the inputs e g cube country city zip will produce country city zip country city country zip city zip country city zip this produces 2 n grouping sets grouping alias grouping_id is a special aggregate function that can be used in combination with grouping sets the grouping function takes as parameters a group and returns 0 if the group is included in the grouping for that row or 1 otherwise this is primarily useful because the grouping columns by which we do not aggregate return null which is ambiguous with groups that are actually the value null the grouping or grouping_id function can be used to distinguish these two cases syntax",
- "category": "Query Syntax",
- "url": "../docs/sql/query_syntax/grouping_sets",
- "blurb": "GROUPING SETS , ROLLUP and CUBE can be used in the GROUP BY clause to perform a grouping over multiple dimensions..."
+ "title": "Delete Statement",
+ "text": "the delete statement removes rows from the table identified by the table-name examples -- remove the rows matching the condition i 2 from the database delete from tbl where i 2 -- delete all rows in the table tbl delete from tbl syntax the delete statement removes rows from the table identified by the table-name if the where clause is not present all records in the table are deleted if a where clause is supplied then only those rows for which the where clause results in true are deleted rows for which the expression is false or null are retained the using clause allows deleting based on the content of other tables or subqueries",
+ "category": "Statements",
+ "url": "../docs/sql/statements/delete",
+ "blurb": "The DELETE statement removes rows from the table identified by the table-name. Examples -- remove the rows matching..."
},
{
- "title": "VALUES Clause",
- "text": "the values clause is used to specify a fixed number of rows the values clause can be used as a stand-alone statement as part of the from clause or as input to an insert into statement examples -- generate two rows and directly return them values amsterdam 1 london 2 -- generate two rows as part of a from clause and rename the columns select from values amsterdam 1 london 2 cities name id -- generate two rows and insert them into a table insert into cities values amsterdam 1 london 2 -- create a table directly from a values clause create table cities as select from values amsterdam 1 london 2 cities name id syntax",
- "category": "Query Syntax",
- "url": "../docs/sql/query_syntax/values",
- "blurb": "The VALUES clause is used to specify a fixed number of rows. The VALUES clause can be used as a stand-alone..."
+ "title": "Statements Overview",
+ "text": "more",
+ "category": "Statements",
+ "url": "../docs/sql/statements/overview",
+ "blurb": ""
},
{
- "title": "FROM Clause",
- "text": "the from clause specifies the source of the data on which the remainder of the query should operate logically the from clause is where the query starts execution the from clause can contain a single table a combination of multiple tables that are joined together or another select query inside a subquery node examples -- select all columns from the table called table_name select from table_name -- select all columns from the table called table_name in the schema schema_name select from schema_name table_name -- select the column i from the table function range where the first column of the range function is renamed to i select t i from range 100 as t i -- select all columns from the csv file called test csv select from test csv -- select all columns from a subquery select from select from table_name -- join two tables together select from table_name join other_table on table_name key other_table key -- select a 10 sample from a table select from table_name tablesample 10 -- select a sample of 10 rows from a table select from table_name tablesample 10 rows syntax",
- "category": "Query Syntax",
- "url": "../docs/sql/query_syntax/from",
- "blurb": "The FROM clause can contain a single table, a combination of multiple tables that are joined together, or another SELECT..."
+ "title": "Information Schema",
+ "text": "the views in the information_schema are sql-standard views that describe the catalog entries of the database these views can be filtered to obtain information about a specific column or table database catalog and schema the top level catalog view is information_schema schemata it lists the catalogs and the schemas present in the database and has the following layout column description type example --- --- --- --- catalog_name name of the database that the schema is contained in not yet implemented varchar null schema_name name of the schema varchar main schema_owner name of the owner of the schema not yet implemented varchar null default_character_set_catalog applies to a feature not available in duckdb varchar null default_character_set_schema applies to a feature not available in duckdb varchar null default_character_set_name applies to a feature not available in duckdb varchar null sql_path the file system location of the database currently unimplemented varchar null tables and views the view that describes the catalog information for tables and views is information_schema tables it lists the tables present in the database and has the following layout column description type example --- --- --- --- table_catalog the catalog the table or view belongs to not yet implemented varchar null table_schema the schema the table or view belongs to varchar main table_name the name of the table or view varchar widgets table_type the type of table one of base table local temporary view varchar base table self_referencing_column_name applies to a feature not available in duckdb varchar null reference_generation applies to a feature not available in duckdb varchar null user_defined_type_catalog if the table is a typed table the name of the database that contains the underlying data type always the current database else null currently unimplemented varchar null user_defined_type_schema if the table is a typed table the name of the schema that contains the underlying data type else null currently unimplemented varchar null user_defined_type_name if the table is a typed table the name of the underlying data type else null currently unimplemented varchar null is_insertable_into yes if the table is insertable into no if not base tables are always insertable into views not necessarily varchar yes is_typed yes if the table is a typed table no if not varchar no commit_action not yet implemented varchar no columns the view that describes the catalog information for columns is information_schema columns it lists the column present in the database and has the following layout column description type example --- --- --- --- table_catalog name of the database containing the table not yet implemented varchar null table_schema name of the schema containing the table varchar main table_name name of the table varchar widgets column_name name of the column varchar price ordinal_position ordinal position of the column within the table count starts at 1 integer 5 column_default default expression of the column varchar 1 99 is_nullable yes if the column is possibly nullable no if it is known not nullable varchar yes data_type data type of the column varchar decimal 18 2 character_maximum_length if data_type identifies a character or bit string type the declared maximum length null for all other data types or if no maximum length was declared integer 255 character_octet_length if data_type identifies a character type the maximum possible length in octets bytes of a datum null for all other data types the maximum octet length depends on the declared character maximum length see above and the character encoding integer 1073741824 numeric_precision if data_type identifies a numeric type this column contains the declared or implicit precision of the type for this column the precision indicates the number of significant digits for all other data types this column is null integer 18 numeric_scale if data_type identifies a numeric type this column contains the declared or implicit scale of the type for this column the precision indicates the number of significant digits for all other data types this column is null integer 2 datetime_precision if data_type identifies a date time timestamp or interval type this column contains the declared or implicit fractional seconds precision of the type for this column that is the number of decimal digits maintained following the decimal point in the seconds value no fractional seconds are currently supported in duckdb for all other data types this column is null integer 0 catalog functions several functions are also provided to see details about the schemas that are configured in the database function description example result --- --- --- --- current_schema return the name of the currently active schema default is main current_schema main current_schemas boolean return list of schemas pass a parameter of true to include implicit schemas current_schemas true temp main pg_catalog",
+ "category": "SQL",
+ "url": "../docs/sql/information_schema",
+ "blurb": "The views in the information_schema are SQL-standard views that describe the catalog entries of the database. These..."
},
{
- "title": "SAMPLE Clause",
- "text": "the sample clause allows you to run the query on a sample from the base table this can significantly speed up processing of queries at the expense of accuracy in the result samples can also be used to quickly see a snapshot of the data when exploring a data set the sample clause is applied right after anything in the from clause i e after any joins but before the where clause or any aggregates see the sample page for more information examples -- select a sample of 1 of the addresses table using default system sampling select from addresses using sample 1 -- select a sample of 1 of the addresses table using bernoulli sampling select from addresses using sample 1 bernoulli -- select a sample of 10 rows from the subquery select from select from addresses using sample 10 rows syntax",
- "category": "Query Syntax",
- "url": "../docs/sql/query_syntax/sample",
- "blurb": "The SAMPLE clause allows you to run the query on a sample from the base table. This can significantly speed up..."
+ "title": "IN Operator",
+ "text": "in operator the in operator checks containment of the left expression inside the set of expressions on the right hand side rhs the in operator returns true if the expression is present in the rhs false if the expression is not in the rhs and the rhs has no null values or null if the expression is not in the rhs and the rhs has null values select math in cs math -- true select english in cs math -- false select math in cs math null -- true select english in cs math null -- null not in can be used to check if an element is not present in the set x not in y is equivalent to not x in y the in operator can also be used with a subquery that returns a single column see the subqueries page for more information",
+ "category": "Expressions",
+ "url": "../docs/sql/expressions/in",
+ "blurb": "In Operator The IN operator checks containment of the left expression inside the set of expressions on the right..."
},
{
- "title": "LIMIT Clause",
- "text": "limit is an output modifier logically it is applied at the very end of the query the limit clause restricts the amount of rows fetched the offset clause indicates at which position to start reading the values i e the first offset values are ignored note that while limit can be used without an order by clause the results might not be deterministic without the order by clause this can still be useful however for example when you want to inspect a quick snapshot of the data examples -- select the first 5 rows from the addresses table select from addresses limit 5 -- select the 5 rows from the addresses table starting at position 5 i e ignoring the first 5 rows select from addresses limit 5 offset 5 -- select the top 5 cities with the highest population select city count as population from addresses group by city order by population desc limit 5 syntax",
- "category": "Query Syntax",
- "url": "../docs/sql/query_syntax/limit",
- "blurb": "LIMIT is an output modifier. Logically it is applied at the very end of the query. The LIMIT clause restricts the..."
+ "title": "Casting",
+ "text": "casting refers to the process of changing the type of a row from one type to another the standard sql syntax for this is cast expr as typename duckdb also supports the easier to type shorthand expr typename which is also present in postgresql select cast i as varchar from generate_series 1 3 tbl i -- 1 2 3 select i double from generate_series 1 3 tbl i -- 1 0 2 0 3 0 select cast hello as integer -- conversion error could not convert string hello to int32 select try_cast hello as integer -- null the exact behavior of the cast depends on the source and destination types for example when casting from varchar to any other type the string will be attempted to be converted not all casts are possible for example it is not possible to convert an integer to a date casts may also throw errors when the cast could not be successfully performed for example trying trying to cast the string hello to an integer will result in an error being thrown try_cast can be used when the preferred behavior is not to throw an error but instead to return a null value try_cast will never throw an error and will instead return null if a cast is not possible implicit casting in many situations the system will add casts by itself this is called implicit casting this happens for example when a function is called with an argument that does not match the type of the function but can be casted to the desired type consider the function sin double this function takes as input argument a column of type double however it can be called with an integer as well sin 1 the integer is converted into a double before being passed to the sin function generally implicit casts only cast upwards that is to say we can implicitly cast an integer to a bigint but not the other way around",
+ "category": "Expressions",
+ "url": "../docs/sql/expressions/cast",
+ "blurb": "Casting refers to the process of changing the type of a row from one type to another. The standard SQL syntax for this is..."
},
{
- "title": "UNNEST",
- "text": "the unnest function is used to unnest a list by one level the function can be used as a regular scalar function but only in the select clause unnest is a special function in the sense that it changes the cardinality of the result the result of the unnest function is one tuple per entry in the list when unnest is combined with regular scalar expressions those expressions are repeated for every entry in the list when multiple lists are unnested in the same select clause the lists are unnested side-by-side if one list is longer than the other the shorter list will be padded with null values an empty list and a null list will both unnest to zero elements untyped and types null arguments will both return zero rows examples -- unnest a scalar list generating 3 rows 1 2 3 select unnest 1 2 3 -- unnest a scalar list generating 3 rows 1 10 2 11 3 null select unnest 1 2 3 unnest 10 11 -- unnest a scalar list generating 3 rows 1 10 2 10 3 10 select unnest 1 2 3 10 -- unnest a list column generated from a subquery select unnest l 10 from values 1 2 3 4 5 tbl l -- empty result select unnest -- zero rows untyped null select unnest null -- zero rows typed null select unnest null int",
- "category": "Query Syntax",
- "url": "../docs/sql/query_syntax/unnest",
- "blurb": "The UNNEST function is used to unnest a list by one level. The function can be used as a regular scalar function, but..."
+ "title": "Collations",
+ "text": "collations provide rules for how text should be sorted or compared in the execution engine collations are useful for localization as the rules for how text should be ordered are different for different languages or for different countries these orderings are often incompatible with one another for example in english the letter y comes between x and z however in lithuanian the letter y comes between the i and j for that reason different collations are supported the user must choose which collation they want to use when performing sorting and comparison operations by default the binary collation is used that means that strings are ordered and compared based only on their binary contents this makes sense for standard ascii characters i e the letters a-z and numbers 0-9 but generally does not make much sense for special unicode characters it is however by far the fastest method of performing ordering and comparisons hence it is recommended to stick with the binary collation unless required otherwise using collations in the stand-alone installation of duckdb three collations are included nocase noaccent and nfc the nocase collation compares characters as equal regardless of their casing the noaccent collation compares characters as equal regardless of their accents the nfc collation performs nfc-normalized comparisons see here for more information select hello hello -- false select hello collate nocase hello -- true select hello hëllo -- false select hello collate noaccent hëllo -- true collations can be combined by chaining them using the dot operator note however that not all collations can be combined together in general the nocase collation can be combined with any other collator but most other collations cannot be combined select hello collate nocase hellö -- false select hello collate noaccent hellö -- false select hello collate nocase noaccent hellö -- true default collations the collations we have seen so far have all been specified per expression it is also possible to specify a default collator either on the global database level or on a base table column the pragma default_collation can be used to specify the global default collator this is the collator that will be used if no other one is specified pragma default_collation nocase select hello hello -- true collations can also be specified per-column when creating a table when that column is then used in a comparison the per-column collation is used to perform that comparison create table names name varchar collate noaccent insert into names values hännes select name from names where name hannes -- hännes be careful here however as different collations cannot be combined this can be problematic when you want to compare columns that have a different collation specified select name from names where name hannes collate nocase -- error cannot combine types with different collation create table other_names name varchar collate nocase insert into other_names values hännes select from names other_names where names name other_names name -- error cannot combine types with different collation -- need to manually overwrite the collation select from names other_names where names name collate noaccent nocase other_names name collate noaccent nocase -- hännes hännes icu collations the collations we have seen so far are not region dependent and do not follow any specific regional rules if you wish to follow the rules of a specific region or language you will need to use one of the icu collations for that you need to include the icu extension this can be found in the extension icu folder in the project using the c api the extension can be loaded as follows duckdb db db loadextension icuextension loading this extension will add a number of language and region specific collations to your database these can be queried using pragma collations command or by querying the pragma_collations function pragma collations select from pragma_collations -- af am ar as az be bg bn bo bs bs ca ceb chr cs cy da de de_at dsb dz ee el en en_us en_us eo es et fa fa_af fi fil fo fr fr_ca ga gl gu ha haw he he_il hi hr hsb hu hy id id_id ig is it ja ka kk kl km kn ko kok ku ky lb lkt ln lo lt lv mk ml mn mr ms mt my nb nb_no ne nl nn om or pa pa pa_in pl ps pt ro ru se si sk sl smn sq sr sr sr_ba sr_me sr_rs sr sr_ba sr_rs sv sw ta te th tk to tr ug uk ur uz vi wae wo xh yi yo zh zh zh_cn zh_sg zh zh_hk zh_mo zh_tw zu these collations can then be used as the other collations would be used before they can also be combined with the nocase collation for example to use the german collation rules you could use the following code snippet create table strings s varchar collate de insert into strings values gabel göbel goethe goldmann göthe götz select from strings order by s -- gabel göbel goethe goldmann göthe götz",
+ "category": "Expressions",
+ "url": "../docs/sql/expressions/collations",
+ "blurb": "Collations provide rules for how text should be sorted or compared in the execution engine. Collations are useful for..."
},
{
- "title": "HAVING Clause",
- "text": "the having clause can be used after the group by clause to provide filter criteria after the grouping has been completed in terms of syntax the having clause is identical to the where clause but while the where clause occurs before the grouping the having clause occurs after the grouping examples -- count the number of entries in the addresses table that belong to each different city -- filtering out cities with a count below 50 select city count from addresses group by city having count 50 -- compute the average income per city per street_name -- filtering out cities with an average income bigger than twice the median income select city street_name avg income from addresses group by city street_name having avg income 2 median income syntax",
- "category": "Query Syntax",
- "url": "../docs/sql/query_syntax/having",
- "blurb": "The HAVING clause can be used after the GROUP BY clause to provide filter criteria after the grouping has been..."
+ "title": "Logical Operators",
+ "text": "the following logical operators are available and or and not sql uses a three-valuad logic system with true false and null note that logical operators involving null do not always evaluate to null for example null and false will evaluate to false and null or true will evaluate to true below are the complete truth tables a b a and b a or b --- --- --- --- true true true true true false false true true null null true false false false false false null false null null null null null a not a --- --- true false false true null null the operators and and or are commutative that is you can switch the left and right operand without affecting the result",
+ "category": "Expressions",
+ "url": "../docs/sql/expressions/logical_operators",
+ "blurb": "The following logical operators are available: AND , OR and NOT . SQL uses a three-valuad logic system with TRUE , ..."
},
{
- "title": "ORDER BY Clause",
- "text": "order by is an output modifier logically it is applied at the very end of the query the order by clause sorts the rows on the sorting criteria in either ascending or descending order in addition every order clause can specify whether null values should be moved to the beginning or to the end by default if no modifiers are provided duckdb sorts asc nulls first i e the values are sorted in ascending order and null values are placed first this is identical to the default sort order of sqlite postgresql by default sorts in asc nulls last order the default sort order can be changed using the following pragma statements -- change the default null sorting order to either nulls first and nulls last pragma default_null_order nulls last -- change the default sorting order to either desc or asc pragma default_order desc text is sorted using the binary comparison collation by default which means values are sorted on their binary utf8 values while this works well for ascii text e g for english language data the sorting order can be incorrect for other languages for this purpose duckdb provides collations for more information on collations see the collation page examples -- select the addresses ordered by city name using the default null order and default order select from addresses order by city -- select the addresses ordered by city name in descending order with nulls at the end select from addresses order by city desc nulls last -- order by city and then by zip code both using the default orderings select from addresses order by city zip -- order by city using german collation rules select from addresses order by city collate de syntax",
- "category": "Query Syntax",
- "url": "../docs/sql/query_syntax/orderby",
- "blurb": "ORDER BY is an output modifier. Logically it is applied at the very end of the query. The ORDER BY clause sorts the..."
+ "title": "Comparisons",
+ "text": "comparison operators the table below shows the standard comparison operators whenever either of the input arguments is null the output of the comparison is null operator description example result --- --- --- --- less than 2 3 true greater than 2 3 false less than or equal to 2 3 true greater than or equal to 4 null null equal null null null or not equal 2 2 false the table below shows the standard distinction operators these operators treat null values as equal operator description example result --- --- --- --- is distinct from equal including null 2 is distinct from null true is not distinct from not equal including null null is not distinct from null true between and is not null besides the stanadrd comparison operators there are also the between and is not null operators these behave much like operators but have special syntax mandated by the sql standard they are shown in the table below predicate description --- --- a between x and y equivalent to a x and a y a not between x and y equivalent to a x or a y expression is null true if expression is null false otherwise expression isnull alias for is null non-standard expression is not null false if expression is null true otherwise expression notnull alias for is not null non-standard",
+ "category": "Expressions",
+ "url": "../docs/sql/expressions/comparison_operators",
+ "blurb": "Comparison Operators The table below shows the standard comparison operators. Whenever either of the input arguments..."
},
{
- "title": "Information Schema",
- "text": "the views in the information_schema are sql-standard views that describe the catalog entries of the database these views can be filtered to obtain information about a specific column or table database catalog and schema the top level catalog view is information_schema schemata it lists the catalogs and the schemas present in the database and has the following layout column description type example --- --- --- --- catalog_name name of the database that the schema is contained in not yet implemented varchar null schema_name name of the schema varchar main schema_owner name of the owner of the schema not yet implemented varchar null default_character_set_catalog applies to a feature not available in duckdb varchar null default_character_set_schema applies to a feature not available in duckdb varchar null default_character_set_name applies to a feature not available in duckdb varchar null sql_path the file system location of the database currently unimplemented varchar null tables and views the view that describes the catalog information for tables and views is information_schema tables it lists the tables present in the database and has the following layout column description type example --- --- --- --- table_catalog the catalog the table or view belongs to not yet implemented varchar null table_schema the schema the table or view belongs to varchar main table_name the name of the table or view varchar widgets table_type the type of table one of base table local temporary view varchar base table self_referencing_column_name applies to a feature not available in duckdb varchar null reference_generation applies to a feature not available in duckdb varchar null user_defined_type_catalog if the table is a typed table the name of the database that contains the underlying data type always the current database else null currently unimplemented varchar null user_defined_type_schema if the table is a typed table the name of the schema that contains the underlying data type else null currently unimplemented varchar null user_defined_type_name if the table is a typed table the name of the underlying data type else null currently unimplemented varchar null is_insertable_into yes if the table is insertable into no if not base tables are always insertable into views not necessarily varchar yes is_typed yes if the table is a typed table no if not varchar no commit_action not yet implemented varchar no columns the view that describes the catalog information for columns is information_schema columns it lists the column present in the database and has the following layout column description type example --- --- --- --- table_catalog name of the database containing the table not yet implemented varchar null table_schema name of the schema containing the table varchar main table_name name of the table varchar widgets column_name name of the column varchar price ordinal_position ordinal position of the column within the table count starts at 1 integer 5 column_default default expression of the column varchar 1 99 is_nullable yes if the column is possibly nullable no if it is known not nullable varchar yes data_type data type of the column varchar decimal 18 2 character_maximum_length if data_type identifies a character or bit string type the declared maximum length null for all other data types or if no maximum length was declared integer 255 character_octet_length if data_type identifies a character type the maximum possible length in octets bytes of a datum null for all other data types the maximum octet length depends on the declared character maximum length see above and the character encoding integer 1073741824 numeric_precision if data_type identifies a numeric type this column contains the declared or implicit precision of the type for this column the precision indicates the number of significant digits for all other data types this column is null integer 18 numeric_scale if data_type identifies a numeric type this column contains the declared or implicit scale of the type for this column the precision indicates the number of significant digits for all other data types this column is null integer 2 datetime_precision if data_type identifies a date time timestamp or interval type this column contains the declared or implicit fractional seconds precision of the type for this column that is the number of decimal digits maintained following the decimal point in the seconds value no fractional seconds are currently supported in duckdb for all other data types this column is null integer 0 catalog functions several functions are also provided to see details about the schemas that are configured in the database function description example result --- --- --- --- current_schema return the name of the currently active schema default is main current_schema main current_schemas boolean return list of schemas pass a parameter of true to include implicit schemas current_schemas true temp main pg_catalog",
- "category": "SQL",
- "url": "../docs/sql/information_schema",
- "blurb": "The views in the information_schema are SQL-standard views that describe the catalog entries of the database. These..."
+ "title": "Subqueries",
+ "text": "scalar subquery scalar subqueries are subqueries that return a single value they can be used anywhere where a regular expression can be used if a scalar subquery returns more than a single value the first value returned will be used consider the following table grades grade course --- --- 7 math 9 math 8 cs create table grades grade integer course varchar insert into grades values 7 math 9 math 8 cs we can run the following query to obtain the minimum grade select min grade from grades -- 7 by using a scalar subquery in the where clause we can figure out for which course this grade was obtained select course from grades where grade select min grade from grades -- math exists the exists operator is used to test for the existence of any row inside the subquery it returns either true when the subquery returns one or more records or false otherwise the exists operator is generally the most useful as a correlated subquery however it can be used as an uncorrelated subquery as well for example we can use it to figure out if there are any grades present for a given course select exists select from grades where course math -- true select exists select from grades where course history -- false in operator the in operator checks containment of the left expression inside the result defined by the subquery or the set of expressions on the right hand side rhs the in operator returns true if the expression is present in the rhs false if the expression is not in the rhs and the rhs has no null values or null if the expression is not in the rhs and the rhs has null values we can use the in operator in a similar manner as we used the exists operator select math in select course from grades -- true correlated subqueries all the subqueries presented here so far have been uncorrelated subqueries where the subqueries themselves are entirely self-contained and can be run without the parent query there exists a second type of subqueries called correlated subqueries for correlated subqueries the subquery uses values from the parent subquery conceptually the subqueries are run once for every single row in the parent query perhaps a simple way of envisioning this is that the correlated subquery is a function that is applied to every row in the source data set for example suppose that we want to find the minimum grade for every course we could do that as follows select from grades grades_parent where grade select min grade from grades where grades course grades_parent course -- 7 math 8 cs the subquery uses a column from the parent query grades_parent course conceptually we can see the subquery as a function where the correlated column is a parameter to that function select min grade from grades where course now when we execute this function for each of the rows we can see that for math this will return 7 and for cs it will return 8 we then compare it against the grade for that actual row as a result the row math 9 will be filtered out as 9 7",
+ "category": "Expressions",
+ "url": "../docs/sql/expressions/subqueries",
+ "blurb": "Scalar Subquery Scalar subqueries are subqueries that return a single value. They can be used anywhere where a regular..."
},
{
"title": "Case Statement",
@@ -742,13 +721,6 @@
"url": "../docs/sql/expressions/case",
"blurb": "The CASE statement performs a switch based on a condition. The basic form is identical to the ternary condition used in..."
},
- {
- "title": "Comparisons",
- "text": "comparison operators the table below shows the standard comparison operators whenever either of the input arguments is null the output of the comparison is null operator description example result --- --- --- --- less than 2 3 true greater than 2 3 false less than or equal to 2 3 true greater than or equal to 4 null null equal null null null or not equal 2 2 false the table below shows the standard distinction operators these operators treat null values as equal operator description example result --- --- --- --- is distinct from equal including null 2 is distinct from null true is not distinct from not equal including null null is not distinct from null true between and is not null besides the stanadrd comparison operators there are also the between and is not null operators these behave much like operators but have special syntax mandated by the sql standard they are shown in the table below predicate description --- --- a between x and y equivalent to a x and a y a not between x and y equivalent to a x or a y expression is null true if expression is null false otherwise expression isnull alias for is null non-standard expression is not null false if expression is null true otherwise expression notnull alias for is not null non-standard",
- "category": "Expressions",
- "url": "../docs/sql/expressions/comparison_operators",
- "blurb": "Comparison Operators The table below shows the standard comparison operators. Whenever either of the input arguments..."
- },
{
"title": "Expressions",
"text": "an expression is a combination of values operators and functions expressions are highly composable and range from very simple to arbitrarily complex they can be found in many different parts of sql statements in this section we provide the different types of operators and functions that can be used within expressions more",
@@ -757,242 +729,270 @@
"blurb": "An expression is a combination of values, operators and functions. Expressions are highly composable, and range from very..."
},
{
- "title": "Subqueries",
- "text": "scalar subquery scalar subqueries are subqueries that return a single value they can be used anywhere where a regular expression can be used if a scalar subquery returns more than a single value the first value returned will be used consider the following table grades grade course --- --- 7 math 9 math 8 cs create table grades grade integer course varchar insert into grades values 7 math 9 math 8 cs we can run the following query to obtain the minimum grade select min grade from grades -- 7 by using a scalar subquery in the where clause we can figure out for which course this grade was obtained select course from grades where grade select min grade from grades -- math exists the exists operator is used to test for the existence of any row inside the subquery it returns either true when the subquery returns one or more records or false otherwise the exists operator is generally the most useful as a correlated subquery however it can be used as an uncorrelated subquery as well for example we can use it to figure out if there are any grades present for a given course select exists select from grades where course math -- true select exists select from grades where course history -- false in operator the in operator checks containment of the left expression inside the result defined by the subquery or the set of expressions on the right hand side rhs the in operator returns true if the expression is present in the rhs false if the expression is not in the rhs and the rhs has no null values or null if the expression is not in the rhs and the rhs has null values we can use the in operator in a similar manner as we used the exists operator select math in select course from grades -- true correlated subqueries all the subqueries presented here so far have been uncorrelated subqueries where the subqueries themselves are entirely self-contained and can be run without the parent query there exists a second type of subqueries called correlated subqueries for correlated subqueries the subquery uses values from the parent subquery conceptually the subqueries are run once for every single row in the parent query perhaps a simple way of envisioning this is that the correlated subquery is a function that is applied to every row in the source data set for example suppose that we want to find the minimum grade for every course we could do that as follows select from grades grades_parent where grade select min grade from grades where grades course grades_parent course -- 7 math 8 cs the subquery uses a column from the parent query grades_parent course conceptually we can see the subquery as a function where the correlated column is a parameter to that function select min grade from grades where course now when we execute this function for each of the rows we can see that for math this will return 7 and for cs it will return 8 we then compare it against the grade for that actual row as a result the row math 9 will be filtered out as 9 7",
- "category": "Expressions",
- "url": "../docs/sql/expressions/subqueries",
- "blurb": "Scalar Subquery Scalar subqueries are subqueries that return a single value. They can be used anywhere where a regular..."
+ "title": "Samples",
+ "text": "samples are used to randomly select a subset of a dataset examples -- select a sample of 5 rows from tbl using reservoir sampling select from tbl using sample 5 -- select a sample of 10 of the table using system sampling cluster sampling select from tbl using sample 10 -- select a sample of 10 of the table using bernoulli sampling select from tbl using sample 10 percent bernoulli -- select a sample of 50 rows of the table using reservoir sampling with a fixed seed 100 select from tbl using sample reservoir 50 rows repeatable 100 -- select a sample of 20 of the table using system sampling with a fixed seed 377 select from tbl using sample 10 system 377 -- select a sample of 10 of tbl before the join with tbl2 select from tbl tablesample reservoir 20 tbl2 where tbl i tbl2 i -- select a sample of 10 of tbl after the join with tbl2 select from tbl tbl2 where tbl i tbl2 i using sample reservoir 20 syntax samples allow you to randomly extract a subset of a dataset samples are useful for exploring a dataset faster as often you might not be interested in the exact answers to queries but only in rough indications of what the data looks like and what is in the data samples allow you to get approximate answers to queries faster as they reduce the amount of data that needs to pass through the query engine duckdb supports three different types of sampling methods reservoir bernoulli and system by default duckdb uses reservoir sampling when an exact number of rows is sampled and system sampling when a percentage is specified the sampling methods are described in detail below samples require a sample size which is an indication of how many elements will be sampled from the total population samples can either be given as a percentage 10 or as a fixed number of rows 10 rows all three sampling methods support sampling over a percentage but only reservoir sampling supports sampling a fixed number of rows samples are probablistic that is to say samples can be different between runs unless the seed is specifically specified specifying the seed only guarantees that the sample is the same if multi-threading is not enabled i e pragma threads 1 in the case of multiple threads running over a sample samples are not necessarily consistent even with a fixed seed reservoir reservoir sampling is a stream sampling technique that selects a random sample by keeping a reservoir of size equal to the sample size and randomly replacing elements as more elements come in reservoir sampling allows us to specify exactly how many elements we want in the resulting sample by selecting the size of the reservoir as a result reservoir sampling always outputs the same amount of elements unlike system and bernoulli sampling reservoir sampling is only recommended for small sample sizes and is not recommended for use with percentages that is because reservoir sampling needs to materialize the entire sample and randomly replace tuples within the materialized sample the larger the sample size the higher the performance hit incurred by this process reservoir sampling also incurs an additional performance penalty when multi-processing is used since the reservoir is to be shared amongst the different threads to ensure unbiased sampling this is not a big problem when the reservoir is very small but becomes costly when the sample is large bernoulli bernoulli sampling can only be used when a sampling percentage is specified it is rather straightforward every tuple in the underlying table is included with a chance equal to the specified percentage as a result bernoulli sampling can return a different number of tuples even if the same percentage is specified the amount of rows will generally be more or less equal to the specified percentage of the table but there will be some variance because bernoulli sampling is completely independent there is no shared state there is no penalty for using bernoulli sampling together with multiple threads system system sampling is a variant of bernoulli sampling with one crucial difference every vector is included with a chance equal to the sampling percentage this is a form of cluster sampling system sampling is more efficient than bernoulli sampling as no per-tuple selections have to be performed there is almost no extra overhead for using system sampling whereas bernoulli sampling can add additional cost as it has to perform random number generation for every single tuple system sampling is not suitable for smaller data sets as the granularity of the sampling is on the order of 1000 tuples that means that if system sampling is used for small data sets e g 100 rows either all the data will be filtered out or all the data will be included table samples the tablesample and using sample clauses are identical in terms of syntax and effect with one important difference tablesamples sample directly from the table for which they are specified whereas the sample clause samples after the entire from clause has been resolved this is relevant when there are joins present in the query plan the tablesample clause is essentially equivalent to creating a subquery with the using sample clause i e the following two queries are identical -- sample 20 of tbl before the join select from tbl tablesample reservoir 20 tbl2 where tbl i tbl2 i -- sample 20 of tbl before the join select from select from tbl using sample reservoir 20 tbl tbl2 where tbl i tbl2 i -- sample 20 after the join i e sample 20 of the join result select from tbl tbl2 where tbl i tbl2 i using sample reservoir 20",
+ "category": "SQL",
+ "url": "../docs/sql/samples",
+ "blurb": "Samples are used to randomly select a subset of a dataset. Examples -- select a sample of 5 rows from tbl using..."
},
{
- "title": "Casting",
- "text": "casting refers to the process of changing the type of a row from one type to another the standard sql syntax for this is cast expr as typename duckdb also supports the easier to type shorthand expr typename which is also present in postgresql select cast i as varchar from generate_series 1 3 tbl i -- 1 2 3 select i double from generate_series 1 3 tbl i -- 1 0 2 0 3 0 select cast hello as integer -- conversion error could not convert string hello to int32 select try_cast hello as integer -- null the exact behavior of the cast depends on the source and destination types for example when casting from varchar to any other type the string will be attempted to be converted not all casts are possible for example it is not possible to convert an integer to a date casts may also throw errors when the cast could not be successfully performed for example trying trying to cast the string hello to an integer will result in an error being thrown try_cast can be used when the preferred behavior is not to throw an error but instead to return a null value try_cast will never throw an error and will instead return null if a cast is not possible implicit casting in many situations the system will add casts by itself this is called implicit casting this happens for example when a function is called with an argument that does not match the type of the function but can be casted to the desired type consider the function sin double this function takes as input argument a column of type double however it can be called with an integer as well sin 1 the integer is converted into a double before being passed to the sin function generally implicit casts only cast upwards that is to say we can implicitly cast an integer to a bigint but not the other way around",
- "category": "Expressions",
- "url": "../docs/sql/expressions/cast",
- "blurb": "Casting refers to the process of changing the type of a row from one type to another. The standard SQL syntax for this is..."
+ "title": "Date Types",
+ "text": "name aliases description ------- -------- -------------------------------- date calendar date year month day a date specifies a combination of year month and day duckdb follows the sql standard s lead by counting dates exclusively in the gregorian calendar even for years before that calendar was in use dates can be created using the date keyword where the data must be formatted according to the iso 8601 format yyyy-mm-dd -- 20 september 1992 select date 1992-09-20 special values there are also three special date values that can be used on input input string description ------------- ---------------------------------- epoch 1970-01-01 unix system day zero infinity later than all other dates -infinity earlier than all other dates the values infinity and -infinity are specially represented inside the system and will be displayed unchanged but epoch is simply a notational shorthand that will be converted to the date value when read select -infinity date epoch date infinity date negative epoch positive ---------- ----------- --------- -infinity 1970-01-01 infinity functions see date functions",
+ "category": "Data Types",
+ "url": "../docs/sql/data_types/date",
+ "blurb": "A date specifies a combination of year, month and day."
},
{
- "title": "Logical Operators",
- "text": "the following logical operators are available and or and not sql uses a three-valuad logic system with true false and null note that logical operators involving null do not always evaluate to null for example null and false will evaluate to false and null or true will evaluate to true below are the complete truth tables a b a and b a or b --- --- --- --- true true true true true false false true true null null true false false false false false null false null null null null null a not a --- --- true false false true null null the operators and and or are commutative that is you can switch the left and right operand without affecting the result",
- "category": "Expressions",
- "url": "../docs/sql/expressions/logical_operators",
- "blurb": "The following logical operators are available: AND , OR and NOT . SQL uses a three-valuad logic system with TRUE , ..."
+ "title": "Interval Type",
+ "text": "intervals represent a period of time this period can be measured in a variety of units for example years days or seconds see the date part functions docs for a list of available date parts for use with an interval name description --- --- interval period of time an interval can be generated directly or can be a result of a function for example calculating the difference between two timestamps intervals can be used to modify date timestamp or timestamp with time zone data types see the interval operators for details -- each date part can be either singular or plural -- in this example year or years can be used interchangeably -- 1 year select interval 1 year -- 1 year select interval 1 years -- the number used to specify an interval can optionally be wrapped in single quotes -- 28 days select interval 28 days -- the number and date part can optionally be wrapped entirely in single quotes -- 28 days select interval 28 days -- intervals can also be used to specify a time period rather than a date period -- 00 00 30 select interval 30 seconds -- intervals can also be produced as a result of a timestamp operator like subtraction -- these can include a date and time component on the interval -- 1 day 01 00 00 select 2022-01-02 01 00 00 timestamp - 2022-01-01 timestamp -- warning if a decimal value is specified it will be automatically truncated to an integer -- to use more precise values simply use a more granular date part -- in this example use 18 months instead of 1 5 years -- the statement below is equivalent to to_years cast 1 5 as integer -- 1 year select interval 1 5 years --warning this returns 1 year functions see interval functions",
+ "category": "Data Types",
+ "url": "../docs/sql/data_types/interval",
+ "blurb": "An interval specifies a period of time measured in units of a specific date part like years, days, seconds, or others."
},
{
- "title": "Collations",
- "text": "collations provide rules for how text should be sorted or compared in the execution engine collations are useful for localization as the rules for how text should be ordered are different for different languages or for different countries these orderings are often incompatible with one another for example in english the letter y comes between x and z however in lithuanian the letter y comes between the i and j for that reason different collations are supported the user must choose which collation they want to use when performing sorting and comparison operations by default the binary collation is used that means that strings are ordered and compared based only on their binary contents this makes sense for standard ascii characters i e the letters a-z and numbers 0-9 but generally does not make much sense for special unicode characters it is however by far the fastest method of performing ordering and comparisons hence it is recommended to stick with the binary collation unless required otherwise using collations in the stand-alone installation of duckdb three collations are included nocase noaccent and nfc the nocase collation compares characters as equal regardless of their casing the noaccent collation compares characters as equal regardless of their accents the nfc collation performs nfc-normalized comparisons see here for more information select hello hello -- false select hello collate nocase hello -- true select hello hëllo -- false select hello collate noaccent hëllo -- true collations can be combined by chaining them using the dot operator note however that not all collations can be combined together in general the nocase collation can be combined with any other collator but most other collations cannot be combined select hello collate nocase hellö -- false select hello collate noaccent hellö -- false select hello collate nocase noaccent hellö -- true default collations the collations we have seen so far have all been specified per expression it is also possible to specify a default collator either on the global database level or on a base table column the pragma default_collation can be used to specify the global default collator this is the collator that will be used if no other one is specified pragma default_collation nocase select hello hello -- true collations can also be specified per-column when creating a table when that column is then used in a comparison the per-column collation is used to perform that comparison create table names name varchar collate noaccent insert into names values hännes select name from names where name hannes -- hännes be careful here however as different collations cannot be combined this can be problematic when you want to compare columns that have a different collation specified select name from names where name hannes collate nocase -- error cannot combine types with different collation create table other_names name varchar collate nocase insert into other_names values hännes select from names other_names where names name other_names name -- error cannot combine types with different collation -- need to manually overwrite the collation select from names other_names where names name collate noaccent nocase other_names name collate noaccent nocase -- hännes hännes icu collations the collations we have seen so far are not region dependent and do not follow any specific regional rules if you wish to follow the rules of a specific region or language you will need to use one of the icu collations for that you need to include the icu extension this can be found in the extension icu folder in the project using the c api the extension can be loaded as follows duckdb db db loadextension icuextension loading this extension will add a number of language and region specific collations to your database these can be queried using pragma collations command or by querying the pragma_collations function pragma collations select from pragma_collations -- af am ar as az be bg bn bo bs bs ca ceb chr cs cy da de de_at dsb dz ee el en en_us en_us eo es et fa fa_af fi fil fo fr fr_ca ga gl gu ha haw he he_il hi hr hsb hu hy id id_id ig is it ja ka kk kl km kn ko kok ku ky lb lkt ln lo lt lv mk ml mn mr ms mt my nb nb_no ne nl nn om or pa pa pa_in pl ps pt ro ru se si sk sl smn sq sr sr sr_ba sr_me sr_rs sr sr_ba sr_rs sv sw ta te th tk to tr ug uk ur uz vi wae wo xh yi yo zh zh zh_cn zh_sg zh zh_hk zh_mo zh_tw zu these collations can then be used as the other collations would be used before they can also be combined with the nocase collation for example to use the german collation rules you could use the following code snippet create table strings s varchar collate de insert into strings values gabel göbel goethe goldmann göthe götz select from strings order by s -- gabel göbel goethe goldmann göthe götz",
- "category": "Expressions",
- "url": "../docs/sql/expressions/collations",
- "blurb": "Collations provide rules for how text should be sorted or compared in the execution engine. Collations are useful for..."
+ "title": "NULL Values",
+ "text": "null values are special values that are used to represent missing data in sql columns of any type can contain null values logically a null value can be seen as the value of this field is unknown -- insert a null value into a table create table integers i integer insert into integers values null null values have special semantics in many parts of the query as well as in many functions any comparison with a null value returns null including null null you can use is not distinct from to perform an equality comparison where null values compare equal to each other use is not null to check if a value is null select null null -- returns null select null is not distinct from null -- returns true select null is null -- returns true null and functions a function that has input argument as null usually returns null select cos null -- null coalesce is an exception to this coalesce takes any number of arguments and returns for each row the first argument that is not null if all arguments are null coalesce also returns null select coalesce null null 1 -- 1 select coalesce 10 20 -- 10 select coalesce null null -- null null and conjunctions null values have special semantics in and or conjunctions for the ternary logic truth tables see the boolean type documentation null and aggregate functions null values are ignored in most aggregate functions aggregate functions that do not ignore null values include first last list and array_agg to exclude null values from those aggregate functions the filter clause can be used create table integers i integer insert into integers values 1 10 null select min i from integers -- 1 select max i from integers -- 10",
+ "category": "Data Types",
+ "url": "../docs/sql/data_types/nulls",
+ "blurb": "The NULL value represents a missing value."
},
{
- "title": "IN Operator",
- "text": "in operator the in operator checks containment of the left expression inside the set of expressions on the right hand side rhs the in operator returns true if the expression is present in the rhs false if the expression is not in the rhs and the rhs has no null values or null if the expression is not in the rhs and the rhs has null values select math in cs math -- true select english in cs math -- false select math in cs math null -- true select english in cs math null -- null not in can be used to check if an element is not present in the set x not in y is equivalent to not x in y the in operator can also be used with a subquery that returns a single column see the subqueries page for more information",
- "category": "Expressions",
- "url": "../docs/sql/expressions/in",
- "blurb": "In Operator The IN operator checks containment of the left expression inside the set of expressions on the right..."
+ "title": "Numeric Types",
+ "text": "integer types the types tinyint smallint integer bigint and hugeint store whole numbers that is numbers without fractional components of various ranges attempts to store values outside of the allowed range will result in an error the types utinyint usmallint uinteger ubigint store whole unsigned numbers attempts to store negative numbers or values outside of the allowed range will result in an error name aliases min max --- --- --- --- tinyint int1 -128 127 smallint int2 short -32768 32767 integer int4 int signed -2147483648 2147483647 bigint int8 long -9223372036854775808 9223372036854775807 hugeint -170141183460469231731687303715884105727 170141183460469231731687303715884105727 utinyint - 0 255 usmallint - 0 65535 uinteger - 0 4294967295 ubigint - 0 18446744073709551615 the type integer is the common choice as it offers the best balance between range storage size and performance the smallint type is generally only used if disk space is at a premium the bigint and hugeint types are designed to be used when the range of the integer type is insufficient -170141183460469231731687303715884105728 -1 127 is not representable by the internal structure fixed-point decimals the data type decimal width scale represents an exact fixed-point decimal value when creating a value of type decimal the width and scale can be specified to define which size of decimal values can be held in the field the width field determines how many digits can be held and the scale determines the amount of digits after the decimal point for example the type decimal 3 2 can fit the value 1 23 but cannot fit the value 12 3 or the value 1 234 the default width and scale is decimal 18 3 if none are specified internally decimals are represented as integers depending on their specified width width internal size bytes --- --- --- 1-4 int16 2 5-9 int32 4 10-18 int64 8 19-38 int128 16 performance can be impacted by using too large decimals when not required in particular decimal values with a width above 19 are very slow as arithmetic involving the int128 type is much more expensive than operations involving the int32 or int64 types it is therefore recommended to stick with a width of 18 or below unless there is a good reason for why this is insufficient floating-point types the data types real and double precision are inexact variable-precision numeric types in practice these types are usually implementations of ieee standard 754 for binary floating-point arithmetic single and double precision respectively to the extent that the underlying processor operating system and compiler support it name aliases description --- --- --- real float4 single precision floating-point number 4 bytes double float8 double precision floating-point number 8 bytes inexact means that some values cannot be converted exactly to the internal format and are stored as approximations so that storing and retrieving a value might show slight discrepancies managing these errors and how they propagate through calculations is the subject of an entire branch of mathematics and computer science and will not be discussed here except for the following points if you require exact storage and calculations such as for monetary amounts use the numeric type instead if you want to do complicated calculations with these types for anything important especially if you rely on certain behavior in boundary cases infinity underflow you should evaluate the implementation carefully comparing two floating-point values for equality might not always work as expected on most platforms the real type has a range of at least 1e-37 to 1e 37 with a precision of at least 6 decimal digits the double type typically has a range of around 1e-307 to 1e 308 with a precision of at least 15 digits values that are too large or too small will cause an error rounding might take place if the precision of an input number is too high numbers too close to zero that are not representable as distinct from zero will cause an underflow error in addition to ordinary numeric values the floating-point types have several special values infinity -infinity nan these represent the ieee 754 special values infinity negative infinity and not-a-number respectively on a machine whose floating-point arithmetic does not follow ieee 754 these values will probably not work as expected when writing these values as constants in an sql command you must put quotes around them for example update table set x -infinity on input these strings are recognized in a case-insensitive manner functions see numeric functions and operators",
+ "category": "Data Types",
+ "url": "../docs/sql/data_types/numeric",
+ "blurb": "Numeric types are used to store numbers, and come in different shapes and sizes."
},
{
- "title": "Connect",
- "text": "connect or create a database to use duckdb you must first create a connection to a database the exact process varies by client most clients take a parameter pointing to a database file to read and write from the file extension may be anything e g db duckdb etc if the database file does not exist it will be created the special value memory can be used to create an in-memory database where no data is persisted to disk i e all data is lost when you exit the process see the api docs for client-specific details",
- "category": "Docs",
- "url": "../docs/connect",
- "blurb": "Connect or Create a Database To use DuckDB, you must first create a connection to a database. The exact process varies..."
+ "title": "Text Types",
+ "text": "in duckdb strings can be stored in the varchar field name aliases description --- --- --- varchar char bpchar text string variable-length character string varchar n variable-length character string with maximum length n it is possible to supply a maximum length along with the type by initializing a type as varchar n where n is a positive integer note that specifying this length is not required and specifying this length will not improve performance or reduce storage space of the strings in the database specifying a maximum length is useful only for data integrity reasons not for performance reasons in fact the following sql statements are equivalent -- the following statements are equivalent create table strings val varchar 10 -- val has a maximum length of 10 characters create table strings val varchar check length val 10 -- val has a maximum length of 10 characters the varchar field allows storage of unicode characters internally the data is encoded as utf-8 functions see character functions and pattern matching",
+ "category": "Data Types",
+ "url": "../docs/sql/data_types/text",
+ "blurb": "In DuckDB, strings can be stored in the VARCHAR field."
},
{
- "title": "Scala JDBC API",
- "text": "installation the duckdb java jdbc api can be used in scala and can be installed from maven central please see the installation page for details basic api usage scala uses duckdb s jdbc api implements the main parts of the standard java database connectivity jdbc api version 4 0 describing jdbc is beyond the scope of this page see the official documentation for details below we focus on the duckdb-specific parts startup shutdown in scala database connections are created through the standard java sql drivermanager class the driver should auto-register in the drivermanager if that does not work for some reason you can enforce registration like so class forname org duckdb duckdbdriver to create a duckdb connection call drivermanager with the jdbc duckdb jdbc url prefix like so val conn drivermanager getconnection jdbc duckdb when using the jdbc duckdb url alone an in-memory database is created note that for an in-memory database no data is persisted to disk i e all data is lost when you exit the java program if you would like to access or create a persistent database append its file name after the path for example if your database is stored in tmp my_database use the jdbc url jdbc duckdb tmp my_database to create a connection to it it is possible to open a duckdb database file in read-only mode this is for example useful if multiple java processes want to read the same database file at the same time to open an existing database file in read-only mode set the connection property duckdb read_only like so val ro_prop new properties ro_prop setproperty duckdb read_only true val conn_ro drivermanager getconnection jdbc duckdb tmp my_database ro_prop the jdbc drivermanager api is a relatively poor fit for embedded database management systems such as duckdb if you would like to create multiple connections to the same database it would be somewhat logical to just create additional connections with the same url this is however only supported for read-only connections if you would like to create multiple read-write connections to the same database file or the same in-memory database instance you can use the custom duplicate method like so val conn2 duckdbconnection conn duplicate querying duckdb supports the standard jdbc methods to send queries and retreive result sets first a statement object has to be created from the connection this object can then be used to send queries using execute and executequery execute is meant for queries where no results are expected like create table or update etc and executequery is meant to be used for queries that produce results e g select below two examples see also the jdbc statement and resultset documentations create a table val stmt conn createstatement stmt execute create table items item varchar value decimal 10 2 count integer insert two items into the table stmt execute insert into items values jeans 20 0 1 hammer 42 2 2 val rs stmt executequery select from items while rs next system out println rs getstring 1 system out println rs getint 3 rs close jeans 1 hammer 2 duckdb also supports prepared statements as per the jdbc api val p_stmt conn preparestatement insert into test values p_stmt setstring 1 chainsaw p_stmt setdouble 2 500 0 p_stmt setint 3 42 p_stmt execute more calls to execute possible p_stmt close do not use prepared statements to insert large amounts of data into duckdb see the data import documentation for better options",
- "category": "Api",
- "url": "../docs/api/scala",
- "blurb": "Installation The DuckDB Java JDBC API can be used in Scala and can be installed from Maven Central . Please see the ..."
+ "title": "Map",
+ "text": "map data type map s are similar to struct s in that they are an ordered list of entries where a key maps to a value however map s do not need to have the same keys present on each row and thus open additional use cases map s are useful when the schema is unknown beforehand and when adding or removing keys in subsequent rows their flexibility is a key differentiator map s must have a single type for all keys and a single type for all values keys and values can be any type and the type of the keys does not need to match the type of the values ex a map of int s to varchar s map s may also have duplicate keys this is possible and useful because maps are ordered map s are also more forgiving when extracting values as they return an empty list if a key is not found rather than throwing an error as structs do in contrast struct s must have string keys but each key may have a value of a different type struct s may not have duplicate keys see the data types overview for a comparison between nested data types to construct a map use the map function provide a list of keys as the first parameter and a list of values for the second creating maps -- a map with integer keys and varchar values this returns 1 a 5 e select map 1 5 a e -- a map with integer keys and numeric values this returns 1 42 001 5 -32 100 select map 1 5 42 001 -32 1 -- keys and or values can also be nested types -- this returns a b 1 1 2 2 c d 3 3 4 4 select map a b c d 1 1 2 2 3 3 4 4 -- create a table with a map column that has integer keys and double values create table map_table map_col map int double retrieving from maps map s use bracket notation for retrieving values this is due to the variety of types that can be used as a map s key selecting from a map also returns a list rather than an individual value -- use bracket notation to retrieve a list containing the value at a key s location this returns 42 -- note that the expression in bracket notation must match the type of the map s key select map 100 5 42 43 100 -- to retrieve the underlying value use list selection syntax to grab the 0th element -- this returns 42 select map 100 5 42 43 100 0 -- if the element is not in the map an empty list will be returned returns -- note that the expression in bracket notation must match the type of the map s key else an error is returned select map 100 5 42 43 123 -- the element_at function can also be used to retrieve a map value this returns 42 select element_at map 100 5 42 43 100 comparison operators nested types can be compared using all the comparison operators these comparisons can be used in logical expressions for both where and having clauses as well as for creating boolean values the ordering is defined positionally in the same way that words can be ordered in a dictionary null values compare greater than all other values and are considered equal to each other at the top level null nested values obey standard sql null comparison rules comparing a null nested value to a non- null nested value produces a null result comparing nested value members however uses the internal nested value rules for null s and a null nested value member will compare above a non- null nested value member functions see nested functions",
+ "category": "Data Types",
+ "url": "../docs/sql/data_types/map",
+ "blurb": "Map Data Type MAP s are similar to STRUCT s in that they are an ordered list of entries where a key maps to a value...."
},
{
- "title": "Rust API",
- "text": "installation the duckdb rust api can be installed from crate io please see the docs rs for details basic api usage duckdb-rs is an ergonomic wrapper based on the duckdb c api please refer to the readme for details startup shutdown to use duckdb you must first initialize a connection handle using connection open connection open takes as parameter the database file to read and write from if the database file does not exist it will be created the file extension may be db duckdb or anything else you can also use connection open_in_memory to create an in-memory database note that for an in-memory database no data is persisted to disk i e all data is lost when you exit the process use duckdb params connection result let conn connection open_in_memory you can conn close the connection manually or just leave it out of scope we had implement the drop trait which will automatically close the underlining db connection for you querying sql queries can be sent to duckdb using the execute method of connections or we can also prepare the statement and then query on that conn execute insert into person name data values params me name me data let mut stmt conn prepare select id name data from person let person_iter stmt query_map row ok person id row get 0 name row get 1 data row get 2 for person in person_iter println found person person unwrap",
- "category": "Api",
- "url": "../docs/api/rust",
- "blurb": "Installation The DuckDB Rust API can be installed from crate.io . Please see the docs.rs for details. Basic API..."
+ "title": "Boolean Type",
+ "text": "name aliases description --- --- --- boolean bool logical boolean true false the boolean type represents a statement of truth true or false in sql the boolean field can also have a third state unknown which is represented by the sql null value -- select the three possible values of a boolean column select true false null boolean boolean values can be explicitly created using the literals true and false however they are most often created as a result of comparisons or conjunctions for example the comparison i 10 results in a boolean value boolean values can be used in the where and having clauses of a sql statement to filter out tuples from the result in this case tuples for which the predicate evaluates to true will pass the filter and tuples for which the predicate evaluates to false or null will be filtered out consider the following example -- create a table with the value 5 15 and null create table integers i integer insert into integers values 5 15 null -- select all entries where i 10 select from integers where i 10 -- in this case 5 and null are filtered out -- 5 10 false -- null 10 null -- the result is 15 conjunctions the and or conjunctions can be used to combine boolean values below is a truth table for the and conjunction i e x and y x x and true x and false x and null ------- ------- ------- ------- true true false null false false false false null null false null below is a truth table for the or conjunction i e x or y x x or true x or false x or null ------- ------ ------- ------ true true true true false true false null null true null null expressions see logical operators and comparison operators",
+ "category": "Data Types",
+ "url": "../docs/sql/data_types/boolean",
+ "blurb": "The BOOLEAN type represents a statement of truth (true or false)."
},
{
- "title": "C API - Appender",
- "text": "appenders are the most efficient way of loading data into duckdb from within the c interface and are recommended for fast data loading the appender is much faster than using prepared statements or individual insert into statements appends are made in row-wise format for every column a duckdb_append_ type call should be made after which the row should be finished by calling duckdb_appender_end_row after all rows have been appended duckdb_appender_destroy should be used to finalize the appender and clean up the resulting memory note that duckdb_appender_destroy should always be called on the resulting appender even if the function returns duckdberror example duckdb_query con create table people id integer name varchar null duckdb_appender appender if duckdb_appender_create con null people appender duckdberror handle error append the first row 1 mark duckdb_append_int32 appender 1 duckdb_append_varchar appender mark duckdb_appender_end_row appender append the second row 2 hannes duckdb_append_int32 appender 2 duckdb_append_varchar appender hannes duckdb_appender_end_row appender finish appending and flush all the rows to the table duckdb_appender_destroy appender api reference syntax the connection context to create the appender in schema the schema of the table to append to or nullptr for the default schema table the table name to append to out_appender the resulting appender object returns duckdbsuccess on success or duckdberror on failure duckdb_appender_error returns the error message associated with the given appender if the appender has no error message this returns nullptr instead the error message should not be freed it will be de-allocated when duckdb_appender_destroy is called syntax the appender to get the error from returns the error message or nullptr if there is none duckdb_appender_flush flush the appender to the table forcing the cache of the appender to be cleared and the data to be appended to the base table this should generally not be used unless you know what you are doing instead call duckdb_appender_destroy when you are done with the appender syntax the appender to flush returns duckdbsuccess on success or duckdberror on failure duckdb_appender_close close the appender flushing all intermediate state in the appender to the table and closing it for further appends this is generally not necessary call duckdb_appender_destroy instead syntax the appender to flush and close returns duckdbsuccess on success or duckdberror on failure duckdb_appender_destroy close the appender and destroy it flushing all intermediate state in the appender to the table and de-allocating all memory associated with the appender syntax the appender to flush close and destroy returns duckdbsuccess on success or duckdberror on failure duckdb_appender_begin_row a nop function provided for backwards compatibility reasons does nothing only duckdb_appender_end_row is required syntax duckdb_appender_end_row finish the current row of appends after end_row is called the next row can be appended syntax the appender returns duckdbsuccess on success or duckdberror on failure duckdb_append_bool append a bool value to the appender syntax duckdb_append_int8 append an int8_t value to the appender syntax duckdb_append_int16 append an int16_t value to the appender syntax duckdb_append_int32 append an int32_t value to the appender syntax duckdb_append_int64 append an int64_t value to the appender syntax duckdb_append_hugeint append a duckdb_hugeint value to the appender syntax duckdb_append_uint8 append a uint8_t value to the appender syntax duckdb_append_uint16 append a uint16_t value to the appender syntax duckdb_append_uint32 append a uint32_t value to the appender syntax duckdb_append_uint64 append a uint64_t value to the appender syntax duckdb_append_float append a float value to the appender syntax duckdb_append_double append a double value to the appender syntax duckdb_append_date append a duckdb_date value to the appender syntax duckdb_append_time append a duckdb_time value to the appender syntax duckdb_append_timestamp append a duckdb_timestamp value to the appender syntax duckdb_append_interval append a duckdb_interval value to the appender syntax duckdb_append_varchar append a varchar value to the appender syntax duckdb_append_varchar_length append a varchar value to the appender syntax duckdb_append_blob append a blob value to the appender syntax duckdb_append_null append a null value to the appender of any type syntax duckdb_append_data_chunk appends a pre-filled data chunk to the specified appender the types of the data chunk must exactly match the types of the table no casting is performed if the types do not match or the appender is in an invalid state duckdberror is returned if the append is successful duckdbsuccess is returned syntax the appender to append to chunk the data chunk to append returns the return state",
- "category": "C",
- "url": "../docs/api/c/appender",
- "blurb": "Appenders are the most efficient way of loading data into DuckDB from within the C interface, and are recommended for ..."
+ "title": "List",
+ "text": "list data type a list column can have values with different lengths but they must all have the same underlying type list s are typically used to store arrays of numbers but can contain any uniform data type including other list s and struct s list s are similar to postgres s array type duckdb uses the list terminology but some array functions are provided for postgres compatibility see the data types overview for a comparison between nested data types lists can be created using the list_value expr function or the equivalent bracket notation expr the expressions can be constants or arbitrary expressions creating lists -- list of integers select 1 2 3 -- list of strings with a null value select duck goose null heron -- list of lists with null values select duck goose heron null frog toad -- create a list with the list_value function select list_value 1 2 3 -- create a table with an integer list column and a varchar list column create table list_table int_list int varchar_list varchar retrieving from lists retrieving one or more values from a list can be accomplished using brackets and slicing notation or through list functions like list_extract multiple equivalent functions are provided as aliases for compatibility with systems that refer to lists as arrays for example the function array_slice -- retrieve an element from a list using brackets this returns c -- note that we wrap the list creation in parenthesis so that it happens first -- this is only needed in our basic examples here not when working with a list column -- for example this can t be parsed select a b c 1 select a b c 2 -- use a negative index to grab the nth element from the end of the list this returns c select a b c -1 -- any expression that evaluates to an integer can be used to retrieve a list value -- this includes using a column to determine which index to retrieve -- this returns c select a b c 1 1 -- the list_extract function may also be used in place of brackets for selecting individual elements -- this returns c select list_extract a b c 2 -- retrieve multiple list values using a bracketed slice syntax this returns b c select a b c 1 3 -- single sided slices are also supported here grab the first 2 elements this returns a b select a b c 2 -- use a negative index to grab the last 2 elements this returns b c select a b c -2 -- the list_slice function syntax is also supported this returns b c select list_slice a b c 1 3 ordering the ordering is defined positionally null values compare greater than all other values and are considered equal to each other null comparisons at the top level null nested values obey standard sql null comparison rules comparing a null nested value to a non- null nested value produces a null result comparing nested value members however uses the internal nested value rules for null s and a null nested value member will compare above a non- null nested value member functions see nested functions",
+ "category": "Data Types",
+ "url": "../docs/sql/data_types/list",
+ "blurb": "List Data Type A LIST column can have values with different lengths, but they must all have the same underlying type...."
},
{
- "title": "C API - Data Chunks",
- "text": "data chunks represent a horizontal slice of a table they hold a number of vectors that can each hold up to the vector_size rows the vector size can be obtained through the duckdb_vector_size function and is configurable but is usually set to 1024 data chunks and vectors are what duckdb uses natively to store and represent data for this reason the data chunk interface is the most efficient way of interfacing with duckdb be aware however that correctly interfacing with duckdb using the data chunk api does require knowledge of duckdb s internal vector format the primary manner of interfacing with data chunks is by obtaining the internal vectors of the data chunk using the duckdb_data_chunk_get_vector method and subsequently using the duckdb_vector_get_data and duckdb_vector_get_validity methods to read the internal data and the validity mask of the vector for composite types list and struct vectors duckdb_list_vector_get_child and duckdb_struct_vector_get_child should be used to read child vectors api reference syntax an array of types of the data chunk column_count the number of columns returns the data chunk duckdb_destroy_data_chunk destroys the data chunk and de-allocates all memory allocated for that chunk syntax the data chunk to destroy duckdb_data_chunk_reset resets a data chunk clearing the validity masks and setting the cardinality of the data chunk to 0 syntax the data chunk to reset duckdb_data_chunk_get_column_count retrieves the number of columns in a data chunk syntax the data chunk to get the data from returns the number of columns in the data chunk duckdb_data_chunk_get_vector retrieves the vector at the specified column index in the data chunk the pointer to the vector is valid for as long as the chunk is alive it does not need to be destroyed syntax the data chunk to get the data from returns the vector duckdb_data_chunk_get_size retrieves the current number of tuples in a data chunk syntax the data chunk to get the data from returns the number of tuples in the data chunk duckdb_data_chunk_set_size sets the current number of tuples in a data chunk syntax the data chunk to set the size in size the number of tuples in the data chunk duckdb_vector_get_column_type retrieves the column type of the specified vector the result must be destroyed with duckdb_destroy_logical_type syntax the vector get the data from returns the type of the vector duckdb_vector_get_data retrieves the data pointer of the vector the data pointer can be used to read or write values from the vector how to read or write values depends on the type of the vector syntax the vector to get the data from returns the data pointer duckdb_vector_get_validity retrieves the validity mask pointer of the specified vector if all values are valid this function might return null the validity mask is a bitset that signifies null-ness within the data chunk it is a series of uint64_t values where each uint64_t value contains validity for 64 tuples the bit is set to 1 if the value is valid i e not null or 0 if the value is invalid i e null validity of a specific value can be obtained like this idx_t entry_idx row_idx 64 idx_t idx_in_entry row_idx 64 bool is_valid validity_mask entry_idx 1 idx_in_entry alternatively the slower duckdb_validity_row_is_valid function can be used syntax the vector to get the data from returns the pointer to the validity mask or null if no validity mask is present duckdb_vector_ensure_validity_writable ensures the validity mask is writable by allocating it after this function is called duckdb_vector_get_validity will always return non-null this allows null values to be written to the vector regardless of whether a validity mask was present before syntax the vector to alter duckdb_vector_assign_string_element assigns a string element in the vector at the specified location syntax the vector to alter index the row position in the vector to assign the string to str the null-terminated string duckdb_vector_assign_string_element_len assigns a string element in the vector at the specified location syntax the vector to alter index the row position in the vector to assign the string to str the string str_len the length of the string in bytes duckdb_list_vector_get_child retrieves the child vector of a list vector the resulting vector is valid as long as the parent vector is valid syntax the vector returns the child vector duckdb_list_vector_get_size returns the size of the child vector of the list syntax the vector returns the size of the child list duckdb_struct_vector_get_child retrieves the child vector of a struct vector the resulting vector is valid as long as the parent vector is valid syntax the vector index the child index returns the child vector duckdb_validity_row_is_valid returns whether or not a row is valid i e not null in the given validity mask syntax the validity mask as obtained through duckdb_data_chunk_get_validity row the row index returns true if the row is valid false otherwise duckdb_validity_set_row_validity in a validity mask sets a specific row to either valid or invalid note that duckdb_data_chunk_ensure_validity_writable should be called before calling duckdb_data_chunk_get_validity to ensure that there is a validity mask to write to syntax the validity mask as obtained through duckdb_data_chunk_get_validity row the row index valid whether or not to set the row to valid or invalid duckdb_validity_set_row_invalid in a validity mask sets a specific row to invalid equivalent to duckdb_validity_set_row_validity with valid set to false syntax the validity mask row the row index duckdb_validity_set_row_valid in a validity mask sets a specific row to valid equivalent to duckdb_validity_set_row_validity with valid set to true syntax the validity mask row the row index",
- "category": "C",
- "url": "../docs/api/c/data_chunk",
- "blurb": "Data chunks represent a horizontal slice of a table. They hold a number of vectors, that can each hold up to the ..."
+ "title": "Time Zones",
+ "text": "time zone reference list an up-to-date version of this list can be pulled from the pg_timezone_names table function select name abbrev utc_offset from pg_timezone_names order by name name abbrev utc_offset ---------------------------------- ---------------------------------- ------------ act act 09 30 00 aet aet 10 00 00 agt agt -03 00 00 art art 02 00 00 ast ast -09 00 00 africa abidjan africa abidjan 00 00 00 africa accra africa accra 00 00 00 africa addis_ababa eat 03 00 00 africa algiers africa algiers 01 00 00 africa asmara eat 03 00 00 africa asmera eat 03 00 00 africa bamako africa bamako 00 00 00 africa bangui africa bangui 01 00 00 africa banjul africa banjul 00 00 00 africa bissau africa bissau 00 00 00 africa blantyre cat 02 00 00 africa brazzaville africa brazzaville 01 00 00 africa bujumbura cat 02 00 00 africa cairo art 02 00 00 africa casablanca africa casablanca 00 00 00 africa ceuta africa ceuta 01 00 00 africa conakry africa conakry 00 00 00 africa dakar africa dakar 00 00 00 africa dar_es_salaam eat 03 00 00 africa djibouti eat 03 00 00 africa douala africa douala 01 00 00 africa el_aaiun africa el_aaiun 00 00 00 africa freetown africa freetown 00 00 00 africa gaborone cat 02 00 00 africa harare cat 02 00 00 africa johannesburg africa johannesburg 02 00 00 africa juba africa juba 03 00 00 africa kampala eat 03 00 00 africa khartoum africa khartoum 02 00 00 africa kigali cat 02 00 00 africa kinshasa africa kinshasa 01 00 00 africa lagos africa lagos 01 00 00 africa libreville africa libreville 01 00 00 africa lome africa lome 00 00 00 africa luanda africa luanda 01 00 00 africa lubumbashi cat 02 00 00 africa lusaka cat 02 00 00 africa malabo africa malabo 01 00 00 africa maputo cat 02 00 00 africa maseru africa maseru 02 00 00 africa mbabane africa mbabane 02 00 00 africa mogadishu eat 03 00 00 africa monrovia africa monrovia 00 00 00 africa nairobi eat 03 00 00 africa ndjamena africa ndjamena 01 00 00 africa niamey africa niamey 01 00 00 africa nouakchott africa nouakchott 00 00 00 africa ouagadougou africa ouagadougou 00 00 00 africa porto-novo africa porto-novo 01 00 00 africa sao_tome africa sao_tome 00 00 00 africa timbuktu africa timbuktu 00 00 00 africa tripoli libya 02 00 00 africa tunis africa tunis 01 00 00 africa windhoek africa windhoek 02 00 00 america adak america adak -10 00 00 america anchorage ast -09 00 00 america anguilla america anguilla -04 00 00 america antigua america antigua -04 00 00 america araguaina america araguaina -03 00 00 america argentina buenos_aires agt -03 00 00 america argentina catamarca america argentina catamarca -03 00 00 america argentina comodrivadavia america argentina comodrivadavia -03 00 00 america argentina cordoba america argentina cordoba -03 00 00 america argentina jujuy america argentina jujuy -03 00 00 america argentina la_rioja america argentina la_rioja -03 00 00 america argentina mendoza america argentina mendoza -03 00 00 america argentina rio_gallegos america argentina rio_gallegos -03 00 00 america argentina salta america argentina salta -03 00 00 america argentina san_juan america argentina san_juan -03 00 00 america argentina san_luis america argentina san_luis -03 00 00 america argentina tucuman america argentina tucuman -03 00 00 america argentina ushuaia america argentina ushuaia -03 00 00 america aruba america aruba -04 00 00 america asuncion america asuncion -04 00 00 america atikokan america atikokan -05 00 00 america atka america atka -10 00 00 america bahia america bahia -03 00 00 america bahia_banderas america bahia_banderas -06 00 00 america barbados america barbados -04 00 00 america belem america belem -03 00 00 america belize america belize -06 00 00 america blanc-sablon america blanc-sablon -04 00 00 america boa_vista america boa_vista -04 00 00 america bogota america bogota -05 00 00 america boise america boise -07 00 00 america buenos_aires agt -03 00 00 america cambridge_bay america cambridge_bay -07 00 00 america campo_grande america campo_grande -04 00 00 america cancun america cancun -05 00 00 america caracas america caracas -04 00 00 america catamarca america catamarca -03 00 00 america cayenne america cayenne -03 00 00 america cayman america cayman -05 00 00 america chicago cst -06 00 00 america chihuahua america chihuahua -07 00 00 america coral_harbour america coral_harbour -05 00 00 america cordoba america cordoba -03 00 00 america costa_rica america costa_rica -06 00 00 america creston america creston -07 00 00 america cuiaba america cuiaba -04 00 00 america curacao america curacao -04 00 00 america danmarkshavn america danmarkshavn 00 00 00 america dawson america dawson -08 00 00 america dawson_creek america dawson_creek -07 00 00 america denver navajo -07 00 00 america detroit america detroit -05 00 00 america dominica america dominica -04 00 00 america edmonton america edmonton -07 00 00 america eirunepe america eirunepe -05 00 00 america el_salvador america el_salvador -06 00 00 america ensenada america ensenada -08 00 00 america fort_nelson america fort_nelson -07 00 00 america fort_wayne iet -05 00 00 america fortaleza america fortaleza -03 00 00 america glace_bay america glace_bay -04 00 00 america godthab america godthab -03 00 00 america goose_bay america goose_bay -04 00 00 america grand_turk america grand_turk -05 00 00 america grenada america grenada -04 00 00 america guadeloupe america guadeloupe -04 00 00 america guatemala america guatemala -06 00 00 america guayaquil america guayaquil -05 00 00 america guyana america guyana -04 00 00 america halifax america halifax -04 00 00 america havana cuba -05 00 00 america hermosillo america hermosillo -07 00 00 america indiana indianapolis iet -05 00 00 america indiana knox america indiana knox -06 00 00 america indiana marengo america indiana marengo -05 00 00 america indiana petersburg america indiana petersburg -05 00 00 america indiana tell_city america indiana tell_city -06 00 00 america indiana vevay america indiana vevay -05 00 00 america indiana vincennes america indiana vincennes -05 00 00 america indiana winamac america indiana winamac -05 00 00 america indianapolis iet -05 00 00 america inuvik america inuvik -07 00 00 america iqaluit america iqaluit -05 00 00 america jamaica jamaica -05 00 00 america jujuy america jujuy -03 00 00 america juneau america juneau -09 00 00 america kentucky louisville america kentucky louisville -05 00 00 america kentucky monticello america kentucky monticello -05 00 00 america knox_in america knox_in -06 00 00 america kralendijk america kralendijk -04 00 00 america la_paz america la_paz -04 00 00 america lima america lima -05 00 00 america los_angeles pst -08 00 00 america louisville america louisville -05 00 00 america lower_princes america lower_princes -04 00 00 america maceio america maceio -03 00 00 america managua america managua -06 00 00 america manaus america manaus -04 00 00 america marigot america marigot -04 00 00 america martinique america martinique -04 00 00 america matamoros america matamoros -06 00 00 america mazatlan america mazatlan -07 00 00 america mendoza america mendoza -03 00 00 america menominee america menominee -06 00 00 america merida america merida -06 00 00 america metlakatla america metlakatla -09 00 00 america mexico_city america mexico_city -06 00 00 america miquelon america miquelon -03 00 00 america moncton america moncton -04 00 00 america monterrey america monterrey -06 00 00 america montevideo america montevideo -03 00 00 america montreal america montreal -05 00 00 america montserrat america montserrat -04 00 00 america nassau america nassau -05 00 00 america new_york america new_york -05 00 00 america nipigon america nipigon -05 00 00 america nome america nome -09 00 00 america noronha america noronha -02 00 00 america north_dakota beulah america north_dakota beulah -06 00 00 america north_dakota center america north_dakota center -06 00 00 america north_dakota new_salem america north_dakota new_salem -06 00 00 america ojinaga america ojinaga -07 00 00 america panama america panama -05 00 00 america pangnirtung america pangnirtung -05 00 00 america paramaribo america paramaribo -03 00 00 america phoenix pnt -07 00 00 america port-au-prince america port-au-prince -05 00 00 america port_of_spain america port_of_spain -04 00 00 america porto_acre america porto_acre -05 00 00 america porto_velho america porto_velho -04 00 00 america puerto_rico prt -04 00 00 america punta_arenas america punta_arenas -03 00 00 america rainy_river america rainy_river -06 00 00 america rankin_inlet america rankin_inlet -06 00 00 america recife america recife -03 00 00 america regina america regina -06 00 00 america resolute america resolute -06 00 00 america rio_branco america rio_branco -05 00 00 america rosario america rosario -03 00 00 america santa_isabel america santa_isabel -08 00 00 america santarem america santarem -03 00 00 america santiago america santiago -04 00 00 america santo_domingo america santo_domingo -04 00 00 america sao_paulo bet -03 00 00 america scoresbysund america scoresbysund -01 00 00 america shiprock navajo -07 00 00 america sitka america sitka -09 00 00 america st_barthelemy america st_barthelemy -04 00 00 america st_johns cnt -03 30 00 america st_kitts america st_kitts -04 00 00 america st_lucia america st_lucia -04 00 00 america st_thomas america st_thomas -04 00 00 america st_vincent america st_vincent -04 00 00 america swift_current america swift_current -06 00 00 america tegucigalpa america tegucigalpa -06 00 00 america thule america thule -04 00 00 america thunder_bay america thunder_bay -05 00 00 america tijuana america tijuana -08 00 00 america toronto america toronto -05 00 00 america tortola america tortola -04 00 00 america vancouver america vancouver -08 00 00 america virgin america virgin -04 00 00 america whitehorse america whitehorse -08 00 00 america winnipeg america winnipeg -06 00 00 america yakutat america yakutat -09 00 00 america yellowknife america yellowknife -07 00 00 antarctica casey antarctica casey 08 00 00 antarctica davis antarctica davis 07 00 00 antarctica dumontdurville antarctica dumontdurville 10 00 00 antarctica macquarie antarctica macquarie 11 00 00 antarctica mawson antarctica mawson 05 00 00 antarctica mcmurdo nst 12 00 00 antarctica palmer antarctica palmer -03 00 00 antarctica rothera antarctica rothera -03 00 00 antarctica south_pole nst 12 00 00 antarctica syowa antarctica syowa 03 00 00 antarctica troll antarctica troll 00 00 00 antarctica vostok antarctica vostok 06 00 00 arctic longyearbyen arctic longyearbyen 01 00 00 asia aden asia aden 03 00 00 asia almaty asia almaty 06 00 00 asia amman asia amman 02 00 00 asia anadyr asia anadyr 12 00 00 asia aqtau asia aqtau 05 00 00 asia aqtobe asia aqtobe 05 00 00 asia ashgabat asia ashgabat 05 00 00 asia ashkhabad asia ashkhabad 05 00 00 asia atyrau asia atyrau 05 00 00 asia baghdad asia baghdad 03 00 00 asia bahrain asia bahrain 03 00 00 asia baku asia baku 04 00 00 asia bangkok asia bangkok 07 00 00 asia barnaul asia barnaul 07 00 00 asia beirut asia beirut 02 00 00 asia bishkek asia bishkek 06 00 00 asia brunei asia brunei 08 00 00 asia calcutta ist 05 30 00 asia chita asia chita 09 00 00 asia choibalsan asia choibalsan 08 00 00 asia chongqing ctt 08 00 00 asia chungking ctt 08 00 00 asia colombo asia colombo 05 30 00 asia dacca bst 06 00 00 asia damascus asia damascus 02 00 00 asia dhaka bst 06 00 00 asia dili asia dili 09 00 00 asia dubai asia dubai 04 00 00 asia dushanbe asia dushanbe 05 00 00 asia famagusta asia famagusta 02 00 00 asia gaza asia gaza 02 00 00 asia harbin ctt 08 00 00 asia hebron asia hebron 02 00 00 asia ho_chi_minh vst 07 00 00 asia hong_kong hongkong 08 00 00 asia hovd asia hovd 07 00 00 asia irkutsk asia irkutsk 08 00 00 asia istanbul turkey 03 00 00 asia jakarta asia jakarta 07 00 00 asia jayapura asia jayapura 09 00 00 asia jerusalem israel 02 00 00 asia kabul asia kabul 04 30 00 asia kamchatka asia kamchatka 12 00 00 asia karachi plt 05 00 00 asia kashgar asia kashgar 06 00 00 asia kathmandu asia kathmandu 05 45 00 asia katmandu asia katmandu 05 45 00 asia khandyga asia khandyga 09 00 00 asia kolkata ist 05 30 00 asia krasnoyarsk asia krasnoyarsk 07 00 00 asia kuala_lumpur asia kuala_lumpur 08 00 00 asia kuching asia kuching 08 00 00 asia kuwait asia kuwait 03 00 00 asia macao asia macao 08 00 00 asia macau asia macau 08 00 00 asia magadan asia magadan 11 00 00 asia makassar asia makassar 08 00 00 asia manila asia manila 08 00 00 asia muscat asia muscat 04 00 00 asia nicosia asia nicosia 02 00 00 asia novokuznetsk asia novokuznetsk 07 00 00 asia novosibirsk asia novosibirsk 07 00 00 asia omsk asia omsk 06 00 00 asia oral asia oral 05 00 00 asia phnom_penh asia phnom_penh 07 00 00 asia pontianak asia pontianak 07 00 00 asia pyongyang asia pyongyang 09 00 00 asia qatar asia qatar 03 00 00 asia qostanay asia qostanay 06 00 00 asia qyzylorda asia qyzylorda 05 00 00 asia rangoon asia rangoon 06 30 00 asia riyadh asia riyadh 03 00 00 asia saigon vst 07 00 00 asia sakhalin asia sakhalin 11 00 00 asia samarkand asia samarkand 05 00 00 asia seoul rok 09 00 00 asia shanghai ctt 08 00 00 asia singapore singapore 08 00 00 asia srednekolymsk asia srednekolymsk 11 00 00 asia taipei roc 08 00 00 asia tashkent asia tashkent 05 00 00 asia tbilisi asia tbilisi 04 00 00 asia tehran iran 03 30 00 asia tel_aviv israel 02 00 00 asia thimbu asia thimbu 06 00 00 asia thimphu asia thimphu 06 00 00 asia tokyo jst 09 00 00 asia tomsk asia tomsk 07 00 00 asia ujung_pandang asia ujung_pandang 08 00 00 asia ulaanbaatar asia ulaanbaatar 08 00 00 asia ulan_bator asia ulan_bator 08 00 00 asia urumqi asia urumqi 06 00 00 asia ust-nera asia ust-nera 10 00 00 asia vientiane asia vientiane 07 00 00 asia vladivostok asia vladivostok 10 00 00 asia yakutsk asia yakutsk 09 00 00 asia yangon asia yangon 06 30 00 asia yekaterinburg asia yekaterinburg 05 00 00 asia yerevan net 04 00 00 atlantic azores atlantic azores -01 00 00 atlantic bermuda atlantic bermuda -04 00 00 atlantic canary atlantic canary 00 00 00 atlantic cape_verde atlantic cape_verde -01 00 00 atlantic faeroe atlantic faeroe 00 00 00 atlantic faroe atlantic faroe 00 00 00 atlantic jan_mayen atlantic jan_mayen 01 00 00 atlantic madeira atlantic madeira 00 00 00 atlantic reykjavik iceland 00 00 00 atlantic south_georgia atlantic south_georgia -02 00 00 atlantic st_helena atlantic st_helena 00 00 00 atlantic stanley atlantic stanley -03 00 00 australia act aet 10 00 00 australia adelaide australia adelaide 09 30 00 australia brisbane australia brisbane 10 00 00 australia broken_hill australia broken_hill 09 30 00 australia canberra aet 10 00 00 australia currie australia currie 10 00 00 australia darwin act 09 30 00 australia eucla australia eucla 08 45 00 australia hobart australia hobart 10 00 00 australia lhi australia lhi 10 30 00 australia lindeman australia lindeman 10 00 00 australia lord_howe australia lord_howe 10 30 00 australia melbourne australia melbourne 10 00 00 australia nsw aet 10 00 00 australia north act 09 30 00 australia perth australia perth 08 00 00 australia queensland australia queensland 10 00 00 australia south australia south 09 30 00 australia sydney aet 10 00 00 australia tasmania australia tasmania 10 00 00 australia victoria australia victoria 10 00 00 australia west australia west 08 00 00 australia yancowinna australia yancowinna 09 30 00 bet bet -03 00 00 bst bst 06 00 00 brazil acre brazil acre -05 00 00 brazil denoronha brazil denoronha -02 00 00 brazil east bet -03 00 00 brazil west brazil west -04 00 00 cat cat 02 00 00 cet cet 01 00 00 cnt cnt -03 30 00 cst cst -06 00 00 cst6cdt cst6cdt -06 00 00 ctt ctt 08 00 00 canada atlantic canada atlantic -04 00 00 canada central canada central -06 00 00 canada east-saskatchewan canada east-saskatchewan -06 00 00 canada eastern canada eastern -05 00 00 canada mountain canada mountain -07 00 00 canada newfoundland cnt -03 30 00 canada pacific canada pacific -08 00 00 canada saskatchewan canada saskatchewan -06 00 00 canada yukon canada yukon -08 00 00 chile continental chile continental -04 00 00 chile easterisland chile easterisland -06 00 00 cuba cuba -05 00 00 eat eat 03 00 00 ect ect 01 00 00 eet eet 02 00 00 est est -05 00 00 est5edt est5edt -05 00 00 egypt art 02 00 00 eire eire 00 00 00 etc gmt gmt 00 00 00 etc gmt 0 gmt 00 00 00 etc gmt 1 etc gmt 1 -01 00 00 etc gmt 10 etc gmt 10 -10 00 00 etc gmt 11 etc gmt 11 -11 00 00 etc gmt 12 etc gmt 12 -12 00 00 etc gmt 2 etc gmt 2 -02 00 00 etc gmt 3 etc gmt 3 -03 00 00 etc gmt 4 etc gmt 4 -04 00 00 etc gmt 5 etc gmt 5 -05 00 00 etc gmt 6 etc gmt 6 -06 00 00 etc gmt 7 etc gmt 7 -07 00 00 etc gmt 8 etc gmt 8 -08 00 00 etc gmt 9 etc gmt 9 -09 00 00 etc gmt-0 gmt 00 00 00 etc gmt-1 etc gmt-1 01 00 00 etc gmt-10 etc gmt-10 10 00 00 etc gmt-11 etc gmt-11 11 00 00 etc gmt-12 etc gmt-12 12 00 00 etc gmt-13 etc gmt-13 13 00 00 etc gmt-14 etc gmt-14 14 00 00 etc gmt-2 etc gmt-2 02 00 00 etc gmt-3 etc gmt-3 03 00 00 etc gmt-4 etc gmt-4 04 00 00 etc gmt-5 etc gmt-5 05 00 00 etc gmt-6 etc gmt-6 06 00 00 etc gmt-7 etc gmt-7 07 00 00 etc gmt-8 etc gmt-8 08 00 00 etc gmt-9 etc gmt-9 09 00 00 etc gmt0 gmt 00 00 00 etc greenwich gmt 00 00 00 etc uct uct 00 00 00 etc utc uct 00 00 00 etc universal uct 00 00 00 etc zulu uct 00 00 00 europe amsterdam europe amsterdam 01 00 00 europe andorra europe andorra 01 00 00 europe astrakhan europe astrakhan 04 00 00 europe athens europe athens 02 00 00 europe belfast gb 00 00 00 europe belgrade europe belgrade 01 00 00 europe berlin europe berlin 01 00 00 europe bratislava europe bratislava 01 00 00 europe brussels europe brussels 01 00 00 europe bucharest europe bucharest 02 00 00 europe budapest europe budapest 01 00 00 europe busingen europe busingen 01 00 00 europe chisinau europe chisinau 02 00 00 europe copenhagen europe copenhagen 01 00 00 europe dublin eire 00 00 00 europe gibraltar europe gibraltar 01 00 00 europe guernsey gb 00 00 00 europe helsinki europe helsinki 02 00 00 europe isle_of_man gb 00 00 00 europe istanbul turkey 03 00 00 europe jersey gb 00 00 00 europe kaliningrad europe kaliningrad 02 00 00 europe kiev europe kiev 02 00 00 europe kirov europe kirov 03 00 00 europe lisbon portugal 00 00 00 europe ljubljana europe ljubljana 01 00 00 europe london gb 00 00 00 europe luxembourg europe luxembourg 01 00 00 europe madrid europe madrid 01 00 00 europe malta europe malta 01 00 00 europe mariehamn europe mariehamn 02 00 00 europe minsk europe minsk 03 00 00 europe monaco europe monaco 01 00 00 europe moscow w-su 03 00 00 europe nicosia europe nicosia 02 00 00 europe oslo europe oslo 01 00 00 europe paris ect 01 00 00 europe podgorica europe podgorica 01 00 00 europe prague europe prague 01 00 00 europe riga europe riga 02 00 00 europe rome europe rome 01 00 00 europe samara europe samara 04 00 00 europe san_marino europe san_marino 01 00 00 europe sarajevo europe sarajevo 01 00 00 europe saratov europe saratov 04 00 00 europe simferopol europe simferopol 03 00 00 europe skopje europe skopje 01 00 00 europe sofia europe sofia 02 00 00 europe stockholm europe stockholm 01 00 00 europe tallinn europe tallinn 02 00 00 europe tirane europe tirane 01 00 00 europe tiraspol europe tiraspol 02 00 00 europe ulyanovsk europe ulyanovsk 04 00 00 europe uzhgorod europe uzhgorod 02 00 00 europe vaduz europe vaduz 01 00 00 europe vatican europe vatican 01 00 00 europe vienna europe vienna 01 00 00 europe vilnius europe vilnius 02 00 00 europe volgograd europe volgograd 04 00 00 europe warsaw poland 01 00 00 europe zagreb europe zagreb 01 00 00 europe zaporozhye europe zaporozhye 02 00 00 europe zurich europe zurich 01 00 00 factory factory 00 00 00 gb gb 00 00 00 gb-eire gb 00 00 00 gmt gmt 00 00 00 gmt 0 gmt 00 00 00 gmt-0 gmt 00 00 00 gmt0 gmt 00 00 00 greenwich gmt 00 00 00 hst hst -10 00 00 hongkong hongkong 08 00 00 iet iet -05 00 00 ist ist 05 30 00 iceland iceland 00 00 00 indian antananarivo eat 03 00 00 indian chagos indian chagos 06 00 00 indian christmas indian christmas 07 00 00 indian cocos indian cocos 06 30 00 indian comoro eat 03 00 00 indian kerguelen indian kerguelen 05 00 00 indian mahe indian mahe 04 00 00 indian maldives indian maldives 05 00 00 indian mauritius indian mauritius 04 00 00 indian mayotte eat 03 00 00 indian reunion indian reunion 04 00 00 iran iran 03 30 00 israel israel 02 00 00 jst jst 09 00 00 jamaica jamaica -05 00 00 japan jst 09 00 00 kwajalein kwajalein 12 00 00 libya libya 02 00 00 met met 01 00 00 mit mit 13 00 00 mst mst -07 00 00 mst7mdt mst7mdt -07 00 00 mexico bajanorte mexico bajanorte -08 00 00 mexico bajasur mexico bajasur -07 00 00 mexico general mexico general -06 00 00 net net 04 00 00 nst nst 12 00 00 nz nst 12 00 00 nz-chat nz-chat 12 45 00 navajo navajo -07 00 00 plt plt 05 00 00 pnt pnt -07 00 00 prc ctt 08 00 00 prt prt -04 00 00 pst pst -08 00 00 pst8pdt pst8pdt -08 00 00 pacific apia mit 13 00 00 pacific auckland nst 12 00 00 pacific bougainville pacific bougainville 11 00 00 pacific chatham nz-chat 12 45 00 pacific chuuk pacific chuuk 10 00 00 pacific easter pacific easter -06 00 00 pacific efate pacific efate 11 00 00 pacific enderbury pacific enderbury 13 00 00 pacific fakaofo pacific fakaofo 13 00 00 pacific fiji pacific fiji 12 00 00 pacific funafuti pacific funafuti 12 00 00 pacific galapagos pacific galapagos -06 00 00 pacific gambier pacific gambier -09 00 00 pacific guadalcanal sst 11 00 00 pacific guam pacific guam 10 00 00 pacific honolulu pacific honolulu -10 00 00 pacific johnston pacific johnston -10 00 00 pacific kiritimati pacific kiritimati 14 00 00 pacific kosrae pacific kosrae 11 00 00 pacific kwajalein kwajalein 12 00 00 pacific majuro pacific majuro 12 00 00 pacific marquesas pacific marquesas -09 30 00 pacific midway pacific midway -11 00 00 pacific nauru pacific nauru 12 00 00 pacific niue pacific niue -11 00 00 pacific norfolk pacific norfolk 11 00 00 pacific noumea pacific noumea 11 00 00 pacific pago_pago pacific pago_pago -11 00 00 pacific palau pacific palau 09 00 00 pacific pitcairn pacific pitcairn -08 00 00 pacific pohnpei pacific pohnpei 11 00 00 pacific ponape pacific ponape 11 00 00 pacific port_moresby pacific port_moresby 10 00 00 pacific rarotonga pacific rarotonga -10 00 00 pacific saipan pacific saipan 10 00 00 pacific samoa pacific samoa -11 00 00 pacific tahiti pacific tahiti -10 00 00 pacific tarawa pacific tarawa 12 00 00 pacific tongatapu pacific tongatapu 13 00 00 pacific truk pacific truk 10 00 00 pacific wake pacific wake 12 00 00 pacific wallis pacific wallis 12 00 00 pacific yap pacific yap 10 00 00 poland poland 01 00 00 portugal portugal 00 00 00 roc roc 08 00 00 rok rok 09 00 00 sst sst 11 00 00 singapore singapore 08 00 00 systemv ast4 systemv ast4 -04 00 00 systemv ast4adt systemv ast4adt -04 00 00 systemv cst6 systemv cst6 -06 00 00 systemv cst6cdt systemv cst6cdt -06 00 00 systemv est5 systemv est5 -05 00 00 systemv est5edt systemv est5edt -05 00 00 systemv hst10 systemv hst10 -10 00 00 systemv mst7 systemv mst7 -07 00 00 systemv mst7mdt systemv mst7mdt -07 00 00 systemv pst8 systemv pst8 -08 00 00 systemv pst8pdt systemv pst8pdt -08 00 00 systemv yst9 systemv yst9 -09 00 00 systemv yst9ydt systemv yst9ydt -09 00 00 turkey turkey 03 00 00 uct uct 00 00 00 us alaska ast -09 00 00 us aleutian us aleutian -10 00 00 us arizona pnt -07 00 00 us central cst -06 00 00 us east-indiana iet -05 00 00 us eastern us eastern -05 00 00 us hawaii us hawaii -10 00 00 us indiana-starke us indiana-starke -06 00 00 us michigan us michigan -05 00 00 us mountain navajo -07 00 00 us pacific pst -08 00 00 us pacific-new pst -08 00 00 us samoa us samoa -11 00 00 utc uct 00 00 00 universal uct 00 00 00 vst vst 07 00 00 w-su w-su 03 00 00 wet wet 00 00 00 zulu uct 00 00 00",
+ "category": "Data Types",
+ "url": "../docs/sql/data_types/timezones",
+ "blurb": "A reference list for Time Zones."
},
{
- "title": "C API - Table Functions",
- "text": "the table function api can be used to define a table function that can then be called from within duckdb in the from clause of a query api reference the return value should be destroyed with duckdb_destroy_table_function syntax the table function object duckdb_destroy_table_function destroys the given table function object syntax the table function to destroy duckdb_table_function_set_name sets the name of the given table function syntax the table function name the name of the table function duckdb_table_function_add_parameter adds a parameter to the table function syntax the table function type the type of the parameter to add duckdb_table_function_set_extra_info assigns extra information to the table function that can be fetched during binding etc syntax the table function extra_info the extra information destroy the callback that will be called to destroy the bind data if any duckdb_table_function_set_bind sets the bind function of the table function syntax the table function bind the bind function duckdb_table_function_set_init sets the init function of the table function syntax the table function init the init function duckdb_table_function_set_local_init sets the thread-local init function of the table function syntax the table function init the init function duckdb_table_function_set_function sets the main function of the table function syntax the table function function the function duckdb_table_function_supports_projection_pushdown sets whether or not the given table function supports projection pushdown if this is set to true the system will provide a list of all required columns in the init stage through the duckdb_init_get_column_count and duckdb_init_get_column_index functions if this is set to false the default the system will expect all columns to be projected syntax the table function pushdown true if the table function supports projection pushdown false otherwise duckdb_register_table_function register the table function object within the given connection the function requires at least a name a bind function an init function and a main function if the function is incomplete or a function with this name already exists duckdberror is returned syntax the connection to register it in function the function pointer returns whether or not the registration was successful duckdb_bind_get_extra_info retrieves the extra info of the function as set in duckdb_table_function_set_extra_info syntax the info object returns the extra info duckdb_bind_add_result_column adds a result column to the output of the table function syntax the info object name the name of the column type the logical type of the column duckdb_bind_get_parameter_count retrieves the number of regular non-named parameters to the function syntax the info object returns the number of parameters duckdb_bind_get_parameter retrieves the parameter at the given index the result must be destroyed with duckdb_destroy_value syntax the info object index the index of the parameter to get returns the value of the parameter must be destroyed with duckdb_destroy_value duckdb_bind_set_bind_data sets the user-provided bind data in the bind object this object can be retrieved again during execution syntax the info object extra_data the bind data object destroy the callback that will be called to destroy the bind data if any duckdb_bind_set_cardinality sets the cardinality estimate for the table function used for optimization syntax the bind data object is_exact whether or not the cardinality estimate is exact or an approximation duckdb_bind_set_error report that an error has occurred while calling bind syntax the info object error the error message duckdb_init_get_extra_info retrieves the extra info of the function as set in duckdb_table_function_set_extra_info syntax the info object returns the extra info duckdb_init_get_bind_data gets the bind data set by duckdb_bind_set_bind_data during the bind note that the bind data should be considered as read-only for tracking state use the init data instead syntax the info object returns the bind data object duckdb_init_set_init_data sets the user-provided init data in the init object this object can be retrieved again during execution syntax the info object extra_data the init data object destroy the callback that will be called to destroy the init data if any duckdb_init_get_column_count returns the number of projected columns this function must be used if projection pushdown is enabled to figure out which columns to emit syntax the info object returns the number of projected columns duckdb_init_get_column_index returns the column index of the projected column at the specified position this function must be used if projection pushdown is enabled to figure out which columns to emit syntax the info object column_index the index at which to get the projected column index from 0 duckdb_init_get_column_count info returns the column index of the projected column duckdb_init_set_max_threads sets how many threads can process this table function in parallel default 1 syntax the info object max_threads the maximum amount of threads that can process this table function duckdb_init_set_error report that an error has occurred while calling init syntax the info object error the error message duckdb_function_get_extra_info retrieves the extra info of the function as set in duckdb_table_function_set_extra_info syntax the info object returns the extra info duckdb_function_get_bind_data gets the bind data set by duckdb_bind_set_bind_data during the bind note that the bind data should be considered as read-only for tracking state use the init data instead syntax the info object returns the bind data object duckdb_function_get_init_data gets the init data set by duckdb_init_set_init_data during the init syntax the info object returns the init data object duckdb_function_get_local_init_data gets the thread-local init data set by duckdb_init_set_init_data during the local_init syntax the info object returns the init data object duckdb_function_set_error report that an error has occurred while executing the function syntax the info object error the error message",
- "category": "C",
- "url": "../docs/api/c/table_functions",
- "blurb": "The table function API can be used to define a table function that can then be called from within DuckDB in the FROM ..."
+ "title": "Timestamp Type",
+ "text": "timestamps represent points in absolute time usually called instants duckdb represents instants as the number of microseconds µs since 1970-01-01 00 00 00 00 name aliases description --- --- --- timestamp datetime time of day ignores time zone timestamp with time zone timestamptz time of day uses time zone a timestamp specifies a combination of date year month day and a time hour minute second millisecond timestamps can be created using the timestamp keyword where the data must be formatted according to the iso 8601 format yyyy-mm-dd hh mm ss zzzzzz -tt tt -- 11 30 am at 20 september 1992 gmt select timestamp 1992-09-20 11 30 00 -- 2 30 pm at 20 september 1992 gmt select timestamp 1992-09-20 14 30 00 special values there are also three special date values that can be used on input input string valid types description ------------- ---------------------------------- ----------------------------------------------- epoch timestamp timestamptz 1970-01-01 00 00 00 00 unix system time zero infinity timestamp timestamptz later than all other time stamps -infinity timestamp timestamptz earlier than all other time stamps the values infinity and -infinity are specially represented inside the system and will be displayed unchanged but epoch is simply a notational shorthand that will be converted to the time stamp value when read select -infinity timestamp epoch timestamp infinity timestamp negative epoch positive ---------- ------------------- --------- -infinity 1970-01-01 00 00 00 infinity functions see timestamp functions time zones the timestamptz type can be binned into calendar and clock bins using a suitable extension the built in icu extension implements all the binning and arithmetic functions using the international components for unicode time zone and calendar functions to set the time zone to use first load the icu extension the icu extension comes pre-bundled with several duckdb clients including python r jdbc and odbc so this step can be skipped in those cases require icu next use the set timezone command set timezone america los_angeles time binning operations for timestamptz will then be implemented using the given time zone a list of available time zones can be pulled from the pg_timezone_names table function select name abbrev utc_offset from pg_timezone_names order by name you can also find a reference table of available time zones here calendars the icu extension also supports non-gregorian calendars using the set calendar command note that the require icu step is only required if the duckdb client does not bundle the icu extension require icu set calendar japanese time binning operations for timestamptz will then be implemented using the given calendar in this example the era part will now report the japanese imperial era number a list of available calendars can be pulled from the icu_calendar_names table function select name from icu_calendar_names order by 1 settings the current value of the timezone and calendar settings are determined by icu when it starts up they can be looked from in the duckdb_settings table function select from duckdb_settings where name timezone -- america los_angeles select from duckdb_settings where name calendar -- gregorian",
+ "category": "Data Types",
+ "url": "../docs/sql/data_types/timestamp",
+ "blurb": "A timestamp specifies a combination of a date (year, month, day) and a time (hour, minute, second, millisecond)."
},
{
- "title": "C API - Replacement Scans",
- "text": "the replacement scan api can be used to register a callback that is called when a table is read that does not exist in the catalog for example when a query such as select from my_table is executed and my_table does not exist the replacement scan callback will be called with my_table as parameter the replacement scan can then insert a table function with a specific parameter to replace the read of the table api reference syntax the database object to add the replacement scan to replacement the replacement scan callback extra_data extra data that is passed back into the specified callback delete_callback the delete callback to call on the extra data if any duckdb_replacement_scan_set_function_name sets the replacement function name to use if this function is called in the replacement callback the replacement scan is performed if it is not called the replacement callback is not performed syntax the info object function_name the function name to substitute duckdb_replacement_scan_add_parameter adds a parameter to the replacement scan function syntax the info object parameter the parameter to add",
- "category": "C",
- "url": "../docs/api/c/replacement_scans",
- "blurb": "The replacement scan API can be used to register a callback that is called when a table is read that does not exist in..."
+ "title": "Struct",
+ "text": "struct data type conceptually a struct column contains an ordered list of other columns called entries the entries are referenced by name using strings this document refers to those entry names as keys each row in the struct column must have the same keys each key must have the same type of value for each row struct s are typically used to nest multiple columns into a single column and the nested column can be of any type including other struct s and list s struct s are similar to postgres s row type the key difference is that duckdb struct s require the same keys in each row of a struct column this allows duckdb to provide significantly improved performance by fully utilizing its vectorized execution engine and also enforces type consistency for improved correctness duckdb includes a row function as a special way to produce a struct but does not have a row data type see an example below and the nested functions docs for details see the data types overview for a comparison between nested data types structs can be created using the struct_pack name expr function or the equivalent array notation name expr notation the expressions can be constants or arbitrary expressions creating structs -- struct of integers select x 1 y 2 z 3 -- struct of strings with a null value select yes duck maybe goose huh null no heron -- struct with a different type for each key select key1 string key2 1 key3 12 345 -- struct using the struct_pack function -- note the lack of single quotes around the keys and the use of the operator select struct_pack key1 value1 key2 42 -- struct of structs with null values select birds yes duck maybe goose huh null no heron aliens null amphibians yes frog maybe salamander huh dragon no toad -- create a struct from columns and or expressions using the row function -- this returns x 1 v2 2 y a select row x x 1 y from select 1 as x a as y -- if using multiple expressions when creating a struct the row function is optional -- this also returns x 1 v2 2 y a select x x 1 y from select 1 as x a as y adding field s value s to structs -- add to a struct of integers select struct_insert a 1 b 2 c 3 d 4 retrieving from structs retrieving a value from a struct can be accomplished using dot notation bracket notation or through struct functions like struct_extract -- use dot notation to retrieve the value at a key s location this returns 1 -- the subquery generates a struct column a which we then query with a x select a x from select x 1 y 2 z 3 as a -- if key contains a space simply wrap it in double quotes this returns 1 -- note use double quotes not single quotes -- this is because this action is most similar to selecting a column from within the struct select a x space from select x space 1 y 2 z 3 as a -- bracket notation may also be used this returns 1 -- note use single quotes since the goal is to specify a certain string key -- only constant expressions may be used inside the brackets no columns select a x space from select x space 1 y 2 z 3 as a -- the struct_extract function is also equivalent this returns 1 select struct_extract x space 1 y 2 z 3 x space referring to structs with dot notation can be ambiguous with referring to schemas and tables in general duckdb looks for columns first then for struct keys within columns duckdb resolves references in these orders using the first match to occur no dots select part1 from tbl part1 is a column one dot select part1 part2 from tbl part1 is a table part2 is a column part1 is a column part2 is a property of that column two or more dots select part1 part2 part3 from tbl part1 is a schema part2 is a table part3 is a column part1 is a table part2 is a column part3 is a property of that column part1 is a column part2 is a property of that column part3 is a property of that column any extra parts e g part4 part5 etc are always treated as properties creating structs with the row function the row function can be used to automatically convert multiple columns to a single struct column the name of each input column is used as a key and the value of each column becomes the struct s value at that key when converting multiple expressions into a struct the row function name is optional - a set of parenthesis is all that is needed example data table named t1 my_column another_column --- --- 1 a 2 b row function example select row my_column another_column as my_struct_column my_column another_column as identical_struct_column from t1 example output my_struct_column identical_struct_column --- --- my_column 1 another_column a my_column 1 another_column a my_column 2 another_column b my_column 2 another_column b the row function or simplified parenthesis syntax may also be used with arbitrary expressions as input rather than column names in the case of an expression a key will be automatically generated in the format of vn where n is a number that refers to its parameter location in the row function ex v1 v2 etc this can be combined with column names as an input in the same call to the row function this example uses the same input table as above row function example with a column name a constant and an expression as input select row my_column 42 my_column 1 as my_struct_column my_column 42 my_column 1 as identical_struct_column from t1 example output my_struct_column identical_struct_column --- --- my_column 1 v2 42 v3 2 my_column 1 v2 42 v3 2 my_column 2 v2 42 v3 3 my_column 2 v2 42 v3 3 comparison operators nested types can be compared using all the comparison operators these comparisons can be used in logical expressions for both where and having clauses as well as for creating boolean values the ordering is defined positionally in the same way that words can be ordered in a dictionary null values compare greater than all other values and are considered equal to each other at the top level null nested values obey standard sql null comparison rules comparing a null nested value to a non- null nested value produces a null result comparing nested value members however uses the internal nested value rules for null s and a null nested value member will compare above a non- null nested value member functions see nested functions",
+ "category": "Data Types",
+ "url": "../docs/sql/data_types/struct",
+ "blurb": "Struct Data Type Conceptually, a STRUCT column contains an ordered list of other columns called entries. The entries..."
},
{
- "title": "C API - Overview",
- "text": "duckdb implements a custom c api modelled somewhat following the sqlite c api the api is contained in the duckdb h header continue to startup shutdown to get started or check out the full api overview we also provide a sqlite api wrapper which means that if your applications is programmed against the sqlite c api you can re-link to duckdb and it should continue working see the sqlite_api_wrapper folder in our source repository for more information installation the duckdb c api can be installed as part of the libduckdb packages please see the installation page for details pages in this section",
- "category": "C",
- "url": "../docs/api/c/overview",
- "blurb": "DuckDB implements a custom C API modelled somewhat following the SQLite C API. The API is contained in the duckdb.h ..."
+ "title": "Blob Type",
+ "text": "name aliases description --- --- --- blob bytea variable-length binary data the blob b inary l arge ob ject type represents an arbitrary binary object stored in the database system the blob type can contain any type of binary data with no restrictions what the actual bytes represent is opaque to the database system -- create a blob value with a single byte 170 select xaa blob -- create a blob value with three bytes 170 171 172 select xaa xab xac blob -- create a blob value with two bytes 65 66 select ab blob blobs are typically used to store non-textual objects that the database does not provide explicit support for such as images while blobs can hold objects up to 4gb in size typically it is not recommended to store very large objects within the database system in many situations it is better to store the large file on the file system and store the path to the file in the database system in a varchar field functions see blob functions",
+ "category": "Data Types",
+ "url": "../docs/sql/data_types/blob",
+ "blurb": "The blob (Binary Large OBject) type represents an arbitrary binary object stored in the database system."
},
{
- "title": "C API - Query",
- "text": "the duckdb_query method allows sql queries to be run in duckdb from c this method takes two parameters a null-terminated sql query string and a duckdb_result result pointer the result pointer may be null if the application is not interested in the result set or if the query produces no result after the result is consumed the duckdb_destroy_result method should be used to clean up the result elements can be extracted from the duckdb_result object using a variety of methods the duckdb_column_count and duckdb_row_count methods can be used to extract the number of columns and the number of rows respectively duckdb_column_name and duckdb_column_type can be used to extract the names and types of individual columns example duckdb_state state duckdb_result result create a table state duckdb_query con create table integers i integer j integer null if state duckdberror handle error insert three rows into the table state duckdb_query con insert into integers values 3 4 5 6 7 null null if state duckdberror handle error query rows again state duckdb_query con select from integers result if state duckdberror handle error handle the result destroy the result after we are done with it duckdb_destroy_result result value extraction values can be extracted using either the duckdb_column_data duckdb_nullmask_data functions or using the duckdb_value convenience functions the duckdb_column_data duckdb_nullmask_data functions directly hand you a pointer to the result arrays in columnar format and can therefore be very fast the duckdb_value functions perform bounds- and type-checking and will automatically cast values to the desired type this makes them more convenient and easier to use at the expense of being slower see the types page for more information for optimal performance use duckdb_column_data and duckdb_nullmask_data to extract data from the query result the duckdb_value functions perform internal type-checking bounds-checking and casting which makes them slower duckdb_value below is an example that prints the above result to csv format using the duckdb_value_varchar function note that the function is generic we do not need to know about the types of the individual result columns print the above result to csv format using duckdb_value_varchar idx_t row_count duckdb_row_count result idx_t column_count duckdb_column_count result for idx_t row 0 row row_count row for idx_t col 0 col column_count col if col 0 printf auto str_val duckdb_value_varchar result col row printf s str_val duckdb_free str_val printf n duckdb_column_data below is an example that prints the above result to csv format using the duckdb_column_data function note that the function is not generic we do need to know exactly what the types of the result columns are int32_t i_data int32_t duckdb_column_data result 0 int32_t j_data int32_t duckdb_column_data result 1 bool i_mask duckdb_nullmask_data result 0 bool j_mask duckdb_nullmask_data result 1 idx_t row_count duckdb_row_count result for idx_t row 0 row row_count row if i_mask row printf null else printf d i_data row printf if j_mask row printf null else printf d j_data row printf n when using duckdb_column_data be careful that the type matches exactly what you expect it to be as the code directly accesses an internal array there is no type-checking accessing a duckdb_type_integer column as if it was a duckdb_type_bigint column will provide unpredictable results api reference note that after running duckdb_query duckdb_destroy_result must be called on the result object even if the query fails otherwise the error stored within the result will not be freed correctly syntax the connection to perform the query in query the sql query to run out_result the query result returns duckdbsuccess on success or duckdberror on failure duckdb_destroy_result closes the result and de-allocates all memory allocated for that connection syntax the result to destroy duckdb_column_name returns the column name of the specified column the result should not need be freed the column names will automatically be destroyed when the result is destroyed returns null if the column is out of range syntax the result object to fetch the column name from col the column index returns the column name of the specified column duckdb_column_type returns the column type of the specified column returns duckdb_type_invalid if the column is out of range syntax the result object to fetch the column type from col the column index returns the column type of the specified column duckdb_column_logical_type returns the logical column type of the specified column the return type of this call should be destroyed with duckdb_destroy_logical_type returns null if the column is out of range syntax the result object to fetch the column type from col the column index returns the logical column type of the specified column duckdb_column_count returns the number of columns present in a the result object syntax the result object returns the number of columns present in the result object duckdb_row_count returns the number of rows present in a the result object syntax the result object returns the number of rows present in the result object duckdb_rows_changed returns the number of rows changed by the query stored in the result this is relevant only for insert update delete queries for other queries the rows_changed will be 0 syntax the result object returns the number of rows changed duckdb_column_data deprecated prefer using duckdb_result_get_chunk instead returns the data of a specific column of a result in columnar format the function returns a dense array which contains the result data the exact type stored in the array depends on the corresponding duckdb_type as provided by duckdb_column_type for the exact type by which the data should be accessed see the comments in the types section or the duckdb_type enum for example for a column of type duckdb_type_integer rows can be accessed in the following manner int32_t data int32_t duckdb_column_data result 0 printf data for row d d n row data row syntax the result object to fetch the column data from col the column index returns the column data of the specified column duckdb_nullmask_data deprecated prefer using duckdb_result_get_chunk instead returns the nullmask of a specific column of a result in columnar format the nullmask indicates for every row whether or not the corresponding row is null if a row is null the values present in the array provided by duckdb_column_data are undefined int32_t data int32_t duckdb_column_data result 0 bool nullmask duckdb_nullmask_data result 0 if nullmask row printf data for row d null n row else printf data for row d d n row data row syntax the result object to fetch the nullmask from col the column index returns the nullmask of the specified column duckdb_result_error returns the error message contained within the result the error is only set if duckdb_query returns duckdberror the result of this function must not be freed it will be cleaned up when duckdb_destroy_result is called syntax the result object to fetch the error from returns the error of the result",
- "category": "C",
- "url": "../docs/api/c/query",
- "blurb": "The duckdb_query method allows SQL queries to be run in DuckDB from C. This method takes two parameters, a..."
+ "title": "Enum Types",
+ "text": "name description --- --- enum dictionary encoding representing all possible string values of a column enums the enum type represents a dictionary data structure with all possible unique values of a column for example a column storing the days of the week can be an enum holding all possible days enums are particularly interesting for string columns with high cardinality this is because the column only stores a numerical reference to the string in the enum dictionary resulting in immense savings in disk storage and faster query performance enum definition enum types are created using the command create type enum_name as enum value_1 value_2 for example -- creates new user defined type mood as an enum create type mood as enum sad ok happy -- this will fail since the mood type already exists create type mood as enum sad ok happy anxious -- this will fail since enums cannot hold null values create type breed as enum maltese null -- this will fail since enum values must be unique create type breed as enum maltese maltese enum usage after an enum has been created it can be used anywhere a standard built-in type is used for example we can create a table with a column that references the enum -- creates a table person with attributes name string type and current_mood mood type create table person name text current_mood mood -- inserts tuples in the person table insert into person values pedro happy mark null pagliacci sad mr mackey ok -- this will fail since the mood type does not have a quackity-quack value insert into person values hannes quackity-quack -- the string sad is cast to the type mood returning a numerical reference value -- this makes the comparison a numerical comparison instead of a string comparison select from person where current_mood sad ---- pagliacci enum vs strings duckdb enums are automatically cast to varchar types whenever necessary this characteristic allows for enum columns to be used in any varchar function in addition it also allows for comparisons between different enum columns or an enum and a varchar column for example -- regexp_matches is a function that takes a varchar hence current_mood is cast to varchar select regexp_matches current_mood a from person ---- true false true false create type new_mood as enum happy anxious create table person_2 name text current_mood mood future_mood new_mood past_mood varchar -- since current_mood and future_mood are constructed on different enums -- duckdb will cast both enums to strings and perform a string comparison select from person_2 where current_mood future_mood -- since current_mood is an enum -- duckdb will cast the current_mood enum to varchar and perform a string comparison select from person_2 where current_mood past_mood enum removal enum types are stored in the catalog and a catalog dependency is added to each table that uses them it is possible to drop an enum from the catalog using the following command drop type enum_name note that any dependent must be removed before dropping the enum or the enum must be dropped with the additional cascade parameter for example -- this will fail since person has a catalog dependency to the mood type drop type mood drop table person drop table person_2 -- this successfully removes the mood type -- another option would be to drop type mood cascade drops the type and its dependents drop type mood",
+ "category": "Data Types",
+ "url": "../docs/sql/data_types/enum",
+ "blurb": "The ENUM type represents a dictionary data structure with all possible unique values of a column."
},
{
- "title": "C API - Types",
- "text": "duckdb is a strongly typed database system as such every column has a single type specified this type is constant over the entire column that is to say a column that is labeled as an integer column will only contain integer values duckdb also supports columns of composite types for example it is possible to define an array of integers int it is also possible to define types as arbitrary structs row i integer j varchar for that reason native duckdb type objects are not mere enums but a class that can potentially be nested types in the c api are modeled using an enum duckdb_type and a complex class duckdb_logical_type for most primitive types e g integers or varchars the enum is sufficient for more complex types such as lists structs or decimals the logical type must be used typedef enum duckdb_type duckdb_type_invalid duckdb_type_boolean duckdb_type_tinyint duckdb_type_smallint duckdb_type_integer duckdb_type_bigint duckdb_type_utinyint duckdb_type_usmallint duckdb_type_uinteger duckdb_type_ubigint duckdb_type_float duckdb_type_double duckdb_type_timestamp duckdb_type_date duckdb_type_time duckdb_type_interval duckdb_type_hugeint duckdb_type_varchar duckdb_type_blob duckdb_type_decimal duckdb_type_timestamp_s duckdb_type_timestamp_ms duckdb_type_timestamp_ns duckdb_type_enum duckdb_type_list duckdb_type_struct duckdb_type_map duckdb_type_uuid duckdb_type_json duckdb_type the enum type of a column in the result can be obtained using the duckdb_column_type function the logical type of a column can be obtained using the duckdb_column_logical_type function duckdb_value the duckdb_value functions will auto-cast values as required for example it is no problem to use duckdb_value_double on a column of type duckdb_value_int32 the value will be auto-cast and returned as a double note that in certain cases the cast may fail for example this can happen if we request a duckdb_value_int8 and the value does not fit within an int8 value in this case a default value will be returned usually 0 or nullptr the same default value will also be returned if the corresponding value is null the duckdb_value_is_null function can be used to check if a specific value is null or not the exception to the auto-cast rule is the duckdb_value_varchar_internal function this function does not auto-cast and only works for varchar columns the reason this function exists is that the result does not need to be freed note that duckdb_value_varchar and duckdb_value_blob require the result to be de-allocated using duckdb_free duckdb_result_get_chunk the duckdb_result_get_chunk function can be used to read data chunks from a duckdb result set and is the most efficient way of reading data from a duckdb result using the c api it is also the only way of reading data of certain types from a duckdb result for example the duckdb_value functions do not support structural reading of composite types lists or structs or more complex types like enums and decimals for more information about data chunks see the documentation on data chunks api reference this function supersedes all duckdb_value functions as well as the duckdb_column_data and duckdb_nullmask_data functions it results in significantly better performance and should be preferred in newer code-bases if this function is used none of the other result functions can be used and vice versa i e this function cannot be mixed with the legacy result functions use duckdb_result_chunk_count to figure out how many chunks there are in the result syntax the result object to fetch the data chunk from chunk_index the chunk index to fetch from returns the resulting data chunk returns null if the chunk index is out of bounds duckdb_result_chunk_count returns the number of data chunks present in the result syntax the result object returns the resulting data chunk returns null if the chunk index is out of bounds duckdb_value_boolean syntax the boolean value at the specified location or false if the value cannot be converted duckdb_value_int8 syntax the int8_t value at the specified location or 0 if the value cannot be converted duckdb_value_int16 syntax the int16_t value at the specified location or 0 if the value cannot be converted duckdb_value_int32 syntax the int32_t value at the specified location or 0 if the value cannot be converted duckdb_value_int64 syntax the int64_t value at the specified location or 0 if the value cannot be converted duckdb_value_hugeint syntax the duckdb_hugeint value at the specified location or 0 if the value cannot be converted duckdb_value_decimal syntax the duckdb_decimal value at the specified location or 0 if the value cannot be converted duckdb_value_uint8 syntax the uint8_t value at the specified location or 0 if the value cannot be converted duckdb_value_uint16 syntax the uint16_t value at the specified location or 0 if the value cannot be converted duckdb_value_uint32 syntax the uint32_t value at the specified location or 0 if the value cannot be converted duckdb_value_uint64 syntax the uint64_t value at the specified location or 0 if the value cannot be converted duckdb_value_float syntax the float value at the specified location or 0 if the value cannot be converted duckdb_value_double syntax the double value at the specified location or 0 if the value cannot be converted duckdb_value_date syntax the duckdb_date value at the specified location or 0 if the value cannot be converted duckdb_value_time syntax the duckdb_time value at the specified location or 0 if the value cannot be converted duckdb_value_timestamp syntax the duckdb_timestamp value at the specified location or 0 if the value cannot be converted duckdb_value_interval syntax the duckdb_interval value at the specified location or 0 if the value cannot be converted duckdb_value_varchar syntax the char value at the specified location or nullptr if the value cannot be converted the result must be freed with duckdb_free duckdb_value_varchar_internal syntax the char value at the specified location only works on varchar columns and does not auto-cast if the column is not a varchar column this function will return null the result must not be freed duckdb_value_blob syntax the duckdb_blob value at the specified location returns a blob with blob data set to nullptr if the value cannot be converted the resulting blob data must be freed with duckdb_free duckdb_value_is_null syntax returns true if the value at the specified index is null and false otherwise duckdb_from_date decompose a duckdb_date object into year month and date stored as duckdb_date_struct syntax the date object as obtained from a duckdb_type_date column returns the duckdb_date_struct with the decomposed elements duckdb_to_date re-compose a duckdb_date from year month and date duckdb_date_struct syntax the year month and date stored in a duckdb_date_struct returns the duckdb_date element duckdb_from_time decompose a duckdb_time object into hour minute second and microsecond stored as duckdb_time_struct syntax the time object as obtained from a duckdb_type_time column returns the duckdb_time_struct with the decomposed elements duckdb_to_time re-compose a duckdb_time from hour minute second and microsecond duckdb_time_struct syntax the hour minute second and microsecond in a duckdb_time_struct returns the duckdb_time element duckdb_from_timestamp decompose a duckdb_timestamp object into a duckdb_timestamp_struct syntax the ts object as obtained from a duckdb_type_timestamp column returns the duckdb_timestamp_struct with the decomposed elements duckdb_to_timestamp re-compose a duckdb_timestamp from a duckdb_timestamp_struct syntax the de-composed elements in a duckdb_timestamp_struct returns the duckdb_timestamp element duckdb_hugeint_to_double converts a duckdb_hugeint object as obtained from a duckdb_type_hugeint column into a double syntax the hugeint value returns the converted double element duckdb_double_to_hugeint converts a double value to a duckdb_hugeint object if the conversion fails because the double value is too big the result will be 0 syntax the double value returns the converted duckdb_hugeint element duckdb_decimal_to_double converts a duckdb_decimal object as obtained from a duckdb_type_decimal column into a double syntax the decimal value returns the converted double element duckdb_create_logical_type creates a duckdb_logical_type from a standard primitive type the resulting type should be destroyed with duckdb_destroy_logical_type this should not be used with duckdb_type_decimal syntax the primitive type to create returns the logical type duckdb_create_list_type creates a list type from its child type the resulting type should be destroyed with duckdb_destroy_logical_type syntax the child type of list type to create returns the logical type duckdb_create_map_type creates a map type from its key type and value type the resulting type should be destroyed with duckdb_destroy_logical_type syntax the key type and value type of map type to create returns the logical type duckdb_create_decimal_type creates a duckdb_logical_type of type decimal with the specified width and scale the resulting type should be destroyed with duckdb_destroy_logical_type syntax the width of the decimal type scale the scale of the decimal type returns the logical type duckdb_get_type_id retrieves the type class of a duckdb_logical_type syntax the logical type object returns the type id duckdb_decimal_width retrieves the width of a decimal type syntax the logical type object returns the width of the decimal type duckdb_decimal_scale retrieves the scale of a decimal type syntax the logical type object returns the scale of the decimal type duckdb_decimal_internal_type retrieves the internal storage type of a decimal type syntax the logical type object returns the internal type of the decimal type duckdb_enum_internal_type retrieves the internal storage type of an enum type syntax the logical type object returns the internal type of the enum type duckdb_enum_dictionary_size retrieves the dictionary size of the enum type syntax the logical type object returns the dictionary size of the enum type duckdb_enum_dictionary_value retrieves the dictionary value at the specified position from the enum the result must be freed with duckdb_free syntax the logical type object index the index in the dictionary returns the string value of the enum type must be freed with duckdb_free duckdb_list_type_child_type retrieves the child type of the given list type the result must be freed with duckdb_destroy_logical_type syntax the logical type object returns the child type of the list type must be destroyed with duckdb_destroy_logical_type duckdb_map_type_key_type retrieves the key type of the given map type the result must be freed with duckdb_destroy_logical_type syntax the logical type object returns the key type of the map type must be destroyed with duckdb_destroy_logical_type duckdb_map_type_value_type retrieves the value type of the given map type the result must be freed with duckdb_destroy_logical_type syntax the logical type object returns the value type of the map type must be destroyed with duckdb_destroy_logical_type duckdb_struct_type_child_count returns the number of children of a struct type syntax the logical type object returns the number of children of a struct type duckdb_struct_type_child_name retrieves the name of the struct child the result must be freed with duckdb_free syntax the logical type object index the child index returns the name of the struct type must be freed with duckdb_free duckdb_struct_type_child_type retrieves the child type of the given struct type at the specified index the result must be freed with duckdb_destroy_logical_type syntax the logical type object index the child index returns the child type of the struct type must be destroyed with duckdb_destroy_logical_type duckdb_destroy_logical_type destroys the logical type and de-allocates all memory allocated for that type syntax the logical type to destroy",
- "category": "C",
- "url": "../docs/api/c/types",
- "blurb": "DuckDB is a strongly typed database system. As such, every column has a single type specified. This type is constant ..."
+ "title": "Data Types",
+ "text": "general-purpose data types the table below shows all the built-in general-purpose data types the alternatives listed in the aliases column can be used to refer to these types as well however note that the aliases are not part of the sql standard and hence might not be accepted by other database engines name aliases description --- --- --- bigint int8 long signed eight-byte integer boolean bool logical logical boolean true false blob bytea binary varbinary variable-length binary data date calendar date year month day double float8 numeric decimal double precision floating-point number 8 bytes decimal s p fixed-precision floating point number with the given scale and precision hugeint signed sixteen-byte integer integer int4 int signed signed four-byte integer interval date time delta real float4 float single precision floating-point number 4 bytes smallint int2 short signed two-byte integer time time of day no time zone timestamp datetime combination of time and date timestamp with time zone timestamptz combination of time and date that uses the current time zone tinyint int1 signed one-byte integer ubigint unsigned eight-byte integer uinteger unsigned four-byte integer usmallint unsigned two-byte integer utinyint unsigned one-byte integer uuid uuid data type varchar char bpchar text string variable-length character string nested composite types duckdb supports three nested data types list struct and map each supports different use cases and has a different structure name description rules when used in a column build from values define in ddl create --- --- --- --- --- list an ordered sequence of data values of the same type each row must have the same data type within each list but can have any number of elements 1 2 3 int struct a dictionary of multiple named values where each key is a string but the value can be a different type for each key each row must have the same keys i 42 j a struct i int j varchar map a dictionary of multiple named values each key having the same type and each value having the same type keys and values can be any type and can be different types from one another rows may have different keys map 1 2 a b map int varchar nesting list s struct s and map s can be arbitrarily nested to any depth so long as the type rules are observed -- struct with lists select birds duck goose heron aliens null amphibians frog toad -- struct with list of maps select test map 1 5 42 1 45 map 1 5 42 1 45 links to detailed documentation",
+ "category": "Data Types",
+ "url": "../docs/sql/data_types/overview",
+ "blurb": "The table below shows all the built-in general-purpose data types."
},
{
- "title": "C API - Prepared Statements",
- "text": "a prepared statement is a parameterized query the query is prepared with question marks or dollar symbols 1 indicating the parameters of the query values can then be bound to these parameters after which the prepared statement can be executed using those parameters a single query can be prepared once and executed many times prepared statements are useful to easily supply parameters to functions while avoiding string concatenation sql injection attacks speeding up queries that will be executed many times with different parameters duckdb supports prepared statements in the c api with the duckdb_prepare method the duckdb_bind family of functions is used to supply values for subsequent execution of the prepared statement using duckdb_execute_prepared after we are done with the prepared statement it can be cleaned up using the duckdb_destroy_prepare method example duckdb_prepared_statement stmt duckdb_result result if duckdb_prepare con insert into integers values 1 2 stmt duckdberror handle error duckdb_bind_int32 stmt 1 42 the parameter index starts counting at 1 duckdb_bind_int32 stmt 2 43 null as second parameter means no result set is requested duckdb_execute_prepared stmt null duckdb_destroy_prepare stmt we can also query result sets using prepared statements if duckdb_prepare con select from integers where i stmt duckdberror handle error duckdb_bind_int32 stmt 1 42 duckdb_execute_prepared stmt result do something with result clean up duckdb_destroy_result result duckdb_destroy_prepare stmt after calling duckdb_prepare the prepared statement parameters can be inspected using duckdb_nparams and duckdb_param_type in case the prepare fails the error can be obtained through duckdb_prepare_error it is not required that the duckdb_bind family of functions matches the prepared statement parameter type exactly the values will be auto-cast to the required value as required for example calling duckdb_bind_int8 on a parameter type of duckdb_type_integer will work as expected do not use prepared statements to insert large amounts of data into duckdb instead it is recommended to use the appender api reference note that after calling duckdb_prepare the prepared statement should always be destroyed using duckdb_destroy_prepare even if the prepare fails if the prepare fails duckdb_prepare_error can be called to obtain the reason why the prepare failed syntax the connection object query the sql query to prepare out_prepared_statement the resulting prepared statement object returns duckdbsuccess on success or duckdberror on failure duckdb_destroy_prepare closes the prepared statement and de-allocates all memory allocated for the statement syntax the prepared statement to destroy duckdb_prepare_error returns the error message associated with the given prepared statement if the prepared statement has no error message this returns nullptr instead the error message should not be freed it will be de-allocated when duckdb_destroy_prepare is called syntax the prepared statement to obtain the error from returns the error message or nullptr if there is none duckdb_nparams returns the number of parameters that can be provided to the given prepared statement returns 0 if the query was not successfully prepared syntax the prepared statement to obtain the number of parameters for duckdb_param_type returns the parameter type for the parameter at the given index returns duckdb_type_invalid if the parameter index is out of range or the statement was not successfully prepared syntax the prepared statement param_idx the parameter index returns the parameter type duckdb_clear_bindings clear the params bind to the prepared statement syntax duckdb_bind_boolean binds a bool value to the prepared statement at the specified index syntax duckdb_bind_int8 binds an int8_t value to the prepared statement at the specified index syntax duckdb_bind_int16 binds an int16_t value to the prepared statement at the specified index syntax duckdb_bind_int32 binds an int32_t value to the prepared statement at the specified index syntax duckdb_bind_int64 binds an int64_t value to the prepared statement at the specified index syntax duckdb_bind_hugeint binds an duckdb_hugeint value to the prepared statement at the specified index syntax duckdb_bind_uint8 binds an uint8_t value to the prepared statement at the specified index syntax duckdb_bind_uint16 binds an uint16_t value to the prepared statement at the specified index syntax duckdb_bind_uint32 binds an uint32_t value to the prepared statement at the specified index syntax duckdb_bind_uint64 binds an uint64_t value to the prepared statement at the specified index syntax duckdb_bind_float binds an float value to the prepared statement at the specified index syntax duckdb_bind_double binds an double value to the prepared statement at the specified index syntax duckdb_bind_date binds a duckdb_date value to the prepared statement at the specified index syntax duckdb_bind_time binds a duckdb_time value to the prepared statement at the specified index syntax duckdb_bind_timestamp binds a duckdb_timestamp value to the prepared statement at the specified index syntax duckdb_bind_interval binds a duckdb_interval value to the prepared statement at the specified index syntax duckdb_bind_varchar binds a null-terminated varchar value to the prepared statement at the specified index syntax duckdb_bind_varchar_length binds a varchar value to the prepared statement at the specified index syntax duckdb_bind_blob binds a blob value to the prepared statement at the specified index syntax duckdb_bind_null binds a null value to the prepared statement at the specified index syntax duckdb_execute_prepared executes the prepared statement with the given bound parameters and returns a materialized query result this method can be called multiple times for each prepared statement and the parameters can be modified between calls to this function syntax the prepared statement to execute out_result the query result returns duckdbsuccess on success or duckdberror on failure duckdb_execute_prepared_arrow executes the prepared statement with the given bound parameters and returns an arrow query result syntax the prepared statement to execute out_result the query result returns duckdbsuccess on success or duckdberror on failure",
- "category": "C",
- "url": "../docs/api/c/prepared",
- "blurb": "A prepared statement is a parameterized query. The query is prepared with question marks ( ? ) or dollar symbols ( $1 )..."
+ "title": "Date Functions",
+ "text": "this section describes functions and operators for examining and manipulating date values date operators the table below shows the available mathematical operators for date types operator description example result --- --- --- --- addition of days integers date 1992-03-22 5 1992-03-27 addition of an interval date 1992-03-22 interval 5 day 1992-03-27 - subtraction of date s date 1992-03-27 - date 1992-03-22 5 - subtraction of an interval date 1992-03-27 - interval 5 day 1992-03-22 adding to or subtracting from infinite values produces the same infinite value date functions the table below shows the available functions for date types dates can also be manipulated with the timestamp functions through type promotion function description example result --- --- --- --- current_date current date at start of current transaction date_diff part startdate enddate the number of partition boundaries between the dates date_diff month date 1992-09-15 date 1992-11-14 2 datediff part startdate enddate alias of date_diff the number of partition boundaries between the dates datediff month date 1992-09-15 date 1992-11-14 2 date_part part date get the subfield equivalent to extract date_part year date 1992-09-20 1992 datepart part date alias of date_part get the subfield equivalent to extract datepart year date 1992-09-20 1992 date_sub part startdate enddate the number of complete partitions between the dates date_sub month date 1992-09-15 date 1992-11-14 1 datesub part startdate enddate alias of date_sub the number of complete partitions between the dates datesub month date 1992-09-15 date 1992-11-14 1 date_trunc part date truncate to specified precision date_trunc month date 1992-03-07 1992-03-01 datetrunc part date alias of date_trunc truncate to specified precision datetrunc month date 1992-03-07 1992-03-01 dayname date the english name of the weekday dayname date 1992-09-20 sunday isfinite date returns true if the date is finite false otherwise isfinite date 1992-03-07 true isinf date returns true if the date is infinite false otherwise isinf date -infinity true extract part from date get subfield from a date extract year from date 1992-09-20 1992 greatest date date the later of two dates greatest date 1992-09-20 date 1992-03-07 1992-09-20 last_day date the last day of the corresponding month in the date last_day date 1992-09-20 1992-09-30 least date date the earlier of two dates least date 1992-09-20 date 1992-03-07 1992-03-07 make_date bigint bigint bigint the date for the given parts make_date 1992 9 20 1992-09-20 monthname date the english name of the month monthname date 1992-09-20 september strftime date format converts a date to a string according to the format string strftime date 1992-01-01 a -d b y wed 1 january 1992 there are also dedicated extraction functions to get the subfields functions applied to infinite dates will either return the same infinite dates e g greatest or null e g date_part depending on what makes sense in general if the function needs to examine the parts of the infinite date the result will be null",
+ "category": "Functions",
+ "url": "../docs/sql/functions/date",
+ "blurb": "This section describes functions and operators for examining and manipulating date values. Date Operators The table..."
},
{
- "title": "C API - Values",
- "text": "the value class represents a single value of any type api reference syntax the value to destroy duckdb_create_varchar creates a value from a null-terminated string syntax the null-terminated string returns the value this must be destroyed with duckdb_destroy_value duckdb_create_varchar_length creates a value from a string syntax the text length the length of the text returns the value this must be destroyed with duckdb_destroy_value duckdb_create_int64 creates a value from an int64 syntax the bigint value returns the value this must be destroyed with duckdb_destroy_value duckdb_get_varchar obtains a string representation of the given value the result must be destroyed with duckdb_free syntax the value returns the string value this must be destroyed with duckdb_free duckdb_get_int64 obtains an int64 of the given value syntax the value returns the int64 value or 0 if no conversion is possible",
- "category": "C",
- "url": "../docs/api/c/value",
- "blurb": "The value class represents a single value of any type. API Reference Syntax The value to destroy. ..."
+ "title": "Interval Functions",
+ "text": "this section describes functions and operators for examining and manipulating interval values interval operators the table below shows the available mathematical operators for interval types operator description example result --- --- --- --- addition of an interval interval 1 hour interval 5 hour interval 6 hour addition to a date date 1992-03-22 interval 5 day 1992-03-27 addition to a timestamp timestamp 1992-03-22 01 02 03 interval 5 day 1992-03-27 01 02 03 addition to a time time 01 02 03 interval 5 hour 06 02 03 - subtraction of an interval interval 5 hour - interval 1 hour interval 4 hour - subtraction from a date date 1992-03-27 - interval 5 day 1992-03-22 - subtraction from a timestamp timestamp 1992-03-27 01 02 03 - interval 5 day 1992-03-22 01 02 03 - subtraction from a time time 06 02 03 - interval 5 hour 01 02 03 interval functions the table below shows the available scalar functions for interval types function description example result --- --- --- --- date_part part interval get subfield equivalent to extract date_part year interval 14 months 1 extract part from interval get subfield from a date extract month from interval 14 months 2 to_years integer construct a year interval to_years 5 interval 5 year to_months integer construct a month interval to_months 5 interval 5 month to_days integer construct a day interval to_days 5 interval 5 day to_hours integer construct a hour interval to_hours 5 interval 5 hour to_minutes integer construct a minute interval to_minutes 5 interval 5 minute to_seconds integer construct a second interval to_seconds 5 interval 5 second to_milliseconds integer construct a millisecond interval to_milliseconds 5 interval 5 millisecond to_microseconds integer construct a microsecond interval to_microseconds 5 interval 5 microsecond only the documented date parts are defined for intervals",
+ "category": "Functions",
+ "url": "../docs/sql/functions/interval",
+ "blurb": "This section describes functions and operators for examining and manipulating INTERVAL values. Interval Operators ..."
},
{
- "title": "C API - Configuration",
- "text": "configuration options can be provided to change different settings of the database system note that many of these settings can be changed later on using pragma statements as well the configuration object should be created filled with values and passed to duckdb_open_ext example duckdb_database db duckdb_config config create the configuration object if duckdb_create_config config duckdberror handle error set some configuration options duckdb_set_config config access_mode read_write or read_only duckdb_set_config config threads 8 duckdb_set_config config max_memory 8gb duckdb_set_config config default_order desc open the database using the configuration if duckdb_open_ext null db config null duckdberror handle error cleanup the configuration object duckdb_destroy_config config run queries cleanup duckdb_close db api reference this will always succeed unless there is a malloc failure syntax the result configuration object returns duckdbsuccess on success or duckdberror on failure duckdb_config_count this returns the total amount of configuration options available for usage with duckdb_get_config_flag this should not be called in a loop as it internally loops over all the options syntax the amount of config options available duckdb_get_config_flag obtains a human-readable name and description of a specific configuration option this can be used to e g display configuration options this will succeed unless index is out of range i e duckdb_config_count the result name or description must not be freed syntax the index of the configuration option between 0 and duckdb_config_count out_name a name of the configuration flag out_description a description of the configuration flag returns duckdbsuccess on success or duckdberror on failure duckdb_set_config sets the specified option for the specified configuration the configuration option is indicated by name to obtain a list of config options see duckdb_get_config_flag in the source code configuration options are defined in config cpp this can fail if either the name is invalid or if the value provided for the option is invalid syntax the configuration object to set the option on name the name of the configuration flag to set option the value to set the configuration flag to returns duckdbsuccess on success or duckdberror on failure duckdb_destroy_config destroys the specified configuration option and de-allocates all memory allocated for the object syntax the configuration object to destroy",
- "category": "C",
- "url": "../docs/api/c/config",
- "blurb": "Configuration options can be provided to change different settings of the database system. Note that many of these ..."
+ "title": "Numeric Functions",
+ "text": "numeric operators the table below shows the available mathematical operators for numeric types operator description example result --- --- --- --- addition 2 3 5 - subtraction 2 - 3 -1 multiplication 2 3 6 division 4 2 2 modulo remainder 5 4 1 exponent 3 4 81 exponent alias for 3 4 81 bitwise and 91 15 11 bitwise or 32 3 35 bitwise shift left 1 4 16 bitwise shift right 8 2 2 factorial of x computes the product of the current integer and all integers below it 4 24 the modulo bitwise and factorial operators work only on integral data types whereas the others are available for all numeric data types numeric functions the table below shows the available mathematical functions function description example result --- --- --- --- abs x absolute value abs -17 4 17 4 acos x computes the arccosine of x acos 0 5 1 0471975511965976 asin x computes the arcsine of x asin 0 5 0 5235987755982989 atan x computes the arctangent of x atan 0 5 0 4636476090008061 atan2 x y computes the arctangent x y atan2 0 5 0 5 0 7853981633974483 bit_count x returns the number of bits that are set bit_count 31 5 cbrt x returns the cube root of the number cbrt 8 2 ceil x rounds the number up ceil 17 4 18 ceiling x rounds the number up alias of ceil ceiling 17 4 18 chr x returns a character which is corresponding the the ascii code value or unicode code point chr 65 a cos x computes the cosine of x cos 90 -0 4480736161291701 cot x computes the cotangent of x cot 0 5 1 830487721712452 degrees x converts radians to degrees degrees pi 180 even x round to next even number by rounding away from zero even 2 9 4 factorial x see operator computes the product of the current integer and all integers below it factorial 4 24 floor x rounds the number down floor 17 4 17 gamma x interpolation of x-1 factorial so decimal inputs are allowed gamma 5 5 52 34277778455352 greatest x1 x2 selects the largest value greatest 3 2 4 4 4 isfinite x returns true if the floating point value is finite false otherwise isfinite 5 5 true isinf x returns true if the floating point value is infinite false otherwise isinf infinity true isnan x returns true if the floating point value is not a number false otherwise isnan nan true least x1 x2 selects the smallest value least 3 2 4 4 2 lgamma x computes the log of the gamma function lgamma 2 0 ln x computes the natural logarithm of x ln 2 0 693 log x computes the 10-log of x log 100 2 log2 x computes the 2-log of x log2 8 3 log10 x alias of log computes the 10-log of x log10 1000 3 nextafter x y return the next floating point value after x in the direction of y nextafter 1 float 2 float 1 0000001 pi returns the value of pi pi 3 141592653589793 pow x y computes x to the power of y pow 2 3 8 power x y alias of pow computes x to the power of y power 2 3 8 radians x converts degrees to radians radians 90 1 5707963267948966 random returns a random number between 0 and 1 random various round v numeric s int round to s decimal places round 42 4332 2 42 43 setseed x sets the seed to be used for the random function setseed 0 42 sin x computes the sin of x sin 90 0 8939966636005579 sign x returns the sign of x as -1 0 or 1 sign -349 -1 sqrt x returns the square root of the number sqrt 9 3 xor x bitwise xor xor 17 5 20 tan x computes the tangent of x tan 90 -1 995200412208242 absolute value parentheses optional if operating on a column -2 2",
+ "category": "Functions",
+ "url": "../docs/sql/functions/numeric",
+ "blurb": "Numeric Operators The table below shows the available mathematical operators for numeric types. | Operator |..."
},
{
- "title": "C API - Startup & Shutdown",
- "text": "to use duckdb you must first initialize a duckdb_database handle using duckdb_open duckdb_open takes as parameter the database file to read and write from the special value null nullptr can be used to create an in-memory database note that for an in-memory database no data is persisted to disk i e all data is lost when you exit the process with the duckdb_database handle you can create one or many duckdb_connection using duckdb_connect while individual connections are thread-safe they will be locked during querying it is therefore recommended that each thread uses its own connection to allow for the best parallel performance all duckdb_connection s have to explicitly be disconnected with duckdb_disconnect and the duckdb_database has to be explicitly closed with duckdb_close to avoid memory and file handle leaking example duckdb_database db duckdb_connection con if duckdb_open null db duckdberror handle error if duckdb_connect db con duckdberror handle error run queries cleanup duckdb_disconnect con duckdb_close db api reference syntax path to the database file on disk or nullptr or memory to open an in-memory database out_database the result database object returns duckdbsuccess on success or duckdberror on failure duckdb_open_ext extended version of duckdb_open creates a new database or opens an existing database file stored at the the given path syntax path to the database file on disk or nullptr or memory to open an in-memory database out_database the result database object config optional configuration used to start up the database system out_error if set and the function returns duckdberror this will contain the reason why the start-up failed note that the error must be freed using duckdb_free returns duckdbsuccess on success or duckdberror on failure duckdb_close closes the specified database and de-allocates all memory allocated for that database this should be called after you are done with any database allocated through duckdb_open note that failing to call duckdb_close in case of e g a program crash will not cause data corruption still it is recommended to always correctly close a database object after you are done with it syntax the database object to shut down duckdb_connect opens a connection to a database connections are required to query the database and store transactional state associated with the connection syntax the database file to connect to out_connection the result connection object returns duckdbsuccess on success or duckdberror on failure duckdb_disconnect closes the specified connection and de-allocates all memory allocated for that connection syntax the connection to close",
- "category": "C",
- "url": "../docs/api/c/connect",
- "blurb": "To use DuckDB, you must first initialize a duckdb_database handle using duckdb_open() . duckdb_open() takes as..."
+ "title": "Text Functions",
+ "text": "this section describes functions and operators for examining and manipulating string values denotes a space character function description example result --- --- --- --- string string string concatenation duck db duckdb string index alias for array_extract duckdb 4 k string begin end alias for array_slice missing arguments are interprete as null s duckdb 4 duck array_extract list index extract a single character using a 1-based index array_extract duckdb 2 u array_slice list begin end extract a string using slice conventions null s are interpreted as the bounds of the string negative values are accepted array_slice duckdb 5 null db ascii string returns an integer that represents the unicode code point of the first character of the string ascii ω 937 base64 blob convert a blob to a base64 encoded string alias of to_base64 base64 a blob qq bit_length string number of bits in a string bit_length abc 24 concat string concatenate many strings together concat hello world hello world concat_ws separator string concatenate strings together separated by the specified separator concat_ws banana apple melon banana apple melon contains string search_string return true if search_string is found within string contains abc a true format format parameters formats a string using fmt syntax format benchmark took seconds csv 42 benchmark csv took 42 seconds from_base64 string convert a base64 encoded string to a character string from_base64 qq a instr string search_string return location of first occurrence of search_string in string counting from 1 returns 0 if no match found instr test test es 2 lcase string alias of lower convert string to lower case lcase hello hello left string count extract the left-most count characters left hello 2 he length string number of characters in string length hello 5 string like target returns true if the string matches the like specifier see pattern matching hello like lo true like_escape string like_specifier escape_character returns true if the string matches the like_specifier see pattern matching escape_character is used to search for wildcard characters in the string like_escape a c a c true list_element string index an alias for array_extract list_element duckdb 2 u list_extract string index an alias for array_extract list_extract duckdb 2 u lower string convert string to lower case lower hello hello lpad string count character pads the string with the character from the left until it has count characters lpad hello 10 hello ltrim string removes any spaces from the left side of the string ltrim test test ltrim string characters removes any occurrences of any of the characters from the left side of the string ltrim test test md5 value returns the md5 hash of the value md5 123 202cb962ac59075b964b07152d234b70 nfc_normalize string convert string to unicode nfc normalized string useful for comparisons and ordering if text data is mixed between nfc normalized and not nfc_normalize arde ch arde ch not_like_escape string like_specifier escape_character returns false if the string matches the like_specifier see pattern matching escape_character is used to search for wildcard characters in the string like_escape a c a c true ord string return ascii character code of the leftmost character in a string ord ü 252 position search_string in string return location of first occurrence of search_string in string counting from 1 returns 0 if no match found position b in abc 2 prefix string search_string return true if string starts with search_string prefix abc ab true printf format parameters formats a string using printf syntax printf benchmark s took d seconds csv 42 benchmark csv took 42 seconds regexp_full_match string regex returns true if the entire string matches the regex see pattern matching regexp_full_match anabanana an false regexp_matches string regex returns true if a part of string matches the regex see pattern matching regexp_matches anabanana an true regexp_replace string regex replacement modifiers replaces the first occurrence of regex with the replacement use g modifier to replace all occurrences instead see pattern matching select regexp_replace hello lo - he-lo regexp_split_to_array string regex alias of string_split_regex splits the string along the regex regexp_split_to_array hello world 42 hello world 42 repeat string count repeats the string count number of times repeat a 5 aaaaa replace string source target replaces any occurrences of the source with target in string replace hello l - he--o reverse string reverses the string reverse hello olleh right string count extract the right-most count characters right hello 3 llo rpad string count character pads the string with the character from the right until it has count characters rpad hello 10 hello rtrim string removes any spaces from the right side of the string rtrim test test rtrim string characters removes any occurrences of any of the characters from the right side of the string rtrim test test string similar to regex returns true if the string matches the regex identical to regexp_full_match see pattern matching hello similar to l false strlen string number of bytes in string length 1 strpos string search_string alias of instr return location of first occurrence of search_string in string counting from 1 returns 0 if no match found strpos test test es 2 strip_accents string strips accents from string strip_accents mühleisen muhleisen str_split string separator alias of string_split splits the string along the separator str_split hello world hello world str_split_regex string regex alias of string_split_regex splits the string along the regex str_split_regex hello world 42 hello world 42 string_split string separator splits the string along the separator string_split hello world hello world string_split_regex string regex splits the string along the regex string_split_regex hello world 42 hello world 42 string_to_array string separator alias of string_split splits the string along the separator string_to_array hello world hello world substr string start length alias of substring extract substring of length characters starting from character start note that a start value of 1 refers to the first character of the string substr hello 2 2 el substring string start length extract substring of length characters starting from character start note that a start value of 1 refers to the first character of the string substring hello 2 2 el suffix string search_string return true if string ends with search_string suffix abc bc true strpos string characters alias of instr return location of first occurrence of characters in string counting from 1 returns 0 if no match found strpos test test es 2 to_base64 blob convert a blob to a base64 encoded string alias of base64 to_base64 a blob qq trim string removes any spaces from either side of the string trim test test trim string characters removes any occurrences of any of the characters from either side of the string trim test test ucase string alias of upper convert string to upper case ucase hello hello unicode string returns the unicode code of the first character of the string unicode ü 252 upper string convert string to upper case upper hello hello text similarity functions these functions are used to measure the similarity of two strings using various metrics function description example result --- --- --- --- editdist3 string string alias of levenshtein for sqlite compatibility the minimum number of single-character edits insertions deletions or substitutions required to change one string to the other different case is considered different editdist3 duck db 3 hamming string string the number of positions with different characters for 2 strings of equal length different case is considered different hamming duck luck 1 jaccard string string the jaccard similarity between two strings different case is considered different returns a number between 0 and 1 jaccard duck luck 0 6 jaro_similarity string string the jaro similarity between two strings different case is considered different returns a number between 0 and 1 jaro_similarity duck duckdb 0 88 jaro_winkler_similarity string string the jaro-winkler similarity between two strings different case is considered different returns a number between 0 and 1 jaro_winkler_similarity duck duckdb 0 93 levenshtein string string the minimum number of single-character edits insertions deletions or substitutions required to change one string to the other different case is considered different levenshtein duck db 3 mismatches string string the number of positions with different characters for 2 strings of equal length different case is considered different mismatches duck luck 1",
+ "category": "Functions",
+ "url": "../docs/sql/functions/char",
+ "blurb": "This section describes functions and operators for examining and manipulating string values. ␣ denotes a space..."
},
{
- "title": "C API - Complete API",
- "text": "api reference open connect syntax path to the database file on disk or nullptr or memory to open an in-memory database out_database the result database object returns duckdbsuccess on success or duckdberror on failure duckdb_open_ext extended version of duckdb_open creates a new database or opens an existing database file stored at the the given path syntax path to the database file on disk or nullptr or memory to open an in-memory database out_database the result database object config optional configuration used to start up the database system out_error if set and the function returns duckdberror this will contain the reason why the start-up failed note that the error must be freed using duckdb_free returns duckdbsuccess on success or duckdberror on failure duckdb_close closes the specified database and de-allocates all memory allocated for that database this should be called after you are done with any database allocated through duckdb_open note that failing to call duckdb_close in case of e g a program crash will not cause data corruption still it is recommended to always correctly close a database object after you are done with it syntax the database object to shut down duckdb_connect opens a connection to a database connections are required to query the database and store transactional state associated with the connection syntax the database file to connect to out_connection the result connection object returns duckdbsuccess on success or duckdberror on failure duckdb_disconnect closes the specified connection and de-allocates all memory allocated for that connection syntax the connection to close duckdb_create_config initializes an empty configuration object that can be used to provide start-up options for the duckdb instance through duckdb_open_ext this will always succeed unless there is a malloc failure syntax the result configuration object returns duckdbsuccess on success or duckdberror on failure duckdb_config_count this returns the total amount of configuration options available for usage with duckdb_get_config_flag this should not be called in a loop as it internally loops over all the options syntax the amount of config options available duckdb_get_config_flag obtains a human-readable name and description of a specific configuration option this can be used to e g display configuration options this will succeed unless index is out of range i e duckdb_config_count the result name or description must not be freed syntax the index of the configuration option between 0 and duckdb_config_count out_name a name of the configuration flag out_description a description of the configuration flag returns duckdbsuccess on success or duckdberror on failure duckdb_set_config sets the specified option for the specified configuration the configuration option is indicated by name to obtain a list of config options see duckdb_get_config_flag in the source code configuration options are defined in config cpp this can fail if either the name is invalid or if the value provided for the option is invalid syntax the configuration object to set the option on name the name of the configuration flag to set option the value to set the configuration flag to returns duckdbsuccess on success or duckdberror on failure duckdb_destroy_config destroys the specified configuration option and de-allocates all memory allocated for the object syntax the configuration object to destroy duckdb_query executes a sql query within a connection and stores the full materialized result in the out_result pointer if the query fails to execute duckdberror is returned and the error message can be retrieved by calling duckdb_result_error note that after running duckdb_query duckdb_destroy_result must be called on the result object even if the query fails otherwise the error stored within the result will not be freed correctly syntax the connection to perform the query in query the sql query to run out_result the query result returns duckdbsuccess on success or duckdberror on failure duckdb_destroy_result closes the result and de-allocates all memory allocated for that connection syntax the result to destroy duckdb_column_name returns the column name of the specified column the result should not need be freed the column names will automatically be destroyed when the result is destroyed returns null if the column is out of range syntax the result object to fetch the column name from col the column index returns the column name of the specified column duckdb_column_type returns the column type of the specified column returns duckdb_type_invalid if the column is out of range syntax the result object to fetch the column type from col the column index returns the column type of the specified column duckdb_column_logical_type returns the logical column type of the specified column the return type of this call should be destroyed with duckdb_destroy_logical_type returns null if the column is out of range syntax the result object to fetch the column type from col the column index returns the logical column type of the specified column duckdb_column_count returns the number of columns present in a the result object syntax the result object returns the number of columns present in the result object duckdb_row_count returns the number of rows present in a the result object syntax the result object returns the number of rows present in the result object duckdb_rows_changed returns the number of rows changed by the query stored in the result this is relevant only for insert update delete queries for other queries the rows_changed will be 0 syntax the result object returns the number of rows changed duckdb_column_data deprecated prefer using duckdb_result_get_chunk instead returns the data of a specific column of a result in columnar format the function returns a dense array which contains the result data the exact type stored in the array depends on the corresponding duckdb_type as provided by duckdb_column_type for the exact type by which the data should be accessed see the comments in the types section or the duckdb_type enum for example for a column of type duckdb_type_integer rows can be accessed in the following manner int32_t data int32_t duckdb_column_data result 0 printf data for row d d n row data row syntax the result object to fetch the column data from col the column index returns the column data of the specified column duckdb_nullmask_data deprecated prefer using duckdb_result_get_chunk instead returns the nullmask of a specific column of a result in columnar format the nullmask indicates for every row whether or not the corresponding row is null if a row is null the values present in the array provided by duckdb_column_data are undefined int32_t data int32_t duckdb_column_data result 0 bool nullmask duckdb_nullmask_data result 0 if nullmask row printf data for row d null n row else printf data for row d d n row data row syntax the result object to fetch the nullmask from col the column index returns the nullmask of the specified column duckdb_result_error returns the error message contained within the result the error is only set if duckdb_query returns duckdberror the result of this function must not be freed it will be cleaned up when duckdb_destroy_result is called syntax the result object to fetch the error from returns the error of the result duckdb_result_get_chunk fetches a data chunk from the duckdb_result this function should be called repeatedly until the result is exhausted this function supersedes all duckdb_value functions as well as the duckdb_column_data and duckdb_nullmask_data functions it results in significantly better performance and should be preferred in newer code-bases if this function is used none of the other result functions can be used and vice versa i e this function cannot be mixed with the legacy result functions use duckdb_result_chunk_count to figure out how many chunks there are in the result syntax the result object to fetch the data chunk from chunk_index the chunk index to fetch from returns the resulting data chunk returns null if the chunk index is out of bounds duckdb_result_chunk_count returns the number of data chunks present in the result syntax the result object returns the resulting data chunk returns null if the chunk index is out of bounds duckdb_value_boolean syntax the boolean value at the specified location or false if the value cannot be converted duckdb_value_int8 syntax the int8_t value at the specified location or 0 if the value cannot be converted duckdb_value_int16 syntax the int16_t value at the specified location or 0 if the value cannot be converted duckdb_value_int32 syntax the int32_t value at the specified location or 0 if the value cannot be converted duckdb_value_int64 syntax the int64_t value at the specified location or 0 if the value cannot be converted duckdb_value_hugeint syntax the duckdb_hugeint value at the specified location or 0 if the value cannot be converted duckdb_value_decimal syntax the duckdb_decimal value at the specified location or 0 if the value cannot be converted duckdb_value_uint8 syntax the uint8_t value at the specified location or 0 if the value cannot be converted duckdb_value_uint16 syntax the uint16_t value at the specified location or 0 if the value cannot be converted duckdb_value_uint32 syntax the uint32_t value at the specified location or 0 if the value cannot be converted duckdb_value_uint64 syntax the uint64_t value at the specified location or 0 if the value cannot be converted duckdb_value_float syntax the float value at the specified location or 0 if the value cannot be converted duckdb_value_double syntax the double value at the specified location or 0 if the value cannot be converted duckdb_value_date syntax the duckdb_date value at the specified location or 0 if the value cannot be converted duckdb_value_time syntax the duckdb_time value at the specified location or 0 if the value cannot be converted duckdb_value_timestamp syntax the duckdb_timestamp value at the specified location or 0 if the value cannot be converted duckdb_value_interval syntax the duckdb_interval value at the specified location or 0 if the value cannot be converted duckdb_value_varchar syntax the char value at the specified location or nullptr if the value cannot be converted the result must be freed with duckdb_free duckdb_value_varchar_internal syntax the char value at the specified location only works on varchar columns and does not auto-cast if the column is not a varchar column this function will return null the result must not be freed duckdb_value_blob syntax the duckdb_blob value at the specified location returns a blob with blob data set to nullptr if the value cannot be converted the resulting blob data must be freed with duckdb_free duckdb_value_is_null syntax returns true if the value at the specified index is null and false otherwise duckdb_malloc allocate size bytes of memory using the duckdb internal malloc function any memory allocated in this manner should be freed using duckdb_free syntax the number of bytes to allocate returns a pointer to the allocated memory region duckdb_free free a value returned from duckdb_malloc duckdb_value_varchar or duckdb_value_blob syntax the memory region to de-allocate duckdb_vector_size the internal vector size used by duckdb this is the amount of tuples that will fit into a data chunk created by duckdb_create_data_chunk syntax the vector size duckdb_from_date decompose a duckdb_date object into year month and date stored as duckdb_date_struct syntax the date object as obtained from a duckdb_type_date column returns the duckdb_date_struct with the decomposed elements duckdb_to_date re-compose a duckdb_date from year month and date duckdb_date_struct syntax the year month and date stored in a duckdb_date_struct returns the duckdb_date element duckdb_from_time decompose a duckdb_time object into hour minute second and microsecond stored as duckdb_time_struct syntax the time object as obtained from a duckdb_type_time column returns the duckdb_time_struct with the decomposed elements duckdb_to_time re-compose a duckdb_time from hour minute second and microsecond duckdb_time_struct syntax the hour minute second and microsecond in a duckdb_time_struct returns the duckdb_time element duckdb_from_timestamp decompose a duckdb_timestamp object into a duckdb_timestamp_struct syntax the ts object as obtained from a duckdb_type_timestamp column returns the duckdb_timestamp_struct with the decomposed elements duckdb_to_timestamp re-compose a duckdb_timestamp from a duckdb_timestamp_struct syntax the de-composed elements in a duckdb_timestamp_struct returns the duckdb_timestamp element duckdb_hugeint_to_double converts a duckdb_hugeint object as obtained from a duckdb_type_hugeint column into a double syntax the hugeint value returns the converted double element duckdb_double_to_hugeint converts a double value to a duckdb_hugeint object if the conversion fails because the double value is too big the result will be 0 syntax the double value returns the converted duckdb_hugeint element duckdb_decimal_to_double converts a duckdb_decimal object as obtained from a duckdb_type_decimal column into a double syntax the decimal value returns the converted double element duckdb_prepare create a prepared statement object from a query note that after calling duckdb_prepare the prepared statement should always be destroyed using duckdb_destroy_prepare even if the prepare fails if the prepare fails duckdb_prepare_error can be called to obtain the reason why the prepare failed syntax the connection object query the sql query to prepare out_prepared_statement the resulting prepared statement object returns duckdbsuccess on success or duckdberror on failure duckdb_destroy_prepare closes the prepared statement and de-allocates all memory allocated for the statement syntax the prepared statement to destroy duckdb_prepare_error returns the error message associated with the given prepared statement if the prepared statement has no error message this returns nullptr instead the error message should not be freed it will be de-allocated when duckdb_destroy_prepare is called syntax the prepared statement to obtain the error from returns the error message or nullptr if there is none duckdb_nparams returns the number of parameters that can be provided to the given prepared statement returns 0 if the query was not successfully prepared syntax the prepared statement to obtain the number of parameters for duckdb_param_type returns the parameter type for the parameter at the given index returns duckdb_type_invalid if the parameter index is out of range or the statement was not successfully prepared syntax the prepared statement param_idx the parameter index returns the parameter type duckdb_clear_bindings clear the params bind to the prepared statement syntax duckdb_bind_boolean binds a bool value to the prepared statement at the specified index syntax duckdb_bind_int8 binds an int8_t value to the prepared statement at the specified index syntax duckdb_bind_int16 binds an int16_t value to the prepared statement at the specified index syntax duckdb_bind_int32 binds an int32_t value to the prepared statement at the specified index syntax duckdb_bind_int64 binds an int64_t value to the prepared statement at the specified index syntax duckdb_bind_hugeint binds an duckdb_hugeint value to the prepared statement at the specified index syntax duckdb_bind_uint8 binds an uint8_t value to the prepared statement at the specified index syntax duckdb_bind_uint16 binds an uint16_t value to the prepared statement at the specified index syntax duckdb_bind_uint32 binds an uint32_t value to the prepared statement at the specified index syntax duckdb_bind_uint64 binds an uint64_t value to the prepared statement at the specified index syntax duckdb_bind_float binds an float value to the prepared statement at the specified index syntax duckdb_bind_double binds an double value to the prepared statement at the specified index syntax duckdb_bind_date binds a duckdb_date value to the prepared statement at the specified index syntax duckdb_bind_time binds a duckdb_time value to the prepared statement at the specified index syntax duckdb_bind_timestamp binds a duckdb_timestamp value to the prepared statement at the specified index syntax duckdb_bind_interval binds a duckdb_interval value to the prepared statement at the specified index syntax duckdb_bind_varchar binds a null-terminated varchar value to the prepared statement at the specified index syntax duckdb_bind_varchar_length binds a varchar value to the prepared statement at the specified index syntax duckdb_bind_blob binds a blob value to the prepared statement at the specified index syntax duckdb_bind_null binds a null value to the prepared statement at the specified index syntax duckdb_execute_prepared executes the prepared statement with the given bound parameters and returns a materialized query result this method can be called multiple times for each prepared statement and the parameters can be modified between calls to this function syntax the prepared statement to execute out_result the query result returns duckdbsuccess on success or duckdberror on failure duckdb_execute_prepared_arrow executes the prepared statement with the given bound parameters and returns an arrow query result syntax the prepared statement to execute out_result the query result returns duckdbsuccess on success or duckdberror on failure duckdb_pending_prepared executes the prepared statement with the given bound parameters and returns a pending result the pending result represents an intermediate structure for a query that is not yet fully executed the pending result can be used to incrementally execute a query returning control to the client between tasks note that after calling duckdb_pending_prepared the pending result should always be destroyed using duckdb_destroy_pending even if this function returns duckdberror syntax the prepared statement to execute out_result the pending query result returns duckdbsuccess on success or duckdberror on failure duckdb_destroy_pending closes the pending result and de-allocates all memory allocated for the result syntax the pending result to destroy duckdb_pending_error returns the error message contained within the pending result the result of this function must not be freed it will be cleaned up when duckdb_destroy_pending is called syntax the pending result to fetch the error from returns the error of the pending result duckdb_pending_execute_task executes a single task within the query returning whether or not the query is ready if this returns duckdb_pending_result_ready the duckdb_execute_pending function can be called to obtain the result if this returns duckdb_pending_result_not_ready the duckdb_pending_execute_task function should be called again if this returns duckdb_pending_error an error occurred during execution the error message can be obtained by calling duckdb_pending_error on the pending_result syntax the pending result to execute a task within returns the state of the pending result after the execution duckdb_execute_pending fully execute a pending query result returning the final query result if duckdb_pending_execute_task has been called until duckdb_pending_result_ready was returned this will return fast otherwise all remaining tasks must be executed first syntax the pending result to execute out_result the result object returns duckdbsuccess on success or duckdberror on failure duckdb_destroy_value destroys the value and de-allocates all memory allocated for that type syntax the value to destroy duckdb_create_varchar creates a value from a null-terminated string syntax the null-terminated string returns the value this must be destroyed with duckdb_destroy_value duckdb_create_varchar_length creates a value from a string syntax the text length the length of the text returns the value this must be destroyed with duckdb_destroy_value duckdb_create_int64 creates a value from an int64 syntax the bigint value returns the value this must be destroyed with duckdb_destroy_value duckdb_get_varchar obtains a string representation of the given value the result must be destroyed with duckdb_free syntax the value returns the string value this must be destroyed with duckdb_free duckdb_get_int64 obtains an int64 of the given value syntax the value returns the int64 value or 0 if no conversion is possible duckdb_create_logical_type creates a duckdb_logical_type from a standard primitive type the resulting type should be destroyed with duckdb_destroy_logical_type this should not be used with duckdb_type_decimal syntax the primitive type to create returns the logical type duckdb_create_list_type creates a list type from its child type the resulting type should be destroyed with duckdb_destroy_logical_type syntax the child type of list type to create returns the logical type duckdb_create_map_type creates a map type from its key type and value type the resulting type should be destroyed with duckdb_destroy_logical_type syntax the key type and value type of map type to create returns the logical type duckdb_create_decimal_type creates a duckdb_logical_type of type decimal with the specified width and scale the resulting type should be destroyed with duckdb_destroy_logical_type syntax the width of the decimal type scale the scale of the decimal type returns the logical type duckdb_get_type_id retrieves the type class of a duckdb_logical_type syntax the logical type object returns the type id duckdb_decimal_width retrieves the width of a decimal type syntax the logical type object returns the width of the decimal type duckdb_decimal_scale retrieves the scale of a decimal type syntax the logical type object returns the scale of the decimal type duckdb_decimal_internal_type retrieves the internal storage type of a decimal type syntax the logical type object returns the internal type of the decimal type duckdb_enum_internal_type retrieves the internal storage type of an enum type syntax the logical type object returns the internal type of the enum type duckdb_enum_dictionary_size retrieves the dictionary size of the enum type syntax the logical type object returns the dictionary size of the enum type duckdb_enum_dictionary_value retrieves the dictionary value at the specified position from the enum the result must be freed with duckdb_free syntax the logical type object index the index in the dictionary returns the string value of the enum type must be freed with duckdb_free duckdb_list_type_child_type retrieves the child type of the given list type the result must be freed with duckdb_destroy_logical_type syntax the logical type object returns the child type of the list type must be destroyed with duckdb_destroy_logical_type duckdb_map_type_key_type retrieves the key type of the given map type the result must be freed with duckdb_destroy_logical_type syntax the logical type object returns the key type of the map type must be destroyed with duckdb_destroy_logical_type duckdb_map_type_value_type retrieves the value type of the given map type the result must be freed with duckdb_destroy_logical_type syntax the logical type object returns the value type of the map type must be destroyed with duckdb_destroy_logical_type duckdb_struct_type_child_count returns the number of children of a struct type syntax the logical type object returns the number of children of a struct type duckdb_struct_type_child_name retrieves the name of the struct child the result must be freed with duckdb_free syntax the logical type object index the child index returns the name of the struct type must be freed with duckdb_free duckdb_struct_type_child_type retrieves the child type of the given struct type at the specified index the result must be freed with duckdb_destroy_logical_type syntax the logical type object index the child index returns the child type of the struct type must be destroyed with duckdb_destroy_logical_type duckdb_destroy_logical_type destroys the logical type and de-allocates all memory allocated for that type syntax the logical type to destroy duckdb_create_data_chunk creates an empty datachunk with the specified set of types syntax an array of types of the data chunk column_count the number of columns returns the data chunk duckdb_destroy_data_chunk destroys the data chunk and de-allocates all memory allocated for that chunk syntax the data chunk to destroy duckdb_data_chunk_reset resets a data chunk clearing the validity masks and setting the cardinality of the data chunk to 0 syntax the data chunk to reset duckdb_data_chunk_get_column_count retrieves the number of columns in a data chunk syntax the data chunk to get the data from returns the number of columns in the data chunk duckdb_data_chunk_get_vector retrieves the vector at the specified column index in the data chunk the pointer to the vector is valid for as long as the chunk is alive it does not need to be destroyed syntax the data chunk to get the data from returns the vector duckdb_data_chunk_get_size retrieves the current number of tuples in a data chunk syntax the data chunk to get the data from returns the number of tuples in the data chunk duckdb_data_chunk_set_size sets the current number of tuples in a data chunk syntax the data chunk to set the size in size the number of tuples in the data chunk duckdb_vector_get_column_type retrieves the column type of the specified vector the result must be destroyed with duckdb_destroy_logical_type syntax the vector get the data from returns the type of the vector duckdb_vector_get_data retrieves the data pointer of the vector the data pointer can be used to read or write values from the vector how to read or write values depends on the type of the vector syntax the vector to get the data from returns the data pointer duckdb_vector_get_validity retrieves the validity mask pointer of the specified vector if all values are valid this function might return null the validity mask is a bitset that signifies null-ness within the data chunk it is a series of uint64_t values where each uint64_t value contains validity for 64 tuples the bit is set to 1 if the value is valid i e not null or 0 if the value is invalid i e null validity of a specific value can be obtained like this idx_t entry_idx row_idx 64 idx_t idx_in_entry row_idx 64 bool is_valid validity_mask entry_idx 1 idx_in_entry alternatively the slower duckdb_validity_row_is_valid function can be used syntax the vector to get the data from returns the pointer to the validity mask or null if no validity mask is present duckdb_vector_ensure_validity_writable ensures the validity mask is writable by allocating it after this function is called duckdb_vector_get_validity will always return non-null this allows null values to be written to the vector regardless of whether a validity mask was present before syntax the vector to alter duckdb_vector_assign_string_element assigns a string element in the vector at the specified location syntax the vector to alter index the row position in the vector to assign the string to str the null-terminated string duckdb_vector_assign_string_element_len assigns a string element in the vector at the specified location syntax the vector to alter index the row position in the vector to assign the string to str the string str_len the length of the string in bytes duckdb_list_vector_get_child retrieves the child vector of a list vector the resulting vector is valid as long as the parent vector is valid syntax the vector returns the child vector duckdb_list_vector_get_size returns the size of the child vector of the list syntax the vector returns the size of the child list duckdb_struct_vector_get_child retrieves the child vector of a struct vector the resulting vector is valid as long as the parent vector is valid syntax the vector index the child index returns the child vector duckdb_validity_row_is_valid returns whether or not a row is valid i e not null in the given validity mask syntax the validity mask as obtained through duckdb_data_chunk_get_validity row the row index returns true if the row is valid false otherwise duckdb_validity_set_row_validity in a validity mask sets a specific row to either valid or invalid note that duckdb_data_chunk_ensure_validity_writable should be called before calling duckdb_data_chunk_get_validity to ensure that there is a validity mask to write to syntax the validity mask as obtained through duckdb_data_chunk_get_validity row the row index valid whether or not to set the row to valid or invalid duckdb_validity_set_row_invalid in a validity mask sets a specific row to invalid equivalent to duckdb_validity_set_row_validity with valid set to false syntax the validity mask row the row index duckdb_validity_set_row_valid in a validity mask sets a specific row to valid equivalent to duckdb_validity_set_row_validity with valid set to true syntax the validity mask row the row index duckdb_create_table_function creates a new empty table function the return value should be destroyed with duckdb_destroy_table_function syntax the table function object duckdb_destroy_table_function destroys the given table function object syntax the table function to destroy duckdb_table_function_set_name sets the name of the given table function syntax the table function name the name of the table function duckdb_table_function_add_parameter adds a parameter to the table function syntax the table function type the type of the parameter to add duckdb_table_function_set_extra_info assigns extra information to the table function that can be fetched during binding etc syntax the table function extra_info the extra information destroy the callback that will be called to destroy the bind data if any duckdb_table_function_set_bind sets the bind function of the table function syntax the table function bind the bind function duckdb_table_function_set_init sets the init function of the table function syntax the table function init the init function duckdb_table_function_set_local_init sets the thread-local init function of the table function syntax the table function init the init function duckdb_table_function_set_function sets the main function of the table function syntax the table function function the function duckdb_table_function_supports_projection_pushdown sets whether or not the given table function supports projection pushdown if this is set to true the system will provide a list of all required columns in the init stage through the duckdb_init_get_column_count and duckdb_init_get_column_index functions if this is set to false the default the system will expect all columns to be projected syntax the table function pushdown true if the table function supports projection pushdown false otherwise duckdb_register_table_function register the table function object within the given connection the function requires at least a name a bind function an init function and a main function if the function is incomplete or a function with this name already exists duckdberror is returned syntax the connection to register it in function the function pointer returns whether or not the registration was successful duckdb_bind_get_extra_info retrieves the extra info of the function as set in duckdb_table_function_set_extra_info syntax the info object returns the extra info duckdb_bind_add_result_column adds a result column to the output of the table function syntax the info object name the name of the column type the logical type of the column duckdb_bind_get_parameter_count retrieves the number of regular non-named parameters to the function syntax the info object returns the number of parameters duckdb_bind_get_parameter retrieves the parameter at the given index the result must be destroyed with duckdb_destroy_value syntax the info object index the index of the parameter to get returns the value of the parameter must be destroyed with duckdb_destroy_value duckdb_bind_set_bind_data sets the user-provided bind data in the bind object this object can be retrieved again during execution syntax the info object extra_data the bind data object destroy the callback that will be called to destroy the bind data if any duckdb_bind_set_cardinality sets the cardinality estimate for the table function used for optimization syntax the bind data object is_exact whether or not the cardinality estimate is exact or an approximation duckdb_bind_set_error report that an error has occurred while calling bind syntax the info object error the error message duckdb_init_get_extra_info retrieves the extra info of the function as set in duckdb_table_function_set_extra_info syntax the info object returns the extra info duckdb_init_get_bind_data gets the bind data set by duckdb_bind_set_bind_data during the bind note that the bind data should be considered as read-only for tracking state use the init data instead syntax the info object returns the bind data object duckdb_init_set_init_data sets the user-provided init data in the init object this object can be retrieved again during execution syntax the info object extra_data the init data object destroy the callback that will be called to destroy the init data if any duckdb_init_get_column_count returns the number of projected columns this function must be used if projection pushdown is enabled to figure out which columns to emit syntax the info object returns the number of projected columns duckdb_init_get_column_index returns the column index of the projected column at the specified position this function must be used if projection pushdown is enabled to figure out which columns to emit syntax the info object column_index the index at which to get the projected column index from 0 duckdb_init_get_column_count info returns the column index of the projected column duckdb_init_set_max_threads sets how many threads can process this table function in parallel default 1 syntax the info object max_threads the maximum amount of threads that can process this table function duckdb_init_set_error report that an error has occurred while calling init syntax the info object error the error message duckdb_function_get_extra_info retrieves the extra info of the function as set in duckdb_table_function_set_extra_info syntax the info object returns the extra info duckdb_function_get_bind_data gets the bind data set by duckdb_bind_set_bind_data during the bind note that the bind data should be considered as read-only for tracking state use the init data instead syntax the info object returns the bind data object duckdb_function_get_init_data gets the init data set by duckdb_init_set_init_data during the init syntax the info object returns the init data object duckdb_function_get_local_init_data gets the thread-local init data set by duckdb_init_set_init_data during the local_init syntax the info object returns the init data object duckdb_function_set_error report that an error has occurred while executing the function syntax the info object error the error message duckdb_add_replacement_scan add a replacement scan definition to the specified database syntax the database object to add the replacement scan to replacement the replacement scan callback extra_data extra data that is passed back into the specified callback delete_callback the delete callback to call on the extra data if any duckdb_replacement_scan_set_function_name sets the replacement function name to use if this function is called in the replacement callback the replacement scan is performed if it is not called the replacement callback is not performed syntax the info object function_name the function name to substitute duckdb_replacement_scan_add_parameter adds a parameter to the replacement scan function syntax the info object parameter the parameter to add duckdb_appender_create creates an appender object syntax the connection context to create the appender in schema the schema of the table to append to or nullptr for the default schema table the table name to append to out_appender the resulting appender object returns duckdbsuccess on success or duckdberror on failure duckdb_appender_error returns the error message associated with the given appender if the appender has no error message this returns nullptr instead the error message should not be freed it will be de-allocated when duckdb_appender_destroy is called syntax the appender to get the error from returns the error message or nullptr if there is none duckdb_appender_flush flush the appender to the table forcing the cache of the appender to be cleared and the data to be appended to the base table this should generally not be used unless you know what you are doing instead call duckdb_appender_destroy when you are done with the appender syntax the appender to flush returns duckdbsuccess on success or duckdberror on failure duckdb_appender_close close the appender flushing all intermediate state in the appender to the table and closing it for further appends this is generally not necessary call duckdb_appender_destroy instead syntax the appender to flush and close returns duckdbsuccess on success or duckdberror on failure duckdb_appender_destroy close the appender and destroy it flushing all intermediate state in the appender to the table and de-allocating all memory associated with the appender syntax the appender to flush close and destroy returns duckdbsuccess on success or duckdberror on failure duckdb_appender_begin_row a nop function provided for backwards compatibility reasons does nothing only duckdb_appender_end_row is required syntax duckdb_appender_end_row finish the current row of appends after end_row is called the next row can be appended syntax the appender returns duckdbsuccess on success or duckdberror on failure duckdb_append_bool append a bool value to the appender syntax duckdb_append_int8 append an int8_t value to the appender syntax duckdb_append_int16 append an int16_t value to the appender syntax duckdb_append_int32 append an int32_t value to the appender syntax duckdb_append_int64 append an int64_t value to the appender syntax duckdb_append_hugeint append a duckdb_hugeint value to the appender syntax duckdb_append_uint8 append a uint8_t value to the appender syntax duckdb_append_uint16 append a uint16_t value to the appender syntax duckdb_append_uint32 append a uint32_t value to the appender syntax duckdb_append_uint64 append a uint64_t value to the appender syntax duckdb_append_float append a float value to the appender syntax duckdb_append_double append a double value to the appender syntax duckdb_append_date append a duckdb_date value to the appender syntax duckdb_append_time append a duckdb_time value to the appender syntax duckdb_append_timestamp append a duckdb_timestamp value to the appender syntax duckdb_append_interval append a duckdb_interval value to the appender syntax duckdb_append_varchar append a varchar value to the appender syntax duckdb_append_varchar_length append a varchar value to the appender syntax duckdb_append_blob append a blob value to the appender syntax duckdb_append_null append a null value to the appender of any type syntax duckdb_append_data_chunk appends a pre-filled data chunk to the specified appender the types of the data chunk must exactly match the types of the table no casting is performed if the types do not match or the appender is in an invalid state duckdberror is returned if the append is successful duckdbsuccess is returned syntax the appender to append to chunk the data chunk to append returns the return state duckdb_query_arrow executes a sql query within a connection and stores the full materialized result in an arrow structure if the query fails to execute duckdberror is returned and the error message can be retrieved by calling duckdb_query_arrow_error note that after running duckdb_query_arrow duckdb_destroy_arrow must be called on the result object even if the query fails otherwise the error stored within the result will not be freed correctly syntax the connection to perform the query in query the sql query to run out_result the query result returns duckdbsuccess on success or duckdberror on failure duckdb_query_arrow_schema fetch the internal arrow schema from the arrow result syntax the result to fetch the schema from out_schema the output schema returns duckdbsuccess on success or duckdberror on failure duckdb_query_arrow_array fetch an internal arrow array from the arrow result this function can be called multiple time to get next chunks which will free the previous out_array so consume the out_array before calling this function again syntax the result to fetch the array from out_array the output array returns duckdbsuccess on success or duckdberror on failure duckdb_arrow_column_count returns the number of columns present in a the arrow result object syntax the result object returns the number of columns present in the result object duckdb_arrow_row_count returns the number of rows present in a the arrow result object syntax the result object returns the number of rows present in the result object duckdb_arrow_rows_changed returns the number of rows changed by the query stored in the arrow result this is relevant only for insert update delete queries for other queries the rows_changed will be 0 syntax the result object returns the number of rows changed duckdb_query_arrow_error returns the error message contained within the result the error is only set if duckdb_query_arrow returns duckdberror the error message should not be freed it will be de-allocated when duckdb_destroy_arrow is called syntax the result object to fetch the nullmask from returns the error of the result duckdb_destroy_arrow closes the result and de-allocates all memory allocated for the arrow result syntax the result to destroy duckdb_execute_tasks execute duckdb tasks on this thread will return after max_tasks have been executed or if there are no more tasks present syntax the database object to execute tasks for max_tasks the maximum amount of tasks to execute duckdb_create_task_state creates a task state that can be used with duckdb_execute_tasks_state to execute tasks until duckdb_finish_execution is called on the state duckdb_destroy_state should be called on the result in order to free memory syntax the database object to create the task state for returns the task state that can be used with duckdb_execute_tasks_state duckdb_execute_tasks_state execute duckdb tasks on this thread the thread will keep on executing tasks forever until duckdb_finish_execution is called on the state multiple threads can share the same duckdb_task_state syntax the task state of the executor duckdb_execute_n_tasks_state execute duckdb tasks on this thread the thread will keep on executing tasks until either duckdb_finish_execution is called on the state max_tasks tasks have been executed or there are no more tasks to be executed multiple threads can share the same duckdb_task_state syntax the task state of the executor max_tasks the maximum amount of tasks to execute returns the amount of tasks that have actually been executed duckdb_finish_execution finish execution on a specific task syntax the task state to finish execution duckdb_task_state_is_finished check if the provided duckdb_task_state has finished execution syntax the task state to inspect returns whether or not duckdb_finish_execution has been called on the task state duckdb_destroy_task_state destroys the task state returned from duckdb_create_task_state note that this should not be called while there is an active duckdb_execute_tasks_state running on the task state syntax the task state to clean up",
- "category": "C",
- "url": "../docs/api/c/api",
- "blurb": "API Reference Open/Connect Syntax Path to the database file on disk, or nullptr or :memory: to open an..."
+ "title": "Date Format",
+ "text": "the strftime and strptime functions can be used to convert between dates timestamps and strings this is often required when parsing csv files displaying output to the user or transferring information between programs because there are many possible date representations these functions accept a format string that describes how the date or timestamp should be structured strftime examples strftime timestamp format converts timestamps or dates to strings according to the specified pattern select strftime date 1992-03-02 d m y -- 02 03 1992 select strftime timestamp 1992-03-02 20 32 45 a -d b y - i m s p -- monday 2 march 1992 - 08 32 45 pm strptime examples strptime string format converts strings to timestamps according to the specified pattern select strptime 02 03 1992 d m y -- 1992-03-02 00 00 00 select strptime monday 2 march 1992 - 08 32 45 pm a -d b y - i m s p -- 1992-03-02 20 32 45 csv parsing the date formats can also be specified during csv parsing either in the copy statement or in the read_csv function this can be done by either specifying a dateformat or a timestampformat or both dateformat will be used for converting dates and timestampformat will be used for converting timestamps below are some examples for how to use this -- in copy statement copy dates from test csv dateformat d m y timestampformat a -d b y - i m s p -- in read_csv function select from read_csv test csv dateformat m d y format specifiers below is a full list of all available format specifiers specifier description example --- --- --- --- a abbreviated weekday name sun mon a full weekday name sunday monday w weekday as a decimal number 0 1 6 d day of the month as a zero-padded decimal 01 02 31 -d day of the month as a decimal number 1 2 30 b abbreviated month name jan feb dec b full month name january february m month as a zero-padded decimal number 01 02 12 -m month as a decimal number 1 2 12 y year without century as a zero-padded decimal number 00 01 99 -y year without century as a decimal number 0 1 99 y year with century as a decimal number 2013 2019 etc h hour 24-hour clock as a zero-padded decimal number 00 01 23 -h hour 24-hour clock as a decimal number 0 1 23 i hour 12-hour clock as a zero-padded decimal number 01 02 12 -i hour 12-hour clock as a decimal number 1 2 12 p locale s am or pm am pm m minute as a zero-padded decimal number 00 01 59 -m minute as a decimal number 0 1 59 s second as a zero-padded decimal number 00 01 59 -s second as a decimal number 0 1 59 g millisecond as a decimal number zero-padded on the left 000 - 999 f microsecond as a decimal number zero-padded on the left 000000 - 999999 z utc offset in the form hhmm or -hhmm z time zone name j day of the year as a zero-padded decimal number 001 002 366 -j day of the year as a decimal number 1 2 366 u week number of the year sunday as the first day of the week 00 01 53 w week number of the year monday as the first day of the week 00 01 53 c iso date and time representation 1992-03-02 10 30 20 x iso date representation 1992-03-02 x iso time representation 10 30 20 a literal character",
+ "category": "Functions",
+ "url": "../docs/sql/functions/dateformat",
+ "blurb": "The strftime and strptime functions can be used to convert between dates/timestamps and strings. This is often..."
},
{
- "title": "Client APIs Overview",
- "text": "there are various client apis for duckdb duckdb s native api is c with official wrappers available for c python r java node js webassembly wasm odbc api julia and a command line interface cli there are also contributed third-party duckdb wrappers for ruby go c and rust pages in this section",
- "category": "Api",
- "url": "../docs/api/overview",
- "blurb": "There are various client APIs for DuckDB. DuckDB's native API is C++ , with official wrappers available for C , ..."
+ "title": "Pattern Matching",
+ "text": "pattern matching there are four separate approaches to pattern matching provided by duckdb the traditional sql like operator the more recent similar to operator added in sql 1999 a glob operator and posix-style regular expressions like the like expression returns true if the string matches the supplied pattern as expected the not like expression returns false if like returns true and vice versa an equivalent expression is not string like pattern if pattern does not contain percent signs or underscores then the pattern only represents the string itself in that case like acts like the equals operator an underscore _ in pattern stands for matches any single character a percent sign matches any sequence of zero or more characters like pattern matching always covers the entire string therefore if it s desired to match a sequence anywhere within a string the pattern must start and end with a percent sign some examples abc like abc -- true abc like a -- true abc like _b_ -- true abc like c -- false abc like c -- false abc like c -- true abc not like c -- false the keyword ilike can be used instead of like to make the match case-insensitive according to the active locale abc ilike c -- true abc not ilike c -- false to search within a string for a character that is a wildcard or _ the pattern must use an escape clause and an escape character to indicate the wildcard should be treated as a literal character instead of a wildcard see an example below additionally the function like_escape has the same functionality as a like expression with an escape clause but using function syntax see the text functions docs for details --search for strings with a then a literal percent sign then c a c like a c escape -- true azc like a c escape -- false --case insensitive ilike with escape a c ilike a c escape --true there are also alternative characters that can be used as keywords in place of like expressions these enhance postgres compatibility like-style postgres-style --- --- like not like ilike not ilike similar to the similar to operator returns true or false depending on whether its pattern matches the given string it is similar to like except that it interprets the pattern using a regular expression like like the similar to operator succeeds only if its pattern matches the entire string this is unlike common regular expression behavior where the pattern can match any part of the string a regular expression is a character sequence that is an abbreviated definition of a set of strings a regular set a string is said to match a regular expression if it is a member of the regular set described by the regular expression as with like pattern characters match string characters exactly unless they are special characters in the regular expression language but regular expressions use different special characters than like does some examples abc similar to abc -- true abc similar to a -- false abc similar to b d -- true abc similar to b c -- false abc not similar to abc -- false there are also alternative characters that can be used as keywords in place of similar to expressions these follow posix syntax similar to-style posix-style --- --- similar to not similar to glob the glob operator returns true or false if the string matches the glob pattern the glob operator is most commonly used when searching for filenames that follow a specific pattern for example a specific file extension use the question mark wildcard to match any single character and use the asterisk to match zero or more characters in addition use bracket syntax to match any single character contained within the brackets or within the character range specified by the brackets an exclamation mark may be used inside the first bracket to search for a character that is not contained within the brackets to learn more visit the glob programming wikipedia page some examples best txt glob txt -- true best txt glob txt -- true best txt glob txt -- false best txt glob abc est txt -- true best txt glob a-z est txt -- true -- the bracket syntax is case sensitive best txt glob a-z est txt -- false best txt glob a-za-z est txt -- true -- the applies to all characters within the brackets best txt glob a-za-z est txt -- false -- to negate a glob operator negate the entire expression -- not glob is not valid syntax not best txt glob txt -- false three tildes may also be used in place of the glob keyword glob-style symbolic-style --- --- glob regular expressions function description --- --- regexp_matches string pattern returns true if string contains the regexp pattern false otherwise regexp_replace string pattern replacement if string contains the regexp pattern replaces the matching part with replacement regexp_extract string pattern idx if string contains the regexp pattern returns the capturing group specified by optional parameter idx the regexp_matches function is similar to the similar to operator however it does not require the entire string to match instead regexp_matches returns true if the string merely contains the pattern unless the special tokens and are used to anchor the regular expression to the start and end of the string below are some examples regexp_matches abc abc -- true regexp_matches abc abc -- true regexp_matches abc a -- true regexp_matches abc a -- false regexp_matches abc b d -- true regexp_matches abc b c -- true regexp_matches abc b c -- false the regexp_replace function can be used to replace the part of a string that matches the regexp pattern with a replacement string the notation d where d is a number indicating the group can be used to refer to groups captured in the regular expression in the replacement string below are some examples regexp_replace abc b c x -- axc regexp_replace abc b c 1 1 1 1 -- abbbbc regexp_replace abc c 1e -- abe regexp_replace abc a b 2 1 -- bac the regexp_extract function is used to extract a part of a string that matches the regexp pattern a specific capturing group within the pattern can be extracted using the idx parameter if idx is not specified it defaults to 0 extracting the first match with the whole pattern regexp_extract abc b -- abc regexp_extract abc b 0 -- abc regexp_extract abc b 1 -- empty regexp_extract abc a-z b 1 -- a regexp_extract abc a-z b 2 -- b",
+ "category": "Functions",
+ "url": "../docs/sql/functions/patternmatching",
+ "blurb": "Pattern Matching There are four separate approaches to pattern matching provided by DuckDB: the traditional SQL LIKE ..."
},
{
- "title": "R API",
- "text": "installation the duckdb r api can be installed using install packages please see the installation page for details basic api usage the standard duckdb r api implements the dbi interface for r if you are not familiar with dbi yet see here for an introduction startup shutdown to use duckdb you must first create a connection object that represents the database the connection object takes as parameter the database file to read and write from if the database file does not exist it will be created the file extension may be db duckdb or anything else the special value memory the default can be used to create an in-memory database note that for an in-memory database no data is persisted to disk i e all data is lost when you exit the r process if you would like to connect to an existing database in read-only mode set the read_only flag to true read-only mode is required if multiple r processes want to access the same database file at the same time library dbi to start an in-memory database con dbconnect duckdb duckdb dbdir memory to use a database file not shared between processes con dbconnect duckdb duckdb dbdir my-db duckdb read_only false to use a database file shared between processes con dbconnect duckdb duckdb dbdir my-db duckdb read_only true connections are closed implicitly when they go out of scope or if they are explicitly closed using dbdisconnect to shut down the database instance associated with the connection use dbdisconnect con shutdown true querying duckdb supports the standard dbi methods to send queries and retreive result sets dbexecute is meant for queries where no results are expected like create table or update etc and dbgetquery is meant to be used for queries that produce results e g select below an example create a table dbexecute con create table items item varchar value decimal 10 2 count integer insert two items into the table dbexecute con insert into items values jeans 20 0 1 hammer 42 2 2 retrieve the items again res dbgetquery con select from items print res item value count 1 jeans 20 0 1 2 hammer 42 2 2 duckdb also supports prepared statements in the r api with the dbexecute and dbgetquery methods here is an example prepared statement parameters are given as a list dbexecute con insert into items values list laptop 2000 1 if you want to reuse a prepared statement multiple times use dbsendstatement and dbbind stmt dbsendstatement con insert into items values dbbind stmt list iphone 300 2 dbbind stmt list android 3 5 1 dbclearresult stmt query the database using a prepared statement res dbgetquery con select item from items where value list 400 print res item 1 laptop do not use prepared statements to insert large amounts of data into duckdb see below for better options efficient transfer to write a r data frame into duckdb use the standard dbi function dbwritetable this creates a table in duckdb and populates it with the data frame contents for example dbwritetable con iris_table iris res dbgetquery con select from iris_table limit 1 print res sepal length sepal width petal length petal width species 1 5 1 3 5 1 4 0 2 setosa it is also possible to register a r data frame as a virtual table comparable to a sql view this does not actually transfer data into duckdb yet below is an example duckdb duckdb_register con iris_view iris res dbgetquery con select from iris_view limit 1 print res sepal length sepal width petal length petal width species 1 5 1 3 5 1 4 0 2 setosa duckdb keeps a reference to the r data frame after registration this prevents the data frame from being garbage-collected the reference is cleared when the connection is closed but can also be cleared manually using the duckdb duckdb_unregister method also refer to the data import documentation for more options of efficiently importing data dbplyr duckdb also plays well with the dbplyr dplyr packages for programmatic query construction from r here is an example library dbi library dplyr con - dbconnect duckdb duckdb duckdb duckdb_register con flights nycflights13 flights tbl con flights group_by dest summarise delay mean dep_time",
- "category": "Api",
- "url": "../docs/api/r",
- "blurb": "Installation The DuckDB R API can be installed using install.packages . Please see the installation page for..."
+ "title": "Date Parts",
+ "text": "the date_part and date_diff and date_trunc functions can be used to manipulate the fields of temporal types the fields are specified as strings that contain the part name of the field part specifiers below is a full list of all available date part specifiers the examples are the corresponding parts of the timestamp 2021-08-03 11 59 44 123456 usable as date part specifiers and in intervals specifier description synonyms example --- --- --- --- year gregorian year y years 2021 month gregorian month mon months mons 8 day gregorian day days d dayofmonth 3 decade gregorian decade decades 202 century gregorian century centuries 21 millennium gregorian millennium millenia millenium 3 quarter quarter of the year 1-4 quarters 3 microseconds sub-minute microseconds microsecond 44123456 milliseconds sub-minute milliseconds millisecond ms msec msecs 44123 second seconds seconds s 44 minute minutes minutes m 59 hour hours hours h 11 usable in date part specifiers only specifier description synonyms example --- --- --- --- epoch seconds since 1970-01-01 1627991984 dayofweek day of the week sunday 0 saturday 6 weekday dow 2 isodow iso day of the week monday 1 sunday 7 2 isoyear iso year number starts on monday of week containing jan 4th 2021 week week number weeks w 31 yearweek iso year and week number in yyyyww format 202131 dayofyear day of the year 1-365 366 doy 215 era gregorian era ce ad bce bc 1 timezone time zone offset in seconds 0 timezone_hour time zone offset hour portion 0 timezone_minute time zone offset minute portion 0 note that the time zone parts are all zero unless a time zone plugin such as icu has been installed to support timestamp with time zone part functions there are dedicated extraction functions to get certain subfields function description example result --- --- --- --- year date year year date 1992-02-15 1992 month date month month date 1992-02-15 2 day date day day date 1992-02-15 15 dayofmonth date day synonym dayofmonth date 1992-02-15 15 decade date decade year 10 decade date 1992-02-15 199 century date century century date 1992-02-15 20 millennium date millennium millennium date 1992-02-15 2 quarter date quarter quarter date 1992-02-15 1 microsecond date sub-minute microseconds microsecond timestamp 2021-08-03 11 59 44 123456 44123456 millisecond date sub-minute milliseconds millisecond timestamp 2021-08-03 11 59 44 123456 44123 second date seconds second timestamp 2021-08-03 11 59 44 123456 44 minute date minutes minute timestamp 2021-08-03 11 59 44 123456 59 hour date hours hour timestamp 2021-08-03 11 59 44 123456 11 epoch date seconds since 1970-01-01 epoch date 1992-02-15 698112000 dayofweek date numeric weekday sunday 0 saturday 6 dayofweek date 1992-02-15 6 weekday date numeric weekday synonym sunday 0 saturday 6 weekday date 1992-02-15 6 isodow date numeric iso weekday monday 1 sunday 7 isodow date 1992-02-15 6 isoyear date iso year number starts on monday of week containing jan 4th isoyear date 2022-01-01 2021 week date iso week week date 1992-02-15 7 weekofyear date iso week synonym weekofyear date 1992-02-15 7 yearweek date bigint of combined iso year number and 2-digit version of iso week number yearweek date 1992-02-15 199207 dayofyear date numeric iso weekday monday 1 sunday 7 isodow date 1992-02-15 46 quarter date quarter quarter date 1992-02-15 1 era date calendar era era date 0044-03-15 bc 0 timezone date time zone offset in minutes timezone date 1992-02-15 0 timezone_hour date time zone offset hour portion timezone_hour date 1992-02-15 0 timezone_minute date time zone offset minutes portion timezone_minute date 1992-02-15 0",
+ "category": "Functions",
+ "url": "../docs/sql/functions/datepart",
+ "blurb": "The date_part and date_diff and date_trunc functions can be used to manipulate the fields of temporal types. The..."
},
{
- "title": "WebAssembly API",
- "text": "getting started with duckdb-wasm a great starting point is to read the duckdb-wasm launch blog post another great resource is the github repository for details see the full duckdb-wasm api documentation",
- "category": "Api",
- "url": "../docs/api/wasm",
- "blurb": "Getting Started with DuckDB-WASM A great starting point is to read the DuckDB-WASM launch blog post ! Another great..."
+ "title": "Timestamp Functions",
+ "text": "this section describes functions and operators for examining and manipulating timestamp values timestamp operators the table below shows the available mathematical operators for timestamp types operator description example result --- --- --- --- addition of an interval timestamp 1992-03-22 01 02 03 interval 5 day 1992-03-27 01 02 03 - subtraction of timestamp s timestamp 1992-03-27 - timestamp 1992-03-22 5 days - subtraction of an interval timestamp 1992-03-27 01 02 03 - interval 5 day 1992-03-22 01 02 03 adding to or subtracting from infinite values produces the same infinite value timestamp functions the table below shows the available scalar functions for timestamp types function description example result --- --- --- --- age timestamp timestamp subtract arguments resulting in the time difference between the two timestamps age timestamp 2001-04-10 timestamp 1992-09-20 8 years 6 months 20 days age timestamp subtract from current_date age timestamp 1992-09-20 29 years 1 month 27 days 12 39 00 844 century timestamp extracts the century of a timestamp century timestamp 1992-03-22 20 current_timestamp current date and time start of current transaction date_diff part startdate enddate the number of partition boundaries between the timestamps date_diff hour timestamp 1992-09-30 23 59 59 timestamp 1992-10-01 01 58 00 2 date_part part timestamp get subfield equivalent to extract date_part minute timestamp 1992-09-20 20 38 40 38 date_sub part startdate enddate the number of complete partitions between the timestamps date_sub hour timestamp 1992-09-30 23 59 59 timestamp 1992-10-01 01 58 00 1 date_trunc part timestamp truncate to specified precision date_trunc hour timestamp 1992-09-20 20 38 40 1992-09-20 20 00 00 dayname timestamp the english name of the weekday dayname timestamp 1992-03-22 sunday epoch_ms ms converts ms since epoch to a timestamp epoch_ms 701222400000 1992-03-22 00 00 00 extract field from timestamp get subfield from a timestamp extract hour from timestamp 1992-09-20 20 38 48 20 greatest timestamp timestamp the later of two timestamps greatest timestamp 1992-09-20 20 38 48 timestamp 1992-03-22 01 02 03 1234 1992-09-20 20 38 48 isfinite timestamp returns true if the timestamp is finite false otherwise isfinite timestamp 1992-03-07 true isinf timestamp returns true if the timestamp is infinite false otherwise isinf timestamp -infinity true last_day timestamp the last day of the month last_day timestamp 1992-03-22 01 02 03 1234 1992-03-31 least timestamp timestamp the earlier of two timestamps least timestamp 1992-09-20 20 38 48 timestamp 1992-03-22 01 02 03 1234 1992-03-22 01 02 03 1234 make_timestamp bigint bigint bigint bigint bigint double the timestamp for the given parts make_timestamp 1992 9 20 13 34 27 123456 1992-09-20 13 34 27 123456 monthname timestamp the english name of the month monthname timestamp 1992-09-20 september now current date and time start of current transaction strftime timestamp format converts timestamp to string according to the format string strftime timestamp 1992-01-01 20 38 40 a -d b y - i m s p wed 1 january 1992 - 08 38 40 pm strptime text format converts string to timestamp according to the format string strptime wed 1 january 1992 - 08 38 40 pm a -d b y - i m s p 1992-01-01 20 38 40 to_timestamp sec converts sec since epoch to a timestamp to_timestamp 701222400 1992-03-22 00 00 00 there are also dedicated extraction functions to get the subfields functions applied to infinite dates will either return the same infinite dates e g greatest or null e g date_part depending on what makes sense in general if the function needs to examine the parts of the infinite date the result will be null timestamp table functions the table below shows the available table functions for timestamp types function description example --- --- --- --- generate_series timestamp timestamp interval generate a table of timestamps in the closed range stepping by the interval generate_series timestamp 2001-04-10 timestamp 2001-04-11 interval 30 minute range timestamp timestamp interval generate a table of timestamps in the half open range stepping by the interval range timestamp 2001-04-10 timestamp 2001-04-11 interval 30 minute infinite values are not allowed as table function bounds",
+ "category": "Functions",
+ "url": "../docs/sql/functions/timestamp",
+ "blurb": "This section describes functions and operators for examining and manipulating TIMESTAMP values. Timestamp Operators ..."
},
{
- "title": "Python Client API",
- "text": "",
- "category": "Reference",
- "url": "../docs/api/python/reference/index",
- "blurb": ""
+ "title": "Time Functions",
+ "text": "this section describes functions and operators for examining and manipulating time values time operators the table below shows the available mathematical operators for time types operator description example result --- --- --- --- addition of an interval time 01 02 03 interval 5 hour 06 02 03 - subtraction of an interval time 06 02 03 - interval 5 hour 01 02 03 time functions the table below shows the available scalar functions for time types function description example result --- --- --- --- current_time current time start of current transaction date_diff part starttime endtime the number of partition boundaries between the times date_diff hour time 01 02 03 time 06 01 03 5 date_part part time get subfield equivalent to extract date_part minute time 14 21 13 21 date_sub part starttime endtime the number of complete partitions between the times date_sub hour time 01 02 03 time 06 01 03 4 extract part from time get subfield from a time extract hour from time 14 21 13 14 make_time bigint bigint double the time for the given parts make_time 13 34 27 123456 13 34 27 123456 the only date parts that are defined for times are epoch hours minutes seconds milliseconds and microseconds",
+ "category": "Functions",
+ "url": "../docs/sql/functions/time",
+ "blurb": "This section describes functions and operators for examining and manipulating TIME values. Time Operators The table..."
},
{
- "title": "Python API",
- "text": "installation the duckdb python api can be installed using pip pip install duckdb please see the installation page for details it is also possible to install duckdb using conda conda install python-duckdb -c conda-forge basic api usage the standard duckdb python api provides a sql interface compliant with the db-api 2 0 specification described by pep 249 similar to the sqlite python api startup shutdown to use the module you must first create a connection object that represents the database the connection object takes as parameter the database file to read and write from if the database file does not exist it will be created the file extension may be db duckdb or anything else the special value memory the default can be used to create an in-memory database note that for an in-memory database no data is persisted to disk i e all data is lost when you exit the python process if you would like to connect to an existing database in read-only mode you can set the read_only flag to true read-only mode is required if multiple python processes want to access the same database file at the same time import duckdb to start an in-memory database con duckdb connect database memory to use a database file not shared between processes con duckdb connect database my-db duckdb read_only false to use a database file shared between processes con duckdb connect database my-db duckdb read_only true if you want to create a second connection to an existing database you can use the cursor method this might be useful for example to allow parallel threads running queries independently a single connection is thread-safe but is locked for the duration of the queries effectively serializing database access in this case connections are closed implicitly when they go out of scope or if they are explicitly closed using close once the last connection to a database instance is closed the database instance is closed as well querying sql queries can be sent to duckdb using the execute method of connections once a query has been executed results can be retrieved using the fetchone and fetchall methods on the connection below is a short example create a table con execute create table items item varchar value decimal 10 2 count integer insert two items into the table con execute insert into items values jeans 20 0 1 hammer 42 2 2 retrieve the items again con execute select from items print con fetchall jeans 20 0 1 hammer 42 2 2 the description property of the connection object contains the column names as per the standard duckdb also supports prepared statements in the api with the execute and executemany methods in this case the parameters are passed as an additional parameter after a query that contains placeholders here is an example insert a row using prepared statements con execute insert into items values laptop 2000 1 insert several rows using prepared statements con executemany insert into items values chainsaw 500 10 iphone 300 2 query the database using a prepared statement con execute select item from items where value 400 print con fetchall laptop chainsaw do not use executemany to insert large amounts of data into duckdb see below for better options efficient transfer transferring large datasets to and from duckdb uses a separate api built around numpy and pandas or apache arrow this api works with entire columns of data instead of scalar values and is therefore far more efficient by default duckdb will automatically be able to query a pandas dataframe or arrow object that is stored in a python variable by name duckdb supports querying multiple types of apache arrow objects including tables datasets recordbatchreaders and scanners see the python guides for more examples duckdb also supports registering a dataframe or arrow object as a virtual table comparable to a sql view this is useful when querying a dataframe arrow object that is stored in another way as a class variable or a value in a dictionary below is a pandas example import pandas as pd test_df pd dataframe from_dict i 1 2 3 4 j one two three four con execute select from test_df con fetchall 1 one 2 two 3 three 4 four if your pandas dataframe is stored in another location here is an example of manually registering it import pandas as pd my_dictionary my_dictionary test_df pd dataframe from_dict i 1 2 3 4 j one two three four con register test_df_view my_dictionary test_df con execute select from test_df_view con fetchall 1 one 2 two 3 three 4 four pyarrow tables work in much the same way import pyarrow as pa arrow_table pa table from_pydict i 1 2 3 4 j one two three four con execute select from arrow_table con fetchall you can also create a persistent table in duckdb from the contents of the dataframe or the view con execute create table test_df_table as select from test_df duckdb keeps a reference to the pandas dataframe or arrow object after registration this prevents them from being garbage-collected by the python runtime the reference is cleared when the connection is closed but can also be cleared manually using the unregister method when retrieving the data from duckdb back into python the standard method of calling fetchall is inefficient as individual python objects need to be created for every value in the result set when retrieving a lot of data this can become very slow duckdb s python client provides multiple additional methods that can be used to efficiently retrieve data numpy fetchnumpy fetches the data as a dictionary of numpy arrays pandas df fetches the data as a pandas dataframe fetchdf is an alias of df fetch_df is an alias of df fetch_df_chunk vector_multiple fetches a portion of the results into a dataframe the number of rows returned in each chunk is the vector size 1024 by default vector_multiple 1 by default apache arrow arrow fetches the data as an arrow table fetch_arrow_table is an alias of arrow fetch_record_batch chunk_size returns an arrow record batch reader with chunk_size rows per batch below are some examples using this functionality see the python guides for more examples fetch as pandas dataframe df con execute select from items fetchdf print df item value count 0 jeans 20 0 1 1 hammer 42 2 2 2 laptop 2000 0 1 3 chainsaw 500 0 10 4 iphone 300 0 2 fetch as dictionary of numpy arrays arr con execute select from items fetchnumpy print arr item masked_array data jeans hammer laptop chainsaw iphone mask false false false false false fill_value dtype object value masked_array data 20 0 42 2 2000 0 500 0 300 0 mask false false false false false fill_value 1e 20 count masked_array data 1 2 1 10 2 mask false false false false false fill_value 999999 dtype int32 fetch as an arrow table converting to pandas afterwards just for pretty printing tbl con execute select from items fetch_arrow_table print tbl to_pandas item value count 0 jeans 20 00 1 1 hammer 42 20 2 2 laptop 2000 00 1 3 chainsaw 500 00 10 4 iphone 300 00 2 also refer to the data import documentation for more options of efficiently importing data relational api in addition to the sql api duckdb supports a programmatic method to construct queries see https github com duckdb duckdb blob master examples python duckdb-python py for an example",
- "category": "Python",
- "url": "../docs/api/python/overview",
- "blurb": "Installation The DuckDB Python API can be installed using pip : pip install duckdb . Please see the installation..."
+ "title": "Utility Functions",
+ "text": "utility functions the functions below are difficult to categorize into specific function types and are broadly useful function description example result --- --- --- --- alias column return the name of the column alias column1 column1 coalesce expr return the first expression that evaluates to a non- null value accepts 1 or more parameters each expression can be a column literal value function result or many others coalesce null null default_string default_string current_schema return the name of the currently active schema default is main current_schema main current_schemas boolean return list of schemas pass a parameter of true to include implicit schemas current_schemas true temp main pg_catalog current_setting setting_name return the current value of the configuration setting current_setting access_mode automatic currval sequence_name return the current value of the sequence note that nextval must be called at least once prior to calling currval currval my_sequence_name 1 gen_random_uuid alias of uuid return a random uuid similar to this eeccb8c5-9943-b2bb-bb5e-222f4e14b687 gen_random_uuid various icu_sort_key string collator surrogate key used to sort special characters according to the specific locale collator parameter is optional valid only when icu extension is installed icu_sort_key ö de 460145960106 md5 string return an md5 one-way hash of the string md5 123 202cb962ac59075b964b07152d234b70 nextval sequence_name return the following value of the sequence nextval my_sequence_name 2 pg_typeof expression returns the lower case name of the data type of the result of the expression for postgres compatibility pg_typeof abc varchar stats expression returns a string with statistics about the expression expression can be a column constant or sql expression stats 5 min 5 max 5 has null false txid_current returns the current transaction s id a bigint it will assign a new one if the current transaction does not have one already txid_current various typeof expression returns the name of the data type of the result of the expression typeof abc varchar uuid return a random uuid similar to this eeccb8c5-9943-b2bb-bb5e-222f4e14b687 uuid various version return the currently active version of duckdb in this format v0 3 2 version various",
+ "category": "Functions",
+ "url": "../docs/sql/functions/utility",
+ "blurb": "Utility Functions The functions below are difficult to categorize into specific function types and are broadly useful. ..."
},
{
- "title": "Node.js API",
- "text": "duckdb node bindings this package provides a node js api for duckdb the sqlite for analytics the api for this client is somewhat compliant to the sqlite node js client for easier transition load the package and create a database object var duckdb require duckdb var db new duckdb database memory or a file name for a persistent db then you can run a query db all select 42 as fortytwo function err res if err throw err console log res 0 fortytwo other available methods are each where the callback is invoked for each row run to execute a single statement without results and exec which can execute several sql commands at once but also does not return results all those commands can work with prepared statements taking the values for the parameters as additional arguments for example like so db all select integer as fortytwo string as hello 42 hello world function err res if err throw err console log res 0 fortytwo console log res 0 hello however these are all shorthands for something much more elegant a database can have multiple connection s those are created using db connect var con db connect you can create multiple connections each with their own transaction context connection objects also contain shorthands to directly call run all and each with parameters and callbacks respectively for example con all select 42 as fortytwo function err res if err throw err console log res 0 fortytwo from connections you can create prepared statements and only that using con prepare var stmt con prepare select integer as fortytwo to execute this statement you can call for example all on the stmt object stmt all 42 function err res if err throw err console log res 0 fortytwo you can also execute the prepared statement multiple times this is for example useful to fill a table with data con run create table a i integer var stmt con prepare insert into a values for var i 0 i 10 i stmt run i stmt finalize con all select from a function err res if err throw err console log res prepare can also take a callback which gets the prepared statement as an argument var stmt con prepare select integer as fortytwo function err stmt stmt all 42 function err res if err throw err console log res 0 fortytwo",
- "category": "Nodejs",
- "url": "../docs/api/nodejs/overview",
- "blurb": "DuckDB Node Bindings This package provides a node.js API for DuckDB , the SQLite for Analytics. The API for this..."
+ "title": "Nested Functions",
+ "text": "this section describes functions and operators for examining and manipulating nested values there are three nested data types lists structs and maps list functions in the descriptions l is the three element list 4 5 6 function description example result -------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ --------------------------------------- ------------- list index bracket notation serves as an alias for list_extract l 3 6 list_extract list index extract the index th 1-based value from the list list_extract l 3 6 list_element list index alias for list_extract list_element l 3 6 array_extract list index alias for list_extract array_extract l 3 6 list begin end bracket notation with colon is an alias for list_slice missing arguments are interpreted as null s l 2 3 5 6 list_slice list begin end extract a sublist using slice conventions null s are interpreted as the bounds of the list negative values are accepted list_slice l 2 null 5 6 array_slice list begin end alias for list_slice array_slice l 2 null 5 6 array_pop_front list returns the list without the first element array_pop_front l 5 6 array_pop_back list returns the list without the last element array_pop_back l 4 5 list_value any create a list containing the argument values list_value 4 5 6 4 5 6 list_pack any alias for list_value list_pack 4 5 6 4 5 6 len list return the length of the list len 1 2 3 3 array_length list alias for len array_length 1 2 3 3 unnest list unnests a list by one level note that this is a special function that alters the cardinality of the result see the unnest page for more details unnest 1 2 3 1 2 3 list_concat list1 list2 concatenates two lists list_concat 2 3 4 5 6 2 3 4 5 6 list_cat list1 list2 alias for list_concat list_cat 2 3 4 5 6 2 3 4 5 6 array_concat list1 list2 alias for list_concat array_concat 2 3 4 5 6 2 3 4 5 6 array_cat list1 list2 alias for list_concat array_cat 2 3 4 5 6 2 3 4 5 6 list_prepend element list prepends element to list list_prepend 3 4 5 6 3 4 5 6 array_prepend element list alias for list_prepend array_prepend 3 4 5 6 3 4 5 6 array_push_front list element alias for list_prepend array_push_front l 3 3 4 5 6 list_append list element appends element to list list_append 2 3 4 2 3 4 array_append list element alias for list_append array_append 2 3 4 2 3 4 array_push_back list element alias for list_append array_push_back l 7 4 5 6 7 list_contains list element returns true if the list contains the element list_contains 1 2 null 1 true list_has list element alias for list_contains list_has 1 2 null 1 true array_contains list element alias for list_contains array_contains 1 2 null 1 true array_has list element alias for list_contains array_has 1 2 null 1 true list_position list element returns the index of the element if the list contains the element list_contains 1 2 null 2 2 list_indexof list element alias for list_position list_indexof 1 2 null 2 2 array_position list element alias for list_position array_position 1 2 null 2 2 array_indexof list element alias for list_position array_indexof 1 2 null 2 2 list_aggregate list name executes the aggregate function name on the elements of list see the list aggregates section for more details list_aggregate 1 2 null min 1 list_aggr list name alias for list_aggregate list_aggr 1 2 null min 1 array_aggregate list name alias for list_aggregate array_aggregate 1 2 null min 1 array_aggr list name alias for list_aggregate array_aggr 1 2 null min 1 list_sort list sorts the elements of the list see the sorting lists section for more details about the sorting order and the null sorting order list_sort 3 6 1 2 1 2 3 6 array_sort list alias for list_sort array_sort 3 6 1 2 1 2 3 6 list_reverse_sort list sorts the elements of the list in reverse order see the sorting lists section for more details about the null sorting order list_reverse_sort 3 6 1 2 6 3 2 1 array_reverse_sort list alias for list_reverse_sort array_reverse_sort 3 6 1 2 6 3 2 1 list_transform list lambda returns a list that is the result of applying the lambda function to each element of the input list see the lambda functions section for more details list_transform l x - x 1 5 6 7 array_transform list lambda alias for list_transform array_transform l x - x 1 5 6 7 list_apply list lambda alias for list_transform list_apply l x - x 1 5 6 7 array_apply list lambda alias for list_transform array_apply l x - x 1 5 6 7 list_filter list lambda constructs a list from those elements of the input list for which the lambda function returns true see the lambda functions section for more details list_filter l x - x 4 5 6 array_filter list lambda alias for list_filter array_filter l x - x 4 5 6 struct functions function description example result --- --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- --- --- struct entry dot notation serves as an alias for struct_extract i 3 s string s string struct entry bracket notation serves as an alias for struct_extract i 3 s string s string row any create a struct containing the argument values if the values are column references the entry name will be the column name otherwise it will be the string vn where n is the 1-based position of the argument row i i 4 i 4 i 3 v2 3 v3 0 struct_extract struct entry extract the named entry from the struct struct_extract s i 4 struct_pack name any create a struct containing the argument values the entry name will be the bound variable name struct_pack i 4 s string i 3 s string struct_insert struct name any add field s value s to an existing struct with the argument values the entry name s will be the bound variable name s struct_insert a 1 b 2 a 1 b 2 map functions function description example result --- --- --- --- map entry alias for element_at map 100 5 a b 100 a element_at map key return a list containing the value for a given key or an empty list if the key is not contained in the map the type of the key provided in the second parameter must match the type of the map s keys else an error is returned element_at map 100 5 42 43 100 42 map_extract map key alias of element_at return a list containing the value for a given key or an empty list if the key is not contained in the map the type of the key provided in the second parameter must match the type of the map s keys else an error is returned map_extract map 100 5 42 43 100 42 cardinality map return the size of the map or the number of entries in the map cardinality map 4 2 a b 2 map returns an empty map map range functions the functions range and generate_series create a list of values in the range between start and stop the start parameter is inclusive for the range function the stop parameter is exclusive while for generate_series it is inclusive based on the number of arguments the following variants exist range start stop step range start stop range stop generate_series start stop step generate_series start stop generate_series stop the default value of start is 0 and the default value of step is 1 select range 5 -- 0 1 2 3 4 select range 2 5 -- 2 3 4 select range 2 5 3 -- 2 select generate_series 5 -- 0 1 2 3 4 5 select generate_series 2 5 -- 2 3 4 5 select generate_series 2 5 3 -- 2 5 date ranges are also supported select from range date 1992-01-01 date 1992-03-01 interval 1 month range 1992-01-01 00 00 00 1992-02-01 00 00 00 list aggregates the function list_aggregate allows the execution of arbitrary existing aggregate functions on the elements of a list its first argument is the list column its second argument is the aggregate function name e g min histogram or sum select list_aggregate 1 2 -4 null min -- -4 select list_aggregate 2 4 8 42 sum -- 56 select list_aggregate 1 2 null 2 10 3 last -- 2 10 3 the following is a list of existing rewrites rewrites simplify the use of the list aggregate function by only taking the list column as their argument list_avg list_var_samp list_var_pop list_stddev_pop list_stddev_samp list_sem list_approx_count_distinct list_bit_xor list_bit_or list_bit_and list_bool_and list_bool_or list_count list_entropy list_last list_first list_kurtosis list_min list_max list_product list_skewness list_sum list_string_agg list_mode list_median list_mad and list_histogram select list_min 1 2 -4 null -- -4 select list_sum 2 4 8 42 -- 56 select list_last 1 2 null 2 10 3 -- 2 10 3 sorting lists the function list_sort sorts the elements of a list either in ascending or descending order in addition it allows to provide whether null values should be moved to the beginning or to the end of the list by default if no modifiers are provided duckdb sorts asc nulls first i e the values are sorted in ascending order and null values are placed first this is identical to the default sort order of sqlite the default sort order can be changed using these pragma statements list_sort leaves it open to the user whether they want to use the default sort order or a custom order list_sort takes up to two additional optional parameters the second parameter provides the sort order and can be either asc or desc the third parameter provides the null sort order and can be either nulls first or nulls last -- default sort order and default null sort order select list_sort 1 3 null 5 null -5 ---- null null -5 1 3 5 -- only providing the sort order select list_sort 1 3 null 2 asc ---- null 1 2 3 -- providing the sort order and the null sort order select list_sort 1 3 null 2 desc nulls first ---- null 3 2 1 list_reverse_sort has an optional second parameter providing the null sort order it can be either nulls first or nulls last -- default null sort order select list_sort 1 3 null 5 null -5 ---- null null -5 1 3 5 -- providing the null sort order select list_reverse_sort 1 3 null 2 nulls last ---- 3 2 1 null lambda functions parameter1 parameter2 - expression if the lambda function has only one parameter then the brackets can be omitted the parameters can have any names param - param 1 duck - contains concat duck db duck x y - x y transform list_transform list lambda returns a list that is the result of applying the lambda function to each element of the input list the lambda function must have exactly one left-hand side parameter the return type of the lambda function defines the type of the list elements -- incrementing each list element by one select list_transform 1 2 null 3 x - x 1 ---- 2 3 null 4 -- transforming strings select list_transform duck a b duck - concat duck db ---- duckdb adb bdb -- combining lambda functions with other functions select list_transform 5 null 6 x - coalesce x 0 1 ---- 6 1 7 filter list_filter list lambda constructs a list from those elements of the input list for which the lambda function returns true the lambda function must have exactly one left-hand side parameter and its return type must be of type boolean -- filter out negative values select list_filter 5 -6 null 7 x - x 0 ---- 5 7 -- divisible by 2 and 5 select list_filter list_filter 2 4 3 1 20 10 3 30 x - x 2 0 y - y 5 0 ---- 20 10 30 -- in combination with range to construct lists select list_filter 1 2 3 4 x - x 1 from range 4 ---- 1 2 3 4 2 3 4 3 4 4 lambda functions can be arbitrarily nested -- nested lambda functions to get all squares of even list elements select list_transform list_filter 0 1 2 3 4 5 x - x 2 0 y - y y ---- 0 4 16 generate_subscripts the generate_subscript arr dim function generates indexes along the dim th dimension of array arr select generate_subscripts 4 5 6 1 as i i 1 2 3 related functions there are also aggregate functions list and histogram that produces lists and lists of structs unnest is used to unnest a list by one level",
+ "category": "Functions",
+ "url": "../docs/sql/functions/nested",
+ "blurb": "This section describes functions and operators for examining and manipulating nested values. There are three nested data..."
},
{
- "title": "NodeJS API",
- "text": "a name module_duckdb a duckdb summary these jsdoc annotations are still a work in progress - feedback and suggestions are welcome duckdb connection run sql params callback code void code all sql params callback code void code each sql params callback code void code stream sql params register name return_type fun code void code prepare sql params callback code statement code exec sql params callback code void code register_bulk name return_type callback code void code unregister name return_type callback code void code statement get run sql params callback code void code all sql params callback code void code each sql params callback code void code finalize sql params callback code void code stream sql params queryresult nextchunk asynciterator database close callback code void code close_internal callback code void code wait callback code void code serialize callback code void code parallelize callback code void code connect path code connection code interrupt callback code void code prepare sql code statement code run sql params callback code void code each sql params callback code void code all sql params callback code void code exec sql params callback code void code register name return_type fun code this code unregister name code this code get error code number code open_readonly code number code open_readwrite code number code open_create code number code open_fullmutex code number code open_sharedcache code number code open_privatecache code number code a name module_duckdb connection a duckdb connection kind inner class of code duckdb code connection run sql params callback code void code all sql params callback code void code each sql params callback code void code stream sql params register name return_type fun code void code prepare sql params callback code statement code exec sql params callback code void code register_bulk name return_type callback code void code unregister name return_type callback code void code a name module_duckdb connection run a connection run sql params callback code void code run a sql statement and trigger a callback when done kind instance method of code connection code param type --- --- sql params code code callback a name module_duckdb connection all a connection all sql params callback code void code run a sql query and triggers the callback once for all result rows kind instance method of code connection code param type --- --- sql params code code callback a name module_duckdb connection each a connection each sql params callback code void code runs a sql query and triggers the callback for each result row kind instance method of code connection code param type --- --- sql params code code callback a name module_duckdb connection stream a connection stream sql params kind instance method of code connection code param type --- --- sql params code code a name module_duckdb connection register a connection register name return_type fun code void code register a user defined function kind instance method of code connection code note this follows the wasm udfs somewhat but is simpler because we can pass data much more cleanly param --- name return_type fun a name module_duckdb connection prepare a connection prepare sql params callback code statement code prepare a sql query for execution kind instance method of code connection code param type --- --- sql params code code callback a name module_duckdb connection exec a connection exec sql params callback code void code execute a sql query kind instance method of code connection code param type --- --- sql params code code callback a name module_duckdb connection register_bulk a connection register _ bulk name return_type callback code void code register a user defined function kind instance method of code connection code param --- name return_type callback a name module_duckdb connection unregister a connection unregister name return_type callback code void code unregister a user defined function kind instance method of code connection code param --- name return_type callback a name module_duckdb statement a duckdb statement kind inner class of code duckdb code statement get run sql params callback code void code all sql params callback code void code each sql params callback code void code finalize sql params callback code void code stream sql params a name module_duckdb statement get a statement get not implemented kind instance method of code statement code a name module_duckdb statement run a statement run sql params callback code void code kind instance method of code statement code param type --- --- sql params code code callback a name module_duckdb statement all a statement all sql params callback code void code kind instance method of code statement code param type --- --- sql params code code callback a name module_duckdb statement each a statement each sql params callback code void code kind instance method of code statement code param type --- --- sql params code code callback a name module_duckdb statement finalize a statement finalize sql params callback code void code kind instance method of code statement code param type --- --- sql params code code callback a name module_duckdb statement stream a statement stream sql params kind instance method of code statement code param type --- --- sql params code code a name module_duckdb queryresult a duckdb queryresult kind inner class of code duckdb code queryresult nextchunk asynciterator a name module_duckdb queryresult nextchunk a queryresult nextchunk kind instance method of code queryresult code returns data chunk a name module_duckdb queryresult asynciterator a queryresult asynciterator kind instance method of code queryresult code a name module_duckdb database a duckdb database main database interface kind inner property of code duckdb code database close callback code void code close_internal callback code void code wait callback code void code serialize callback code void code parallelize callback code void code connect path code connection code interrupt callback code void code prepare sql code statement code run sql params callback code void code each sql params callback code void code all sql params callback code void code exec sql params callback code void code register name return_type fun code this code unregister name code this code get a name module_duckdb database close a database close callback code void code closes database instance kind instance method of code database code param --- callback a name module_duckdb database close_internal a database close _ internal callback code void code internal method do not use call connection close instead kind instance method of code database code param --- callback a name module_duckdb database wait a database wait callback code void code triggers callback when all scheduled database tasks have completed kind instance method of code database code param --- callback a name module_duckdb database serialize a database serialize callback code void code todo what does this do kind instance method of code database code param --- callback a name module_duckdb database parallelize a database parallelize callback code void code todo what does this do kind instance method of code database code param --- callback a name module_duckdb database connect a database connect path code connection code create a new database connection kind instance method of code database code param description --- --- path the database to connect to either a file path or memory a name module_duckdb database interrupt a database interrupt callback code void code supposedly interrupt queries but currently does not do anything kind instance method of code database code param --- callback a name module_duckdb database prepare a database prepare sql code statement code prepare a sql query for execution kind instance method of code database code param --- sql a name module_duckdb database run a database run sql params callback code void code convenience method for connection run using a built-in default connection kind instance method of code database code param type --- --- sql params code code callback a name module_duckdb database each a database each sql params callback code void code convenience method for connection each using a built-in default connection kind instance method of code database code param type --- --- sql params code code callback a name module_duckdb database all a database all sql params callback code void code convenience method for connection apply using a built-in default connection kind instance method of code database code param type --- --- sql params code code callback a name module_duckdb database exec a database exec sql params callback code void code convenience method for connection exec using a built-in default connection kind instance method of code database code param type --- --- sql params code code callback a name module_duckdb database register a database register name return_type fun code this code convenience method for connection register using a built-in default connection kind instance method of code database code param --- name return_type fun a name module_duckdb database unregister a database unregister name code this code convenience method for connection unregister using a built-in default connection kind instance method of code database code param --- name a name module_duckdb database get a database get not implemented kind instance method of code database code a name module_duckdb error a duckdb error code number code check that errno attribute equals this to check for a duckdb error kind inner constant of code duckdb code a name module_duckdb open_readonly a duckdb open _ readonly code number code open database in readonly mode kind inner constant of code duckdb code a name module_duckdb open_readwrite a duckdb open _ readwrite code number code currently ignored kind inner constant of code duckdb code a name module_duckdb open_create a duckdb open _ create code number code currently ignored kind inner constant of code duckdb code a name module_duckdb open_fullmutex a duckdb open _ fullmutex code number code currently ignored kind inner constant of code duckdb code a name module_duckdb open_sharedcache a duckdb open _ sharedcache code number code currently ignored kind inner constant of code duckdb code a name module_duckdb open_privatecache a duckdb open _ privatecache code number code currently ignored kind inner constant of code duckdb code",
- "category": "Nodejs",
- "url": "../docs/api/nodejs/reference",
- "blurb": " duckdb Summary : these jsdoc annotations are still a work in progress - feedback and..."
+ "title": "Blob Functions",
+ "text": "this section describes functions and operators for examining and manipulating blob values function description example result --- --- --- --- blob blob blob concatenation xaa blob xbb blob xaabb decode blob convert blob to varchar fails if blob is not valid utf-8 decode xc3 xbc blob ü encode string convert varchar to blob converts utf-8 characters into literal encoding encode my_string_with_ü my_string_with_ xc3 xbc octet_length blob number of bytes in blob octet_length xaabb blob 2",
+ "category": "Functions",
+ "url": "../docs/sql/functions/blob",
+ "blurb": "This section describes functions and operators for examining and manipulating blob values. | Function | Description |..."
},
{
- "title": "CLI API",
- "text": "installation the duckdb cli command line interface is a single dependency free executable it is precompiled for windows mac and linux please see the installation page under the cli tab or download the version for your environment from the duckdb github releases page in the assets section for pre-release versions you may compile from source or download the executable file that is produced from github actions linux mac windows the duckdb cli is based on the sqlite command line shell so cli-client-specific functionality is similar to what is described in the sqlite documentation although duckdb s sql syntax follows postgresql conventions getting started once the cli executable has been downloaded unzip it and save it to any directory navigate to that directory in a terminal and enter the command duckdb to run the executable if in a powershell or posix shell environment use the command duckdb instead to see additional command line options to use when starting the cli use the command duckdb --help by default the cli will open a temporary in-memory database to open or create a persistent database simply include a path as a command line argument like duckdb path to my_database duckdb this path can point to an existing database or to a file that does not yet exist and duckdb will open or create a database at that location as needed the file may have any arbitrary extension but db or duckdb are two common choices you will see a prompt like the below with a d on the final line v0 3 4 662041e2b enter help for usage hints connected to a transient in-memory database use open filename to reopen on a persistent database d once the cli has been opened enter a sql statement followed by a semicolon then hit enter and it will be executed results will be displayed in a table in the terminal if a semicolon is omitted hitting enter will allow for multi-line sql statements to be entered d select quack as my_column my_column ----------- quack d select nicely formatted quack as my_column excited quacking as another_column my_column another_column ------------------------ ------------------ nicely formatted quack excited quacking the cli supports all of duckdb s rich sql syntax including select create and alter statements etc to exit the cli press ctrl-d if your platform supports it otherwise press ctrl-c if using a persistent database it will automatically checkpoint save the latest edits to disk and close this will remove the wal file the write-ahead-log and consolidate all of your data into the single file database special commands dot commands in addition to sql syntax special dot commands may be entered that are specific to the cli client to use one of these commands begin the line with a period immediately followed by the name of the command you wish to execute additional arguments to the command are entered space separated after the command if an argument must contain a space either single or double quotes may be used to wrap that parameter dot commands must be entered on a single line and no whitespace may occur before the period no semicolon is required at the end of the line to see available commands use the help command d help auth on off show authorizer callbacks backup db file backup db default main to file bail on off stop after hitting an error default off binary on off turn binary output on or off default off cd directory change the working directory to directory changes on off show number of rows changed by sql check glob fail if output since testcase does not match clone newdb clone data into newdb from the existing database databases list names and files of attached databases dbconfig op val list or change sqlite3_db_config options dbinfo db show status information about the database dump table render database content as sql echo on off turn command echo on or off eqp on off full enable or disable automatic explain query plan excel display the output of next command in spreadsheet exit code exit this program with return-code code expert experimental suggest indexes for queries explain on off auto change the explain formatting mode default auto filectrl cmd run various sqlite3_file_control operations fullschema --indent show schema and the content of sqlite_stat tables headers on off turn display of headers on or off help -all pattern show help text for pattern import file table import data from file into table imposter index table create imposter table table on index index indexes table show names of indexes limit limit val display or change the value of an sqlite_limit lint options report potential schema issues log file off turn logging on or off file can be stderr stdout mode mode table set output mode nullvalue string use string in place of null values once options file output for the next sql command only to file open options file close existing database and reopen file output file send output to file or stdout if file is omitted parameter cmd manage sql parameter bindings print string print literal string progress n invoke progress handler after every n opcodes prompt main continue replace the standard prompts quit exit this program read file read input from file restore db file restore content of db default main from file save file write in-memory database into file scanstats on off turn sqlite3_stmt_scanstatus metrics on or off schema pattern show the create statements matching pattern selftest options run tests defined in the selftest table separator col row change the column and row separators sha3sum compute a sha3 hash of database content shell cmd args run cmd args in a system shell show show the current values for various settings stats on off show stats or turn stats on or off system cmd args run cmd args in a system shell tables table list names of tables matching like pattern table testcase name begin redirecting output to testcase-out txt testctrl cmd run various sqlite3_test_control operations timeout ms try opening locked tables for ms milliseconds timer on off turn sql timer on or off trace options output each sql statement as it is run vfsinfo aux information about the top-level vfs vfslist list all available vfses vfsname aux print the name of the vfs stack width num1 num2 set minimum column widths for columnar output note that the above list of methods is extensive and duckdb supports only a subset of the commands that are displayed please file a github issue if a command that is central to your workflow is not yet supported as an example of passing an argument to a dot command the help text may be filtered by passing in a text string as the second argument d help sh sha3sum compute a sha3 hash of database content shell cmd args run cmd args in a system shell show show the current values for various settings output formats the mode command may be used to change the appearance of the tables returned in the terminal output in addition to customizing the appearance these modes have additional benefits this can be useful for presenting duckdb output elsewhere by redirecting the terminal output to a file for example see writing results to a file section below using the insert mode will build a series of sql statements that can be used to insert the data at a later point the markdown mode is particularly useful for building documentation ascii box csv column html insert json jsonlines line list markdown quote table tabs tcl trash d mode markdown d select quacking intensifies as incoming_ducks incoming_ducks ---------------------- quacking intensifies the output appearance can also be adjusted with the separator command if using an export mode that relies on a separator csv or tabs for example the separator will be reset when the mode is changed for example mode csv will set the separator to a comma using separator will then convert the output to be pipe separated d mode csv d select 1 as col_1 2 as col_2 union all select 10 as col1 20 as col_2 col_1 col_2 1 2 10 20 d separator d select 1 as col_1 2 as col_2 union all select 10 as col1 20 as col_2 col_1 col_2 1 2 10 20 prepared statements the duckdb cli supports executing prepared statements in addition to normal select statements to create a prepared statement use the prepare statement d prepare s1 as select from my_table where my_column 1 or my_column 2 to run the prepared statement with parameters use the execute statement d execute s1 42 101 querying the database schema all duckdb clients support querying the database schema with sql but the cli has additional dot commands that can make it easier to understand the contents of a database the tables command will return a list of tables in the database it has an optional argument that will filter the results according to a like pattern d create table swimmers as select duck as animal d create table fliers as select duck as animal d create table walkers as select duck as animal d tables fliers swimmers walkers for example to filter to only tables that contain an l use the like pattern l d tables l fliers walkers the schema command will show all of the sql statements used to define the schema of the database d schema create table fliers animal varchar create table swimmers animal varchar create table walkers animal varchar opening database files in addition to connecting to a database when opening the cli a new database connection can be made by using the open command if no additional parameters are supplied a new in-memory database connection is created this database will not be persisted when the cli connection is closed d open the open command optionally accepts several options but the final parameter can be used to indicate a path to a persistent database or where one should be created the special string memory can also be used to open a temporary in-memory database d open persistent duckdb one important option accepted by open is the --readonly flag this disallows any editing of the database to open in read only mode the database must already exist this also means that a new in-memory database can t be opened in read only mode since in-memory databases are created upon connection d open --readonly preexisting duckdb writing results to a file by default the duckdb cli sends results to the terminal s standard output however this can be modified using either the output or once commands pass in the desired output file location as a parameter the once command will only output the next set of results and then revert to standard out but output will redirect all subsequent output to that file location note that each result will overwrite the entire file at that destination to revert back to standard output enter output with no file parameter in this example the output format is changed to markdown the destination is identified as a markdown file and then duckdb will write the output of the sql statement to that file output is then reverted to standard output using output with no parameter d mode markdown d output my_results md d select taking flight as output_column d output d select back to the terminal as displayed_column the file my_results md will then contain output_column --------------- taking flight the terminal will then display displayed_column ---------------------- back to the terminal a common output format is csv or comma separated values duckdb supports sql syntax to export data as csv or parquet but the cli-specific commands may be used to write a csv instead if desired d mode csv d once my_output_file csv d select 1 as col_1 2 as col_2 union all select 10 as col1 20 as col_2 the file my_output_file csv will then contain col_1 col_2 1 2 10 20 by passing special options flags to the once command query results can also be sent to a temporary file and automatically opened in the user s default program use either the -e flag for a text file opened in the default text editor or the -x flag for a csv file opened in the default spreadsheet editor this is useful for more detailed inspection of query results especially if there is a relatively large result set the excel command is equivalent to once -x d once -e d select quack as hello the results then open in the default text file editor of the system for example import data from csv duckdb supports sql syntax to directly query or import csv files but the cli-specific commands may be used to import a csv instead if desired the import command takes two arguments and also supports several options the first argument is the path to the csv file and the second is the name of the duckdb table to create since duckdb requires stricter typing than sqlite upon which the duckdb cli is based the destination table must be created before using the import command to automatically detect the schema and create a table from a csv see the read_csv_auto examples in the import docs in this example a csv file is generated by changing to csv mode and setting an output file location d mode csv d output import_example csv d select 1 as col_1 2 as col_2 union all select 10 as col1 20 as col_2 now that the csv has been written a table can be created with the desired schema and the csv can be imported the output is reset to the terminal to avoid continuing to edit the output file specified above the --skip n option is used to ignore the first row of data since it is a header row and the table has already been created with the correct column names d mode csv d output d create table test_table col_1 int col_2 int d import import_example csv test_table --skip 1 note that the import command utilizes the current mode and separator settings when identifying the structure of the data to import the --csv option can be used to override that behavior d import import_example csv test_table --skip 1 --csv reading sql from a file the duckdb cli can read both sql commands and dot commands from an external file intead of the terminal using the read command this allows for a number of commands to be run in sequence and allows command sequences to be saved and reused the read command requires only one argument the path to the file containing the sql and or commands to execute after running the commands in the file control will revert back to the terminal output from the execution of that file is governed by the same output and once commands that have been discussed previously this allows the output to be displayed back to the terminal as in the first example below or out to another file as in the second example in this example the file select_example sql is located in the same directory as duckdb exe and contains the following sql statement select from generate_series 5 to execute it from the cli the read command is used d read select_example sql the output below is returned to the terminal by default but can be adjusted using the output or once commands generate_series ----------------- 0 1 2 3 4 5 multiple commands including both sql and dot commands can also be run in a single read command in this example the file write_markdown_to_file sql is located in the same directory as duckdb exe and contains the following commands mode markdown output series md select from generate_series 5 to execute it from the cli the read command is used as before d read write_markdown_to_file sql in this case no output is returned to the terminal instead the file series md is created or replaced if it already existed with the markdown-formatted results shown here generate_series ----------------- 0 1 2 3 4 5 loading extensions the cli does not use the sqlite shell s load command instead directly execute duckdb s sql install and load commands as you would other sql statements see the extension docs for details d install fts d load fts",
- "category": "Api",
- "url": "../docs/api/cli",
- "blurb": "Installation The DuckDB CLI (Command Line Interface) is a single, dependency free executable. It is precompiled for..."
+ "title": "Functions",
+ "text": "functions are query functions duckdb_functions table function shows the list of functions currently built into the system d select distinct on function_name function_name function_type return_type parameters parameter_types from duckdb_functions where function_type scalar limit 10 function_name function_type return_type parameters parameter_types log10 scalar double col0 double mod scalar tinyint col0 col1 tinyint tinyint date_diff scalar bigint col0 col1 col2 varchar date date writefile scalar varchar regexp_replace scalar varchar col0 col1 col2 col3 varchar varchar varchar varchar age scalar interval col0 timestamp age scalar interval col0 col1 timestamp timestamp datediff scalar bigint col0 col1 col2 varchar date date map scalar map year scalar bigint col0 timestamp with time zone currently the description and parameter names of functions are still missing more",
+ "category": "Functions",
+ "url": "../docs/sql/functions/overview",
+ "blurb": "Functions are ... Query functions duckdb_functions table function shows the list of functions currently built into..."
},
{
- "title": "ODBC API - Overview",
- "text": "odbc api the odbc open database connectivity is a c-style api that provides access to different flavors of database management systems dbmss the odbc api consists of the driver manager dm and the odbc drivers the dm is part of the system library e g unixodbc which manages the communications between the user applications and the odbc drivers typically applications are linked against the dm which uses data source name dsn to look up the correct odbc driver the odbc driver is a dbms implementation of the odbc api which handles all the internals of that dbms the dm maps user application calls of odbc functions to the correct odbc driver that performs the specified function and returns the proper values duckdb odbc driver duckdb supports the odbc version 3 0 according to the core interface conformance we release the odbc driver as assets for linux and windows users can download them from the latest release of duckdb operating system pages in this section",
- "category": "Odbc",
- "url": "../docs/api/odbc/overview",
- "blurb": "ODBC API The ODBC (Open Database Connectivity) is a C-style API that provides access to different flavors of Database..."
+ "title": "SQL Introduction",
+ "text": "here we provide an overview of how to perform simple operations in sql this tutorial is only intended to give you an introduction and is in no way a complete tutorial on sql this tutorial is adapted from the postgresql tutorial in the examples that follow we assume that you have installed the duckdb command line interface cli shell see here for information on how to install the cli if you build from the source tree you can launch the cli from the build directory build release duckdb launching the shell should give you the following prompt duckdb 5fb6fe57ab enter help for usage hints connected to a transient in-memory database use open filename to reopen on a persistent database d by launching the database like this an in-memory database is launched that means that no data is persisted on disk to persist data on disk you should also pass a database path to the shell the database will then be stored at that path and can be reloaded from disk later concepts duckdb is a relational database management system rdbms that means it is a system for managing data stored in relations a relation is essentially a mathematical term for a table each table is a named collection of rows each row of a given table has the same set of named columns and each column is of a specific data type tables themselves are stored inside schemas and a collection of schemas constitutes the entire database that you can access creating a new table you can create a new table by specifying the table name along with all column names and their types create table weather city varchar temp_lo integer -- minimum temperature on a day temp_hi integer -- maximum temperature on a day prcp real date date you can enter this into the shell with the line breaks the command is not terminated until the semicolon white space i e spaces tabs and newlines can be used freely in sql commands that means you can type the command aligned differently than above or even all on one line two dash characters -- introduce comments whatever follows them is ignored up to the end of the line sql is case insensitive about key words and identifiers except when identifiers are double-quoted to preserve the case not done above in the sql command we first specify the type of command that we want to perform create table after that follows the parameters for the command first the table name weather is given then the column names and column types follow city varchar specifies that the table has a column called city that is of type varchar varchar specifies a data type that can store text of arbitrary length the temperature fields are stored in an integer type a type that stores integer numbers i e whole numbers without a decimal point real columns store single precision floating-point numbers i e numbers with a decimal point date stores a date i e year month day combination date only stores the specific day not a time associated with that day duckdb supports the standard sql types integer smallint real double decimal char n varchar n date time and timestamp the second example will store cities and their associated geographical location create table cities name varchar lat decimal lon decimal finally it should be mentioned that if you don t need a table any longer or want to recreate it differently you can remove it using the following command drop table tablename populating a table with rows the insert statement is used to populate a table with rows insert into weather values san francisco 46 50 0 25 1994-11-27 constants that are not numeric values e g text and dates must be surrounded by single quotes as in the example input dates for the date type must be formatted as yyyy-mm-dd we can insert into the cities table in the same manner insert into cities values san francisco -194 0 53 0 the syntax used so far requires you to remember the order of the columns an alternative syntax allows you to list the columns explicitly insert into weather city temp_lo temp_hi prcp date values san francisco 43 57 0 0 1994-11-29 you can list the columns in a different order if you wish or even omit some columns e g if the prcp is unknown insert into weather date city temp_hi temp_lo values 1994-11-29 hayward 54 37 many developers consider explicitly listing the columns better style than relying on the order implicitly please enter all the commands shown above so you have some data to work with in the following sections you could also have used copy to load large amounts of data from csv files this is usually faster because the copy command is optimized for this application while allowing less flexibility than insert an example would be copy weather from home user weather csv where the file name for the source file must be available on the machine running the process there are many other ways of loading data into duckdb see the corresponding documentation section for more information querying a table to retrieve data from a table the table is queried a sql select statement is used to do this the statement is divided into a select list the part that lists the columns to be returned a table list the part that lists the tables from which to retrieve the data and an optional qualification the part that specifies any restrictions for example to retrieve all the rows of table weather type select from weather here is a shorthand for all columns so the same result would be had with select city temp_lo temp_hi prcp date from weather the output should be city temp_lo temp_hi prcp date --------------- --------- ---------- ------ ------------ san francisco 46 50 0 25 1994-11-27 san francisco 43 57 0 0 1994-11-29 hayward 37 54 1994-11-29 3 rows you can write expressions not just simple column references in the select list for example you can do select city temp_hi temp_lo 2 as temp_avg date from weather this should give city temp_avg date --------------- ---------- ------------ san francisco 48 1994-11-27 san francisco 50 1994-11-29 hayward 45 1994-11-29 3 rows notice how the as clause is used to relabel the output column the as clause is optional a query can be qualified by adding a where clause that specifies which rows are wanted the where clause contains a boolean truth value expression and only rows for which the boolean expression is true are returned the usual boolean operators and or and not are allowed in the qualification for example the following retrieves the weather of san francisco on rainy days select from weather where city san francisco and prcp 0 0 result city temp_lo temp_hi prcp date --------------- --------- --------- ------ ------------ san francisco 46 50 0 25 1994-11-27 you can request that the results of a query be returned in sorted order select from weather order by city city temp_lo temp_hi prcp date --------------- --------- --------- ------ ------------ hayward 37 54 1994-11-29 san francisco 43 57 0 0 1994-11-29 san francisco 46 50 0 25 1994-11-27 in this example the sort order isn t fully specified and so you might get the san francisco rows in either order but you d always get the results shown above if you do select from weather order by city temp_lo you can request that duplicate rows be removed from the result of a query select distinct city from weather city --------------- hayward san francisco 2 rows here again the result row ordering might vary you can ensure consistent results by using distinct and order by together select distinct city from weather order by city joins between tables thus far our queries have only accessed one table at a time queries can access multiple tables at once or access the same table in such a way that multiple rows of the table are being processed at the same time a query that accesses multiple rows of the same or different tables at one time is called a join query as an example say you wish to list all the weather records together with the location of the associated city to do that we need to compare the city column of each row of the weather table with the name column of all rows in the cities table and select the pairs of rows where these values match this would be accomplished by the following query select from weather cities where city name city temp_lo temp_hi prcp date name lon lat --------------- --------- --------- ------ ------------ --------------- ----- ---- san francisco 46 50 0 25 1994-11-27 san francisco -194 53 san francisco 43 57 0 1994-11-29 san francisco -194 53 2 rows observe two things about the result set there is no result row for the city of hayward this is because there is no matching entry in the cities table for hayward so the join ignores the unmatched rows in the weather table we will see shortly how this can be fixed there are two columns containing the city name this is correct because the lists of columns from the weather and cities tables are concatenated in practice this is undesirable though so you will probably want to list the output columns explicitly rather than using select city temp_lo temp_hi prcp date lon lat from weather cities where city name since the columns all had different names the parser automatically found which table they belong to if there were duplicate column names in the two tables you d need to qualify the column names to show which one you meant as in select weather city weather temp_lo weather temp_hi weather prcp weather date cities lon cities lat from weather cities where cities name weather city it is widely considered good style to qualify all column names in a join query so that the query won t fail if a duplicate column name is later added to one of the tables join queries of the kind seen thus far can also be written in this alternative form select from weather inner join cities on weather city cities name this syntax is not as commonly used as the one above but we show it here to help you understand the following topics now we will figure out how we can get the hayward records back in what we want the query to do is to scan the weather table and for each row to find the matching cities row s if no matching row is found we want some empty values to be substituted for the cities table s columns this kind of query is called an outer join the joins we have seen so far are inner joins the command looks like this select from weather left outer join cities on weather city cities name city temp_lo temp_hi prcp date name lon lat --------------- --------- --------- ------ ------------ --------------- ----- ---- san francisco 46 50 0 25 1994-11-27 san francisco -194 53 san francisco 43 57 0 1994-11-29 san francisco -194 53 hayward 37 54 1994-11-29 3 rows this query is called a left outer join because the table mentioned on the left of the join operator will have each of its rows in the output at least once whereas the table on the right will only have those rows output that match some row of the left table when outputting a left-table row for which there is no right-table match empty null values are substituted for the right-table columns aggregate functions like most other relational database products duckdb supports aggregate functions an aggregate function computes a single result from multiple input rows for example there are aggregates to compute the count sum avg average max maximum and min minimum over a set of rows as an example we can find the highest low-temperature reading anywhere with select max temp_lo from weather max ----- 46 1 row if we wanted to know what city or cities that reading occurred in we might try select city from weather where temp_lo max temp_lo -- wrong but this will not work since the aggregate max cannot be used in the where clause this restriction exists because the where clause determines which rows will be included in the aggregate calculation so obviously it has to be evaluated before aggregate functions are computed however as is often the case the query can be restated to accomplish the desired result here by using a subquery select city from weather where temp_lo select max temp_lo from weather city --------------- san francisco 1 row this is ok because the subquery is an independent computation that computes its own aggregate separately from what is happening in the outer query aggregates are also very useful in combination with group by clauses for example we can get the maximum low temperature observed in each city with select city max temp_lo from weather group by city city max --------------- ----- hayward 37 san francisco 46 2 rows which gives us one output row per city each aggregate result is computed over the table rows matching that city we can filter these grouped rows using having select city max temp_lo from weather group by city having max temp_lo 40 city max --------- ----- hayward 37 1 row which gives us the same results for only the cities that have all temp_lo values below 40 finally if we only care about cities whose names begin with s we can use the like operator select city max temp_lo from weather where city like s -- 1 group by city having max temp_lo 40 more information about the like operator can be found here it is important to understand the interaction between aggregates and sql s where and having clauses the fundamental difference between where and having is this where selects input rows before groups and aggregates are computed thus it controls which rows go into the aggregate computation whereas having selects group rows after groups and aggregates are computed thus the where clause must not contain aggregate functions it makes no sense to try to use an aggregate to determine which rows will be inputs to the aggregates on the other hand the having clause always contains aggregate functions in the previous example we can apply the city name restriction in where since it needs no aggregate this is more efficient than adding the restriction to having because we avoid doing the grouping and aggregate calculations for all rows that fail the where check updates you can update existing rows using the update command suppose you discover the temperature readings are all off by 2 degrees after november 28 you can correct the data as follows update weather set temp_hi temp_hi - 2 temp_lo temp_lo - 2 where date 1994-11-28 look at the new state of the data select from weather city temp_lo temp_hi prcp date --------------- --------- --------- ------ ------------ san francisco 46 50 0 25 1994-11-27 san francisco 41 55 0 1994-11-29 hayward 35 52 1994-11-29 3 rows deletions rows can be removed from a table using the delete command suppose you are no longer interested in the weather of hayward then you can do the following to delete those rows from the table delete from weather where city hayward all weather records belonging to hayward are removed select from weather city temp_lo temp_hi prcp date --------------- --------- --------- ------ ------------ san francisco 46 50 0 25 1994-11-27 san francisco 41 55 0 1994-11-29 2 rows one should be wary of statements of the form delete from tablename without a qualification delete will remove all rows from the given table leaving it empty the system will not request confirmation before doing this",
+ "category": "SQL",
+ "url": "../docs/sql/introduction",
+ "blurb": "Here we provide an overview of how to perform simple operations in SQL. This tutorial is only intended to give you an..."
},
{
- "title": "ODBC API - Linux",
- "text": "driver manager unixodbc a driver manager is required to manage communication between applications and the odbc driver we tested and support the unixodbc that is a complete odbc driver manager for linux users can install it from the command line debian so flavors sudo apt get install unixodbc fedora so flavors sudo yum install unixodbc or sudo dnf install unixodbc step 1 download odbc driver duckdb releases the odbc driver as asset for linux download it from a href https github com duckdb duckdb releases download v site currentduckdbversion duckdb_odbc-linux-amd64 zip odbc linux asset a that contains the following artifacts libduckdb_odbc so the duckdb driver compiled to ubuntu 16 04 unixodbc_setup sh a setup script to aid the configuration on linux step 2 extracting odbc artifacts run unzip to extract the files to a permanent directory mkdir duckdb_odbc unzip duckdb_odbc-linux-amd64 zip -d duckdb_odbc step 3 configuring with unixodbc the unixodbc_setup sh script aids the configuration of the duckdb odbc driver it is based on the unixodbc package that provides some commands to handle the odbc setup and test like odbcinst and isql in a terminal window change to the duckdb_odbc permanent directory and run the following commands with level options -u or -s either to configure duckdb odbc user-level odbc setup -u the -u option based on the user home directory to setup the odbc init files unixodbc_setup sh -u p s the default configuration consists of a database memory system-level odbc setup -s the -s changes the system level files that will be visible for all users because of that it requires root privileges sudo unixodbc_setup sh -s p s the default configuration consists of a database memory show usage --help the option --help shows the usage of unixodbc_setup sh that provides alternative options for a customer configuration like -db and -d unixodbc_setup sh --help usage unixodbc_setup sh level options example unixodbc_setup sh -u -db database_path -d driver_path libduckdb_odbc so level -s system-level using sudo to configure duckdb odbc at the system-level changing the files etc odbc inst ini -u user-level configuring the duckdb odbc at the user-level changing the files odbc inst ini options -db database_path the duckdb database file path the default is memory if not provided -d driver_path the driver file path i e the path for libduckdb_odbc so the default is using the base script directory step 4 optional configure the odbc driver the odbc setup on linux is based on files the well-known odbc ini and odbcinst ini these files can be placed at the system etc directory or at the user home directory home user shortcut as the dm prioritizes the user configuration files and then the system files the odbc ini file the odbc ini contains the dsns for the drivers which can have specific knobs an example of odbc ini with duckdb would be duckdb driver duckdb driver database memory duckdb between the brackets is a dsn for the duckdb driver it describes the driver s name and other configurations will be placed at the odbcinst ini database it describes the database name used by duckdb and it can also be a file path to a db in the system the odbcinst ini file the odbcinst ini contains general configurations for the odbc installed drivers in the system a driver section starts with the driver name between brackets and then it follows specific configuration knobs belonging to that driver an example of odbcinst ini with the duckdb driver would be odbc trace yes tracefile tmp odbctrace duckdb driver driver home user duckdb_odbc libduckdb_odbc so odbc it is the dm configuration section trace it enables the odbc trace file using the option yes tracefile the absolute system file path for the odbc trace file duckdb driver the section of the duckdb installed driver driver the absolute system file path of the duckdb driver",
- "category": "Odbc",
- "url": "../docs/api/odbc/linux",
- "blurb": "Driver Manager: unixODBC A driver manager is required to manage communication between applications and the ODBC driver...."
+ "title": "Sitemap",
+ "text": "",
+ "category": "Docs",
+ "url": "../docs/index",
+ "blurb": ""
},
{
- "title": "ODBC API - Windows",
- "text": "windows odbc driver manager odbccp32 dll the microsoft windows requires an odbc driver manager to manage communication between applications and the odbc drivers the dm on windows is provided in a dll file odbccp32 dll and other files and tools for detailed information checkout out the common odbc component files step 1 download odbc driver duckdb releases the odbc driver as asset for windows download it from a href https github com duckdb duckdb releases download v site currentduckdbversion duckdb_odbc-windows-amd64 zip windows asset a that contains the following artifacts duckdb_odbc dll the duckdb driver compiled for windows duckdb_odbc_setup dll a setup dll used by the windows odbc data source administrator tool odbc_install exe a installation script to aid the configuration on windows step 2 extracting odbc artifacts unzip the file to a permanent directory e g duckdb_odbc an example with powershell and unzip command would be mkdir duckdb_odbc unzip duckdb_odbc-linux-amd64 zip -d duckdb_odbc step 3 odbc windows installer the odbc_install exe aids the configuration of the duckdb odbc driver on windows it depends on the odbccp32 dll that provides functions to configure the odbc registry entries inside the permanent directory e g duckdb_odbc double-click on the odbc_install exe windows administrator privileges is required in case of a non-administrator a user account control shall display step 4 configure the odbc driver the odbc_install exe adds a default dsn configuration into the odbc registries with a default database memory dsn windows setup after the installation it is possible to change the default dsn configuration or add a new one using the windows odbc data source administrator tool odbcad32 exe it also can be launched thought the windows start default duckdb dsn in the the windows odbc data source administrator tool at system dsn tab is placed the default installed dsn for duckdb windows odbc config tool changing duckdb dsn selecting the default dsn i e duckdb or add a new configuration the following setup window will display duckdb windows dsn setup for now it is possible to set the dsn and the database file path associated with that dsn more detailed windows setup the odbc setup on windows is based on registry keys see registry entries for odbc components the odbc entries can be placed at the current user registry key hkcu or the system registry key hklm we have tested and used the system entries based on hklm- software- odbc the odbc_install exe changes this entry that has two subkeys odbc ini and odbcinst ini the odbc ini is where users usually insert dsn registry entries for the drivers for example the dsn registry for duckdb would look like this hklm- software- odbc- odbc ini- duckdb the odbcinst ini contains one entry for each odbc driver and other keys predefined for windows odbc configuration",
- "category": "Odbc",
- "url": "../docs/api/odbc/windows",
- "blurb": "Windows ODBC Driver Manager: Odbccp32.dll The Microsoft Windows requires an ODBC Driver Manager to manage communication..."
+ "title": "JSON",
+ "text": "the json extension is a loadable extension that implements sql functions that are useful for reading values from existing json and creating new json data json type the json extension makes use of the json logical type the json logical type is interpreted as json i e parsed in json functions rather than interpreted as varchar i e a regular string all json creation functions return values of this type json functions the following scalar json functions can be used to gain information about the stored json values with the exception of json_valid json all json functions produce an error when invalid json is supplied we support two kinds of notations to describe locations within json json pointer and jsonpath function description --- --- json json parse and minify json json_valid json return whether json is valid json json_array_length json path return the number of elements in the json array json or 0 if it is not a json array if path is specified return the number of elements in the json array at the given path if path is a list the result will be list of array lengths json_type json path return the type of the supplied json which is one of object array bigint ubigint varchar boolean null if path is specified return the type of the element at the given path if path is a list the result will be list of types json_structure json return the structure of json throws an error if the structure is inconsistent incompatible types in an array examples create table example j json insert into example values family anatidae species duck goose swan null select json j from example -- family anatidae species duck goose swan null select json_valid j from example -- true select json_valid -- false select json_array_length duck goose swan null -- 4 select json_array_length j species from example -- 4 select json_array_length j species from example -- 4 select json_array_length j species from example -- 4 select json_array_length j species from example -- 4 select json_type j from example -- object select json_structure j from example -- family varchar species varchar select json_structure duck family anatidae -- invalid input error inconsistent json structure json extraction functions there are two extraction functions which have their respective operators the operators can only be used if the string is stored as the json logical type these functions supports the same two location notations as the previous functions function alias operator description --- --- --- json_extract json path json_extract_path - extract json from json at the given path if path is a list the result will be a list of json json_extract_string json path json_extract_path_text - extract varchar from json at the given path if path is a list the result will be a list of varchar examples create table example j json insert into example values family anatidae species duck goose swan null select json_extract j family from example -- anatidae select j- family from example -- anatidae select j- species 0 from example -- duck select j- species - 0 from example -- duck select j- species - 0 1 from example -- duck goose select json_extract_string j family from example -- anatidae select j- family from example -- anatidae select j- species 0 from example -- duck select j- species - 0 from example -- duck select j- species - 0 1 from example -- duck goose json creation functions the following functions are used to create json function description --- --- to_json any create json from a value of any type our list is converted to a json array and our struct and map are converted to a json object json_quote any alias for to_json array_to_json list alias for to_json that only accepts list row_to_json list alias for to_json that only accepts struct json_array any create a json array from any number of values json_object key value create a json object from any number of key value pairs json_merge_patch json json merge two json documents together read_json_objects filename read 1 or more newline-delimited json objects from a file read_ndjson_objects filename alias for read_json_objects examples select to_json duck -- duck select to_json 1 2 3 -- 1 2 3 select to_json duck 42 -- duck 42 select to_json map duck 42 -- duck 42 select json_array 42 duck null -- 42 duck null select json_object duck 42 -- duck 42 select json_merge_patch duck 42 goose 123 -- goose 123 duck 42 json aggregate functions there are three json aggregate functions function description --- --- json_group_array any return a json array with all values of any in the aggregation json_group_object key value return a json object with all key value pairs in the aggregation json_group_structure json return the combined json_structure of all json in the aggregation examples create table example k varchar v integer insert into example values duck 42 goose 7 select json_group_array v from example -- 42 7 select json_group_object k v from example -- duck 42 goose 7 drop table example create table example j json insert into example values family anatidae species duck goose coolness 42 42 family canidae species labrador bulldog hair true select json_group_structure j from example -- family varchar species varchar coolness double hair boolean transforming json in many cases it is inefficient to extract values from json one-by-one instead we can extract all values at once transforming json to the nested types list and struct function description --- --- json_transform json structure transform json according to the specified structure from_json json structure alias for json_transform json_transform_strict json structure same as json_transform but throws an error when type casting fails from_json_strict json structure alias for json_transform_strict the structure argument is json of the same form as returned by json_structure the structure argument can be modified to transform the json into the desired structure and types it is possible to extract fewer key value pairs than are present in the json and it is also possible to extract more missing keys become null examples create table example j json insert into example values family anatidae species duck goose coolness 42 42 family canidae species labrador bulldog hair true select json_transform j family varchar coolness double from example -- family anatidae coolness 42 420000 -- family canidae coolness null select json_transform j family tinyint coolness decimal 4 2 from example -- family null coolness 42 42 -- family null coolness null select json_transform_strict j family tinyint coolness double from example -- invalid input error failed to cast value anatidae",
+ "category": "Extensions",
+ "url": "../docs/extensions/json",
+ "blurb": "The json extension is a loadable extension that implements SQL functions that are useful for reading values from..."
},
{
- "title": "Julia Package",
- "text": "duckdb julia package the duckdb julia package provides a high-performance front-end for duckdb much like sqlite duckdb runs in-process within the julia client and provides a dbinterface front-end the package also supports multi-threaded execution it uses julia threads tasks for this purpose if you wish to run queries in parallel you must launch julia with multi-threading support by e g setting the julia_num_threads environment variable installation pkg add duckdb julia using duckdb basics create a new in-memory database con dbinterface connect duckdb db memory create a table dbinterface execute con create table integers i integer insert data using a prepared statement stmt dbinterface prepare con insert into integers values dbinterface execute stmt 42 query the database results dbinterface execute con select 42 a print results scanning dataframes the duckdb julia package also provides support for querying julia dataframes note that the dataframes are directly read by duckdb - they are not inserted or copied into the database itself if you wish to load data from a dataframe into a duckdb table you can run a create table as or insert into query using duckdb using dataframes create a new in-memory dabase con dbinterface connect duckdb db create a dataframe df dataframe a 1 2 3 b 42 84 42 register it as a view in the database duckdb register_data_frame con df my_df run a sql query over the dataframe results dbinterface execute con select from my_df print results original julia connector credits to kimmolinna for the original duckdb julia connector",
- "category": "Api",
- "url": "../docs/api/julia",
- "blurb": "DuckDB Julia Package The DuckDB Julia package provides a high-performance front-end for DuckDB. Much like SQLite,..."
+ "title": "HTTPFS",
+ "text": "the httpfs extension is a loadable extension implementing a file system that allows reading remote writing remote files for pure http s only file reading is supported for object storage using the s3 api the httpfs extension supports reading writing globbing files http s with the httpfs extension it is possible to directly query files over http s this currently works for csv and parquet files select from https domain tld file extension for csv files files will be downloaded entirely in most cases due to the row-based nature of the format for parquet files duckdb can use a combination of the parquet metadata and http range requests to only download the parts of the file that are actually required by the query for example the query select column_a from https domain tld file parquet will only read the parquet metadata and the data for the column_a column in some cases even no actual data needs to be read at all as they only require reading the metadata select count from https domain tld file parquet scanning multiple files over http s is also supported select from parquet_scan https domain tld file1 parquet https domain tld file2 parquet s3 the httpfs extension supports reading writing globbing files on object storage servers using the s3 api requirements the httpfs filesystem is tested with aws s3 minio and google cloud other services that implement the s3 api should also work but not all features may be supported below is a list of which parts of the s3 api are required for each httpfs feature feature required s3 api features --- --- public file reads http range requests private file reads secret key or session token authentication file glob listobjectv2 file writes multipart upload configuration to be able to read or write from s3 the correct region should be set set s3_region us-east-1 optionally the endpoint can be configured in case a non-aws object storage server is used set s3_endpoint domain tld port switching between path-style and vhost-style urls see aws docs is possible using set s3_url_style path however note that this may also require updating the endpoint for example for aws s3 it is required to change the endpoint to s3 region amazonaws com after configuring the correct endpoint and region public files can be read to also read private files authentication credentials can be added set s3_access_key_id aws access key id set s3_secret_access_key aws secret access key alternatively session tokens are also supported and can be used instead set s3_session_token aws session token reading reading files from s3 is now as simple as select from s3 bucket file extension multiple files are also possible for example select from parquet_scan s3 bucket file1 parquet s3 bucket file2 parquet glob file globbing is implemented using the listobjectv2 api call and allows to use filesystem-like glob patterns to match multiple files for example select from parquet_scan s3 bucket parquet this query matches all files in the root of the bucket with the parquet extension several features for matching are supported such as to match any number of any character for any single character or 0-9 for a single character in a range of characters select count from parquet_scan s3 bucket folder 100 t 0-9 parquet a useful feature when using globs is the filename option which adds a column with the file that a row originated from select from parquet_scan s3 bucket parquet filename 1 could for example result in column_a column_b filename --- --- --- 1 examplevalue1 s3 bucket file1 parquet 2 examplevalue1 s3 bucket file2 parquet hive partitioning duckdb also offers support for the hive partitioning scheme in the hive partitioning scheme data is partitioned in separate files the columns by which the data is partitioned are not actually in the files but are encoded in the file path so for example let us consider three parquet files hive paritioned by year s3 bucket year 2012 file parquet s3 bucket year 2013 file parquet s3 bucket year 2014 file parquet if scanning these files with the hive_partitioning option enabled select from parquet_scan s3 bucket file parquet hive_partitioning 1 could result in column_a column_b year --- --- --- 1 examplevalue1 2012 2 examplevalue2 2013 3 examplevalue3 2014 note that the year column does not actually exist in the parquet files it is parsed from the filenames within duckdb however these columns behave just like regular columns for example filters can be applied on hive partition columns select from parquet_scan s3 bucket file parquet hive_partitioning 1 where year 2013 writing writing to s3 uses the multipart upload api this allows duckdb to robustly upload files at high speed writing to s3 works for both csv and parquet copy table_name to s3 bucket file extension configuration some additional configuration options exist for the s3 upload though the default values should suffice for most use cases setting description --- --- s3_uploader_max_parts_per_file used for part size calculation see aws docs s3_uploader_max_filesize used for part size calculation see aws docs s3_uploader_thread_limit maximum number of uploader threads",
+ "category": "Extensions",
+ "url": "../docs/extensions/httpfs",
+ "blurb": "The httpfs extension is a loadable extension implementing a file system that allows reading remote/writing remote ..."
},
{
- "title": "C++ API",
- "text": "installation the duckdb c api can be installed as part of the libduckdb packages please see the installation page for details basic api usage duckdb implements a custom c api this is built around the abstractions of a database instance duckdb class multiple connection s to the database instance and queryresult instances as the result of queries the header file for the c api is duckdb hpp the standard source distribution of libduckdb contains an amalgamation of the duckdb sources which combine all sources into two files duckdb hpp and duckdb cpp the duckdb hpp header is much larger in this case regardless of whether you are using the amalgamation or not just include duckdb hpp startup shutdown to use duckdb you must first initialize a duckdb instance using its constructor duckdb takes as parameter the database file to read and write from the special value nullptr can be used to create an in-memory database note that for an in-memory database no data is persisted to disk i e all data is lost when you exit the process the second parameter to the duckdb constructor is an optional dbconfig object in dbconfig you can set various database parameters for example the read write mode or memory limits the duckdb constructor may throw exceptions for example if the database file is not usable with the duckdb instance you can create one or many connection instances using the connection constructor while connections should be thread-safe they will be locked during querying it is therefore recommended that each thread uses its own connection if you are in a multithreaded environment duckdb db nullptr connection con db querying connections expose the query method to send a sql query string to duckdb from c query fully materializes the query result as a materializedqueryresult in memory before returning at which point the query result can be consumed there is also a streaming api for queries see further below create a table con query create table integers i integer j integer insert three rows into the table con query insert into integers values 3 4 5 6 7 null materializedqueryresult result con query select from integers if result- success cerr result- error the materializedqueryresult instance contains firstly two fields that indicate whether the query was successful query will not throw exceptions under normal circumstances instead invalid queries or other issues will lead to the success boolean field in the query result instance to be set to false in this case an error message may be available in error as a string if successful other fields are set the type of statement that was just executed e g statementtype insert_statement is contained in statement_type the high-level sql type types of the result set columns are in sql_types and the low-level data representation types are in types the names of the result columns are in the names string vector in case multiple result sets are returned for example because the result set contained multiple statements the result set can be chained using the next field todo duckdb also supports prepared statements in the c api with the prepare method this returns an instance of preparedstatement this instance can be used to execute the prepared statement with parameters below is an example std unique_ptr preparedstatement prepare con prepare select count from a where i 1 std unique_ptr queryresult result prepare- execute 12 do not use prepared statements to insert large amounts of data into duckdb see the data import documentation for better options streaming queries udf api the udf api is exposed in duckdb connection through the methods createscalarfunction and createvectorizedfunction and variants these methods created udfs into the temporary schema temp_schema of the owner connection that is the only one allowed to use and change them createscalarfunction the user can code an ordinary scalar function and invoke the createscalarfunction to register and afterward use the udf in a select statement for instance bool bigger_than_four int value return value 4 connection createscalarfunction bool int bigger_than_four bigger_than_four connection query select bigger_than_four i from values 3 5 tbl i - print the createscalarfunction methods automatically creates vectorized scalar udfs so they are as efficient as built-in functions we have two variants of this method interface as follows 1 template typename tr typename args nbsp nbsp nbsp void createscalarfunction string name tr udf_func args template parameters tr is the return type of the udf function args are the arguments up to 3 for the udf function this method only supports until ternary functions name is the name to register the udf function udf_func is a pointer to the udf function this method automatically discovers from the template typenames the corresponding sqltypes bool sqltype boolean int8_t sqltype tinyint int16_t sqltype smallint int32_t sqltype integer int64_t sqltype bigint float sqltype float double sqltype double string_t sqltype varchar in duckdb some primitive types e g int32_t are mapped to the same sqltype integer time and date then for disambiguation the users can use the following overloaded method 2 template typename tr typename args nbsp nbsp nbsp void createscalarfunction string name vector sqltype args sqltype ret_type tr udf_func args an example of use would be int32_t udf_date int32_t a return a con query create table dates d date con query insert into dates values 1992-01-01 con createscalarfunction int32_t int32_t udf_date sqltype date sqltype date udf_date con query select udf_date d from dates - print template parameters tr is the return type of the udf function args are the arguments up to 3 for the udf function this method only supports until ternary functions name is the name to register the udf function args are the sqltype arguments that the function uses which should match with the template args types ret_type is the sqltype of return of the function which should match with the template tr type udf_func is a pointer to the udf function this function checks the template types against the sqltypes passed as arguments and they must match as follow sqltypeid boolean bool sqltypeid tinyint int8_t sqltypeid smallint int16_t sqltypeid date sqltypeid time sqltypeid integer int32_t sqltypeid bigint sqltypeid timestamp int64_t sqltypeid float sqltypeid double sqltypeid decimal double sqltypeid varchar sqltypeid char sqltypeid blob string_t sqltypeid varbinary blob_t createvectorizedfunction the createvectorizedfunction methods register a vectorized udf such as this vectorized function copies the input values to the result vector template typename type static void udf_vectorized datachunk args expressionstate state vector result set the result vector type result vector_type vectortype flat_vector get a raw array from the result auto result_data flatvector getdata type result get the solely input vector auto input args data 0 now get an orrified vector vectordata vdata input orrify args size vdata get a raw array from the orrified input auto input_data type vdata data handling the data for idx_t i 0 i args size i auto idx vdata sel- get_index i if vdata nullmask idx continue result_data i input_data idx con query create table integers i integer con query insert into integers values 1 2 3 999 con createvectorizedfunction int int udf_vectorized_int udf_vectorized int con query select udf_vectorized_int i from integers - print the vectorized udf is a pointer of the type scalar_function_t typedef std function void datachunk args expressionstate expr vector result scalar_function_t args is a datachunk that holds a set of input vectors for the udf that all have the same length expr is an expressionstate that provides information to the query s expression state result is a vector to store the result values there are different vector types to handle in a vectorized udf constantvector dictionaryvector flatvector listvector stringvector structvector sequencevector the general api of the createvectorizedfunction method is as follows 1 template typename tr typename args nbsp nbsp nbsp void createvectorizedfunction string name scalar_function_t udf_func sqltype varargs sqltype invalid template parameters tr is the return type of the udf function args are the arguments up to 3 for the udf function name is the name to register the udf function udf_func is a vectorized udf function varargs the type of varargs to support or sqltypeid invalid default value if the function does not accept variable length arguments this method automatically discovers from the template typenames the corresponding sqltypes bool sqltype boolean int8_t sqltype tinyint int16_t sqltype smallint int32_t sqltype integer int64_t sqltype bigint float sqltype float double sqltype double string_t sqltype varchar 2 template typename tr typename args nbsp nbsp nbsp void createvectorizedfunction string name vector sqltype args sqltype ret_type scalar_function_t udf_func sqltype varargs sqltype invalid todo",
- "category": "Api",
- "url": "../docs/api/cpp",
- "blurb": "Installation The DuckDB C++ API can be installed as part of the libduckdb packages. Please see the installation page..."
+ "title": "Postgres Scanner",
+ "text": "see here for usage",
+ "category": "Extensions",
+ "url": "../docs/extensions/postgres_scanner",
+ "blurb": "See here for usage"
},
{
- "title": "Java JDBC API",
- "text": "installation the duckdb java jdbc api can be installed from maven central please see the installation page for details basic api usage duckdb s jdbc api implements the main parts of the standard java database connectivity jdbc api version 4 0 describing jdbc is beyond the scope of this page see the official documentation for details below we focus on the duckdb-specific parts startup shutdown in jdbc database connections are created through the standard java sql drivermanager class the driver should auto-register in the drivermanager if that does not work for some reason you can enforce registration like so class forname org duckdb duckdbdriver to create a duckdb connection call drivermanager with the jdbc duckdb jdbc url prefix like so connection conn drivermanager getconnection jdbc duckdb when using the jdbc duckdb url alone an in-memory database is created note that for an in-memory database no data is persisted to disk i e all data is lost when you exit the java program if you would like to access or create a persistent database append its file name after the path for example if your database is stored in tmp my_database use the jdbc url jdbc duckdb tmp my_database to create a connection to it it is possible to open a duckdb database file in read-only mode this is for example useful if multiple java processes want to read the same database file at the same time to open an existing database file in read-only mode set the connection property duckdb read_only like so properties ro_prop new properties ro_prop setproperty duckdb read_only true connection conn_ro drivermanager getconnection jdbc duckdb tmp my_database ro_prop the jdbc drivermanager api is a relatively poor fit for embedded database management systems such as duckdb if you would like to create multiple connections to the same database it would be somewhat logical to just create additional connections with the same url this is however only supported for read-only connections if you would like to create multiple read-write connections to the same database file or the same in-memory database instance you can use the custom duplicate method like so connection conn2 duckdbconnection conn duplicate querying duckdb supports the standard jdbc methods to send queries and retreive result sets first a statement object has to be created from the connection this object can then be used to send queries using execute and executequery execute is meant for queries where no results are expected like create table or update etc and executequery is meant to be used for queries that produce results e g select below two examples see also the jdbc statement and resultset documentations create a table statement stmt conn createstatement stmt execute create table items item varchar value decimal 10 2 count integer insert two items into the table stmt execute insert into items values jeans 20 0 1 hammer 42 2 2 resultset rs stmt executequery select from items while rs next system out println rs getstring 1 system out println rs getint 3 rs close jeans 1 hammer 2 duckdb also supports prepared statements as per the jdbc api preparedstatement p_stmt conn preparestatement insert into test values p_stmt setstring 1 chainsaw p_stmt setdouble 2 500 0 p_stmt setint 3 42 p_stmt execute more calls to execute possible p_stmt close do not use prepared statements to insert large amounts of data into duckdb see the data import documentation for better options",
- "category": "Api",
- "url": "../docs/api/java",
- "blurb": "Installation The DuckDB Java JDBC API can be installed from Maven Central . Please see the installation page for..."
+ "title": "Substrait",
+ "text": "see repo for usage",
+ "category": "Extensions",
+ "url": "../docs/extensions/substrait",
+ "blurb": "See repo for usage"
+ },
+ {
+ "title": "Full Text Search",
+ "text": "full text search is an extension to duckdb that allows for search through strings similar to sqlite s fts5 extension api the extension adds two pragma statements to duckdb one to create and one to drop an index additionally a scalar macro stem is added which is used internally by the extension pragma create_fts_index create_fts_index input_table input_id input_values stemmer porter stopwords english ignore a-z strip_accents true lower true overwrite false pragma that creates a fts index for the specified table name type description -- -- -- input_table varchar qualified name of specified table e g table_name or main table_name input_id varchar column name of document identifier e g document_identifier input_values varchar column names of the text fields to be indexed vararg e g text_field_1 text_field_2 text_field_n or for all columns in input_table of type varchar stemmer varchar the type of stemmer to be used one of arabic basque catalan danish dutch english finnish french german greek hindi hungarian indonesian irish italian lithuanian nepali norwegian porter portuguese romanian russian serbian spanish swedish tamil turkish or none if no stemming is to be used defaults to porter stopwords varchar qualified name of table containing a single varchar column containing the desired stopwords or none if no stopwords are to be used defaults to english for a pre-defined list of 571 english stopwords ignore varchar regular expression of patterns to be ignored defaults to a-z ignoring all escaped and non-alphabetic lowercase characters strip_accents boolean whether to remove accents e g convert á to a defaults to true lower boolean whether to convert all text to lowercase defaults to true overwrite boolean whether to overwrite an existing index on a table defaults to false this pragma builds the index under a newly created schema the schema will be named after the input table if an index is created on table main table_name then the schema will be named fts_main_table_name pragma drop_fts_index drop_fts_index input_table drops a fts index for the specified table name type description -- -- -- input_table varchar qualified name of input table e g table_name or main table_name match_bm25 match_bm25 input_id query_string fields null k 1 2 b 0 75 conjunctive 0 when an index is built this retrieval macro is created that can be used to search the index name type description -- -- -- input_id varchar column name of document identifier e g document_identifier query_string varchar the string to search the index for fields varchar comma-separarated list of fields to search in e g text_field_2 text_field_n defaults to null to search all indexed fields k double parameter k sub 1 sub in the okapi bm25 retrieval model defaults to 1 2 b double parameter b in the okapi bm25 retrieval model defaults to 0 75 conjunctive boolean whether to make the query conjunctive i e all terms in the query string must be present in order for a document to be retrieved stem stem input_string stemmer reduces words to their base used internally by the extension name type description -- -- -- input_string varchar the column or constant to be stemmed stemmer varchar the type of stemmer to be used one of arabic basque catalan danish dutch english finnish french german greek hindi hungarian indonesian irish italian lithuanian nepali norwegian porter portuguese romanian russian serbian spanish swedish tamil turkish or none if no stemming is to be used example usage -- create a table and fill it with text data create table documents document_identifier varchar text_content varchar author varchar doc_version integer insert into documents values doc1 the mallard is a dabbling duck that breeds throughout the temperate hannes mühleisen 3 doc2 the cat is a domestic species of small carnivorous mammal laurens kuiper 2 -- build the index make both the text_content and author columns searchable pragma create_fts_index documents document_identifier text_content author -- search the author field index for documents that are written by hannes - this retrieves doc1 select text_content score from select fts_main_documents match_bm25 document_identifier muhleisen fields author as score from documents sq where score is not null and doc_version 2 order by score desc -- search for documents about small cats - this retrieves doc2 select text_content score from select fts_main_documents match_bm25 document_identifier small cats as score from documents sq where score is not null order by score desc caveats note that the fts index will not update automatically when input table changes a workaround of this limitation can be recreating the index to refresh",
+ "category": "Extensions",
+ "url": "../docs/extensions/full_text_search",
+ "blurb": "Full Text Search is an extension to DuckDB that allows for search through strings, similar to SQLite's FTS5 extension. ..."
+ },
+ {
+ "title": "SQLite Scanner",
+ "text": "see here for usage",
+ "category": "Extensions",
+ "url": "../docs/extensions/sqlite_scanner",
+ "blurb": "See here for usage"
+ },
+ {
+ "title": "Extensions",
+ "text": "duckdb has a number of extensions available for use not all of them are included by default in every distribution but duckdb has a mechanism that allows for remote installation remote installation if a given extensions is not available with your distribution you can do the following to make it available install fts load fts if you are using the python api client you can install and load them with the load_extension name str and install_extension name str methods unsigned extensions all verified extensions are signed if you wish to load your own extensions or extensions from untrusted third-parties you ll need to enable the allow_unsigned_extensions flag to load unsigned extensions using the cli you ll need to pass the -unsigned flag to it on startup listing extensions you can check the list of core and installed extensions with the following query select from duckdb_extensions all available extensions extension name description ---------------- -------------------------------------------------------------------- fts adds support for full-text search indexes httpfs adds support for reading and writing files over a http s connection icu adds support for time zones and collations using the icu library json adds support for json operations parquet adds support for reading and writing parquet files postgres_scanner adds support for reading from a postgres database sqlite_scanner adds support for reading sqlite database files substrait adds support for the substrait integration tpcds adds tpc-ds data generation and query support tpch adds tpc-h data generation and query support pages in this section",
+ "category": "Extensions",
+ "url": "../docs/extensions/overview",
+ "blurb": "DuckDB has a number of extensions available for use. Not all of them are included by default in every distribution, but..."
+ },
+ {
+ "title": "CSV Import/Export",
+ "text": "copy moves data between duckdb tables and external comma separated value csv files csv import copy from imports data into duckdb from an external csv file into an existing table the data is appended to whatever data is in the table already the amount of columns inside the file must match the amount of columns in the table table_name and the contents of the columns must be convertible to the column types of the table in case this is not possible an error will be thrown if a list of columns is specified copy will only copy the data in the specified columns from the file if there are any columns in the table that are not in the column list copy from will insert the default values for those columns -- copy the contents of a comma-separated file test csv without a header into the table test copy test from test csv -- copy the contents of a comma-separated file with a header into the category table copy category from categories csv header -- copy the contents of lineitem tbl into the lineitem table where the contents are delimited by a pipe character copy lineitem from lineitem tbl delimiter -- read the contents of a comma-separated file names csv into the name column of the category table any other columns of this table are filled with their default value copy category name from names csv syntax csv export copy to exports data from duckdb to an external csv file it has mostly the same set of options as copy from however in the case of copy to the options specify how the csv file should be written to disk any csv file created by copy to can be copied back into the database by using copy from with the same set of options the copy to function can be called specifying either a table name or a query when a table name is specified the contents of the entire table will be written into the resulting csv file when a query is specified the query is executed and the result of the query is written to the resulting file -- copy the contents of the lineitem table to the file lineitem tbl where the columns are delimited by a pipe character including a header line copy lineitem to lineitem tbl delimiter header -- copy the l_orderkey column of the lineitem table to the file orderkey tbl copy lineitem l_orderkey to orderkey tbl delimiter -- copy the result of a query to the file query csv including a header with column names copy select 42 as a hello as b to query csv with header 1 delimiter syntax parameters name description --- --- table_name the name optionally schema-qualified of an existing table column_name an optional list of columns to be copied if no column list is specified all columns of the table will be copied filename the path and name of the input or output file boolean specifies whether the selected option should be turned on or off you can write true on or 1 to enable the option and false off or 0 to disable it the boolean value can also be omitted in which case true is assumed format if this option is used its value must be csv with any other format an error will be thrown delimiter specifies the string that separates columns within each row line of the file the default value is a comma null specifies the string that represents a null value the default is an empty string please note that in copy from both an unquoted empty string and a quoted empty string represent a null value if any other null string is specified again both its quoted and its unquoted appearance represent a null value copy to does not quote null values on output even if force_quote is true header specifies that the file contains a header line with the names of each column in the file thus copy from ignores the first line when importing data whereas on output copy to the first line of the file contains the column names of the exported columns quote specifies the quoting string to be used when a data value is quoted the default is double-quote escape specifies the string that should appear before a data character sequence that matches the quote value the default is the same as the quote value so that the quoting string is doubled if it appears in the data force_quote forces quoting to be used for all non-null values in each specified column null output is never quoted if is specified non-null values will be quoted in all columns this option is allowed only in copy to force_not_null do not match the specified columns values against the null string in the default case where the null string is empty this means that empty values will be read as zero-length strings rather than nulls this option is allowed only in copy from encoding if this option is used its value must be utf8 with any other encoding an error will be thrown notes it is recommended that the file name used in copy always be specified as an absolute path the values in each record are separated by the delimiter string if the value contains the delimiter string the quote string the null string a carriage return or line feed character then the whole value is prefixed and suffixed by the quote string and any occurrence within the value of a quote string or the escape string is preceded by the escape string you can also use force_quote to force quotes when outputting non-null values in specific columns the csv format has no standard way to distinguish a null value from an empty string you can use force_not_null to prevent null input comparisons for specific columns in csv format all characters are significant a quoted value surrounded by white space or any characters other than delimiter will include those characters this can cause errors if you import data from a system that pads csv lines with white space out to some fixed width if such a situation arises you might need to preprocess the csv file to remove the trailing white space before importing the data into duckdb",
+ "category": "Docs",
+ "url": "../docs/csv_import",
+ "blurb": "COPY moves data between DuckDB tables and external Comma Separated Value (CSV) files. CSV Import COPY ... FROM ..."
}
]
}
diff --git a/docs/api/c/api.md b/docs/api/c/api.md
index a5524e012be..67591a00b98 100644
--- a/docs/api/c/api.md
+++ b/docs/api/c/api.md
@@ -10,7 +10,10 @@ selected: API Reference
duckdb_state duckdb_open_ext(const char *path, duckdb_database *out_database, duckdb_config config, char **out_error);
void duckdb_close(duckdb_database *database);
duckdb_state duckdb_connect(duckdb_database database, duckdb_connection *out_connection);
+void duckdb_interrupt(duckdb_connection connection);
+double duckdb_query_progress(duckdb_connection connection);
void duckdb_disconnect(duckdb_connection *connection);
+const char *duckdb_library_version();
### **Configuration**
duckdb_state duckdb_create_config(duckdb_config *out_config);
@@ -34,6 +37,7 @@ selected: API Reference
### **Result Functions**
duckdb_data_chunk duckdb_result_get_chunk(duckdb_result result, idx_t chunk_index);
+bool duckdb_result_is_streaming(duckdb_result result);
idx_t duckdb_result_chunk_count(duckdb_result result);
bool duckdb_value_boolean(duckdb_result *result, idx_t col, idx_t row);
int8_t duckdb_value_int8(duckdb_result *result, idx_t col, idx_t row);
@@ -54,6 +58,7 @@ selected: API Reference
duckdb_interval duckdb_value_interval(duckdb_result *result, idx_t col, idx_t row);
char *duckdb_value_varchar(duckdb_result *result, idx_t col, idx_t row);
char *duckdb_value_varchar_internal(duckdb_result *result, idx_t col, idx_t row);
+duckdb_string duckdb_value_string_internal(duckdb_result *result, idx_t col, idx_t row);
duckdb_blob duckdb_value_blob(duckdb_result *result, idx_t col, idx_t row);
bool duckdb_value_is_null(duckdb_result *result, idx_t col, idx_t row);
@@ -61,6 +66,7 @@ selected: API Reference
### **Date/Time/Timestamp Helpers**
duckdb_date_struct duckdb_from_date(duckdb_date date);
@@ -73,6 +79,7 @@ selected: API Reference
### **Hugeint Helpers**
### **Decimal Helpers**
double duckdb_decimal_to_double(duckdb_decimal val);
@@ -84,12 +91,15 @@ selected: API Reference
idx_t duckdb_nparams(duckdb_prepared_statement prepared_statement);
duckdb_type duckdb_param_type(duckdb_prepared_statement prepared_statement, idx_t param_idx);
duckdb_state duckdb_clear_bindings(duckdb_prepared_statement prepared_statement);
+duckdb_state duckdb_bind_value(duckdb_prepared_statement prepared_statement, idx_t param_idx, duckdb_value val);
+duckdb_state duckdb_bind_parameter_index(duckdb_prepared_statement prepared_statement, idx_t *param_idx_out, const char *name);
duckdb_state duckdb_bind_boolean(duckdb_prepared_statement prepared_statement, idx_t param_idx, bool val);
duckdb_state duckdb_bind_int8(duckdb_prepared_statement prepared_statement, idx_t param_idx, int8_t val);
duckdb_state duckdb_bind_int16(duckdb_prepared_statement prepared_statement, idx_t param_idx, int16_t val);
duckdb_state duckdb_bind_int32(duckdb_prepared_statement prepared_statement, idx_t param_idx, int32_t val);
duckdb_state duckdb_bind_int64(duckdb_prepared_statement prepared_statement, idx_t param_idx, int64_t val);
duckdb_state duckdb_bind_hugeint(duckdb_prepared_statement prepared_statement, idx_t param_idx, duckdb_hugeint val);
+duckdb_state duckdb_bind_decimal(duckdb_prepared_statement prepared_statement, idx_t param_idx, duckdb_decimal val);
duckdb_state duckdb_bind_uint8(duckdb_prepared_statement prepared_statement, idx_t param_idx, uint8_t val);
duckdb_state duckdb_bind_uint16(duckdb_prepared_statement prepared_statement, idx_t param_idx, uint16_t val);
duckdb_state duckdb_bind_uint32(duckdb_prepared_statement prepared_statement, idx_t param_idx, uint32_t val);
@@ -106,13 +116,23 @@ selected: API Reference
duckdb_state duckdb_bind_null(duckdb_prepared_statement prepared_statement, idx_t param_idx);
duckdb_state duckdb_execute_prepared(duckdb_prepared_statement prepared_statement, duckdb_result *out_result);
duckdb_state duckdb_execute_prepared_arrow(duckdb_prepared_statement prepared_statement, duckdb_arrow *out_result);
+duckdb_state duckdb_arrow_scan(duckdb_connection connection, const char *table_name, duckdb_arrow_stream arrow);
+duckdb_state duckdb_arrow_array_scan(duckdb_connection connection, const char *table_name, duckdb_arrow_schema arrow_schema, duckdb_arrow_array arrow_array, duckdb_arrow_stream *out_stream);
+
+### **Extract Statements**
+
### **Pending Result Interface**
### **Value Interface**
void duckdb_destroy_value(duckdb_value *value);
@@ -126,6 +146,8 @@ selected: API Reference
duckdb_logical_type duckdb_create_logical_type(duckdb_type type);
duckdb_logical_type duckdb_create_list_type(duckdb_logical_type type);
duckdb_logical_type duckdb_create_map_type(duckdb_logical_type key_type, duckdb_logical_type value_type);
+duckdb_logical_type duckdb_create_union_type(duckdb_logical_type member_types, const char **member_names, idx_t member_count);
+duckdb_logical_type duckdb_create_struct_type(duckdb_logical_type *member_types, const char **member_names, idx_t member_count);
duckdb_logical_type duckdb_create_decimal_type(uint8_t width, uint8_t scale);
duckdb_type duckdb_get_type_id(duckdb_logical_type type);
uint8_t duckdb_decimal_width(duckdb_logical_type type);
@@ -140,6 +162,9 @@ selected: API Reference
idx_t duckdb_struct_type_child_count(duckdb_logical_type type);
char *duckdb_struct_type_child_name(duckdb_logical_type type, idx_t index);
duckdb_logical_type duckdb_struct_type_child_type(duckdb_logical_type type, idx_t index);
+idx_t duckdb_union_type_member_count(duckdb_logical_type type);
+char *duckdb_union_type_member_name(duckdb_logical_type type, idx_t index);
+duckdb_logical_type duckdb_union_type_member_type(duckdb_logical_type type, idx_t index);
void duckdb_destroy_logical_type(duckdb_logical_type *type);
### **Data Chunk Interface**
@@ -160,6 +185,8 @@ selected: API Reference
void duckdb_vector_assign_string_element_len(duckdb_vector vector, idx_t index, const char *str, idx_t str_len);
duckdb_vector duckdb_list_vector_get_child(duckdb_vector vector);
idx_t duckdb_list_vector_get_size(duckdb_vector vector);
+duckdb_state duckdb_list_vector_set_size(duckdb_vector vector, idx_t size);
+duckdb_state duckdb_list_vector_reserve(duckdb_vector vector, idx_t required_capacity);
duckdb_vector duckdb_struct_vector_get_child(duckdb_vector vector, idx_t index);
### **Validity Mask Functions**
@@ -173,6 +200,7 @@ selected: API Reference
void duckdb_destroy_table_function(duckdb_table_function *table_function);
void duckdb_table_function_set_name(duckdb_table_function table_function, const char *name);
void duckdb_table_function_add_parameter(duckdb_table_function table_function, duckdb_logical_type type);
+void duckdb_table_function_add_named_parameter(duckdb_table_function table_function, const char *name, duckdb_logical_type type);
void duckdb_table_function_set_extra_info(duckdb_table_function table_function, void *extra_info, duckdb_delete_callback_t destroy);
void duckdb_table_function_set_bind(duckdb_table_function table_function, duckdb_table_function_bind_t bind);
void duckdb_table_function_set_init(duckdb_table_function table_function, duckdb_table_function_init_t init);
@@ -186,6 +214,7 @@ selected: API Reference
void duckdb_bind_add_result_column(duckdb_bind_info info, const char *name, duckdb_logical_type type);
idx_t duckdb_bind_get_parameter_count(duckdb_bind_info info);
duckdb_value duckdb_bind_get_parameter(duckdb_bind_info info, idx_t index);
+duckdb_value duckdb_bind_get_named_parameter(duckdb_bind_info info, const char *name);
void duckdb_bind_set_bind_data(duckdb_bind_info info, void *bind_data, duckdb_delete_callback_t destroy);
void duckdb_bind_set_cardinality(duckdb_bind_info info, idx_t cardinality, bool is_exact);
void duckdb_bind_set_error(duckdb_bind_info info, const char *error);
@@ -210,6 +239,7 @@ selected: API Reference
### **Appender**
duckdb_state duckdb_appender_create(duckdb_connection connection, const char *schema, const char *table, duckdb_appender *out_appender);
@@ -244,6 +274,7 @@ selected: API Reference
### **Arrow Interface**
+### **Streaming Result Interface**
+
### duckdb_open
---
-Creates a new database or opens an existing database file stored at the the given path.
+Creates a new database or opens an existing database file stored at the given path.
If no path is given a new in-memory database is created instead.
+The instantiated database should be closed with 'duckdb_close'
#### Syntax
---
@@ -288,7 +324,7 @@ The result database object.
### duckdb_open_ext
---
-Extended version of duckdb_open. Creates a new database or opens an existing database file stored at the the given path.
+Extended version of duckdb_open. Creates a new database or opens an existing database file stored at the given path.
#### Syntax
---
@@ -345,6 +381,7 @@ The database object to shut down.
---
Opens a connection to a database. Connections are required to query the database, and store transactional state
associated with the connection.
+The instantiated connection should be closed using 'duckdb_disconnect'
#### Syntax
---
@@ -367,6 +404,45 @@ The result connection object.
+### duckdb_interrupt
+---
+Interrupt running query
+
+#### Syntax
+---
+void duckdb_interrupt(
+ duckdb_connection connection
+);
+
+#### Parameters
+---
+* `connection`
+
+The connection to interruot
+
+
+
+### duckdb_query_progress
+---
+Get progress of the running query
+
+#### Syntax
+---
+double duckdb_query_progress(
+ duckdb_connection connection
+);
+
+#### Parameters
+---
+* `connection`
+
+The working connection
+* `returns`
+
+-1 if no progress or a percentage of the progress
+
+
+
### duckdb_disconnect
---
Closes the specified connection and de-allocates all memory allocated for that connection.
@@ -385,6 +461,20 @@ The connection to close.
+### duckdb_library_version
+---
+Returns the version of the linked DuckDB, with a version postfix for dev versions
+
+Usually used for developing C extensions that must return this for a compatibility check.
+
+#### Syntax
+---
+const char *duckdb_library_version(
+
+);
+
+
+
### duckdb_create_config
---
Initializes an empty configuration object that can be used to provide start-up options for the DuckDB instance
@@ -816,6 +906,8 @@ The error of the result.
---
Fetches a data chunk from the duckdb_result. This function should be called repeatedly until the result is exhausted.
+The result must be destroyed with `duckdb_destroy_data_chunk`.
+
This function supersedes all `duckdb_value` functions, as well as the `duckdb_column_data` and `duckdb_nullmask_data`
functions. It results in significantly better performance, and should be preferred in newer code-bases.
@@ -845,6 +937,27 @@ The resulting data chunk. Returns `NULL` if the chunk index is out of bounds.
+### duckdb_result_is_streaming
+---
+Checks if the type of the internal result is StreamQueryResult.
+
+#### Syntax
+---
+bool duckdb_result_is_streaming(
+ duckdb_result result
+);
+
+#### Parameters
+---
+* `result`
+
+The result object to check.
+* `returns`
+
+Whether or not the result object is of the type StreamQueryResult
+
+
+
### duckdb_result_chunk_count
---
Returns the number of data chunks present in the result.
@@ -862,7 +975,7 @@ Returns the number of data chunks present in the result.
The result object
* `returns`
-The resulting data chunk. Returns `NULL` if the chunk index is out of bounds.
+Number of data chunks present in the result.
@@ -1184,10 +1297,13 @@ The duckdb_interval value at the specified location, or 0 if the value cannot be
#### Parameters
---
+* `DEPRECATED`
+
+use duckdb_value_string instead. This function does not work correctly if the string contains null bytes.
* `returns`
-The char* value at the specified location, or nullptr if the value cannot be converted.
-The result must be freed with `duckdb_free`.
+The text value at the specified location as a null-terminated string, or nullptr if the value cannot be
+converted. The result must be freed with `duckdb_free`.
@@ -1203,6 +1319,35 @@ The result must be freed with `duckdb_free`.
#### Parameters
---
+* `DEPRECATED`
+
+use duckdb_value_string_internal instead. This function does not work correctly if the string contains
+null bytes.
+* `returns`
+
+The char* value at the specified location. ONLY works on VARCHAR columns and does not auto-cast.
+If the column is NOT a VARCHAR column this function will return NULL.
+
+The result must NOT be freed.
+
+
+
+### duckdb_value_string_internal
+---
+#### Syntax
+---
+duckdb_string duckdb_value_string_internal(
+ duckdb_result *result,
+ idx_t col,
+ idx_t row
+);
+
+#### Parameters
+---
+* `DEPRECATED`
+
+use duckdb_value_string_internal instead. This function does not work correctly if the string contains
+null bytes.
* `returns`
The char* value at the specified location. ONLY works on VARCHAR columns and does not auto-cast.
@@ -1308,6 +1453,20 @@ The vector size.
+### duckdb_string_is_inlined
+---
+Whether or not the duckdb_string_t value is inlined.
+This means that the data of the string does not have a separate allocation.
+
+
+#### Syntax
+---
+bool duckdb_string_is_inlined(
+ duckdb_string_t string
+);
+
+
+
### duckdb_from_date
---
Decompose a `duckdb_date` object into year, month and date (stored as `duckdb_date_struct`).
@@ -1478,6 +1637,31 @@ The converted `duckdb_hugeint` element.
+### duckdb_double_to_decimal
+---
+Converts a double value to a duckdb_decimal object.
+
+If the conversion fails because the double value is too big, or the width/scale are invalid the result will be 0.
+
+#### Syntax
+---
+duckdb_decimal duckdb_double_to_decimal(
+ double val,
+ uint8_t width,
+ uint8_t scale
+);
+
+#### Parameters
+---
+* `val`
+
+The double value.
+* `returns`
+
+The converted `duckdb_decimal` element.
+
+
+
### duckdb_decimal_to_double
---
Converts a duckdb_decimal object (as obtained from a `DUCKDB_TYPE_DECIMAL` column) into a double.
@@ -1634,6 +1818,34 @@ Clear the params bind to the prepared statement.
+### duckdb_bind_value
+---
+Binds a value to the prepared statement at the specified index.
+
+#### Syntax
+---
+duckdb_state duckdb_bind_value(
+ duckdb_prepared_statement prepared_statement,
+ idx_t param_idx,
+ duckdb_value val
+);
+
+
+
+### duckdb_bind_parameter_index
+---
+Retrieve the index of the parameter for the prepared statement, identified by name
+
+#### Syntax
+---
+duckdb_state duckdb_bind_parameter_index(
+ duckdb_prepared_statement prepared_statement,
+ idx_t *param_idx_out,
+ const char *name
+);
+
+
+
### duckdb_bind_boolean
---
Binds a bool value to the prepared statement at the specified index.
@@ -1706,7 +1918,7 @@ Binds an int64_t value to the prepared statement at the specified index.
### duckdb_bind_hugeint
---
-Binds an duckdb_hugeint value to the prepared statement at the specified index.
+Binds a duckdb_hugeint value to the prepared statement at the specified index.
#### Syntax
---
@@ -1718,6 +1930,20 @@ Binds an duckdb_hugeint value to the prepared statement at the specified index.
+### duckdb_bind_decimal
+---
+Binds a duckdb_decimal value to the prepared statement at the specified index.
+
+#### Syntax
+---
+duckdb_state duckdb_bind_decimal(
+ duckdb_prepared_statement prepared_statement,
+ idx_t param_idx,
+ duckdb_decimal val
+);
+
+
+
### duckdb_bind_uint8
---
Binds an uint8_t value to the prepared statement at the specified index.
@@ -1776,7 +2002,7 @@ Binds an uint64_t value to the prepared statement at the specified index.
### duckdb_bind_float
---
-Binds an float value to the prepared statement at the specified index.
+Binds a float value to the prepared statement at the specified index.
#### Syntax
---
@@ -1790,7 +2016,7 @@ Binds an float value to the prepared statement at the specified index.
### duckdb_bind_double
---
-Binds an double value to the prepared statement at the specified index.
+Binds a double value to the prepared statement at the specified index.
#### Syntax
---
@@ -1968,6 +2194,176 @@ The query result.
+### duckdb_arrow_scan
+---
+Scans the Arrow stream and creates a view with the given name.
+
+#### Syntax
+---
+duckdb_state duckdb_arrow_scan(
+ duckdb_connection connection,
+ const char *table_name,
+ duckdb_arrow_stream arrow
+);
+
+#### Parameters
+---
+* `connection`
+
+The connection on which to execute the scan.
+* `table_name`
+
+Name of the temporary view to create.
+* `arrow`
+
+Arrow stream wrapper.
+* `returns`
+
+`DuckDBSuccess` on success or `DuckDBError` on failure.
+
+
+
+### duckdb_arrow_array_scan
+---
+Scans the Arrow array and creates a view with the given name.
+
+#### Syntax
+---
+duckdb_state duckdb_arrow_array_scan(
+ duckdb_connection connection,
+ const char *table_name,
+ duckdb_arrow_schema arrow_schema,
+ duckdb_arrow_array arrow_array,
+ duckdb_arrow_stream *out_stream
+);
+
+#### Parameters
+---
+* `connection`
+
+The connection on which to execute the scan.
+* `table_name`
+
+Name of the temporary view to create.
+* `arrow_schema`
+
+Arrow schema wrapper.
+* `arrow_array`
+
+Arrow array wrapper.
+* `out_stream`
+
+Output array stream that wraps around the passed schema, for releasing/deleting once done.
+* `returns`
+
+`DuckDBSuccess` on success or `DuckDBError` on failure.
+
+
+
+### duckdb_extract_statements
+---
+Extract all statements from a query.
+Note that after calling `duckdb_extract_statements`, the extracted statements should always be destroyed using
+`duckdb_destroy_extracted`, even if no statements were extracted.
+If the extract fails, `duckdb_extract_statements_error` can be called to obtain the reason why the extract failed.
+#### Syntax
+---
+idx_t duckdb_extract_statements(
+ duckdb_connection connection,
+ const char *query,
+ duckdb_extracted_statements *out_extracted_statements
+);
+
+#### Parameters
+---
+* `connection`
+
+The connection object
+* `query`
+
+The SQL query to extract
+* `out_extracted_statements`
+
+The resulting extracted statements object
+* `returns`
+
+The number of extracted statements or 0 on failure.
+
+
+
+### duckdb_prepare_extracted_statement
+---
+Prepare an extracted statement.
+Note that after calling `duckdb_prepare_extracted_statement`, the prepared statement should always be destroyed using
+`duckdb_destroy_prepare`, even if the prepare fails.
+If the prepare fails, `duckdb_prepare_error` can be called to obtain the reason why the prepare failed.
+#### Syntax
+---
+duckdb_state duckdb_prepare_extracted_statement(
+ duckdb_connection connection,
+ duckdb_extracted_statements extracted_statements,
+ idx_t index,
+ duckdb_prepared_statement *out_prepared_statement
+);
+
+#### Parameters
+---
+* `connection`
+
+The connection object
+* `extracted_statements`
+
+The extracted statements object
+* `index`
+
+The index of the extracted statement to prepare
+* `out_prepared_statement`
+
+The resulting prepared statement object
+* `returns`
+
+`DuckDBSuccess` on success or `DuckDBError` on failure.
+
+
+
+### duckdb_extract_statements_error
+---
+Returns the error message contained within the extracted statements.
+The result of this function must not be freed. It will be cleaned up when `duckdb_destroy_extracted` is called.
+#### Syntax
+---
+const char *duckdb_extract_statements_error(
+ duckdb_extracted_statements extracted_statements
+);
+
+#### Parameters
+---
+* `result`
+
+The extracted statements to fetch the error from.
+* `returns`
+
+The error of the extracted statements.
+
+
+
+### duckdb_destroy_extracted
+---
+De-allocates all memory allocated for the extracted statements.
+#### Syntax
+---
+void duckdb_destroy_extracted(
+ duckdb_extracted_statements *extracted_statements
+);
+
+#### Parameters
+---
+* `extracted_statements`
+
+The extracted statements to destroy.
+
+
+
### duckdb_pending_prepared
---
Executes the prepared statement with the given bound parameters, and returns a pending result.
@@ -1998,6 +2394,36 @@ The pending query result.
+### duckdb_pending_prepared_streaming
+---
+Executes the prepared statement with the given bound parameters, and returns a pending result.
+This pending result will create a streaming duckdb_result when executed.
+The pending result represents an intermediate structure for a query that is not yet fully executed.
+
+Note that after calling `duckdb_pending_prepared_streaming`, the pending result should always be destroyed using
+`duckdb_destroy_pending`, even if this function returns DuckDBError.
+
+#### Syntax
+---
+duckdb_state duckdb_pending_prepared_streaming(
+ duckdb_prepared_statement prepared_statement,
+ duckdb_pending_result *out_result
+);
+
+#### Parameters
+---
+* `prepared_statement`
+
+The prepared statement to execute.
+* `out_result`
+
+The pending query result.
+* `returns`
+
+`DuckDBSuccess` on success or `DuckDBError` on failure.
+
+
+
### duckdb_destroy_pending
---
Closes the pending result and de-allocates all memory allocated for the result.
@@ -2087,10 +2513,32 @@ Otherwise, all remaining tasks must be executed first.
The pending result to execute.
* `out_result`
-The result object.
+The result object.
+* `returns`
+
+`DuckDBSuccess` on success or `DuckDBError` on failure.
+
+
+
+### duckdb_pending_execution_is_finished
+---
+Returns whether a duckdb_pending_state is finished executing. For example if `pending_state` is
+DUCKDB_PENDING_RESULT_READY, this function will return true.
+
+#### Syntax
+---
+bool duckdb_pending_execution_is_finished(
+ duckdb_pending_state pending_state
+);
+
+#### Parameters
+---
+* `pending_state`
+
+The pending state on which to decide whether to finish execution.
* `returns`
-`DuckDBSuccess` on success or `DuckDBError` on failure.
+Boolean indicating pending execution should be considered finished.
@@ -2291,6 +2739,63 @@ The logical type.
+### duckdb_create_union_type
+---
+Creates a UNION type from the passed types array
+The resulting type should be destroyed with `duckdb_destroy_logical_type`.
+
+#### Syntax
+---
+duckdb_logical_type duckdb_create_union_type(
+ duckdb_logical_type member_types,
+ const char **member_names,
+ idx_t member_count
+);
+
+#### Parameters
+---
+* `types`
+
+The array of types that the union should consist of.
+* `type_amount`
+
+The size of the types array.
+* `returns`
+
+The logical type.
+
+
+
+### duckdb_create_struct_type
+---
+Creates a STRUCT type from the passed member name and type arrays.
+The resulting type should be destroyed with `duckdb_destroy_logical_type`.
+
+#### Syntax
+---
+duckdb_logical_type duckdb_create_struct_type(
+ duckdb_logical_type *member_types,
+ const char **member_names,
+ idx_t member_count
+);
+
+#### Parameters
+---
+* `member_types`
+
+The array of types that the struct should consist of.
+* `member_names`
+
+The array of names that the struct should consist of.
+* `member_count`
+
+The number of members that were specified for both arrays.
+* `returns`
+
+The logical type.
+
+
+
### duckdb_create_decimal_type
---
Creates a `duckdb_logical_type` of type decimal with the specified width and scale
@@ -2614,6 +3119,81 @@ The child type of the struct type. Must be destroyed with `duckdb_destroy_logica
+### duckdb_union_type_member_count
+---
+Returns the number of members that the union type has.
+
+#### Syntax
+---
+idx_t duckdb_union_type_member_count(
+ duckdb_logical_type type
+);
+
+#### Parameters
+---
+* `type`
+
+The logical type (union) object
+* `returns`
+
+The number of members of a union type.
+
+
+
+### duckdb_union_type_member_name
+---
+Retrieves the name of the union member.
+
+The result must be freed with `duckdb_free`
+
+#### Syntax
+---
+char *duckdb_union_type_member_name(
+ duckdb_logical_type type,
+ idx_t index
+);
+
+#### Parameters
+---
+* `type`
+
+The logical type object
+* `index`
+
+The child index
+* `returns`
+
+The name of the union member. Must be freed with `duckdb_free`.
+
+
+
+### duckdb_union_type_member_type
+---
+Retrieves the child type of the given union member at the specified index.
+
+The result must be freed with `duckdb_destroy_logical_type`
+
+#### Syntax
+---
+duckdb_logical_type duckdb_union_type_member_type(
+ duckdb_logical_type type,
+ idx_t index
+);
+
+#### Parameters
+---
+* `type`
+
+The logical type object
+* `index`
+
+The child index
+* `returns`
+
+The child type of the union member. Must be destroyed with `duckdb_destroy_logical_type`.
+
+
+
### duckdb_destroy_logical_type
---
Destroys the logical type and de-allocates all memory allocated for that type.
@@ -2985,6 +3565,56 @@ The size of the child list
+### duckdb_list_vector_set_size
+---
+Sets the total size of the underlying child-vector of a list vector.
+
+#### Syntax
+---
+duckdb_state duckdb_list_vector_set_size(
+ duckdb_vector vector,
+ idx_t size
+);
+
+#### Parameters
+---
+* `vector`
+
+The list vector.
+* `size`
+
+The size of the child list.
+* `returns`
+
+The duckdb state. Returns DuckDBError if the vector is nullptr.
+
+
+
+### duckdb_list_vector_reserve
+---
+Sets the total capacity of the underlying child-vector of a list.
+
+#### Syntax
+---
+duckdb_state duckdb_list_vector_reserve(
+ duckdb_vector vector,
+ idx_t required_capacity
+);
+
+#### Parameters
+---
+* `vector`
+
+The list vector.
+* `required_capacity`
+
+the total capacity to reserve.
+* `return`
+
+The duckdb state. Returns DuckDBError if the vector is nullptr.
+
+
+
### duckdb_struct_vector_get_child
---
Retrieves the child vector of a struct vector.
@@ -3027,7 +3657,7 @@ Returns whether or not a row is valid (i.e. not NULL) in the given validity mask
---
* `validity`
-The validity mask, as obtained through `duckdb_data_chunk_get_validity`
+The validity mask, as obtained through `duckdb_vector_get_validity`
* `row`
The row index
@@ -3041,7 +3671,7 @@ true if the row is valid, false otherwise
---
In a validity mask, sets a specific row to either valid or invalid.
-Note that `duckdb_data_chunk_ensure_validity_writable` should be called before calling `duckdb_data_chunk_get_validity`,
+Note that `duckdb_vector_ensure_validity_writable` should be called before calling `duckdb_vector_get_validity`,
to ensure that there is a validity mask to write to.
#### Syntax
@@ -3056,7 +3686,7 @@ to ensure that there is a validity mask to write to.
---
* `validity`
-The validity mask, as obtained through `duckdb_data_chunk_get_validity`.
+The validity mask, as obtained through `duckdb_vector_get_validity`.
* `row`
The row index
@@ -3196,6 +3826,32 @@ The type of the parameter to add.
+### duckdb_table_function_add_named_parameter
+---
+Adds a named parameter to the table function.
+
+#### Syntax
+---
+void duckdb_table_function_add_named_parameter(
+ duckdb_table_function table_function,
+ const char *name,
+ duckdb_logical_type type
+);
+
+#### Parameters
+---
+* `table_function`
+
+The table function
+* `name`
+
+The name of the parameter
+* `type`
+
+The type of the parameter to add.
+
+
+
### duckdb_table_function_set_extra_info
---
Assigns extra information to the table function that can be fetched during binding, etc.
@@ -3460,6 +4116,33 @@ The value of the parameter. Must be destroyed with `duckdb_destroy_value`.
+### duckdb_bind_get_named_parameter
+---
+Retrieves a named parameter with the given name.
+
+The result must be destroyed with `duckdb_destroy_value`.
+
+#### Syntax
+---
+duckdb_value duckdb_bind_get_named_parameter(
+ duckdb_bind_info info,
+ const char *name
+);
+
+#### Parameters
+---
+* `info`
+
+The info object
+* `name`
+
+The name of the parameter
+* `returns`
+
+The value of the parameter. Must be destroyed with `duckdb_destroy_value`.
+
+
+
### duckdb_bind_set_bind_data
---
Sets the user-provided bind data in the bind object. This object can be retrieved again during execution.
@@ -3880,6 +4563,28 @@ The parameter to add.
+### duckdb_replacement_scan_set_error
+---
+Report that an error has occurred while executing the replacement scan.
+
+#### Syntax
+---
+void duckdb_replacement_scan_set_error(
+ duckdb_replacement_scan_info info,
+ const char *error
+);
+
+#### Parameters
+---
+* `info`
+
+The info object
+* `error`
+
+The error message
+
+
+
### duckdb_appender_create
---
Creates an appender object.
@@ -4389,6 +5094,31 @@ The output schema.
+### duckdb_prepared_arrow_schema
+---
+Fetch the internal arrow schema from the prepared statement.
+
+#### Syntax
+---
+duckdb_state duckdb_prepared_arrow_schema(
+ duckdb_prepared_statement prepared,
+ duckdb_arrow_schema *out_schema
+);
+
+#### Parameters
+---
+* `result`
+
+The prepared statement to fetch the schema from.
+* `out_schema`
+
+The output schema.
+* `returns`
+
+`DuckDBSuccess` on success or `DuckDBError` on failure.
+
+
+
### duckdb_query_arrow_array
---
Fetch an internal arrow array from the arrow result.
@@ -4682,3 +5412,52 @@ The task state to clean up
+### duckdb_execution_is_finished
+---
+Returns true if execution of the current query is finished.
+
+#### Syntax
+---
+bool duckdb_execution_is_finished(
+ duckdb_connection con
+);
+
+#### Parameters
+---
+* `con`
+
+The connection on which to check
+
+
+
+### duckdb_stream_fetch_chunk
+---
+Fetches a data chunk from the (streaming) duckdb_result. This function should be called repeatedly until the result is
+exhausted.
+
+The result must be destroyed with `duckdb_destroy_data_chunk`.
+
+This function can only be used on duckdb_results created with 'duckdb_pending_prepared_streaming'
+
+If this function is used, none of the other result functions can be used and vice versa (i.e. this function cannot be
+mixed with the legacy result functions or the materialized result functions).
+
+It is not known beforehand how many chunks will be returned by this result.
+
+#### Syntax
+---
+duckdb_data_chunk duckdb_stream_fetch_chunk(
+ duckdb_result result
+);
+
+#### Parameters
+---
+* `result`
+
+The result object to fetch the data chunk from.
+* `returns`
+
+The resulting data chunk. Returns `NULL` if the result has an error.
+
+
+
diff --git a/docs/api/c/connect.md b/docs/api/c/connect.md
index 79a32084a0b..29295f6415e 100644
--- a/docs/api/c/connect.md
+++ b/docs/api/c/connect.md
@@ -33,12 +33,16 @@ duckdb_close(&db);
duckdb_state duckdb_open_ext(const char *path, duckdb_database *out_database, duckdb_config config, char **out_error);
void duckdb_close(duckdb_database *database);
duckdb_state duckdb_connect(duckdb_database database, duckdb_connection *out_connection);
+void duckdb_interrupt(duckdb_connection connection);
+double duckdb_query_progress(duckdb_connection connection);
void duckdb_disconnect(duckdb_connection *connection);
+const char *duckdb_library_version();
### duckdb_open
---
-Creates a new database or opens an existing database file stored at the the given path.
+Creates a new database or opens an existing database file stored at the given path.
If no path is given a new in-memory database is created instead.
+The instantiated database should be closed with 'duckdb_close'
#### Syntax
---
@@ -63,7 +67,7 @@ The result database object.
### duckdb_open_ext
---
-Extended version of duckdb_open. Creates a new database or opens an existing database file stored at the the given path.
+Extended version of duckdb_open. Creates a new database or opens an existing database file stored at the given path.
#### Syntax
---
@@ -120,6 +124,7 @@ The database object to shut down.
---
Opens a connection to a database. Connections are required to query the database, and store transactional state
associated with the connection.
+The instantiated connection should be closed using 'duckdb_disconnect'
#### Syntax
---
@@ -142,6 +147,45 @@ The result connection object.
+### duckdb_interrupt
+---
+Interrupt running query
+
+#### Syntax
+---
+void duckdb_interrupt(
+ duckdb_connection connection
+);
+
+#### Parameters
+---
+* `connection`
+
+The connection to interruot
+
+
+
+### duckdb_query_progress
+---
+Get progress of the running query
+
+#### Syntax
+---
+double duckdb_query_progress(
+ duckdb_connection connection
+);
+
+#### Parameters
+---
+* `connection`
+
+The working connection
+* `returns`
+
+-1 if no progress or a percentage of the progress
+
+
+
### duckdb_disconnect
---
Closes the specified connection and de-allocates all memory allocated for that connection.
@@ -160,3 +204,17 @@ The connection to close.
+### duckdb_library_version
+---
+Returns the version of the linked DuckDB, with a version postfix for dev versions
+
+Usually used for developing C extensions that must return this for a compatibility check.
+
+#### Syntax
+---
+const char *duckdb_library_version(
+
+);
+
+
+
diff --git a/docs/api/c/data_chunk.md b/docs/api/c/data_chunk.md
index 92c5e7af142..9c65ce24942 100644
--- a/docs/api/c/data_chunk.md
+++ b/docs/api/c/data_chunk.md
@@ -30,6 +30,8 @@ The primary manner of interfacing with data chunks is by obtaining the internal
void duckdb_vector_assign_string_element_len(duckdb_vector vector, idx_t index, const char *str, idx_t str_len);
duckdb_vector duckdb_list_vector_get_child(duckdb_vector vector);
idx_t duckdb_list_vector_get_size(duckdb_vector vector);
+duckdb_state duckdb_list_vector_set_size(duckdb_vector vector, idx_t size);
+duckdb_state duckdb_list_vector_reserve(duckdb_vector vector, idx_t required_capacity);
duckdb_vector duckdb_struct_vector_get_child(duckdb_vector vector, idx_t index);
#### Validity Mask Functions
@@ -391,6 +393,56 @@ The size of the child list
+### duckdb_list_vector_set_size
+---
+Sets the total size of the underlying child-vector of a list vector.
+
+#### Syntax
+---
+duckdb_state duckdb_list_vector_set_size(
+ duckdb_vector vector,
+ idx_t size
+);
+
+#### Parameters
+---
+* `vector`
+
+The list vector.
+* `size`
+
+The size of the child list.
+* `returns`
+
+The duckdb state. Returns DuckDBError if the vector is nullptr.
+
+
+
+### duckdb_list_vector_reserve
+---
+Sets the total capacity of the underlying child-vector of a list.
+
+#### Syntax
+---
+duckdb_state duckdb_list_vector_reserve(
+ duckdb_vector vector,
+ idx_t required_capacity
+);
+
+#### Parameters
+---
+* `vector`
+
+The list vector.
+* `required_capacity`
+
+the total capacity to reserve.
+* `return`
+
+The duckdb state. Returns DuckDBError if the vector is nullptr.
+
+
+
### duckdb_struct_vector_get_child
---
Retrieves the child vector of a struct vector.
@@ -433,7 +485,7 @@ Returns whether or not a row is valid (i.e. not NULL) in the given validity mask
---
* `validity`
-The validity mask, as obtained through `duckdb_data_chunk_get_validity`
+The validity mask, as obtained through `duckdb_vector_get_validity`
* `row`
The row index
@@ -447,7 +499,7 @@ true if the row is valid, false otherwise
---
In a validity mask, sets a specific row to either valid or invalid.
-Note that `duckdb_data_chunk_ensure_validity_writable` should be called before calling `duckdb_data_chunk_get_validity`,
+Note that `duckdb_vector_ensure_validity_writable` should be called before calling `duckdb_vector_get_validity`,
to ensure that there is a validity mask to write to.
#### Syntax
@@ -462,7 +514,7 @@ to ensure that there is a validity mask to write to.
---
* `validity`
-The validity mask, as obtained through `duckdb_data_chunk_get_validity`.
+The validity mask, as obtained through `duckdb_vector_get_validity`.
* `row`
The row index
diff --git a/docs/api/c/prepared.md b/docs/api/c/prepared.md
index e023a4f3afa..bcc88381058 100644
--- a/docs/api/c/prepared.md
+++ b/docs/api/c/prepared.md
@@ -53,12 +53,15 @@ It is not required that the `duckdb_bind` family of functions matches the prepar
idx_t duckdb_nparams(duckdb_prepared_statement prepared_statement);
duckdb_type duckdb_param_type(duckdb_prepared_statement prepared_statement, idx_t param_idx);
duckdb_state duckdb_clear_bindings(duckdb_prepared_statement prepared_statement);
+duckdb_state duckdb_bind_value(duckdb_prepared_statement prepared_statement, idx_t param_idx, duckdb_value val);
+duckdb_state duckdb_bind_parameter_index(duckdb_prepared_statement prepared_statement, idx_t *param_idx_out, const char *name);
duckdb_state duckdb_bind_boolean(duckdb_prepared_statement prepared_statement, idx_t param_idx, bool val);
duckdb_state duckdb_bind_int8(duckdb_prepared_statement prepared_statement, idx_t param_idx, int8_t val);
duckdb_state duckdb_bind_int16(duckdb_prepared_statement prepared_statement, idx_t param_idx, int16_t val);
duckdb_state duckdb_bind_int32(duckdb_prepared_statement prepared_statement, idx_t param_idx, int32_t val);
duckdb_state duckdb_bind_int64(duckdb_prepared_statement prepared_statement, idx_t param_idx, int64_t val);
duckdb_state duckdb_bind_hugeint(duckdb_prepared_statement prepared_statement, idx_t param_idx, duckdb_hugeint val);
+duckdb_state duckdb_bind_decimal(duckdb_prepared_statement prepared_statement, idx_t param_idx, duckdb_decimal val);
duckdb_state duckdb_bind_uint8(duckdb_prepared_statement prepared_statement, idx_t param_idx, uint8_t val);
duckdb_state duckdb_bind_uint16(duckdb_prepared_statement prepared_statement, idx_t param_idx, uint16_t val);
duckdb_state duckdb_bind_uint32(duckdb_prepared_statement prepared_statement, idx_t param_idx, uint32_t val);
@@ -75,6 +78,8 @@ It is not required that the `duckdb_bind` family of functions matches the prepar
duckdb_state duckdb_bind_null(duckdb_prepared_statement prepared_statement, idx_t param_idx);
duckdb_state duckdb_execute_prepared(duckdb_prepared_statement prepared_statement, duckdb_result *out_result);
duckdb_state duckdb_execute_prepared_arrow(duckdb_prepared_statement prepared_statement, duckdb_arrow *out_result);
+duckdb_state duckdb_arrow_scan(duckdb_connection connection, const char *table_name, duckdb_arrow_stream arrow);
+duckdb_state duckdb_arrow_array_scan(duckdb_connection connection, const char *table_name, duckdb_arrow_schema arrow_schema, duckdb_arrow_array arrow_array, duckdb_arrow_stream *out_stream);
### duckdb_prepare
---
@@ -211,6 +216,34 @@ Clear the params bind to the prepared statement.
+### duckdb_bind_value
+---
+Binds a value to the prepared statement at the specified index.
+
+#### Syntax
+---
+duckdb_state duckdb_bind_value(
+ duckdb_prepared_statement prepared_statement,
+ idx_t param_idx,
+ duckdb_value val
+);
+
+
+
+### duckdb_bind_parameter_index
+---
+Retrieve the index of the parameter for the prepared statement, identified by name
+
+#### Syntax
+---
+duckdb_state duckdb_bind_parameter_index(
+ duckdb_prepared_statement prepared_statement,
+ idx_t *param_idx_out,
+ const char *name
+);
+
+
+
### duckdb_bind_boolean
---
Binds a bool value to the prepared statement at the specified index.
@@ -283,7 +316,7 @@ Binds an int64_t value to the prepared statement at the specified index.
### duckdb_bind_hugeint
---
-Binds an duckdb_hugeint value to the prepared statement at the specified index.
+Binds a duckdb_hugeint value to the prepared statement at the specified index.
#### Syntax
---
@@ -295,6 +328,20 @@ Binds an duckdb_hugeint value to the prepared statement at the specified index.
+### duckdb_bind_decimal
+---
+Binds a duckdb_decimal value to the prepared statement at the specified index.
+
+#### Syntax
+---
+duckdb_state duckdb_bind_decimal(
+ duckdb_prepared_statement prepared_statement,
+ idx_t param_idx,
+ duckdb_decimal val
+);
+
+
+
### duckdb_bind_uint8
---
Binds an uint8_t value to the prepared statement at the specified index.
@@ -353,7 +400,7 @@ Binds an uint64_t value to the prepared statement at the specified index.
### duckdb_bind_float
---
-Binds an float value to the prepared statement at the specified index.
+Binds a float value to the prepared statement at the specified index.
#### Syntax
---
@@ -367,7 +414,7 @@ Binds an float value to the prepared statement at the specified index.
### duckdb_bind_double
---
-Binds an double value to the prepared statement at the specified index.
+Binds a double value to the prepared statement at the specified index.
#### Syntax
---
@@ -545,3 +592,69 @@ The query result.
+### duckdb_arrow_scan
+---
+Scans the Arrow stream and creates a view with the given name.
+
+#### Syntax
+---
+duckdb_state duckdb_arrow_scan(
+ duckdb_connection connection,
+ const char *table_name,
+ duckdb_arrow_stream arrow
+);
+
+#### Parameters
+---
+* `connection`
+
+The connection on which to execute the scan.
+* `table_name`
+
+Name of the temporary view to create.
+* `arrow`
+
+Arrow stream wrapper.
+* `returns`
+
+`DuckDBSuccess` on success or `DuckDBError` on failure.
+
+
+
+### duckdb_arrow_array_scan
+---
+Scans the Arrow array and creates a view with the given name.
+
+#### Syntax
+---
+duckdb_state duckdb_arrow_array_scan(
+ duckdb_connection connection,
+ const char *table_name,
+ duckdb_arrow_schema arrow_schema,
+ duckdb_arrow_array arrow_array,
+ duckdb_arrow_stream *out_stream
+);
+
+#### Parameters
+---
+* `connection`
+
+The connection on which to execute the scan.
+* `table_name`
+
+Name of the temporary view to create.
+* `arrow_schema`
+
+Arrow schema wrapper.
+* `arrow_array`
+
+Arrow array wrapper.
+* `out_stream`
+
+Output array stream that wraps around the passed schema, for releasing/deleting once done.
+* `returns`
+
+`DuckDBSuccess` on success or `DuckDBError` on failure.
+
+
+
diff --git a/docs/api/c/replacement_scans.md b/docs/api/c/replacement_scans.md
index 6ed0803dc62..3fd4707ddcc 100644
--- a/docs/api/c/replacement_scans.md
+++ b/docs/api/c/replacement_scans.md
@@ -10,6 +10,7 @@ The replacement scan API can be used to register a callback that is called when
### duckdb_add_replacement_scan
---
@@ -86,3 +87,25 @@ The parameter to add.
+### duckdb_replacement_scan_set_error
+---
+Report that an error has occurred while executing the replacement scan.
+
+#### Syntax
+---
+void duckdb_replacement_scan_set_error(
+ duckdb_replacement_scan_info info,
+ const char *error
+);
+
+#### Parameters
+---
+* `info`
+
+The info object
+* `error`
+
+The error message
+
+
+
diff --git a/docs/api/c/table_functions.md b/docs/api/c/table_functions.md
index 53f05630927..ee7120411df 100644
--- a/docs/api/c/table_functions.md
+++ b/docs/api/c/table_functions.md
@@ -11,6 +11,7 @@ The table function API can be used to define a table function that can then be c
void duckdb_destroy_table_function(duckdb_table_function *table_function);
void duckdb_table_function_set_name(duckdb_table_function table_function, const char *name);
void duckdb_table_function_add_parameter(duckdb_table_function table_function, duckdb_logical_type type);
+void duckdb_table_function_add_named_parameter(duckdb_table_function table_function, const char *name, duckdb_logical_type type);
void duckdb_table_function_set_extra_info(duckdb_table_function table_function, void *extra_info, duckdb_delete_callback_t destroy);
void duckdb_table_function_set_bind(duckdb_table_function table_function, duckdb_table_function_bind_t bind);
void duckdb_table_function_set_init(duckdb_table_function table_function, duckdb_table_function_init_t init);
@@ -24,6 +25,7 @@ The table function API can be used to define a table function that can then be c
void duckdb_bind_add_result_column(duckdb_bind_info info, const char *name, duckdb_logical_type type);
idx_t duckdb_bind_get_parameter_count(duckdb_bind_info info);
duckdb_value duckdb_bind_get_parameter(duckdb_bind_info info, idx_t index);
+duckdb_value duckdb_bind_get_named_parameter(duckdb_bind_info info, const char *name);
void duckdb_bind_set_bind_data(duckdb_bind_info info, void *bind_data, duckdb_delete_callback_t destroy);
void duckdb_bind_set_cardinality(duckdb_bind_info info, idx_t cardinality, bool is_exact);
void duckdb_bind_set_error(duckdb_bind_info info, const char *error);
@@ -126,6 +128,32 @@ The type of the parameter to add.
+### duckdb_table_function_add_named_parameter
+---
+Adds a named parameter to the table function.
+
+#### Syntax
+---
+void duckdb_table_function_add_named_parameter(
+ duckdb_table_function table_function,
+ const char *name,
+ duckdb_logical_type type
+);
+
+#### Parameters
+---
+* `table_function`
+
+The table function
+* `name`
+
+The name of the parameter
+* `type`
+
+The type of the parameter to add.
+
+
+
### duckdb_table_function_set_extra_info
---
Assigns extra information to the table function that can be fetched during binding, etc.
@@ -390,6 +418,33 @@ The value of the parameter. Must be destroyed with `duckdb_destroy_value`.
+### duckdb_bind_get_named_parameter
+---
+Retrieves a named parameter with the given name.
+
+The result must be destroyed with `duckdb_destroy_value`.
+
+#### Syntax
+---
+duckdb_value duckdb_bind_get_named_parameter(
+ duckdb_bind_info info,
+ const char *name
+);
+
+#### Parameters
+---
+* `info`
+
+The info object
+* `name`
+
+The name of the parameter
+* `returns`
+
+The value of the parameter. Must be destroyed with `duckdb_destroy_value`.
+
+
+
### duckdb_bind_set_bind_data
---
Sets the user-provided bind data in the bind object. This object can be retrieved again during execution.
diff --git a/docs/api/c/types.md b/docs/api/c/types.md
index 8ded87a2eca..418fa44cfee 100644
--- a/docs/api/c/types.md
+++ b/docs/api/c/types.md
@@ -69,6 +69,7 @@ For more information about data chunks, see the [documentation on data chunks](d
## **API Reference**
duckdb_data_chunk duckdb_result_get_chunk(duckdb_result result, idx_t chunk_index);
+bool duckdb_result_is_streaming(duckdb_result result);
idx_t duckdb_result_chunk_count(duckdb_result result);
bool duckdb_value_boolean(duckdb_result *result, idx_t col, idx_t row);
int8_t duckdb_value_int8(duckdb_result *result, idx_t col, idx_t row);
@@ -89,6 +90,7 @@ For more information about data chunks, see the [documentation on data chunks](d
duckdb_interval duckdb_value_interval(duckdb_result *result, idx_t col, idx_t row);
char *duckdb_value_varchar(duckdb_result *result, idx_t col, idx_t row);
char *duckdb_value_varchar_internal(duckdb_result *result, idx_t col, idx_t row);
+duckdb_string duckdb_value_string_internal(duckdb_result *result, idx_t col, idx_t row);
duckdb_blob duckdb_value_blob(duckdb_result *result, idx_t col, idx_t row);
bool duckdb_value_is_null(duckdb_result *result, idx_t col, idx_t row);
@@ -103,6 +105,7 @@ For more information about data chunks, see the [documentation on data chunks](d
#### Hugeint Helpers
#### Decimal Helpers
double duckdb_decimal_to_double(duckdb_decimal val);
@@ -111,6 +114,8 @@ For more information about data chunks, see the [documentation on data chunks](d
duckdb_logical_type duckdb_create_logical_type(duckdb_type type);
duckdb_logical_type duckdb_create_list_type(duckdb_logical_type type);
duckdb_logical_type duckdb_create_map_type(duckdb_logical_type key_type, duckdb_logical_type value_type);
+duckdb_logical_type duckdb_create_union_type(duckdb_logical_type member_types, const char **member_names, idx_t member_count);
+duckdb_logical_type duckdb_create_struct_type(duckdb_logical_type *member_types, const char **member_names, idx_t member_count);
duckdb_logical_type duckdb_create_decimal_type(uint8_t width, uint8_t scale);
duckdb_type duckdb_get_type_id(duckdb_logical_type type);
uint8_t duckdb_decimal_width(duckdb_logical_type type);
@@ -125,12 +130,17 @@ For more information about data chunks, see the [documentation on data chunks](d
idx_t duckdb_struct_type_child_count(duckdb_logical_type type);
char *duckdb_struct_type_child_name(duckdb_logical_type type, idx_t index);
duckdb_logical_type duckdb_struct_type_child_type(duckdb_logical_type type, idx_t index);
+idx_t duckdb_union_type_member_count(duckdb_logical_type type);
+char *duckdb_union_type_member_name(duckdb_logical_type type, idx_t index);
+duckdb_logical_type duckdb_union_type_member_type(duckdb_logical_type type, idx_t index);
void duckdb_destroy_logical_type(duckdb_logical_type *type);
### duckdb_result_get_chunk
---
Fetches a data chunk from the duckdb_result. This function should be called repeatedly until the result is exhausted.
+The result must be destroyed with `duckdb_destroy_data_chunk`.
+
This function supersedes all `duckdb_value` functions, as well as the `duckdb_column_data` and `duckdb_nullmask_data`
functions. It results in significantly better performance, and should be preferred in newer code-bases.
@@ -160,6 +170,27 @@ The resulting data chunk. Returns `NULL` if the chunk index is out of bounds.
+### duckdb_result_is_streaming
+---
+Checks if the type of the internal result is StreamQueryResult.
+
+#### Syntax
+---
+bool duckdb_result_is_streaming(
+ duckdb_result result
+);
+
+#### Parameters
+---
+* `result`
+
+The result object to check.
+* `returns`
+
+Whether or not the result object is of the type StreamQueryResult
+
+
+
### duckdb_result_chunk_count
---
Returns the number of data chunks present in the result.
@@ -177,7 +208,7 @@ Returns the number of data chunks present in the result.
The result object
* `returns`
-The resulting data chunk. Returns `NULL` if the chunk index is out of bounds.
+Number of data chunks present in the result.
@@ -499,10 +530,13 @@ The duckdb_interval value at the specified location, or 0 if the value cannot be
#### Parameters
---
+* `DEPRECATED`
+
+use duckdb_value_string instead. This function does not work correctly if the string contains null bytes.
* `returns`
-The char* value at the specified location, or nullptr if the value cannot be converted.
-The result must be freed with `duckdb_free`.
+The text value at the specified location as a null-terminated string, or nullptr if the value cannot be
+converted. The result must be freed with `duckdb_free`.
@@ -518,6 +552,35 @@ The result must be freed with `duckdb_free`.
#### Parameters
---
+* `DEPRECATED`
+
+use duckdb_value_string_internal instead. This function does not work correctly if the string contains
+null bytes.
+* `returns`
+
+The char* value at the specified location. ONLY works on VARCHAR columns and does not auto-cast.
+If the column is NOT a VARCHAR column this function will return NULL.
+
+The result must NOT be freed.
+
+
+
+### duckdb_value_string_internal
+---
+#### Syntax
+---
+duckdb_string duckdb_value_string_internal(
+ duckdb_result *result,
+ idx_t col,
+ idx_t row
+);
+
+#### Parameters
+---
+* `DEPRECATED`
+
+use duckdb_value_string_internal instead. This function does not work correctly if the string contains
+null bytes.
* `returns`
The char* value at the specified location. ONLY works on VARCHAR columns and does not auto-cast.
@@ -734,6 +797,31 @@ The converted `duckdb_hugeint` element.
+### duckdb_double_to_decimal
+---
+Converts a double value to a duckdb_decimal object.
+
+If the conversion fails because the double value is too big, or the width/scale are invalid the result will be 0.
+
+#### Syntax
+---
+duckdb_decimal duckdb_double_to_decimal(
+ double val,
+ uint8_t width,
+ uint8_t scale
+);
+
+#### Parameters
+---
+* `val`
+
+The double value.
+* `returns`
+
+The converted `duckdb_decimal` element.
+
+
+
### duckdb_decimal_to_double
---
Converts a duckdb_decimal object (as obtained from a `DUCKDB_TYPE_DECIMAL` column) into a double.
@@ -824,6 +912,63 @@ The logical type.
+### duckdb_create_union_type
+---
+Creates a UNION type from the passed types array
+The resulting type should be destroyed with `duckdb_destroy_logical_type`.
+
+#### Syntax
+---
+duckdb_logical_type duckdb_create_union_type(
+ duckdb_logical_type member_types,
+ const char **member_names,
+ idx_t member_count
+);
+
+#### Parameters
+---
+* `types`
+
+The array of types that the union should consist of.
+* `type_amount`
+
+The size of the types array.
+* `returns`
+
+The logical type.
+
+
+
+### duckdb_create_struct_type
+---
+Creates a STRUCT type from the passed member name and type arrays.
+The resulting type should be destroyed with `duckdb_destroy_logical_type`.
+
+#### Syntax
+---
+duckdb_logical_type duckdb_create_struct_type(
+ duckdb_logical_type *member_types,
+ const char **member_names,
+ idx_t member_count
+);
+
+#### Parameters
+---
+* `member_types`
+
+The array of types that the struct should consist of.
+* `member_names`
+
+The array of names that the struct should consist of.
+* `member_count`
+
+The number of members that were specified for both arrays.
+* `returns`
+
+The logical type.
+
+
+
### duckdb_create_decimal_type
---
Creates a `duckdb_logical_type` of type decimal with the specified width and scale
@@ -1147,6 +1292,81 @@ The child type of the struct type. Must be destroyed with `duckdb_destroy_logica
+### duckdb_union_type_member_count
+---
+Returns the number of members that the union type has.
+
+#### Syntax
+---
+idx_t duckdb_union_type_member_count(
+ duckdb_logical_type type
+);
+
+#### Parameters
+---
+* `type`
+
+The logical type (union) object
+* `returns`
+
+The number of members of a union type.
+
+
+
+### duckdb_union_type_member_name
+---
+Retrieves the name of the union member.
+
+The result must be freed with `duckdb_free`
+
+#### Syntax
+---
+char *duckdb_union_type_member_name(
+ duckdb_logical_type type,
+ idx_t index
+);
+
+#### Parameters
+---
+* `type`
+
+The logical type object
+* `index`
+
+The child index
+* `returns`
+
+The name of the union member. Must be freed with `duckdb_free`.
+
+
+
+### duckdb_union_type_member_type
+---
+Retrieves the child type of the given union member at the specified index.
+
+The result must be freed with `duckdb_destroy_logical_type`
+
+#### Syntax
+---
+duckdb_logical_type duckdb_union_type_member_type(
+ duckdb_logical_type type,
+ idx_t index
+);
+
+#### Parameters
+---
+* `type`
+
+The logical type object
+* `index`
+
+The child index
+* `returns`
+
+The child type of the union member. Must be destroyed with `duckdb_destroy_logical_type`.
+
+
+
### duckdb_destroy_logical_type
---
Destroys the logical type and de-allocates all memory allocated for that type.
diff --git a/docs/api/nodejs/reference.md b/docs/api/nodejs/reference.md
index 6f527069cec..d2c48b13f01 100644
--- a/docs/api/nodejs/reference.md
+++ b/docs/api/nodejs/reference.md
@@ -3,31 +3,60 @@ layout: docu
title: NodeJS API
selected: Client APIs
---
+## Modules
+
+
+- duckdb
+
+
+
+## Typedefs
+
+
+- ColumnInfo :
object
+
+- TypeInfo :
object
+
+- DuckDbError :
object
+
+- HTTPError :
object
+
+
+
## duckdb
-**Summary**: these jsdoc annotations are still a work in progress - feedback and suggestions are welcome!
+**Summary**: DuckDB is an embeddable SQL OLAP Database Management System
* [duckdb](#module_duckdb)
* [~Connection](#module_duckdb..Connection)
* [.run(sql, ...params, callback)](#module_duckdb..Connection+run) ⇒ void
* [.all(sql, ...params, callback)](#module_duckdb..Connection+all) ⇒ void
+ * [.arrowIPCAll(sql, ...params, callback)](#module_duckdb..Connection+arrowIPCAll) ⇒ void
+ * [.arrowIPCStream(sql, ...params, callback)](#module_duckdb..Connection+arrowIPCStream) ⇒
* [.each(sql, ...params, callback)](#module_duckdb..Connection+each) ⇒ void
* [.stream(sql, ...params)](#module_duckdb..Connection+stream)
- * [.register(name, return_type, fun)](#module_duckdb..Connection+register) ⇒ void
+ * [.register_udf(name, return_type, fun)](#module_duckdb..Connection+register_udf) ⇒ void
* [.prepare(sql, ...params, callback)](#module_duckdb..Connection+prepare) ⇒ Statement
* [.exec(sql, ...params, callback)](#module_duckdb..Connection+exec) ⇒ void
- * [.register_bulk(name, return_type, callback)](#module_duckdb..Connection+register_bulk) ⇒ void
- * [.unregister(name, return_type, callback)](#module_duckdb..Connection+unregister) ⇒ void
+ * [.register_udf_bulk(name, return_type, callback)](#module_duckdb..Connection+register_udf_bulk) ⇒ void
+ * [.unregister_udf(name, return_type, callback)](#module_duckdb..Connection+unregister_udf) ⇒ void
+ * [.register_buffer(name, array, force, callback)](#module_duckdb..Connection+register_buffer) ⇒ void
+ * [.unregister_buffer(name, callback)](#module_duckdb..Connection+unregister_buffer) ⇒ void
+ * [.close(callback)](#module_duckdb..Connection+close) ⇒ void
* [~Statement](#module_duckdb..Statement)
+ * [.sql](#module_duckdb..Statement+sql) ⇒
* [.get()](#module_duckdb..Statement+get)
* [.run(sql, ...params, callback)](#module_duckdb..Statement+run) ⇒ void
* [.all(sql, ...params, callback)](#module_duckdb..Statement+all) ⇒ void
+ * [.arrowIPCAll(sql, ...params, callback)](#module_duckdb..Statement+arrowIPCAll) ⇒ void
* [.each(sql, ...params, callback)](#module_duckdb..Statement+each) ⇒ void
* [.finalize(sql, ...params, callback)](#module_duckdb..Statement+finalize) ⇒ void
* [.stream(sql, ...params)](#module_duckdb..Statement+stream)
+ * [.columns()](#module_duckdb..Statement+columns) ⇒ [Array.<ColumnInfo>](#ColumnInfo)
* [~QueryResult](#module_duckdb..QueryResult)
* [.nextChunk()](#module_duckdb..QueryResult+nextChunk) ⇒
+ * [.nextIpcBuffer()](#module_duckdb..QueryResult+nextIpcBuffer) ⇒
* [.asyncIterator()](#module_duckdb..QueryResult+asyncIterator)
* [~Database](#module_duckdb..Database)
* [.close(callback)](#module_duckdb..Database+close) ⇒ void
@@ -39,11 +68,17 @@ selected: Client APIs
* [.interrupt(callback)](#module_duckdb..Database+interrupt) ⇒ void
* [.prepare(sql)](#module_duckdb..Database+prepare) ⇒ Statement
* [.run(sql, ...params, callback)](#module_duckdb..Database+run) ⇒ void
+ * [.scanArrowIpc(sql, ...params, callback)](#module_duckdb..Database+scanArrowIpc) ⇒ void
* [.each(sql, ...params, callback)](#module_duckdb..Database+each) ⇒ void
* [.all(sql, ...params, callback)](#module_duckdb..Database+all) ⇒ void
+ * [.arrowIPCAll(sql, ...params, callback)](#module_duckdb..Database+arrowIPCAll) ⇒ void
+ * [.arrowIPCStream(sql, ...params, callback)](#module_duckdb..Database+arrowIPCStream) ⇒ void
* [.exec(sql, ...params, callback)](#module_duckdb..Database+exec) ⇒ void
- * [.register(name, return_type, fun)](#module_duckdb..Database+register) ⇒ this
- * [.unregister(name)](#module_duckdb..Database+unregister) ⇒ this
+ * [.register_udf(name, return_type, fun)](#module_duckdb..Database+register_udf) ⇒ this
+ * [.register_buffer(name)](#module_duckdb..Database+register_buffer) ⇒ this
+ * [.unregister_buffer(name)](#module_duckdb..Database+unregister_buffer) ⇒ this
+ * [.unregister_udf(name)](#module_duckdb..Database+unregister_udf) ⇒ this
+ * [.registerReplacementScan(fun)](#module_duckdb..Database+registerReplacementScan) ⇒ this
* [.get()](#module_duckdb..Database+get)
* [~ERROR](#module_duckdb..ERROR) : number
* [~OPEN_READONLY](#module_duckdb..OPEN_READONLY) : number
@@ -61,13 +96,18 @@ selected: Client APIs
* [~Connection](#module_duckdb..Connection)
* [.run(sql, ...params, callback)](#module_duckdb..Connection+run) ⇒ void
* [.all(sql, ...params, callback)](#module_duckdb..Connection+all) ⇒ void
+ * [.arrowIPCAll(sql, ...params, callback)](#module_duckdb..Connection+arrowIPCAll) ⇒ void
+ * [.arrowIPCStream(sql, ...params, callback)](#module_duckdb..Connection+arrowIPCStream) ⇒
* [.each(sql, ...params, callback)](#module_duckdb..Connection+each) ⇒ void
* [.stream(sql, ...params)](#module_duckdb..Connection+stream)
- * [.register(name, return_type, fun)](#module_duckdb..Connection+register) ⇒ void
+ * [.register_udf(name, return_type, fun)](#module_duckdb..Connection+register_udf) ⇒ void
* [.prepare(sql, ...params, callback)](#module_duckdb..Connection+prepare) ⇒ Statement
* [.exec(sql, ...params, callback)](#module_duckdb..Connection+exec) ⇒ void
- * [.register_bulk(name, return_type, callback)](#module_duckdb..Connection+register_bulk) ⇒ void
- * [.unregister(name, return_type, callback)](#module_duckdb..Connection+unregister) ⇒ void
+ * [.register_udf_bulk(name, return_type, callback)](#module_duckdb..Connection+register_udf_bulk) ⇒ void
+ * [.unregister_udf(name, return_type, callback)](#module_duckdb..Connection+unregister_udf) ⇒ void
+ * [.register_buffer(name, array, force, callback)](#module_duckdb..Connection+register_buffer) ⇒ void
+ * [.unregister_buffer(name, callback)](#module_duckdb..Connection+unregister_buffer) ⇒ void
+ * [.close(callback)](#module_duckdb..Connection+close) ⇒ void
@@ -95,6 +135,34 @@ Run a SQL query and triggers the callback once for all result rows
| ...params | \* |
| callback | |
+
+
+#### connection.arrowIPCAll(sql, ...params, callback) ⇒ void
+Run a SQL query and serialize the result into the Apache Arrow IPC format (requires arrow extension to be loaded)
+
+**Kind**: instance method of [Connection](#module_duckdb..Connection)
+
+| Param | Type |
+| --- | --- |
+| sql | |
+| ...params | \* |
+| callback | |
+
+
+
+#### connection.arrowIPCStream(sql, ...params, callback) ⇒
+Run a SQL query, returns a IpcResultStreamIterator that allows streaming the result into the Apache Arrow IPC format
+(requires arrow extension to be loaded)
+
+**Kind**: instance method of [Connection](#module_duckdb..Connection)
+**Returns**: Promise
+
+| Param | Type |
+| --- | --- |
+| sql | |
+| ...params | \* |
+| callback | |
+
#### connection.each(sql, ...params, callback) ⇒ void
@@ -118,9 +186,9 @@ Runs a SQL query and triggers the callback for each result row
| sql | |
| ...params | \* |
-
+
-#### connection.register(name, return_type, fun) ⇒ void
+#### connection.register\_udf(name, return_type, fun) ⇒ void
Register a User Defined Function
**Kind**: instance method of [Connection](#module_duckdb..Connection)
@@ -158,9 +226,9 @@ Execute a SQL query
| ...params | \* |
| callback | |
-
+
-#### connection.register\_bulk(name, return_type, callback) ⇒ void
+#### connection.register\_udf\_bulk(name, return_type, callback) ⇒ void
Register a User Defined Function
**Kind**: instance method of [Connection](#module_duckdb..Connection)
@@ -171,9 +239,9 @@ Register a User Defined Function
| return_type |
| callback |
-
+
-#### connection.unregister(name, return_type, callback) ⇒ void
+#### connection.unregister\_udf(name, return_type, callback) ⇒ void
Unregister a User Defined Function
**Kind**: instance method of [Connection](#module_duckdb..Connection)
@@ -184,19 +252,66 @@ Unregister a User Defined Function
| return_type |
| callback |
+
+
+#### connection.register\_buffer(name, array, force, callback) ⇒ void
+Register a Buffer to be scanned using the Apache Arrow IPC scanner
+(requires arrow extension to be loaded)
+
+**Kind**: instance method of [Connection](#module_duckdb..Connection)
+
+| Param |
+| --- |
+| name |
+| array |
+| force |
+| callback |
+
+
+
+#### connection.unregister\_buffer(name, callback) ⇒ void
+Unregister the Buffer
+
+**Kind**: instance method of [Connection](#module_duckdb..Connection)
+
+| Param |
+| --- |
+| name |
+| callback |
+
+
+
+#### connection.close(callback) ⇒ void
+Closes connection
+
+**Kind**: instance method of [Connection](#module_duckdb..Connection)
+
+| Param |
+| --- |
+| callback |
+
### duckdb~Statement
**Kind**: inner class of [duckdb](#module_duckdb)
* [~Statement](#module_duckdb..Statement)
+ * [.sql](#module_duckdb..Statement+sql) ⇒
* [.get()](#module_duckdb..Statement+get)
* [.run(sql, ...params, callback)](#module_duckdb..Statement+run) ⇒ void
* [.all(sql, ...params, callback)](#module_duckdb..Statement+all) ⇒ void
+ * [.arrowIPCAll(sql, ...params, callback)](#module_duckdb..Statement+arrowIPCAll) ⇒ void
* [.each(sql, ...params, callback)](#module_duckdb..Statement+each) ⇒ void
* [.finalize(sql, ...params, callback)](#module_duckdb..Statement+finalize) ⇒ void
* [.stream(sql, ...params)](#module_duckdb..Statement+stream)
+ * [.columns()](#module_duckdb..Statement+columns) ⇒ [Array.<ColumnInfo>](#ColumnInfo)
+
+
+#### statement.sql ⇒
+**Kind**: instance property of [Statement](#module_duckdb..Statement)
+**Returns**: sql contained in statement
+**Field**:
#### statement.get()
@@ -225,6 +340,17 @@ Not implemented
| ...params | \* |
| callback | |
+
+
+#### statement.arrowIPCAll(sql, ...params, callback) ⇒ void
+**Kind**: instance method of [Statement](#module_duckdb..Statement)
+
+| Param | Type |
+| --- | --- |
+| sql | |
+| ...params | \* |
+| callback | |
+
#### statement.each(sql, ...params, callback) ⇒ void
@@ -257,6 +383,11 @@ Not implemented
| sql | |
| ...params | \* |
+
+
+#### statement.columns() ⇒ [Array.<ColumnInfo>](#ColumnInfo)
+**Kind**: instance method of [Statement](#module_duckdb..Statement)
+**Returns**: [Array.<ColumnInfo>](#ColumnInfo) - - Array of column names and types
### duckdb~QueryResult
@@ -264,11 +395,20 @@ Not implemented
* [~QueryResult](#module_duckdb..QueryResult)
* [.nextChunk()](#module_duckdb..QueryResult+nextChunk) ⇒
+ * [.nextIpcBuffer()](#module_duckdb..QueryResult+nextIpcBuffer) ⇒
* [.asyncIterator()](#module_duckdb..QueryResult+asyncIterator)
#### queryResult.nextChunk() ⇒
+**Kind**: instance method of [QueryResult](#module_duckdb..QueryResult)
+**Returns**: data chunk
+
+
+#### queryResult.nextIpcBuffer() ⇒
+Function to fetch the next result blob of an Arrow IPC Stream in a zero-copy way.
+(requires arrow extension to be loaded)
+
**Kind**: instance method of [QueryResult](#module_duckdb..QueryResult)
**Returns**: data chunk
@@ -282,6 +422,14 @@ Main database interface
**Kind**: inner property of [duckdb](#module_duckdb)
+| Param | Description |
+| --- | --- |
+| path | path to database file or :memory: for in-memory database |
+| access_mode | access mode |
+| config | the configuration object |
+| callback | callback function |
+
+
* [~Database](#module_duckdb..Database)
* [.close(callback)](#module_duckdb..Database+close) ⇒ void
* [.close_internal(callback)](#module_duckdb..Database+close_internal) ⇒ void
@@ -292,11 +440,17 @@ Main database interface
* [.interrupt(callback)](#module_duckdb..Database+interrupt) ⇒ void
* [.prepare(sql)](#module_duckdb..Database+prepare) ⇒ Statement
* [.run(sql, ...params, callback)](#module_duckdb..Database+run) ⇒ void
+ * [.scanArrowIpc(sql, ...params, callback)](#module_duckdb..Database+scanArrowIpc) ⇒ void
* [.each(sql, ...params, callback)](#module_duckdb..Database+each) ⇒ void
* [.all(sql, ...params, callback)](#module_duckdb..Database+all) ⇒ void
+ * [.arrowIPCAll(sql, ...params, callback)](#module_duckdb..Database+arrowIPCAll) ⇒ void
+ * [.arrowIPCStream(sql, ...params, callback)](#module_duckdb..Database+arrowIPCStream) ⇒ void
* [.exec(sql, ...params, callback)](#module_duckdb..Database+exec) ⇒ void
- * [.register(name, return_type, fun)](#module_duckdb..Database+register) ⇒ this
- * [.unregister(name)](#module_duckdb..Database+unregister) ⇒ this
+ * [.register_udf(name, return_type, fun)](#module_duckdb..Database+register_udf) ⇒ this
+ * [.register_buffer(name)](#module_duckdb..Database+register_buffer) ⇒ this
+ * [.unregister_buffer(name)](#module_duckdb..Database+unregister_buffer) ⇒ this
+ * [.unregister_udf(name)](#module_duckdb..Database+unregister_udf) ⇒ this
+ * [.registerReplacementScan(fun)](#module_duckdb..Database+registerReplacementScan) ⇒ this
* [.get()](#module_duckdb..Database+get)
@@ -335,7 +489,7 @@ Triggers callback when all scheduled database tasks have completed.
#### database.serialize(callback) ⇒ void
-TODO: what does this do?
+Currently a no-op. Provided for SQLite compatibility
**Kind**: instance method of [Database](#module_duckdb..Database)
@@ -346,7 +500,7 @@ TODO: what does this do?
#### database.parallelize(callback) ⇒ void
-TODO: what does this do?
+Currently a no-op. Provided for SQLite compatibility
**Kind**: instance method of [Database](#module_duckdb..Database)
@@ -400,11 +554,22 @@ Convenience method for Connection#run using a built-in default connection
| ...params | \* |
| callback | |
+
+
+#### database.scanArrowIpc(sql, ...params, callback) ⇒ void
+Convenience method for Connection#scanArrowIpc using a built-in default connection
+
+**Kind**: instance method of [Database](#module_duckdb..Database)
+
+| Param | Type |
+| --- | --- |
+| sql | |
+| ...params | \* |
+| callback | |
+
#### database.each(sql, ...params, callback) ⇒ void
-Convenience method for Connection#each using a built-in default connection
-
**Kind**: instance method of [Database](#module_duckdb..Database)
| Param | Type |
@@ -426,11 +591,35 @@ Convenience method for Connection#apply using a built-in default connection
| ...params | \* |
| callback | |
+
+
+#### database.arrowIPCAll(sql, ...params, callback) ⇒ void
+Convenience method for Connection#arrowIPCAll using a built-in default connection
+
+**Kind**: instance method of [Database](#module_duckdb..Database)
+
+| Param | Type |
+| --- | --- |
+| sql | |
+| ...params | \* |
+| callback | |
+
+
+
+#### database.arrowIPCStream(sql, ...params, callback) ⇒ void
+Convenience method for Connection#arrowIPCStream using a built-in default connection
+
+**Kind**: instance method of [Database](#module_duckdb..Database)
+
+| Param | Type |
+| --- | --- |
+| sql | |
+| ...params | \* |
+| callback | |
+
#### database.exec(sql, ...params, callback) ⇒ void
-Convenience method for Connection#exec using a built-in default connection
-
**Kind**: instance method of [Database](#module_duckdb..Database)
| Param | Type |
@@ -439,10 +628,12 @@ Convenience method for Connection#exec using a built-in default connection
| ...params | \* |
| callback | |
-
+
+
+#### database.register\_udf(name, return_type, fun) ⇒ this
+Register a User Defined Function
-#### database.register(name, return_type, fun) ⇒ this
-Convenience method for Connection#register using a built-in default connection
+Convenience method for Connection#register_udf
**Kind**: instance method of [Database](#module_duckdb..Database)
@@ -452,10 +643,12 @@ Convenience method for Connection#register using a built-in default connection
| return_type |
| fun |
-
+
-#### database.unregister(name) ⇒ this
-Convenience method for Connection#unregister using a built-in default connection
+#### database.register\_buffer(name) ⇒ this
+Register a buffer containing serialized data to be scanned from DuckDB.
+
+Convenience method for Connection#unregister_buffer
**Kind**: instance method of [Database](#module_duckdb..Database)
@@ -463,6 +656,43 @@ Convenience method for Connection#unregister using a built-in default connection
| --- |
| name |
+
+
+#### database.unregister\_buffer(name) ⇒ this
+Unregister a Buffer
+
+Convenience method for Connection#unregister_buffer
+
+**Kind**: instance method of [Database](#module_duckdb..Database)
+
+| Param |
+| --- |
+| name |
+
+
+
+#### database.unregister\_udf(name) ⇒ this
+Unregister a UDF
+
+Convenience method for Connection#unregister_udf
+
+**Kind**: instance method of [Database](#module_duckdb..Database)
+
+| Param |
+| --- |
+| name |
+
+
+
+#### database.registerReplacementScan(fun) ⇒ this
+Register a table replace scan function
+
+**Kind**: instance method of [Database](#module_duckdb..Database)
+
+| Param | Description |
+| --- | --- |
+| fun | Replacement scan function |
+
#### database.get()
@@ -511,3 +741,53 @@ Currently ignored
Currently ignored
**Kind**: inner constant of [duckdb](#module_duckdb)
+
+
+## ColumnInfo : object
+**Kind**: global typedef
+**Properties**
+
+| Name | Type | Description |
+| --- | --- | --- |
+| name | string | Column name |
+| type | [TypeInfo](#TypeInfo) | Column type |
+
+
+
+## TypeInfo : object
+**Kind**: global typedef
+**Properties**
+
+| Name | Type | Description |
+| --- | --- | --- |
+| id | string | Type ID |
+| [alias] | string | SQL type alias |
+| sql_type | string | SQL type name |
+
+
+
+## DuckDbError : object
+**Kind**: global typedef
+**Properties**
+
+| Name | Type | Description |
+| --- | --- | --- |
+| errno | number | -1 for DuckDB errors |
+| message | string | Error message |
+| code | string | 'DUCKDB_NODEJS_ERROR' for DuckDB errors |
+| errorType | string | DuckDB error type code (eg, HTTP, IO, Catalog) |
+
+
+
+## HTTPError : object
+**Kind**: global typedef
+**Extends**: [DuckDbError](#DuckDbError)
+**Properties**
+
+| Name | Type | Description |
+| --- | --- | --- |
+| statusCode | number | HTTP response status code |
+| reason | string | HTTP response reason |
+| response | string | HTTP response body |
+| headers | object | HTTP headers |
+
diff --git a/docs/api/python/reference/index.md b/docs/api/python/reference/index.md
index 9022ce9a8ca..8b1dce7f85c 100644
--- a/docs/api/python/reference/index.md
+++ b/docs/api/python/reference/index.md
@@ -8,10 +8,9 @@ title: Python Client API
-
DuckDB is an embeddable SQL OLAP Database Management System
-
+
-
-duckdb.threadsafety bool
+duckdb.threadsafety bool
-
Indicates that this package is threadsafe
@@ -20,7 +19,7 @@ title: Python Client API
-
-duckdb.apilevel int
+duckdb.apilevel int
-
Indicates which Python DBAPI version this package implements
@@ -29,7 +28,7 @@ title: Python Client API
-
-duckdb.paramstyle str
+duckdb.paramstyle str
-
Indicates which parameter style duckdb supports
@@ -38,94 +37,191 @@ title: Python Client API
-
-duckdb.default_connection duckdb.DuckDBPyConnection
+duckdb.default_connection duckdb.DuckDBPyConnection
-
The connection that is used by default if you don’t explicitly pass one to the root methods in this module
+
+-
+class duckdb.BinaryValue(object: Any)
+
+-
+
Bases: Value
+
+
+
-
-exception duckdb.BinderException
+exception duckdb.BinderException
+
+-
+
Bases: ProgrammingError
+
+
+
+
+-
+class duckdb.BitValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+class duckdb.BlobValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+class duckdb.BooleanValue(object: Any)
-
-
Bases: ProgrammingError
+Bases: Value
+
+-
+duckdb.CaseExpression(condition: duckdb.duckdb.Expression, value: duckdb.duckdb.Expression) → duckdb.duckdb.Expression
+
+
+
+
-
-exception duckdb.CastException
+exception duckdb.CastException
-
-
Bases: DataError
+Bases: DataError
-
-exception duckdb.CatalogException
+exception duckdb.CatalogException
+
+-
+
Bases: ProgrammingError
+
+
+
+
+-
+duckdb.ColumnExpression(name: str) → duckdb.duckdb.Expression
-
-
Bases: ProgrammingError
+Create a column reference from the provided column name
-
-exception duckdb.ConnectionException
+exception duckdb.ConnectionException
+
+-
+
Bases: OperationalError
+
+
+
+
+-
+duckdb.ConstantExpression(value: object) → duckdb.duckdb.Expression
-
-
Bases: OperationalError
+Create a constant expression from the provided value
-
-exception duckdb.ConstraintException
+exception duckdb.ConstraintException
-
-
Bases: IntegrityError
+Bases: IntegrityError
-
-exception duckdb.ConversionException
+exception duckdb.ConversionException
-
-
Bases: DataError
+Bases: DataError
-
-exception duckdb.DataError
+exception duckdb.DataError
+
+-
+
Bases: Error
+
+
+
+
+-
+class duckdb.DateValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+class duckdb.DecimalValue(object: Any, width: int, scale: int)
+
+-
+
Bases: Value
+
+
+
+
+-
+class duckdb.DoubleValue(object: Any)
-
-
Bases: Error
+Bases: Value
-
-class duckdb.DuckDBPyConnection
+class duckdb.DuckDBPyConnection
-
Bases: pybind11_object
-
-append(self: duckdb.DuckDBPyConnection, table_name: str, df: pandas.DataFrame) → duckdb.DuckDBPyConnection
+append(self: duckdb.duckdb.DuckDBPyConnection, table_name: str, df: pandas.DataFrame, *, by_name: bool = False) → duckdb.duckdb.DuckDBPyConnection
+
+-
+
Append the passed DataFrame to the named table
+
+
+
+
+-
+array_type(self: duckdb.duckdb.DuckDBPyConnection, type: duckdb.duckdb.typing.DuckDBPyType) → duckdb.duckdb.typing.DuckDBPyType
-
-
Append the passed Data.Frame to the named table
+Create an array type object of ‘type’
-
-arrow(self: duckdb.DuckDBPyConnection, chunk_size: int = 1000000) → pyarrow.lib.Table
+arrow(self: duckdb.duckdb.DuckDBPyConnection, rows_per_batch: int = 1000000) → pyarrow.lib.Table
-
Fetch a result as Arrow table following execute()
@@ -134,7 +230,7 @@ title: Python Client API
-
-begin(self: duckdb.DuckDBPyConnection) → duckdb.DuckDBPyConnection
+begin(self: duckdb.duckdb.DuckDBPyConnection) → duckdb.duckdb.DuckDBPyConnection
-
Start a new transaction
@@ -143,7 +239,7 @@ title: Python Client API
-
-close(self: duckdb.DuckDBPyConnection) → None
+close(self: duckdb.duckdb.DuckDBPyConnection) → None
-
Close the connection
@@ -152,25 +248,43 @@ title: Python Client API
-
-commit(self: duckdb.DuckDBPyConnection) → duckdb.DuckDBPyConnection
+commit(self: duckdb.duckdb.DuckDBPyConnection) → duckdb.duckdb.DuckDBPyConnection
-
Commit changes performed within a transaction
+
+-
+create_function(self: duckdb.duckdb.DuckDBPyConnection, name: str, function: function, return_type: object = None, parameters: duckdb.duckdb.typing.DuckDBPyType = None, *, type: duckdb.duckdb.functional.PythonUDFType = <PythonUDFType.NATIVE: 0>, null_handling: duckdb.duckdb.functional.FunctionNullHandling = 0, exception_handling: duckdb.duckdb.PythonExceptionHandling = 0, side_effects: bool = False) → duckdb.duckdb.DuckDBPyConnection
+
+-
+
Create a DuckDB function out of the passing in python function so it can be used in queries
+
+
+
-
-cursor(self: duckdb.DuckDBPyConnection) → duckdb.DuckDBPyConnection
+cursor(self: duckdb.duckdb.DuckDBPyConnection) → duckdb.duckdb.DuckDBPyConnection
-
Create a duplicate of the current connection
+
+-
+decimal_type(self: duckdb.duckdb.DuckDBPyConnection, width: int, scale: int) → duckdb.duckdb.typing.DuckDBPyType
+
+-
+
Create a decimal type with ‘width’ and ‘scale’
+
+
+
-
-property description
+property description
-
Get result set attributes, mainly column names
@@ -179,33 +293,43 @@ title: Python Client API
-
-df(*args, **kwargs)
+df(self: duckdb.duckdb.DuckDBPyConnection, *, date_as_object: bool = False) → pandas.DataFrame
-
-
Overloaded function.
-
-df(self: duckdb.DuckDBPyConnection) -> pandas.DataFrame
-
-Fetch a result as Data.Frame following execute()
-
-df(self: duckdb.DuckDBPyConnection, df: pandas.DataFrame) -> duckdb.DuckDBPyRelation
-
-Create a relation object from the Data.Frame in df. This is an alias of from_df
+Fetch a result as DataFrame following execute()
+
+
+
+
+-
+dtype(self: duckdb.duckdb.DuckDBPyConnection, type_str: str) → duckdb.duckdb.typing.DuckDBPyType
+
+-
+
Create a type object by parsing the ‘type_str’ string
-
-duplicate(self: duckdb.DuckDBPyConnection) → duckdb.DuckDBPyConnection
+duplicate(self: duckdb.duckdb.DuckDBPyConnection) → duckdb.duckdb.DuckDBPyConnection
-
Create a duplicate of the current connection
+
+-
+enum_type(self: duckdb.duckdb.DuckDBPyConnection, name: str, type: duckdb.duckdb.typing.DuckDBPyType, values: list) → duckdb.duckdb.typing.DuckDBPyType
+
+-
+
Create an enum type of underlying ‘type’, consisting of the list of ‘values’
+
+
+
-
-execute(self: duckdb.DuckDBPyConnection, query: str, parameters: object = None, multiple_parameter_sets: bool = False) → duckdb.DuckDBPyConnection
+execute(self: duckdb.duckdb.DuckDBPyConnection, query: str, parameters: object = None, multiple_parameter_sets: bool = False) → duckdb.duckdb.DuckDBPyConnection
-
Execute the given SQL query, optionally using prepared statements with parameters set
@@ -214,7 +338,7 @@ title: Python Client API
-
-executemany(self: duckdb.DuckDBPyConnection, query: str, parameters: object = None) → duckdb.DuckDBPyConnection
+executemany(self: duckdb.duckdb.DuckDBPyConnection, query: str, parameters: object = None) → duckdb.duckdb.DuckDBPyConnection
-
Execute the given prepared statement multiple times using the list of parameter sets in parameters
@@ -223,7 +347,7 @@ title: Python Client API
-
-fetch_arrow_table(self: duckdb.DuckDBPyConnection, chunk_size: int = 1000000) → pyarrow.lib.Table
+fetch_arrow_table(self: duckdb.duckdb.DuckDBPyConnection, rows_per_batch: int = 1000000) → pyarrow.lib.Table
-
Fetch a result as Arrow table following execute()
@@ -232,16 +356,16 @@ title: Python Client API
-
-fetch_df(self: duckdb.DuckDBPyConnection) → pandas.DataFrame
+fetch_df(self: duckdb.duckdb.DuckDBPyConnection, *, date_as_object: bool = False) → pandas.DataFrame
-
-
Fetch a result as Data.Frame following execute()
+Fetch a result as DataFrame following execute()
-
-fetch_df_chunk(self: duckdb.DuckDBPyConnection, vectors_per_chunk: int = 1) → pandas.DataFrame
+fetch_df_chunk(self: duckdb.duckdb.DuckDBPyConnection, vectors_per_chunk: int = 1, *, date_as_object: bool = False) → pandas.DataFrame
-
Fetch a chunk of the result as Data.Frame following execute()
@@ -250,7 +374,7 @@ title: Python Client API
-
-fetch_record_batch(self: duckdb.DuckDBPyConnection, chunk_size: int = 1000000) → pyarrow.lib.RecordBatchReader
+fetch_record_batch(self: duckdb.duckdb.DuckDBPyConnection, rows_per_batch: int = 1000000) → pyarrow.lib.RecordBatchReader
-
Fetch an Arrow RecordBatchReader following execute()
@@ -259,7 +383,7 @@ title: Python Client API
-
-fetchall(self: duckdb.DuckDBPyConnection) → list
+fetchall(self: duckdb.duckdb.DuckDBPyConnection) → list
-
Fetch all rows from a result following execute
@@ -268,16 +392,16 @@ title: Python Client API
-
-fetchdf(self: duckdb.DuckDBPyConnection) → pandas.DataFrame
+fetchdf(self: duckdb.duckdb.DuckDBPyConnection, *, date_as_object: bool = False) → pandas.DataFrame
-
-
Fetch a result as Data.Frame following execute()
+Fetch a result as DataFrame following execute()
-
-fetchmany(self: duckdb.DuckDBPyConnection, size: int = 1) → list
+fetchmany(self: duckdb.duckdb.DuckDBPyConnection, size: int = 1) → list
-
Fetch the next set of rows from a result following execute
@@ -286,7 +410,7 @@ title: Python Client API
-
-fetchnumpy(self: duckdb.DuckDBPyConnection) → dict
+fetchnumpy(self: duckdb.duckdb.DuckDBPyConnection) → dict
-
Fetch a result as list of NumPy arrays following execute
@@ -295,16 +419,25 @@ title: Python Client API
-
-fetchone(self: duckdb.DuckDBPyConnection) → object
+fetchone(self: duckdb.duckdb.DuckDBPyConnection) → Optional[tuple]
-
Fetch a single row from a result following execute
+
+-
+filesystem_is_registered(self: duckdb.duckdb.DuckDBPyConnection, name: str) → bool
+
+-
+
Check if a filesystem with the provided name is currently registered
+
+
+
-
-from_arrow(self: duckdb.DuckDBPyConnection, arrow_object: object) → duckdb.DuckDBPyRelation
+from_arrow(self: duckdb.duckdb.DuckDBPyConnection, arrow_object: object) → duckdb.duckdb.DuckDBPyRelation
-
Create a relation object from an Arrow object
@@ -313,16 +446,16 @@ title: Python Client API
-
-from_csv_auto(self: duckdb.DuckDBPyConnection, file_name: str) → duckdb.DuckDBPyRelation
+from_csv_auto(self: duckdb.duckdb.DuckDBPyConnection, name: object, *, header: object = None, compression: object = None, sep: object = None, delimiter: object = None, dtype: object = None, na_values: object = None, skiprows: object = None, quotechar: object = None, escapechar: object = None, encoding: object = None, parallel: object = None, date_format: object = None, timestamp_format: object = None, sample_size: object = None, all_varchar: object = None, normalize_names: object = None, filename: object = None, null_padding: object = None, names: object = None) → duckdb.duckdb.DuckDBPyRelation
-
-
Create a relation object from the CSV file in file_name
+Create a relation object from the CSV file in ‘name’
-
-from_df(self: duckdb.DuckDBPyConnection, df: pandas.DataFrame = None) → duckdb.DuckDBPyRelation
+from_df(self: duckdb.duckdb.DuckDBPyConnection, df: pandas.DataFrame = None) → duckdb.duckdb.DuckDBPyRelation
-
Create a relation object from the Data.Frame in df
@@ -331,34 +464,51 @@ title: Python Client API
-
-from_parquet(self: duckdb.DuckDBPyConnection, file_name: str, binary_as_string: bool = False) → duckdb.DuckDBPyRelation
+from_parquet(*args, **kwargs)
-
-
Create a relation object from the Parquet file in file_name
+Overloaded function.
+
+from_parquet(self: duckdb.duckdb.DuckDBPyConnection, file_glob: str, binary_as_string: bool = False, *, file_row_number: bool = False, filename: bool = False, hive_partitioning: bool = False, union_by_name: bool = False, compression: object = None) -> duckdb.duckdb.DuckDBPyRelation
+
+Create a relation object from the Parquet files in file_glob
+
+from_parquet(self: duckdb.duckdb.DuckDBPyConnection, file_globs: List[str], binary_as_string: bool = False, *, file_row_number: bool = False, filename: bool = False, hive_partitioning: bool = False, union_by_name: bool = False, compression: object = None) -> duckdb.duckdb.DuckDBPyRelation
+
+Create a relation object from the Parquet files in file_globs
-
-from_query(self: duckdb.DuckDBPyConnection, query: str, alias: str = 'query_relation') → duckdb.DuckDBPyRelation
+from_query(self: duckdb.duckdb.DuckDBPyConnection, query: str, *, alias: str = '', params: object = None) → duckdb.duckdb.DuckDBPyRelation
-
-
Create a relation object from the given SQL query
+Run a SQL query. If it is a SELECT statement, create a relation object from the given SQL query, otherwise run the query as-is.
-
-from_substrait(self: duckdb.DuckDBPyConnection, proto: bytes) → duckdb.DuckDBPyRelation
+from_substrait(self: duckdb.duckdb.DuckDBPyConnection, proto: bytes) → duckdb.duckdb.DuckDBPyRelation
-
Create a query object from protobuf plan
+
+-
+from_substrait_json(self: duckdb.duckdb.DuckDBPyConnection, json: str) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Create a query object from a JSON protobuf plan
+
+
+
-
-get_substrait(self: duckdb.DuckDBPyConnection, query: str) → duckdb.DuckDBPyRelation
+get_substrait(self: duckdb.duckdb.DuckDBPyConnection, query: str, *, enable_optimizer: bool = True) → duckdb.duckdb.DuckDBPyRelation
-
Serialize a query to protobuf
@@ -367,7 +517,7 @@ title: Python Client API
-
-get_substrait_json(self: duckdb.DuckDBPyConnection, query: str) → duckdb.DuckDBPyRelation
+get_substrait_json(self: duckdb.duckdb.DuckDBPyConnection, query: str, *, enable_optimizer: bool = True) → duckdb.duckdb.DuckDBPyRelation
-
Serialize a query to protobuf on the JSON format
@@ -376,7 +526,7 @@ title: Python Client API
-
-get_table_names(self: duckdb.DuckDBPyConnection, query: str) → Set[str]
+get_table_names(self: duckdb.duckdb.DuckDBPyConnection, query: str) → Set[str]
-
Extract the required table names from a query
@@ -385,7 +535,7 @@ title: Python Client API
-
-install_extension(self: duckdb.DuckDBPyConnection, extension: str, *, force_install: bool = False) → None
+install_extension(self: duckdb.duckdb.DuckDBPyConnection, extension: str, *, force_install: bool = False) → None
-
Install an extension by name
@@ -393,1165 +543,2640 @@ title: Python Client API
--
-load_extension(self: duckdb.DuckDBPyConnection, extension: str) → None
+
-
+interrupt(self: duckdb.duckdb.DuckDBPyConnection) → None
-
-
Load an installed extension
+Interrupt pending operations
--
-query(self: duckdb.DuckDBPyConnection, query: str, alias: str = 'query_relation') → duckdb.DuckDBPyRelation
+
-
+list_filesystems(self: duckdb.duckdb.DuckDBPyConnection) → list
-
-
Run a SQL query. If it is a SELECT statement, create a relation object from the given SQL query, otherwise run the query as-is.
+List registered filesystems, including builtin ones
--
-register(self: duckdb.DuckDBPyConnection, view_name: str, python_object: object) → duckdb.DuckDBPyConnection
+
-
+list_type(self: duckdb.duckdb.DuckDBPyConnection, type: duckdb.duckdb.typing.DuckDBPyType) → duckdb.duckdb.typing.DuckDBPyType
-
-
Register the passed Python Object value for querying with a view
+Create an array type object of ‘type’
--
-rollback(self: duckdb.DuckDBPyConnection) → duckdb.DuckDBPyConnection
+
-
+load_extension(self: duckdb.duckdb.DuckDBPyConnection, extension: str) → None
-
-
Roll back changes performed within a transaction
+Load an installed extension
--
-table(self: duckdb.DuckDBPyConnection, table_name: str) → duckdb.DuckDBPyRelation
+
-
+map_type(self: duckdb.duckdb.DuckDBPyConnection, key: duckdb.duckdb.typing.DuckDBPyType, value: duckdb.duckdb.typing.DuckDBPyType) → duckdb.duckdb.typing.DuckDBPyType
-
-
Create a relation object for the name’d table
+Create a map type object from ‘key_type’ and ‘value_type’
--
-table_function(self: duckdb.DuckDBPyConnection, name: str, parameters: object = None) → duckdb.DuckDBPyRelation
+
-
+pl(self: duckdb.duckdb.DuckDBPyConnection, rows_per_batch: int = 1000000) → duckdb::PolarsDataFrame
-
-
Create a relation object from the name’d table function with given parameters
+Fetch a result as Polars DataFrame following execute()
--
-unregister(self: duckdb.DuckDBPyConnection, view_name: str) → duckdb.DuckDBPyConnection
+
-
+query(self: duckdb.duckdb.DuckDBPyConnection, query: str, *, alias: str = '', params: object = None) → duckdb.duckdb.DuckDBPyRelation
-
-
Unregister the view name
+Run a SQL query. If it is a SELECT statement, create a relation object from the given SQL query, otherwise run the query as-is.
--
-values(self: duckdb.DuckDBPyConnection, values: object) → duckdb.DuckDBPyRelation
+
-
+read_csv(self: duckdb.duckdb.DuckDBPyConnection, name: object, *, header: object = None, compression: object = None, sep: object = None, delimiter: object = None, dtype: object = None, na_values: object = None, skiprows: object = None, quotechar: object = None, escapechar: object = None, encoding: object = None, parallel: object = None, date_format: object = None, timestamp_format: object = None, sample_size: object = None, all_varchar: object = None, normalize_names: object = None, filename: object = None, null_padding: object = None, names: object = None) → duckdb.duckdb.DuckDBPyRelation
-
-
Create a relation object from the passed values
+Create a relation object from the CSV file in ‘name’
--
-view(self: duckdb.DuckDBPyConnection, view_name: str) → duckdb.DuckDBPyRelation
+
-
+read_json(self: duckdb.duckdb.DuckDBPyConnection, name: str, *, columns: Optional[object] = None, sample_size: Optional[object] = None, maximum_depth: Optional[object] = None, records: Optional[str] = None, format: Optional[str] = None) → duckdb.duckdb.DuckDBPyRelation
-
-
Create a relation object for the name’d view
+Create a relation object from the JSON file in ‘name’
+
+-
+read_parquet(*args, **kwargs)
+
+-
+
Overloaded function.
+
+read_parquet(self: duckdb.duckdb.DuckDBPyConnection, file_glob: str, binary_as_string: bool = False, *, file_row_number: bool = False, filename: bool = False, hive_partitioning: bool = False, union_by_name: bool = False, compression: object = None) -> duckdb.duckdb.DuckDBPyRelation
+
+Create a relation object from the Parquet files in file_glob
+
+read_parquet(self: duckdb.duckdb.DuckDBPyConnection, file_globs: List[str], binary_as_string: bool = False, *, file_row_number: bool = False, filename: bool = False, hive_partitioning: bool = False, union_by_name: bool = False, compression: object = None) -> duckdb.duckdb.DuckDBPyRelation
+
+Create a relation object from the Parquet files in file_globs
-
--
-class duckdb.DuckDBPyRelation
-
--
-
Bases: pybind11_object
--
-abs(self: duckdb.DuckDBPyRelation, aggregation_columns: str) → duckdb.DuckDBPyRelation
+
-
+register(self: duckdb.duckdb.DuckDBPyConnection, view_name: str, python_object: object) → duckdb.duckdb.DuckDBPyConnection
-
-
Returns the absolute value for the specified columns.
+Register the passed Python Object value for querying with a view
--
-aggregate(self: duckdb.DuckDBPyRelation, aggr_expr: str, group_expr: str = '') → duckdb.DuckDBPyRelation
+
-
+register_filesystem(self: duckdb.duckdb.DuckDBPyConnection, filesystem: fsspec.AbstractFileSystem) → None
-
-
Compute the aggregate aggr_expr by the optional groups group_expr on the relation
+Register a fsspec compliant filesystem
-
--
-property alias
+
+-
+remove_function(self: duckdb.duckdb.DuckDBPyConnection, name: str) → duckdb.duckdb.DuckDBPyConnection
-
-
Get the name of the current alias
+Remove a previously created function
--
-apply(self: duckdb.DuckDBPyRelation, function_name: str, function_aggr: str, group_expr: str = '', function_parameter: str = '', projected_columns: str = '') → duckdb.DuckDBPyRelation
+
-
+rollback(self: duckdb.duckdb.DuckDBPyConnection) → duckdb.duckdb.DuckDBPyConnection
-
-
Compute the function of a single column or a list of columns by the optional groups on the relation
+Roll back changes performed within a transaction
--
-arrow(self: duckdb.DuckDBPyRelation, batch_size: int = 1000000) → pyarrow.lib.Table
+
-
+row_type(self: duckdb.duckdb.DuckDBPyConnection, fields: object) → duckdb.duckdb.typing.DuckDBPyType
-
-
Execute and fetch all rows as an Arrow Table
+Create a struct type object from ‘fields’
--
-property columns
+
-
+property rowcount
-
-
Get the names of the columns of this relation.
+Get result set row count
--
-count(self: duckdb.DuckDBPyRelation, count_aggr: str, group_expr: str = '') → duckdb.DuckDBPyRelation
+
-
+sql(self: duckdb.duckdb.DuckDBPyConnection, query: str, *, alias: str = '', params: object = None) → duckdb.duckdb.DuckDBPyRelation
-
-
Compute the aggregate count of a single column or a list of columns by the optional groups on the relation
+Run a SQL query. If it is a SELECT statement, create a relation object from the given SQL query, otherwise run the query as-is.
--
-create(self: duckdb.DuckDBPyRelation, table_name: str) → None
+
-
+sqltype(self: duckdb.duckdb.DuckDBPyConnection, type_str: str) → duckdb.duckdb.typing.DuckDBPyType
-
-
Creates a new table named table_name with the contents of the relation object
+Create a type object by parsing the ‘type_str’ string
--
-create_view(self: duckdb.DuckDBPyRelation, view_name: str, replace: bool = True) → duckdb.DuckDBPyRelation
+
-
+string_type(self: duckdb.duckdb.DuckDBPyConnection, collation: str = '') → duckdb.duckdb.typing.DuckDBPyType
-
-
Creates a view named view_name that refers to the relation object
+Create a string type with an optional collation
--
-cummax(self: duckdb.DuckDBPyRelation, aggregation_columns: str) → duckdb.DuckDBPyRelation
+
-
+struct_type(self: duckdb.duckdb.DuckDBPyConnection, fields: object) → duckdb.duckdb.typing.DuckDBPyType
-
-
Returns the cumulative maximum of the aggregate column.
+Create a struct type object from ‘fields’
--
-cummin(self: duckdb.DuckDBPyRelation, aggregation_columns: str) → duckdb.DuckDBPyRelation
+
-
+table(self: duckdb.duckdb.DuckDBPyConnection, table_name: str) → duckdb.duckdb.DuckDBPyRelation
-
-
Returns the cumulative minimum of the aggregate column.
+Create a relation object for the name’d table
--
-cumprod(self: duckdb.DuckDBPyRelation, aggregation_columns: str) → duckdb.DuckDBPyRelation
+
-
+table_function(self: duckdb.duckdb.DuckDBPyConnection, name: str, parameters: object = None) → duckdb.duckdb.DuckDBPyRelation
-
-
Returns the cumulative product of the aggregate column.
+Create a relation object from the name’d table function with given parameters
--
-cumsum(self: duckdb.DuckDBPyRelation, aggregation_columns: str) → duckdb.DuckDBPyRelation
+
-
+tf(self: duckdb.duckdb.DuckDBPyConnection) → dict
-
-
Returns the cumulative sum of the aggregate column.
+Fetch a result as dict of TensorFlow Tensors following execute()
--
-describe(self: duckdb.DuckDBPyRelation) → duckdb.DuckDBPyRelation
+
-
+torch(self: duckdb.duckdb.DuckDBPyConnection) → dict
-
-
Gives basic statistics (e.g., min,max) and if null exists for each column of the relation.
+Fetch a result as dict of PyTorch Tensors following execute()
--
-df(self: duckdb.DuckDBPyRelation) → pandas.DataFrame
+
-
+type(self: duckdb.duckdb.DuckDBPyConnection, type_str: str) → duckdb.duckdb.typing.DuckDBPyType
-
-
Execute and fetch all rows as a pandas DataFrame
+Create a type object by parsing the ‘type_str’ string
--
-distinct(self: duckdb.DuckDBPyRelation) → duckdb.DuckDBPyRelation
+
-
+union_type(self: duckdb.duckdb.DuckDBPyConnection, members: object) → duckdb.duckdb.typing.DuckDBPyType
-
-
Retrieve distinct rows from this relation object
+Create a union type object from ‘members’
-
--
-property dtypes
+
+-
+unregister(self: duckdb.duckdb.DuckDBPyConnection, view_name: str) → duckdb.duckdb.DuckDBPyConnection
-
-
Get the columns types of the result.
+Unregister the view name
--
-except_(self: duckdb.DuckDBPyRelation, other_rel: duckdb.DuckDBPyRelation) → duckdb.DuckDBPyRelation
+
-
+unregister_filesystem(self: duckdb.duckdb.DuckDBPyConnection, name: str) → None
-
-
Create the set except of this relation object with another relation object in other_rel
+Unregister a filesystem
--
-execute(self: duckdb.DuckDBPyRelation) → duckdb::DuckDBPyResult
+
-
+values(self: duckdb.duckdb.DuckDBPyConnection, values: object) → duckdb.duckdb.DuckDBPyRelation
-
-
Transform the relation into a result set
+Create a relation object from the passed values
--
-explain(self: duckdb.DuckDBPyRelation) → str
+
-
+view(self: duckdb.duckdb.DuckDBPyConnection, view_name: str) → duckdb.duckdb.DuckDBPyRelation
-
+-
+
Create a relation object for the name’d view
+
-
--
-fetchall(self: duckdb.DuckDBPyRelation) → object
-
--
-
Execute and fetch all rows as a list of tuples
+
+-
+class duckdb.DuckDBPyRelation
+
+-
+
Bases: pybind11_object
--
-fetchmany(self: duckdb.DuckDBPyRelation, size: int = 1) → object
+
-
+aggregate(self: duckdb.duckdb.DuckDBPyRelation, aggr_expr: str, group_expr: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Execute and fetch the next set of rows as a list of tuples
+Compute the aggregate aggr_expr by the optional groups group_expr on the relation
-
--
-fetchnumpy(self: duckdb.DuckDBPyRelation) → dict
+
+-
+property alias
-
-
Execute and fetch all rows as a Python dict mapping each column to one numpy arrays
+Get the name of the current alias
--
-fetchone(self: duckdb.DuckDBPyRelation) → object
+
-
+any_value(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Execute and fetch a single row as a tuple
+Returns the first non-null value from a given column
--
-filter(self: duckdb.DuckDBPyRelation, filter_expr: str) → duckdb.DuckDBPyRelation
+
-
+apply(self: duckdb.duckdb.DuckDBPyRelation, function_name: str, function_aggr: str, group_expr: str = '', function_parameter: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Filter the relation object by the filter in filter_expr
+Compute the function of a single column or a list of columns by the optional groups on the relation
--
-insert(self: duckdb.DuckDBPyRelation, values: object) → None
+
-
+arg_max(self: duckdb.duckdb.DuckDBPyRelation, arg_column: str, value_column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Inserts the given values into the relation
+Finds the row with the maximum value for a value column and returns the value of that row for an argument column
--
-insert_into(self: duckdb.DuckDBPyRelation, table_name: str) → None
+
-
+arg_min(self: duckdb.duckdb.DuckDBPyRelation, arg_column: str, value_column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Inserts the relation object into an existing table named table_name
+Finds the row with the minimum value for a value column and returns the value of that row for an argument column
--
-intersect(self: duckdb.DuckDBPyRelation, other_rel: duckdb.DuckDBPyRelation) → duckdb.DuckDBPyRelation
+
-
+arrow(self: duckdb.duckdb.DuckDBPyRelation, batch_size: int = 1000000) → pyarrow.lib.Table
-
-
Create the set intersection of this relation object with another relation object in other_rel
+Execute and fetch all rows as an Arrow Table
--
-join(self: duckdb.DuckDBPyRelation, other_rel: duckdb.DuckDBPyRelation, condition: str, how: str = 'inner') → duckdb.DuckDBPyRelation
+
-
+avg(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Join the relation object with another relation object in other_rel using the join condition expression in join_condition. Types supported are ‘inner’ and ‘left’
+Computes the average on a given column
--
-kurt(self: duckdb.DuckDBPyRelation, aggregation_columns: str, group_columns: str = '') → duckdb.DuckDBPyRelation
+
-
+bit_and(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Returns the excess kurtosis of the aggregate column.
+Computes the bitwise AND of all bits present in a given column
--
-limit(self: duckdb.DuckDBPyRelation, n: int, offset: int = 0) → duckdb.DuckDBPyRelation
+
-
+bit_or(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Only retrieve the first n rows from this relation object, starting at offset
+Computes the bitwise OR of all bits present in a given column
--
-mad(self: duckdb.DuckDBPyRelation, aggregation_columns: str, group_columns: str = '') → duckdb.DuckDBPyRelation
+
-
+bit_xor(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Returns the median absolute deviation for the aggregate columns. NULL values are ignored. Temporal types return a positive INTERVAL.
+Computes the bitwise XOR of all bits present in a given column
--
-map(self: duckdb.DuckDBPyRelation, map_function: function) → duckdb.DuckDBPyRelation
+
-
+bitstring_agg(self: duckdb.duckdb.DuckDBPyRelation, column: str, min: Optional[object] = None, max: Optional[object] = None, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Calls the passed function on the relation
+Computes a bitstring with bits set for each distinct value in a given column
--
-max(self: duckdb.DuckDBPyRelation, max_aggr: str, group_expr: str = '') → duckdb.DuckDBPyRelation
+
-
+bool_and(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Compute the aggregate max of a single column or a list of columns by the optional groups on the relation
+Computes the logical AND of all values present in a given column
--
-mean(self: duckdb.DuckDBPyRelation, mean_aggr: str, group_expr: str = '') → duckdb.DuckDBPyRelation
+
-
+bool_or(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Compute the aggregate mean of a single column or a list of columns by the optional groups on the relation
+Computes the logical OR of all values present in a given column
--
-median(self: duckdb.DuckDBPyRelation, median_aggr: str, group_expr: str = '') → duckdb.DuckDBPyRelation
+
-
+close(self: duckdb.duckdb.DuckDBPyRelation) → None
-
-
Compute the aggregate median of a single column or a list of columns by the optional groups on the relation
+Closes the result
-
--
-min(self: duckdb.DuckDBPyRelation, min_aggr: str, group_expr: str = '') → duckdb.DuckDBPyRelation
+
+-
+property columns
-
-
Compute the aggregate min of a single column or a list of columns by the optional groups on the relation
+Return a list containing the names of the columns of the relation.
--
-mode(self: duckdb.DuckDBPyRelation, aggregation_columns: str, group_columns: str = '') → duckdb.DuckDBPyRelation
+
-
+count(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Returns the most frequent value for the aggregate columns. NULL values are ignored.
+Computes the number of elements present in a given column
--
-order(self: duckdb.DuckDBPyRelation, order_expr: str) → duckdb.DuckDBPyRelation
+
-
+create(self: duckdb.duckdb.DuckDBPyRelation, table_name: str) → None
-
-
Reorder the relation object by order_expr
+Creates a new table named table_name with the contents of the relation object
--
-prod(self: duckdb.DuckDBPyRelation, aggregation_columns: str, group_columns: str = '') → duckdb.DuckDBPyRelation
+
-
+create_view(self: duckdb.duckdb.DuckDBPyRelation, view_name: str, replace: bool = True) → duckdb.duckdb.DuckDBPyRelation
-
-
Calculates the product of the aggregate column.
+Creates a view named view_name that refers to the relation object
--
-project(self: duckdb.DuckDBPyRelation, project_expr: str) → duckdb.DuckDBPyRelation
+
-
+cume_dist(self: duckdb.duckdb.DuckDBPyRelation, window_spec: str, projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Project the relation object by the projection in project_expr
+Computes the cumulative distribution within the partition
--
-quantile(self: duckdb.DuckDBPyRelation, q: str, quantile_aggr: str, group_expr: str = '') → duckdb.DuckDBPyRelation
+
-
+dense_rank(self: duckdb.duckdb.DuckDBPyRelation, window_spec: str, projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Compute the quantile of a single column or a list of columns by the optional groups on the relation
+Computes the dense rank within the partition
--
-query(self: duckdb.DuckDBPyRelation, virtual_table_name: str, sql_query: str) → duckdb.DuckDBPyRelation
+
-
+describe(self: duckdb.duckdb.DuckDBPyRelation) → duckdb.duckdb.DuckDBPyRelation
-
-
Run the given SQL query in sql_query on the view named virtual_table_name that refers to the relation object
+Gives basic statistics (e.g., min,max) and if null exists for each column of the relation.
-
--
-record_batch(self: duckdb.DuckDBPyRelation, batch_size: int = 1000000) → pyarrow.lib.RecordBatchReader
+
+-
+property description
-
-
Execute and return an Arrow Record Batch Reader that yields all rows
+Return the description of the result
--
-sem(self: duckdb.DuckDBPyRelation, aggregation_columns: str, group_columns: str = '') → duckdb.DuckDBPyRelation
+
-
+df(self: duckdb.duckdb.DuckDBPyRelation, *, date_as_object: bool = False) → pandas.DataFrame
-
-
Returns the standard error of the mean of the aggregate column.
+Execute and fetch all rows as a pandas DataFrame
--
-set_alias(self: duckdb.DuckDBPyRelation, alias: str) → duckdb.DuckDBPyRelation
+
-
+distinct(self: duckdb.duckdb.DuckDBPyRelation) → duckdb.duckdb.DuckDBPyRelation
-
-
Rename the relation object to new alias
+Retrieve distinct rows from this relation object
--
-property shape
+
-
+property dtypes
-
-
Tuple of # of rows, # of columns in relation.
+Return a list containing the types of the columns of the relation.
--
-skew(self: duckdb.DuckDBPyRelation, aggregation_columns: str, group_columns: str = '') → duckdb.DuckDBPyRelation
+
-
+except_(self: duckdb.duckdb.DuckDBPyRelation, other_rel: duckdb.duckdb.DuckDBPyRelation) → duckdb.duckdb.DuckDBPyRelation
-
-
Returns the skewness of the aggregate column.
+Create the set except of this relation object with another relation object in other_rel
--
-std(self: duckdb.DuckDBPyRelation, std_aggr: str, group_expr: str = '') → duckdb.DuckDBPyRelation
+
-
+execute(self: duckdb.duckdb.DuckDBPyRelation) → duckdb.duckdb.DuckDBPyRelation
-
-
Compute the standard deviation of a single column or a list of columns by the optional groups on the relation
+Transform the relation into a result set
--
-sum(self: duckdb.DuckDBPyRelation, sum_aggr: str, group_expr: str = '') → duckdb.DuckDBPyRelation
+
-
+explain(self: duckdb.duckdb.DuckDBPyRelation, type: duckdb.duckdb.ExplainType = 'standard') → str
- -
-
Compute the aggregate sum of a single column or a list of columns by the optional groups on the relation
-
+
--
-to_arrow_table(self: duckdb.DuckDBPyRelation, batch_size: int = 1000000) → pyarrow.lib.Table
+
-
+favg(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Execute and fetch all rows as an Arrow Table
+Computes the average of all values present in a given column using a more accurate floating point summation (Kahan Sum)
--
-to_df(self: duckdb.DuckDBPyRelation) → pandas.DataFrame
+
-
+fetch_arrow_reader(self: duckdb.duckdb.DuckDBPyRelation, batch_size: int = 1000000) → pyarrow.lib.RecordBatchReader
-
-
Execute and fetch all rows as a pandas DataFrame
+Execute and return an Arrow Record Batch Reader that yields all rows
-
--
-property type
+
+-
+fetch_arrow_table(self: duckdb.duckdb.DuckDBPyRelation, batch_size: int = 1000000) → pyarrow.lib.Table
-
-
Get the type of the relation.
+Execute and fetch all rows as an Arrow Table
-
--
-property types
+
+-
+fetchall(self: duckdb.duckdb.DuckDBPyRelation) → list
-
-
Get the columns types of the result.
+Execute and fetch all rows as a list of tuples
--
-union(self: duckdb.DuckDBPyRelation, union_rel: duckdb.DuckDBPyRelation) → duckdb.DuckDBPyRelation
+
-
+fetchdf(self: duckdb.duckdb.DuckDBPyRelation, *, date_as_object: bool = False) → pandas.DataFrame
-
-
Create the set union of this relation object with another relation object in other_rel
+Execute and fetch all rows as a pandas DataFrame
--
-unique(self: duckdb.DuckDBPyRelation, unique_aggr: str) → duckdb.DuckDBPyRelation
+
-
+fetchmany(self: duckdb.duckdb.DuckDBPyRelation, size: int = 1) → list
-
-
Number of distinct values in a column.
+Execute and fetch the next set of rows as a list of tuples
--
-value_counts(self: duckdb.DuckDBPyRelation, value_counts_aggr: str, group_expr: str = '') → duckdb.DuckDBPyRelation
+
-
+fetchnumpy(self: duckdb.duckdb.DuckDBPyRelation) → dict
-
-
Count number of rows with each unique value of variable
+Execute and fetch all rows as a Python dict mapping each column to one numpy arrays
--
-var(self: duckdb.DuckDBPyRelation, var_aggr: str, group_expr: str = '') → duckdb.DuckDBPyRelation
+
-
+fetchone(self: duckdb.duckdb.DuckDBPyRelation) → Optional[tuple]
-
-
Compute the variance of a single column or a list of columns by the optional groups on the relation
+Execute and fetch a single row as a tuple
--
-write_csv(self: duckdb.DuckDBPyRelation, file_name: str) → None
+
-
+filter(self: duckdb.duckdb.DuckDBPyRelation, filter_expr: object) → duckdb.duckdb.DuckDBPyRelation
-
-
Write the relation object to a CSV file in file_name
+Filter the relation object by the filter in filter_expr
+
+-
+first(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Returns the first value of a given column
-
--
-class duckdb.DuckDBPyResult
-
--
-
Bases: pybind11_object
--
-arrow(self: duckdb.DuckDBPyResult, chunk_size: int = 1000000) → pyarrow.lib.Table
+
-
+first_value(self: duckdb.duckdb.DuckDBPyRelation, column: str, window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Fetch all rows as an Arrow Table
+Computes the first value within the group or partition
--
-close(self: duckdb.DuckDBPyResult) → None
+
-
+fsum(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
+-
+
Computes the sum of all values present in a given column using a more accurate floating point summation (Kahan Sum)
+
--
-description(self: duckdb.DuckDBPyResult) → list
+
-
+geomean(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
+-
+
Computes the geometric mean over all values present in a given column
+
--
-df(self: duckdb.DuckDBPyResult) → pandas.DataFrame
+
-
+histogram(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Fetch all rows as a pandas DataFrame
+Computes the histogram over all values present in a given column
--
-fetch_arrow_reader(self: duckdb.DuckDBPyResult, approx_batch_size: int) → pyarrow.lib.RecordBatchReader
+
-
+insert(self: duckdb.duckdb.DuckDBPyRelation, values: object) → None
-
-
Fetch all rows as an Arrow Record Batch Reader
+Inserts the given values into the relation
--
-fetch_arrow_table(self: duckdb.DuckDBPyResult, chunk_size: int = 1000000) → pyarrow.lib.Table
+
-
+insert_into(self: duckdb.duckdb.DuckDBPyRelation, table_name: str) → None
-
-
Fetch all rows as an Arrow Table
+Inserts the relation object into an existing table named table_name
--
-fetch_df(self: duckdb.DuckDBPyResult) → pandas.DataFrame
+
-
+intersect(self: duckdb.duckdb.DuckDBPyRelation, other_rel: duckdb.duckdb.DuckDBPyRelation) → duckdb.duckdb.DuckDBPyRelation
-
-
Fetch all rows as a pandas DataFrame
+Create the set intersection of this relation object with another relation object in other_rel
--
-fetch_df_chunk(self: duckdb.DuckDBPyResult, num_of_vectors: int = 1) → pandas.DataFrame
+
-
+join(self: duckdb.duckdb.DuckDBPyRelation, other_rel: duckdb.duckdb.DuckDBPyRelation, condition: object, how: str = 'inner') → duckdb.duckdb.DuckDBPyRelation
-
-
Fetch a chunk of rows as a pandas DataFrame
+Join the relation object with another relation object in other_rel using the join condition expression in join_condition. Types supported are ‘inner’ and ‘left’
--
-fetchall(self: duckdb.DuckDBPyResult) → list
+
-
+lag(self: duckdb.duckdb.DuckDBPyRelation, column: str, window_spec: str, offset: int = 1, default_value: str = 'NULL', ignore_nulls: bool = False, projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Fetch all rows as a list of tuples
+Computes the lag within the partition
--
-fetchdf(self: duckdb.DuckDBPyResult) → pandas.DataFrame
+
-
+last(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Fetch all rows as a pandas DataFrame
+Returns the last value of a given column
--
-fetchmany(self: duckdb.DuckDBPyResult, size: int = 1) → list
+
-
+last_value(self: duckdb.duckdb.DuckDBPyRelation, column: str, window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Fetch the next set of rows as a list of tuples
+Computes the last value within the group or partition
--
-fetchnumpy(self: duckdb.DuckDBPyResult) → dict
+
-
+lead(self: duckdb.duckdb.DuckDBPyRelation, column: str, window_spec: str, offset: int = 1, default_value: str = 'NULL', ignore_nulls: bool = False, projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Fetch all rows as a Python dict mapping each column to one numpy arrays
+Computes the lead within the partition
--
-fetchone(self: duckdb.DuckDBPyResult) → object
+
-
+limit(self: duckdb.duckdb.DuckDBPyRelation, n: int, offset: int = 0) → duckdb.duckdb.DuckDBPyRelation
-
-
Fetch a single row as a tuple
+Only retrieve the first n rows from this relation object, starting at offset
+
+-
+list(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Returns a list containing all values present in a given column
-
--
-exception duckdb.Error
+
+-
+map(self: duckdb.duckdb.DuckDBPyRelation, map_function: function, *, schema: Optional[object] = None) → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: Exception
+Calls the passed function on the relation
-
--
-exception duckdb.FatalException
+
+-
+max(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: Error
+Returns the maximum value present in a given column
-
--
-exception duckdb.IOException
+
+-
+mean(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: OperationalError
+Computes the average on a given column
-
--
-exception duckdb.IntegrityError
+
+-
+median(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: Error
+Computes the median over all values present in a given column
-
--
-exception duckdb.InternalError
+
+-
+min(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: Error
+Returns the minimum value present in a given column
-
--
-exception duckdb.InternalException
+
+-
+mode(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: InternalError
+Computes the mode over all values present in a given column
-
--
-exception duckdb.InterruptException
+
+-
+n_tile(self: duckdb.duckdb.DuckDBPyRelation, window_spec: str, num_buckets: int, projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: Error
+Divides the partition as equally as possible into num_buckets
-
--
-exception duckdb.InvalidInputException
+
+-
+nth_value(self: duckdb.duckdb.DuckDBPyRelation, column: str, window_spec: str, offset: int, ignore_nulls: bool = False, projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: ProgrammingError
+Computes the nth value within the partition
-
--
-exception duckdb.InvalidTypeException
+
+-
+order(self: duckdb.duckdb.DuckDBPyRelation, order_expr: str) → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: ProgrammingError
+Reorder the relation object by order_expr
-
--
-exception duckdb.NotImplementedException
+
+-
+percent_rank(self: duckdb.duckdb.DuckDBPyRelation, window_spec: str, projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: NotSupportedError
+Computes the relative rank within the partition
-
--
-exception duckdb.NotSupportedError
+
+-
+pl(self: duckdb.duckdb.DuckDBPyRelation, batch_size: int = 1000000) → duckdb::PolarsDataFrame
-
-
Bases: Error
+Execute and fetch all rows as a Polars DataFrame
-
--
-exception duckdb.OperationalError
+
+-
+product(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: Error
+Returns the product of all values present in a given column
-
--
-exception duckdb.OutOfMemoryException
+
+-
+project(self: duckdb.duckdb.DuckDBPyRelation, *args, **kwargs) → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: OperationalError
+Project the relation object by the projection in project_expr
-
--
-exception duckdb.OutOfRangeException
+
+-
+quantile(self: duckdb.duckdb.DuckDBPyRelation, column: str, q: object = 0.5, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: DataError
+Computes the exact quantile value for a given column
-
--
-exception duckdb.ParserException
+
+-
+quantile_cont(self: duckdb.duckdb.DuckDBPyRelation, column: str, q: object = 0.5, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: ProgrammingError
+Computes the interpolated quantile value for a given column
-
--
-exception duckdb.PermissionException
+
+-
+quantile_disc(self: duckdb.duckdb.DuckDBPyRelation, column: str, q: object = 0.5, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: Error
+Computes the exact quantile value for a given column
-
--
-exception duckdb.ProgrammingError
+
+-
+query(self: duckdb.duckdb.DuckDBPyRelation, virtual_table_name: str, sql_query: str) → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: Error
+Run the given SQL query in sql_query on the view named virtual_table_name that refers to the relation object
-
--
-exception duckdb.SequenceException
+
+-
+rank(self: duckdb.duckdb.DuckDBPyRelation, window_spec: str, projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: Error
+Computes the rank within the partition
-
--
-exception duckdb.SerializationException
+
+-
+rank_dense(self: duckdb.duckdb.DuckDBPyRelation, window_spec: str, projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: OperationalError
+Computes the dense rank within the partition
-
--
-exception duckdb.StandardException
+
+-
+record_batch(self: duckdb.duckdb.DuckDBPyRelation, batch_size: int = 1000000) → pyarrow.lib.RecordBatchReader
-
-
Bases: Error
+Execute and return an Arrow Record Batch Reader that yields all rows
-
--
-exception duckdb.SyntaxException
+
+-
+row_number(self: duckdb.duckdb.DuckDBPyRelation, window_spec: str, projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: ProgrammingError
+Computes the row number within the partition
-
--
-exception duckdb.TransactionException
+
+-
+select(self: duckdb.duckdb.DuckDBPyRelation, *args, **kwargs) → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: OperationalError
+Project the relation object by the projection in project_expr
-
--
-exception duckdb.TypeMismatchException
+
+-
+select_dtypes(self: duckdb.duckdb.DuckDBPyRelation, types: object) → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: DataError
+Select columns from the relation, by filtering based on type(s)
-
--
-exception duckdb.ValueOutOfRangeException
+
+-
+select_types(self: duckdb.duckdb.DuckDBPyRelation, types: object) → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: DataError
+Select columns from the relation, by filtering based on type(s)
-
--
-exception duckdb.Warning
+
+-
+set_alias(self: duckdb.duckdb.DuckDBPyRelation, alias: str) → duckdb.duckdb.DuckDBPyRelation
-
-
Bases: Exception
+Rename the relation object to new alias
-
--
-duckdb.aggregate(df: pandas.DataFrame, aggr_expr: str, group_expr: str = '', connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+
+-
+property shape
-
-
Compute the aggregate aggr_expr by the optional groups group_expr on Data.frame df
+Tuple of # of rows, # of columns in relation.
-
--
-duckdb.alias(df: pandas.DataFrame, alias: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+
+-
+show(self: duckdb.duckdb.DuckDBPyRelation, *, max_width: Optional[int] = None, max_rows: Optional[int] = None, max_col_width: Optional[int] = None, null_value: Optional[str] = None, render_mode: object = None) → None
-
-
Create a relation from Data.Frame df with the passed alias
+Display a summary of the data
-
--
-duckdb.arrow(arrow_object: object, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+
+-
+sort(self: duckdb.duckdb.DuckDBPyRelation, *args) → duckdb.duckdb.DuckDBPyRelation
-
-
Create a relation object from an Arrow object
+Reorder the relation object by the provided expressions
-
--
-duckdb.connect(database: str = ':memory:', read_only: bool = False, config: object = None) → duckdb.DuckDBPyConnection
+
+-
+sql_query(self: duckdb.duckdb.DuckDBPyRelation) → str
-
-
Create a DuckDB database instance. Can take a database file name to read/write persistent data and a read_only flag if no changes are desired
+Get the SQL query that is equivalent to the relation
-
--
-duckdb.df(df: pandas.DataFrame, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+
+-
+std(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Create a relation object from the Data.Frame df
+Computes the sample standard deviation for a given column
-
--
-duckdb.distinct(df: pandas.DataFrame, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+
+-
+stddev(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Compute the distinct rows from Data.Frame df
+Computes the sample standard deviation for a given column
-
--
-duckdb.filter(df: pandas.DataFrame, filter_expr: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+
+-
+stddev_pop(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Filter the Data.Frame df by the filter in filter_expr
+Computes the population standard deviation for a given column
-
--
-duckdb.from_arrow(arrow_object: object, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+
+-
+stddev_samp(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Create a relation object from an Arrow object
+Computes the sample standard deviation for a given column
-
--
-duckdb.from_csv_auto(file_name: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+
+-
+string_agg(self: duckdb.duckdb.DuckDBPyRelation, column: str, sep: str = ',', groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Creates a relation object from the CSV file in file_name
+Concatenates the values present in a given column with a separator
-
--
-duckdb.from_df(df: pandas.DataFrame, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+
+-
+sum(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
-
-
Create a relation object from the Data.Frame df
+Computes the sum of all values present in a given column
-
--
-duckdb.from_parquet(*args, **kwargs)
+
+-
+tf(self: duckdb.duckdb.DuckDBPyRelation) → dict
-
-
Overloaded function.
-
-from_parquet(file_name: str, binary_as_string: bool, connection: duckdb.DuckDBPyConnection = None) -> duckdb.DuckDBPyRelation
-
-Creates a relation object from the Parquet file in file_name
-
-from_parquet(file_name: str, connection: duckdb.DuckDBPyConnection = None) -> duckdb.DuckDBPyRelation
-
-Creates a relation object from the Parquet file in file_name
+Fetch a result as dict of TensorFlow Tensors
-
--
-duckdb.from_query(query: str, alias: str = 'query_relation', connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+
+-
+to_arrow_table(self: duckdb.duckdb.DuckDBPyRelation, batch_size: int = 1000000) → pyarrow.lib.Table
-
-
Create a relation object from the given SQL query
+Execute and fetch all rows as an Arrow Table
-
--
-duckdb.from_substrait(proto: bytes, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+
+-
+to_csv(self: duckdb.duckdb.DuckDBPyRelation, file_name: str, *, sep: object = None, na_rep: object = None, header: object = None, quotechar: object = None, escapechar: object = None, date_format: object = None, timestamp_format: object = None, quoting: object = None, encoding: object = None, compression: object = None) → None
-
-
Creates a query object from the substrait plan
+Write the relation object to a CSV file in ‘file_name’
-
--
-duckdb.get_substrait(query: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+
+-
+to_df(self: duckdb.duckdb.DuckDBPyRelation, *, date_as_object: bool = False) → pandas.DataFrame
-
-
Serialize a query object to protobuf
+Execute and fetch all rows as a pandas DataFrame
-
--
-duckdb.get_substrait_json(query: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+
+-
+to_parquet(self: duckdb.duckdb.DuckDBPyRelation, file_name: str, *, compression: object = None) → None
-
-
Serialize a query object to protobuf
+Write the relation object to a Parquet file in ‘file_name’
-
--
-duckdb.limit(df: pandas.DataFrame, n: int, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+
+-
+to_table(self: duckdb.duckdb.DuckDBPyRelation, table_name: str) → None
-
-
Retrieve the first n rows from the Data.Frame df
+Creates a new table named table_name with the contents of the relation object
-
--
-duckdb.order(df: pandas.DataFrame, order_expr: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+
+-
+to_view(self: duckdb.duckdb.DuckDBPyRelation, view_name: str, replace: bool = True) → duckdb.duckdb.DuckDBPyRelation
-
-
Reorder the Data.Frame df by order_expr
+Creates a view named view_name that refers to the relation object
-
--
-duckdb.project(df: pandas.DataFrame, project_expr: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+
+-
+torch(self: duckdb.duckdb.DuckDBPyRelation) → dict
-
-
Project the Data.Frame df by the projection in project_expr
+Fetch a result as dict of PyTorch Tensors
-
--
-duckdb.query(query: str, alias: str = 'query_relation', connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+
+-
+property type
-
-
Run a SQL query. If it is a SELECT statement, create a relation object from the given SQL query, otherwise run the query as-is.
+Get the type of the relation.
-
+
+-
+property types
+
+-
+
Return a list containing the types of the columns of the relation.
+
+
+
+
+-
+union(self: duckdb.duckdb.DuckDBPyRelation, union_rel: duckdb.duckdb.DuckDBPyRelation) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Create the set union of this relation object with another relation object in other_rel
+
+
+
+
+-
+unique(self: duckdb.duckdb.DuckDBPyRelation, unique_aggr: str) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Number of distinct values in a column.
+
+
+
+
+-
+var(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Computes the sample variance for a given column
+
+
+
+
+-
+var_pop(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Computes the population variance for a given column
+
+
+
+
+-
+var_samp(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Computes the sample variance for a given column
+
+
+
+
+-
+variance(self: duckdb.duckdb.DuckDBPyRelation, column: str, groups: str = '', window_spec: str = '', projected_columns: str = '') → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Computes the sample variance for a given column
+
+
+
+
+-
+write_csv(self: duckdb.duckdb.DuckDBPyRelation, file_name: str, *, sep: object = None, na_rep: object = None, header: object = None, quotechar: object = None, escapechar: object = None, date_format: object = None, timestamp_format: object = None, quoting: object = None, encoding: object = None, compression: object = None) → None
+
+-
+
Write the relation object to a CSV file in ‘file_name’
+
+
+
+
+-
+write_parquet(self: duckdb.duckdb.DuckDBPyRelation, file_name: str, *, compression: object = None) → None
+
+-
+
Write the relation object to a Parquet file in ‘file_name’
+
+
+
+
+
+
+
+-
+exception duckdb.Error
+
+-
+
Bases: Exception
+
+
+
+
+-
+class duckdb.ExplainType
+
+-
+
Bases: pybind11_object
+Members:
+STANDARD
+ANALYZE
+
+-
+ANALYZE = <ExplainType.ANALYZE: 1>
+
+
+
+
+
+-
+STANDARD = <ExplainType.STANDARD: 0>
+
+
+
+
+
+-
+property name
+
+
+
+
+
+-
+property value
+
+
+
+
+
+
+
+
+-
+class duckdb.Expression
+
+-
+
Bases: pybind11_object
+
+-
+alias(self: duckdb.duckdb.Expression, arg0: str) → duckdb.duckdb.Expression
+
+-
+
Create a copy of this expression with the given alias.
+
+- Parameters:
+-
+
name: The alias to use for the expression, this will affect how it can be referenced.
+
+- Returns:
+-
+
Expression: self with an alias.
+
+
+
+
+
+
+-
+asc(self: duckdb.duckdb.Expression) → duckdb.duckdb.Expression
+
+-
+
Set the order by modifier to ASCENDING.
+
+
+
+
+-
+cast(self: duckdb.duckdb.Expression, type: duckdb.duckdb.typing.DuckDBPyType) → duckdb.duckdb.Expression
+
+-
+
Create a CastExpression to type from self
+
+- Parameters:
+-
+
type: The type to cast to
+
+- Returns:
+-
+
CastExpression: self::type
+
+
+
+
+
+
+-
+desc(self: duckdb.duckdb.Expression) → duckdb.duckdb.Expression
+
+-
+
Set the order by modifier to DESCENDING.
+
+
+
+
+-
+isin(self: duckdb.duckdb.Expression, *args) → duckdb.duckdb.Expression
+
+-
+
Return a IN expression comparing self to the input arguments.
+
+- Returns:
+-
+
DuckDBPyExpression: The compare IN expression
+
+
+
+
+
+
+-
+isnotin(self: duckdb.duckdb.Expression, *args) → duckdb.duckdb.Expression
+
+-
+
Return a NOT IN expression comparing self to the input arguments.
+
+- Returns:
+-
+
DuckDBPyExpression: The compare NOT IN expression
+
+
+
+
+
+
+-
+nulls_first(self: duckdb.duckdb.Expression) → duckdb.duckdb.Expression
+
+-
+
Set the NULL order by modifier to NULLS FIRST.
+
+
+
+
+-
+nulls_last(self: duckdb.duckdb.Expression) → duckdb.duckdb.Expression
+
+-
+
Set the NULL order by modifier to NULLS LAST.
+
+
+
+
+-
+otherwise(self: duckdb.duckdb.Expression, value: duckdb.duckdb.Expression) → duckdb.duckdb.Expression
+
+-
+
Add an ELSE <value> clause to the CaseExpression.
+
+- Parameters:
+-
+
value: The value to use if none of the WHEN conditions are met.
+
+- Returns:
+-
+
CaseExpression: self with an ELSE clause.
+
+
+
+
+
+
+-
+show(self: duckdb.duckdb.Expression) → None
+
+-
+
Print the stringified version of the expression.
+
+
+
+
+-
+when(self: duckdb.duckdb.Expression, condition: duckdb.duckdb.Expression, value: duckdb.duckdb.Expression) → duckdb.duckdb.Expression
+
+-
+
Add an additional WHEN <condition> THEN <value> clause to the CaseExpression.
+
+- Parameters:
+-
+
condition: The condition that must be met.
+value: The value to use if the condition is met.
+
+- Returns:
+-
+
CaseExpression: self with an additional WHEN clause.
+
+
+
+
+
+
+
+
+
+-
+exception duckdb.FatalException
+
+-
+
Bases: Error
+
+
+
+
+-
+class duckdb.FloatValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+duckdb.FunctionExpression(function_name: str, *args) → duckdb.duckdb.Expression
+
+
+
+
+
+-
+exception duckdb.HTTPException
+
+-
+
Bases: IOException
+Thrown when an error occurs in the httpfs extension, or whilst downloading an extension.
+
+-
+body: str
+
+
+
+
+
+
+
+
+
+
+-
+reason: str
+
+
+
+
+
+-
+status_code: int
+
+
+
+
+
+
+
+
+-
+class duckdb.HugeIntegerValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+exception duckdb.IOException
+
+-
+
Bases: OperationalError
+
+
+
+
+-
+class duckdb.IntegerValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+exception duckdb.IntegrityError
+
+-
+
Bases: Error
+
+
+
+
+-
+exception duckdb.InternalError
+
+-
+
Bases: Error
+
+
+
+
+-
+exception duckdb.InternalException
+
+-
+
Bases: InternalError
+
+
+
+
+-
+exception duckdb.InterruptException
+
+-
+
Bases: Error
+
+
+
+
+-
+class duckdb.IntervalValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+exception duckdb.InvalidInputException
+
+-
+
Bases: ProgrammingError
+
+
+
+
+-
+exception duckdb.InvalidTypeException
+
+-
+
Bases: ProgrammingError
+
+
+
+
+-
+class duckdb.LongValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+exception duckdb.NotImplementedException
+
+-
+
Bases: NotSupportedError
+
+
+
+
+-
+exception duckdb.NotSupportedError
+
+-
+
Bases: Error
+
+
+
+
+-
+class duckdb.NullValue
+
+-
+
Bases: Value
+
+
+
+
+-
+exception duckdb.OperationalError
+
+-
+
Bases: Error
+
+
+
+
+-
+exception duckdb.OutOfMemoryException
+
+-
+
Bases: OperationalError
+
+
+
+
+-
+exception duckdb.OutOfRangeException
+
+-
+
Bases: DataError
+
+
+
+
+-
+exception duckdb.ParserException
+
+-
+
Bases: ProgrammingError
+
+
+
+
+-
+exception duckdb.PermissionException
+
+-
+
Bases: Error
+
+
+
+
+-
+exception duckdb.ProgrammingError
+
+-
+
Bases: Error
+
+
+
+
+-
+class duckdb.PythonExceptionHandling
+
+-
+
Bases: pybind11_object
+Members:
+DEFAULT
+RETURN_NULL
+
+-
+DEFAULT = <PythonExceptionHandling.DEFAULT: 0>
+
+
+
+
+
+-
+RETURN_NULL = <PythonExceptionHandling.RETURN_NULL: 1>
+
+
+
+
+
+-
+property name
+
+
+
+
+
+-
+property value
+
+
+
+
+
+
+
+
+-
+exception duckdb.SequenceException
+
+-
+
Bases: Error
+
+
+
+
+-
+exception duckdb.SerializationException
+
+-
+
Bases: OperationalError
+
+
+
+
+-
+class duckdb.ShortValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+exception duckdb.StandardException
+
+-
+
Bases: Error
+
+
+
+
+-
+duckdb.StarExpression(*args, **kwargs)
+
+-
+
Overloaded function.
+
+StarExpression(*, exclude: list = []) -> duckdb.duckdb.Expression
+StarExpression() -> duckdb.duckdb.Expression
+
+
+
+
+
+-
+class duckdb.StringValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+exception duckdb.SyntaxException
+
+-
+
Bases: ProgrammingError
+
+
+
+
+-
+class duckdb.TimeTimeZoneValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+class duckdb.TimeValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+class duckdb.TimestampMilisecondValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+class duckdb.TimestampNanosecondValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+class duckdb.TimestampSecondValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+class duckdb.TimestampTimeZoneValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+class duckdb.TimestampValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+exception duckdb.TransactionException
+
+-
+
Bases: OperationalError
+
+
+
+
+-
+exception duckdb.TypeMismatchException
+
+-
+
Bases: DataError
+
+
+
+
+-
+class duckdb.UUIDValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+class duckdb.UnsignedBinaryValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+class duckdb.UnsignedIntegerValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+class duckdb.UnsignedLongValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+class duckdb.UnsignedShortValue(object: Any)
+
+-
+
Bases: Value
+
+
+
+
+-
+class duckdb.Value(object: Any, type: DuckDBPyType)
+
+-
+
Bases: object
+
+
+
+
+-
+exception duckdb.ValueOutOfRangeException
+
+-
+
Bases: DataError
+
+
+
+
+-
+exception duckdb.Warning
+
+-
+
Bases: Exception
+
+
+
+
+-
+duckdb.aggregate(df: pandas.DataFrame, aggr_expr: str, group_expr: str = '', connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Compute the aggregate aggr_expr by the optional groups group_expr on DataFrame df
+
+
+
+
+-
+duckdb.alias(df: pandas.DataFrame, alias: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Create a relation from DataFrame df with the passed alias
+
+
+
+
+-
+duckdb.append(table_name: str, df: pandas.DataFrame, *, by_name: bool = False, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyConnection
+
+-
+
Append the passed DataFrame to the named table
+
+
+
+
+-
+duckdb.array_type(type: duckdb.duckdb.typing.DuckDBPyType, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.typing.DuckDBPyType
+
+-
+
Create an array type object of ‘type’
+
+
+
+
+-
+duckdb.arrow(*args, **kwargs)
+
+-
+
Overloaded function.
+
+arrow(rows_per_batch: int = 1000000, connection: duckdb.DuckDBPyConnection = None) -> pyarrow.lib.Table
+
+Fetch a result as Arrow table following execute()
+
+arrow(arrow_object: object, connection: duckdb.DuckDBPyConnection = None) -> duckdb.duckdb.DuckDBPyRelation
+
+Create a relation object from an Arrow object
+
+
+
+
+-
+duckdb.begin(connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyConnection
+
+-
+
Start a new transaction
+
+
+
+
+-
+duckdb.close(connection: duckdb.DuckDBPyConnection = None) → None
+
+-
+
Close the connection
+
+
+
+
+-
+duckdb.commit(connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyConnection
+
+-
+
Commit changes performed within a transaction
+
+
+
+
+-
+duckdb.connect(database: str = ':memory:', read_only: bool = False, config: dict = None) → duckdb.DuckDBPyConnection
+
+-
+
Create a DuckDB database instance. Can take a database file name to read/write persistent data and a read_only flag if no changes are desired
+
+
+
+
+-
+duckdb.create_function(name: str, function: function, return_type: object = None, parameters: duckdb.duckdb.typing.DuckDBPyType = None, *, type: duckdb.duckdb.functional.PythonUDFType = <PythonUDFType.NATIVE: 0>, null_handling: duckdb.duckdb.functional.FunctionNullHandling = 0, exception_handling: duckdb.duckdb.PythonExceptionHandling = 0, side_effects: bool = False, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyConnection
+
+-
+
Create a DuckDB function out of the passing in python function so it can be used in queries
+
+
+
+
+-
+duckdb.cursor(connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyConnection
+
+-
+
Create a duplicate of the current connection
+
+
+
+
+-
+duckdb.decimal_type(width: int, scale: int, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.typing.DuckDBPyType
+
+-
+
Create a decimal type with ‘width’ and ‘scale’
+
+
+
+
+-
+duckdb.description(connection: duckdb.DuckDBPyConnection = None) → Optional[list]
+
+-
+
Get result set attributes, mainly column names
+
+
+
+
+-
+duckdb.df(*args, **kwargs)
+
+-
+
Overloaded function.
+
+df(*, date_as_object: bool = False, connection: duckdb.DuckDBPyConnection = None) -> pandas.DataFrame
+
+Fetch a result as DataFrame following execute()
+
+df(df: pandas.DataFrame, connection: duckdb.DuckDBPyConnection = None) -> duckdb.duckdb.DuckDBPyRelation
+
+Create a relation object from the DataFrame df
+
+
+
+
+-
+duckdb.distinct(df: pandas.DataFrame, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Compute the distinct rows from DataFrame df
+
+
+
+
+-
+duckdb.dtype(type_str: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.typing.DuckDBPyType
+
+-
+
Create a type object from ‘type_str’
+
+
+
+
+-
+duckdb.duplicate(connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyConnection
+
+-
+
Create a duplicate of the current connection
+
+
+
+
+-
+duckdb.enum_type(name: str, type: duckdb.duckdb.typing.DuckDBPyType, values: list, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.typing.DuckDBPyType
+
+-
+
Create an enum type of underlying ‘type’, consisting of the list of ‘values’
+
+
+
+
+-
+duckdb.execute(query: str, parameters: object = None, multiple_parameter_sets: bool = False, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyConnection
+
+-
+
Execute the given SQL query, optionally using prepared statements with parameters set
+
+
+
+
+-
+duckdb.executemany(query: str, parameters: object = None, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyConnection
+
+-
+
Execute the given prepared statement multiple times using the list of parameter sets in parameters
+
+
+
+
+-
+duckdb.fetch_arrow_table(rows_per_batch: int = 1000000, connection: duckdb.DuckDBPyConnection = None) → pyarrow.lib.Table
+
+-
+
Fetch a result as Arrow table following execute()
+
+
+
+
+-
+duckdb.fetch_df(*, date_as_object: bool = False, connection: duckdb.DuckDBPyConnection = None) → pandas.DataFrame
+
+-
+
Fetch a result as DataFrame following execute()
+
+
+
+
+-
+duckdb.fetch_df_chunk(vectors_per_chunk: int = 1, *, date_as_object: bool = False, connection: duckdb.DuckDBPyConnection = None) → pandas.DataFrame
+
+-
+
Fetch a chunk of the result as DataFrame following execute()
+
+
+
+
+-
+duckdb.fetch_record_batch(rows_per_batch: int = 1000000, connection: duckdb.DuckDBPyConnection = None) → pyarrow.lib.RecordBatchReader
+
+-
+
Fetch an Arrow RecordBatchReader following execute()
+
+
+
+
+-
+duckdb.fetchall(connection: duckdb.DuckDBPyConnection = None) → list
+
+-
+
Fetch all rows from a result following execute
+
+
+
+
+-
+duckdb.fetchdf(*, date_as_object: bool = False, connection: duckdb.DuckDBPyConnection = None) → pandas.DataFrame
+
+-
+
Fetch a result as DataFrame following execute()
+
+
+
+
+-
+duckdb.fetchmany(size: int = 1, connection: duckdb.DuckDBPyConnection = None) → list
+
+-
+
Fetch the next set of rows from a result following execute
+
+
+
+
+-
+duckdb.fetchnumpy(connection: duckdb.DuckDBPyConnection = None) → dict
+
+-
+
Fetch a result as list of NumPy arrays following execute
+
+
+
+
+-
+duckdb.fetchone(connection: duckdb.DuckDBPyConnection = None) → Optional[tuple]
+
+-
+
Fetch a single row from a result following execute
+
+
+
+
+-
+duckdb.filesystem_is_registered(name: str, connection: duckdb.DuckDBPyConnection = None) → bool
+
+-
+
Check if a filesystem with the provided name is currently registered
+
+
+
+
+-
+duckdb.filter(df: pandas.DataFrame, filter_expr: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Filter the DataFrame df by the filter in filter_expr
+
+
+
+
+-
+duckdb.from_arrow(arrow_object: object, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Create a relation object from an Arrow object
+
+
+
+
+-
+duckdb.from_csv_auto(name: object, connection: duckdb.DuckDBPyConnection = None, header: object = None, compression: object = None, sep: object = None, delimiter: object = None, dtype: object = None, na_values: object = None, skiprows: object = None, quotechar: object = None, escapechar: object = None, encoding: object = None, parallel: object = None, date_format: object = None, timestamp_format: object = None, sample_size: object = None, all_varchar: object = None, normalize_names: object = None, filename: object = None, null_padding: object = None, names: object = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Create a relation object from the CSV file in ‘name’
+
+
+
+
+-
+duckdb.from_df(df: pandas.DataFrame, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Create a relation object from the DataFrame df
+
+
+
+
+-
+duckdb.from_parquet(*args, **kwargs)
+
+-
+
Overloaded function.
+
+from_parquet(file_glob: str, binary_as_string: bool = False, *, file_row_number: bool = False, filename: bool = False, hive_partitioning: bool = False, union_by_name: bool = False, compression: object = None, connection: duckdb.DuckDBPyConnection = None) -> duckdb.duckdb.DuckDBPyRelation
+
+Create a relation object from the Parquet files in file_glob
+
+from_parquet(file_globs: List[str], binary_as_string: bool = False, *, file_row_number: bool = False, filename: bool = False, hive_partitioning: bool = False, union_by_name: bool = False, compression: object = None, connection: duckdb.DuckDBPyConnection = None) -> duckdb.duckdb.DuckDBPyRelation
+
+Create a relation object from the Parquet files in file_globs
+
+
+
+
+-
+duckdb.from_query(query: str, alias: str = '', connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Run a SQL query. If it is a SELECT statement, create a relation object from the given SQL query, otherwise run the query as-is.
+
+
+
+
+-
+duckdb.from_substrait(*args, **kwargs)
+
+-
+
Overloaded function.
+
+from_substrait(proto: bytes, connection: duckdb.DuckDBPyConnection = None) -> duckdb.duckdb.DuckDBPyRelation
+
+Creates a query object from the substrait plan
+
+from_substrait(proto: bytes, connection: duckdb.DuckDBPyConnection = None) -> duckdb.duckdb.DuckDBPyRelation
+
+Create a query object from protobuf plan
+
+
+
+
+-
+duckdb.from_substrait_json(json: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Serialize a query object to protobuf
+
+
+
+
+-
+duckdb.get_substrait(*args, **kwargs)
+
+-
+
Overloaded function.
+
+get_substrait(query: str, connection: duckdb.DuckDBPyConnection = None, *, enable_optimizer: bool = True) -> duckdb.duckdb.DuckDBPyRelation
+
+Serialize a query object to protobuf
+
+get_substrait(query: str, connection: duckdb.DuckDBPyConnection = None, *, enable_optimizer: bool = True) -> duckdb.duckdb.DuckDBPyRelation
+
+Serialize a query to protobuf
+
+
+
+
+-
+duckdb.get_substrait_json(*args, **kwargs)
+
+-
+
Overloaded function.
+
+get_substrait_json(query: str, connection: duckdb.DuckDBPyConnection = None, *, enable_optimizer: bool = True) -> duckdb.duckdb.DuckDBPyRelation
+
+Serialize a query object to protobuf
+
+get_substrait_json(query: str, connection: duckdb.DuckDBPyConnection = None, *, enable_optimizer: bool = True) -> duckdb.duckdb.DuckDBPyRelation
+
+Serialize a query to protobuf on the JSON format
+
+
+
+
+-
+duckdb.get_table_names(query: str, connection: duckdb.DuckDBPyConnection = None) → Set[str]
+
+-
+
Extract the required table names from a query
+
+
+
+
+-
+duckdb.install_extension(extension: str, *, force_install: bool = False, connection: duckdb.DuckDBPyConnection = None) → None
+
+-
+
Install an extension by name
+
+
+
+
+-
+duckdb.interrupt(connection: duckdb.DuckDBPyConnection = None) → None
+
+-
+
Interrupt pending operations
+
+
+
+
+-
+duckdb.limit(df: pandas.DataFrame, n: int, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Retrieve the first n rows from the DataFrame df
+
+
+
+
+-
+duckdb.list_filesystems(connection: duckdb.DuckDBPyConnection = None) → list
+
+-
+
List registered filesystems, including builtin ones
+
+
+
+
+-
+duckdb.list_type(type: duckdb.duckdb.typing.DuckDBPyType, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.typing.DuckDBPyType
+
+-
+
Create an array type object of ‘type’
+
+
+
+
+-
+duckdb.load_extension(extension: str, connection: duckdb.DuckDBPyConnection = None) → None
+
+-
+
Load an installed extension
+
+
+
+
+-
+duckdb.map_type(key: duckdb.duckdb.typing.DuckDBPyType, value: duckdb.duckdb.typing.DuckDBPyType, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.typing.DuckDBPyType
+
+-
+
Create a map type object from ‘key_type’ and ‘value_type’
+
+
+
+
+-
+duckdb.order(df: pandas.DataFrame, order_expr: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Reorder the DataFrame df by order_expr
+
+
+
+
+-
+duckdb.pl(rows_per_batch: int = 1000000, connection: duckdb.DuckDBPyConnection = None) → duckdb::PolarsDataFrame
+
+-
+
Fetch a result as Polars DataFrame following execute()
+
+
+
+
+-
+duckdb.project(df: pandas.DataFrame, project_expr: object, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Project the DataFrame df by the projection in project_expr
+
+
+
+
+-
+duckdb.query(query: str, alias: str = '', connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Run a SQL query. If it is a SELECT statement, create a relation object from the given SQL query, otherwise run the query as-is.
+
+
+
+
-
-duckdb.query_df(df: pandas.DataFrame, virtual_table_name: str, sql_query: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyResult
+duckdb.query_df(df: pandas.DataFrame, virtual_table_name: str, sql_query: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Run the given SQL query in sql_query on the view named virtual_table_name that contains the content of DataFrame df
+
+
+
+
+-
+duckdb.read_csv(name: object, connection: duckdb.DuckDBPyConnection = None, header: object = None, compression: object = None, sep: object = None, delimiter: object = None, dtype: object = None, na_values: object = None, skiprows: object = None, quotechar: object = None, escapechar: object = None, encoding: object = None, parallel: object = None, date_format: object = None, timestamp_format: object = None, sample_size: object = None, all_varchar: object = None, normalize_names: object = None, filename: object = None, null_padding: object = None, names: object = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Create a relation object from the CSV file in ‘name’
+
+
+
+
+-
+duckdb.read_json(name: str, connection: duckdb.DuckDBPyConnection = None, columns: Optional[object] = None, sample_size: Optional[object] = None, maximum_depth: Optional[object] = None, records: Optional[str] = None, format: Optional[str] = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Create a relation object from the JSON file in ‘name’
+
+
+
+
+-
+duckdb.read_parquet(*args, **kwargs)
+
+-
+
Overloaded function.
+
+read_parquet(file_glob: str, binary_as_string: bool = False, *, file_row_number: bool = False, filename: bool = False, hive_partitioning: bool = False, union_by_name: bool = False, compression: object = None, connection: duckdb.DuckDBPyConnection = None) -> duckdb.duckdb.DuckDBPyRelation
+
+Create a relation object from the Parquet files in file_glob
+
+read_parquet(file_globs: List[str], binary_as_string: bool = False, *, file_row_number: bool = False, filename: bool = False, hive_partitioning: bool = False, union_by_name: bool = False, compression: object = None, connection: duckdb.DuckDBPyConnection = None) -> duckdb.duckdb.DuckDBPyRelation
+
+Create a relation object from the Parquet files in file_globs
+
+
+
+
+-
+duckdb.register(view_name: str, python_object: object, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyConnection
+
+-
+
Register the passed Python Object value for querying with a view
+
+
+
+
+-
+duckdb.register_filesystem(filesystem: fsspec.AbstractFileSystem, connection: duckdb.DuckDBPyConnection = None) → None
+
+-
+
Register a fsspec compliant filesystem
+
+
+
+
+-
+duckdb.remove_function(name: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyConnection
+
+-
+
Remove a previously created function
+
+
+
+
+-
+duckdb.rollback(connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyConnection
+
+-
+
Roll back changes performed within a transaction
+
+
+
+
+-
+duckdb.row_type(fields: object, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.typing.DuckDBPyType
+
+-
+
Create a struct type object from ‘fields’
+
+
+
+
+-
+duckdb.rowcount(connection: duckdb.DuckDBPyConnection = None) → int
+
+-
+
Get result set row count
+
+
+
+
+-
+duckdb.sql(query: str, alias: str = '', connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Run a SQL query. If it is a SELECT statement, create a relation object from the given SQL query, otherwise run the query as-is.
+
+
+
+
+-
+duckdb.sqltype(type_str: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.typing.DuckDBPyType
+
+-
+
Create a type object from ‘type_str’
+
+
+
+
+-
+duckdb.string_type(collation: str = '', connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.typing.DuckDBPyType
+
+-
+
Create a string type with an optional collation
+
+
+
+
+-
+duckdb.struct_type(fields: object, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.typing.DuckDBPyType
-
-
Run the given SQL query in sql_query on the view named virtual_table_name that contains the content of Data.Frame df
+Create a struct type object from ‘fields’
+
+
+
+
+-
+duckdb.table(table_name: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Create a relation object for the name’d table
+
+
+
+
+-
+duckdb.table_function(name: str, parameters: object = None, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Create a relation object from the name’d table function with given parameters
+
+
+
+
+-
+duckdb.tf(connection: duckdb.DuckDBPyConnection = None) → dict
+
+-
+
Fetch a result as dict of TensorFlow Tensors following execute()
-
-class duckdb.token_type
+class duckdb.token_type
-
Bases: pybind11_object
@@ -1564,56 +3189,56 @@ title: Python Client API
comment
-
-identifier = <token_type.identifier: 0>
+identifier = <token_type.identifier: 0>
-
-keyword = <token_type.keyword: 4>
+keyword = <token_type.keyword: 4>
-
-property name
+property name
-
-numeric_const = <token_type.numeric_const: 1>
+numeric_const = <token_type.numeric_const: 1>
-
-operator = <token_type.operator: 3>
+operator = <token_type.operator: 3>
-
-string_const = <token_type.string_const: 2>
+string_const = <token_type.string_const: 2>
-
-property value
+property value
@@ -1623,28 +3248,90 @@ title: Python Client API
-
-duckdb.tokenize(query: str) → object
+duckdb.tokenize(query: str) → list
-
Tokenizes a SQL string, returning a list of (position, type) tuples that can be used for e.g. syntax highlighting
+
+-
+duckdb.torch(connection: duckdb.DuckDBPyConnection = None) → dict
+
+-
+
Fetch a result as dict of PyTorch Tensors following execute()
+
+
+
+
+-
+duckdb.type(type_str: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.typing.DuckDBPyType
+
+-
+
Create a type object from ‘type_str’
+
+
+
+
+-
+duckdb.union_type(members: object, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.typing.DuckDBPyType
+
+-
+
Create a union type object from ‘members’
+
+
+
+
+-
+duckdb.unregister(view_name: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyConnection
+
+-
+
Unregister the view name
+
+
+
+
+-
+duckdb.unregister_filesystem(name: str, connection: duckdb.DuckDBPyConnection = None) → None
+
+-
+
Unregister a filesystem
+
+
+
-
-duckdb.values(values: object, connection: duckdb.DuckDBPyConnection = None) → duckdb.DuckDBPyRelation
+duckdb.values(*args, **kwargs)
-
+
Overloaded function.
+
+values(values: object, connection: duckdb.DuckDBPyConnection = None) -> duckdb.duckdb.DuckDBPyRelation
+
+Create a relation object from the passed values
+
+values(values: object, connection: duckdb.DuckDBPyConnection = None) -> duckdb.duckdb.DuckDBPyRelation
+
Create a relation object from the passed values
+
+-
+duckdb.view(view_name: str, connection: duckdb.DuckDBPyConnection = None) → duckdb.duckdb.DuckDBPyRelation
+
+-
+
Create a relation object for the name’d view
+
+
+
-
-duckdb.write_csv(df: pandas.DataFrame, file_name: str, connection: duckdb.DuckDBPyConnection = None) → None
+duckdb.write_csv(df: pandas.DataFrame, file_name: str, connection: duckdb.DuckDBPyConnection = None) → None
-
-
Write the Data.Frame df to a CSV file in file_name
+Write the DataFrame df to a CSV file in file_name
diff --git a/docs/sql/configuration.md b/docs/sql/configuration.md
index 0434eb96051..8a3a9259ac3 100644
--- a/docs/sql/configuration.md
+++ b/docs/sql/configuration.md
@@ -28,47 +28,70 @@ SELECT current_setting('access_mode');
Below is a list of all available settings.
-| name | description | input_type | default_value |
-|------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|------------|-----------------|
-| Calendar | The current calendar | VARCHAR | GREGORIAN |
-| TimeZone | The current time zone | VARCHAR | System timezone |
-| access_mode | Access mode of the database (**AUTOMATIC**, **READ_ONLY** or **READ_WRITE**) | VARCHAR | AUTOMATIC |
-| allow_unsigned_extensions | Allow to load extensions with invalid or missing signatures | BOOLEAN | FALSE |
-| binary_as_string | In Parquet files, interpret binary data as a string. | BOOLEAN | |
-| checkpoint_threshold, wal_autocheckpoint | The WAL size threshold at which to automatically trigger a checkpoint (e.g. 1GB) | VARCHAR | 16.7MB |
-| default_collation | The collation setting used when none is specified | VARCHAR | |
-| default_null_order, null_order | Null ordering used when none is specified (**NULLS_FIRST** or **NULLS_LAST**) | VARCHAR | NULLS_FIRST |
-| default_order | The order type used when none is specified (**ASC** or **DESC**) | VARCHAR | ASC |
-| enable_external_access | Allow the database to access external state (through e.g. loading/installing modules, COPY TO/FROM, CSV readers, pandas replacement scans, etc) | BOOLEAN | TRUE |
-| enable_object_cache | Whether or not object cache is used to cache e.g. Parquet metadata | BOOLEAN | FALSE |
-| enable_profiling | Enables profiling, and sets the output format (**JSON**, **QUERY_TREE**, **QUERY_TREE_OPTIMIZER**) | VARCHAR | NULL |
-| enable_progress_bar | Enables the progress bar, printing progress to the terminal for long queries | BOOLEAN | FALSE |
-| explain_output | Output of EXPLAIN statements (**ALL**, **OPTIMIZED_ONLY**, **PHYSICAL_ONLY**) | VARCHAR | PHYSICAL_ONLY |
-| external_threads | The number of external threads that work on DuckDB tasks. | BIGINT | 0 |
-| file_search_path | A comma separated list of directories to search for input files | VARCHAR | |
-| home_directory | Sets the home directory used by the system | VARCHAR | |
-| httpfs_timeout | HTTP timeout read/write/connection/retry (default 30000ms) | UBIGINT | |
-| log_query_path | Specifies the path to which queries should be logged (default: empty string, queries are not logged) | VARCHAR | NULL |
-| max_expression_depth | The maximum expression depth limit in the parser. WARNING: increasing this setting and using very deep expressions might lead to stack overflow errors. | UBIGINT | 1000 |
-| max_memory, memory_limit | The maximum memory of the system (e.g. 1GB) | VARCHAR | 75% of RAM |
-| perfect_ht_threshold | Threshold in bytes for when to use a perfect hash table (default: 12) | BIGINT | 12 |
-| preserve_identifier_case | Whether or not to preserve the identifier case, instead of always lowercasing all non-quoted identifiers | BOOLEAN | TRUE |
-| preserve_insertion_order | Whether or not to preserve insertion order. If set to false the system is allowed to re-order any results that do not contain ORDER BY clauses. | BOOLEAN | TRUE |
-| profile_output, profiling_output | The file to which profile output should be saved, or empty to print to the terminal | VARCHAR | |
-| profiler_history_size | Sets the profiler history size | BIGINT | NULL |
-| profiling_mode | The profiling mode (**STANDARD** or **DETAILED**) | VARCHAR | NULL |
-| progress_bar_time | Sets the time (in milliseconds) how long a query needs to take before we start printing a progress bar | BIGINT | 2000 |
-| s3_access_key_id | S3 Access Key ID | VARCHAR | |
-| s3_endpoint | S3 Endpoint (default 's3.amazonaws.com') | VARCHAR | |
-| s3_region | S3 Region | VARCHAR | |
-| s3_secret_access_key | S3 Access Key | VARCHAR | |
-| s3_session_token | S3 Session Token | VARCHAR | |
-| s3_uploader_max_filesize | S3 Uploader max filesize (between 50GB and 5TB, default 800GB) | VARCHAR | |
-| s3_uploader_max_parts_per_file | S3 Uploader max parts per file (between 1 and 10000, default 10000) | UBIGINT | |
-| s3_uploader_thread_limit | S3 Uploader global thread limit (default 50) | UBIGINT | |
-| s3_url_style | S3 url style ('vhost' (default) or 'path') | VARCHAR | |
-| s3_use_ssl | S3 use SSL (default true) | BOOLEAN | |
-| schema | Sets the default search schema. Equivalent to setting search_path to a single value. | VARCHAR | |
-| search_path | Sets the default search search path as a comma-separated list of values | VARCHAR | |
-| temp_directory | Set the directory to which to write temp files | VARCHAR | |
-| threads, worker_threads | The number of total threads used by the system. | BIGINT | # Cores |
+| name | description | input_type | default_value |
+|------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|------------|------------------|
+| Calendar | The current calendar | VARCHAR | GREGORIAN |
+| TimeZone | The current time zone | VARCHAR | System timezone |
+| access_mode | Access mode of the database (**AUTOMATIC**, **READ_ONLY** or **READ_WRITE**) | VARCHAR | AUTOMATIC |
+| allocator_flush_threshold | Peak allocation threshold at which to flush the allocator after completing a task. | VARCHAR | 134.2MB |
+| allow_unsigned_extensions | Allow to load extensions with invalid or missing signatures | BOOLEAN | FALSE |
+| arrow_large_buffer_size | If arrow buffers for strings, blobs, uuids and bits should be exported using large buffers | BOOLEAN | FALSE |
+| binary_as_string | In Parquet files, interpret binary data as a string. | BOOLEAN | |
+| checkpoint_threshold, wal_autocheckpoint | The WAL size threshold at which to automatically trigger a checkpoint (e.g. 1GB) | VARCHAR | 16.7MB |
+| custom_extension_repository | Overrides the custom endpoint for remote extension installation | VARCHAR | |
+| default_collation | The collation setting used when none is specified | VARCHAR | |
+| default_null_order, null_order | Null ordering used when none is specified (**NULLS_FIRST** or **NULLS_LAST**) | VARCHAR | NULLS_LAST |
+| default_order | The order type used when none is specified (**ASC** or **DESC**) | VARCHAR | ASC |
+| disabled_filesystems | Disable specific file systems preventing access (e.g. LocalFileSystem) | VARCHAR | |
+| enable_external_access | Allow the database to access external state (through e.g. loading/installing modules, COPY TO/FROM, CSV readers, pandas replacement scans, etc) | BOOLEAN | TRUE |
+| enable_fsst_vectors | Allow scans on FSST compressed segments to emit compressed vectors to utilize late decompression | BOOLEAN | FALSE |
+| enable_http_metadata_cache | Whether or not the global http metadata is used to cache HTTP metadata | BOOLEAN | FALSE |
+| enable_object_cache | Whether or not object cache is used to cache e.g. Parquet metadata | BOOLEAN | FALSE |
+| enable_profiling | Enables profiling, and sets the output format (**JSON**, **QUERY_TREE**, **QUERY_TREE_OPTIMIZER**) | VARCHAR | NULL |
+| enable_progress_bar | Enables the progress bar, printing progress to the terminal for long queries | BOOLEAN | FALSE |
+| enable_progress_bar_print | Controls the printing of the progress bar, when 'enable_progress_bar' is true | BOOLEAN | TRUE |
+| experimental_parallel_csv | Whether or not to use the experimental parallel CSV reader | BOOLEAN | NULL |
+| explain_output | Output of EXPLAIN statements (**ALL**, **OPTIMIZED_ONLY**, **PHYSICAL_ONLY**) | VARCHAR | PHYSICAL_ONLY |
+| extension_directory | Set the directory to store extensions in | VARCHAR | |
+| external_threads | The number of external threads that work on DuckDB tasks. | BIGINT | 0 |
+| file_search_path | A comma separated list of directories to search for input files | VARCHAR | |
+| force_download | Forces upfront download of file | BOOLEAN | 0 |
+| home_directory | Sets the home directory used by the system | VARCHAR | |
+| http_retries | HTTP retries on I/O error (default 3) | UBIGINT | 3 |
+| http_retry_backoff | Backoff factor for exponentially increasing retry wait time (default 4) | FLOAT | 4 |
+| http_retry_wait_ms | Time between retries (default 100ms) | UBIGINT | 100 |
+| http_timeout | HTTP timeout read/write/connection/retry (default 30000ms) | UBIGINT | 30000 |
+| immediate_transaction_mode | Whether transactions should be started lazily when needed, or immediately when BEGIN TRANSACTION is called | BOOLEAN | FALSE |
+| integer_division | Whether or not the / operator defaults to integer division, or to floating point division | BOOLEAN | 0 |
+| lock_configuration | Whether or not the configuration can be altered | BOOLEAN | FALSE |
+| log_query_path | Specifies the path to which queries should be logged (default: empty string, queries are not logged) | VARCHAR | NULL |
+| max_expression_depth | The maximum expression depth limit in the parser. WARNING: increasing this setting and using very deep expressions might lead to stack overflow errors. | UBIGINT | 1000 |
+| max_memory, memory_limit | The maximum memory of the system (e.g. 1GB) | VARCHAR | 75% of RAM |
+| ordered_aggregate_threshold | the number of rows to accumulate before sorting, used for tuning | UBIGINT | 262144 |
+| password | The password to use. Ignored for legacy compatibility. | VARCHAR | NULL |
+| perfect_ht_threshold | Threshold in bytes for when to use a perfect hash table (default: 12) | BIGINT | 12 |
+| pivot_filter_threshold | The threshold to switch from using filtered aggregates to LIST with a dedicated pivot operator | BIGINT | 10 |
+| pivot_limit | The maximum number of pivot columns in a pivot statement (default: 100000) | BIGINT | 100000 |
+| prefer_range_joins | Force use of range joins with mixed predicates | BOOLEAN | FALSE |
+| preserve_identifier_case | Whether or not to preserve the identifier case, instead of always lowercasing all non-quoted identifiers | BOOLEAN | TRUE |
+| preserve_insertion_order | Whether or not to preserve insertion order. If set to false the system is allowed to re-order any results that do not contain ORDER BY clauses. | BOOLEAN | TRUE |
+| profile_output, profiling_output | The file to which profile output should be saved, or empty to print to the terminal | VARCHAR | |
+| profiler_history_size | Sets the profiler history size | BIGINT | NULL |
+| profiling_mode | The profiling mode (**STANDARD** or **DETAILED**) | VARCHAR | NULL |
+| progress_bar_time | Sets the time (in milliseconds) how long a query needs to take before we start printing a progress bar | BIGINT | 2000 |
+| s3_access_key_id | S3 Access Key ID | VARCHAR | |
+| s3_endpoint | S3 Endpoint (default 's3.amazonaws.com') | VARCHAR | S3.AMAZONAWS.COM |
+| s3_region | S3 Region | VARCHAR | |
+| s3_secret_access_key | S3 Access Key | VARCHAR | |
+| s3_session_token | S3 Session Token | VARCHAR | |
+| s3_uploader_max_filesize | S3 Uploader max filesize (between 50GB and 5TB, default 800GB) | VARCHAR | 800GB |
+| s3_uploader_max_parts_per_file | S3 Uploader max parts per file (between 1 and 10000, default 10000) | UBIGINT | 10000 |
+| s3_uploader_thread_limit | S3 Uploader global thread limit (default 50) | UBIGINT | 50 |
+| s3_url_compatibility_mode | Disable Globs and Query Parameters on S3 urls | BOOLEAN | 0 |
+| s3_url_style | S3 url style ('vhost' (default) or 'path') | VARCHAR | VHOST |
+| s3_use_ssl | S3 use SSL (default true) | BOOLEAN | 1 |
+| schema | Sets the default search schema. Equivalent to setting search_path to a single value. | VARCHAR | MAIN |
+| search_path | Sets the default search search path as a comma-separated list of values | VARCHAR | |
+| temp_directory | Set the directory to which to write temp files | VARCHAR | |
+| threads, worker_threads | The number of total threads used by the system. | BIGINT | # Cores |
+| username, user | The username to use. Ignored for legacy compatibility. | VARCHAR | NULL |