In fact, gt file obtained through gmail account is broken

[jamesunnc]

Dear,

For category=query, among 120 cases out of 180 are broken, I mean sol_sql column is not a standard sqlite syntax and therefore cannot be executed under sqlite environment.

[BlackSoi1]

Hi @jamesunnc,

Thanks for your interest in the BIRD project.

May I ask which dataset you are referring to? Based on your description, it seems that you may be referring to the BIRD-CRITIC project. However, the only subset using the SQLite dialect that we currently provide is birdsql/bird-critic-1.0-sqlite, which contains 284 query examples.

Could you please share more details about the dataset or split you are looking for?

Regards,
BIRD Team

[jamesunnc]

Hi @jamesunnc,

Thanks for your interest in the BIRD project.

May I ask which dataset you are referring to? Based on your description, it seems that you may be referring to the BIRD-CRITIC project. However, the only subset using the SQLite dialect that we currently provide is birdsql/bird-critic-1.0-sqlite, which contains 284 query examples.

Could you please share more details about the dataset or split you are looking for?

Regards, BIRD Team

Hi, I am using this one https://huggingface.co/datasets/birdsql/livesqlbench-base-lite-sqlite. I think the gt file obtained from email is postgresql but not sqlite one. I think the problem is the subject title. The website shows exactly same subject line on sqlite compared with postgresql. It would be great if a native sqlite gt file can be pulicly available.

"To prevent data leakage through automated crawling, please request access to the ground truth and test cases by emailing 📧 bird.bench25@gmail.com with the subject line [livesqlbench-base-lite GT&Test Cases]. An automated response will provide these data fields."

[BlackSoi1]

Hi @jamesunnc ,

Thanks for pointing this out. We have updated the Hugging Face repo with the correct subject line [livesqlbench-base-lite-SQLite GT&Test Cases] for the SQLite version.

Please try sending the request email again using the updated subject line. You should then receive the native SQLite GT and test case files.

Regards,
BIRD Team

[jamesunnc]

Cool, I've sent a requesting email just now. 358359482 ***@***.***

[jamesunnc]

Just to clarify, please check if bird-critic-sqlite has the correct email subject line as stated inhere: https://huggingface.co/datasets/birdsql/bird-critic-1.0-sqlite "To avoid data leakage by auto-crawling, certain fields (e.g., sol_sql, test_cases) are excluded from the public dataset. For the full dataset, please email: 📧 ***@***.*** with subject tag [bird-critic-1 GT&Test Cases], which will be sent automatically within 30 mins." I don't think the subject line can tell 358359482 ***@***.***

[BlackSoi1]

Hi @jamesunnc ,

Thanks for your reply. For BIRD-Critic, we package all subsets together in a single zip file, so the current subject line should work.

Regards,
BIRD Team

[jamesunnc]

Hi @jamesunnc ,

Thanks for pointing this out. We have updated the Hugging Face repo with the correct subject line [livesqlbench-base-lite-SQLite GT&Test Cases] for the SQLite version.

Please try sending the request email again using the updated subject line. You should then receive the native SQLite GT and test case files.

Regards, BIRD Team

Hi there,
I just execute the sqlite gt file (combined with query) and get 4 error when running sol_sql and corresponding test_cases for management cases. Please see attached zip for details.

Executable Instances with Assertion Errors:

1. archeology_M_2
2. insider_1
3. vaccine_1
4. vaccine_2

log.zip

livesqlbench_gt_fxy_full_simple_report.txt

[BlackSoi1]

Hi @jamesunnc , thank you for reporting this and sharing the logs.

We have fixed these four instances by updating the corresponding GT SQL and/or test cases:

• archeology_M_2
• insider_1
• vaccine_1
• vaccine_2

If you need the updated files, please send us an email and we will share them with you.

Thanks again!
Regards,
BIRD Team