![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "0sTKUefKWTfr",
- "colab_type": "text"
- },
- "source": [
- "# Import Validation Helper\n",
- "This Colab notebook introduces a few tools to check your template MCF, StatVars, and CSV. \n",
- "\n",
- "A summary of features is as follows.\n",
- "\n",
- "* MCF format checking (no improperly defined nodes).\n",
- "* StatVar reference checking (makes sure that all references either exist locally or in the knowledge graph).\n",
- "* TMCF and CSV column valididation.\n",
- "* Description spell checking.\n",
- "* ASCII encoding checking.\n",
- "\n",
- "### Usage summary:\n",
- "1. Runtime -> Run All\n",
- "2. Authenticate with BigQuery in second cell\n",
- "3. Scroll to bottom, select three files to validate from your local computer.\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "6CGTarNhjPuB",
- "colab_type": "text"
- },
- "source": [
- "# 1) At the top of the page, go to \"Runtime -> Run All\"."
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "RHlcR9km48T7",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- "import re\n",
- "import pandas as pd\n",
- "!pip install --upgrade -q pyspellchecker\n",
- "!pip install --upgrade -q pygsheets\n",
- "\n",
- "from spellchecker import SpellChecker\n",
- "import subprocess\n",
- "\n",
- "from google.colab import auth\n",
- "from google.cloud import bigquery\n",
- "import gspread\n",
- "from oauth2client.client import GoogleCredentials\n"
- ],
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "6f4ZjU27iEM9",
- "colab_type": "text"
- },
- "source": [
- "# 2) Authenticate BQ here.\n",
- "BigQuery is used to check your used references against the KG."
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "KZx5kuFd_76D",
- "colab_type": "code",
- "cellView": "both",
- "colab": {}
- },
- "source": [
- "#@title Do you have BigQuery Access? (Internal Googler)\n",
- "bq_access = True\n",
- "\n",
- "if bq_access:\n",
- " auth.authenticate_user()"
- ],
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "OtqIpPaUiMZF",
- "colab_type": "text"
- },
- "source": [
- "## Helper Functions"
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "ZfqqK8zg-384",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- "# Setup BQ client\n",
- "client = None\n",
- "if bq_access:\n",
- " project_id = \"google.com:datcom-store-dev\"\n",
- " client = bigquery.Client(project=project_id)\n",
- "\n",
- "# Setup logging\n",
- "# gc = gspread.authorize(GoogleCredentials.get_application_default())\n",
- "\n",
- "# Enum definition\n",
- "from enum import Enum\n",
- "class PrecheckError(Enum):\n",
- " CRITICAL = \"Critical\"\n",
- " WARN = \"Warn\"\n",
- "\n",
- "# Helpers\n",
- "cache = {}\n",
- "\n",
- "def validateNodeStructure(mcf_contents):\n",
- " # See if node has been processed\n",
- " hash_of_contents = \"validateNodeStructure_\" + str(hash(mcf_contents))\n",
- " if hash_of_contents in cache:\n",
- " return cache[hash_of_contents]\n",
- "\n",
- " # Nodes in an MCF file are separated by a blank line \n",
- " mcf_nodes_text = mcf_contents.split(\"\\n\\n\")\n",
- "\n",
- " # Lines in an MCF file are separated as property: constraint\n",
- " mcf_line = re.compile(r\"^(\\w+): (.*)$\")\n",
- "\n",
- " mcf_nodes = []\n",
- " errors = []\n",
- "\n",
- " for node in mcf_nodes_text:\n",
- " current_mcf_node = {}\n",
- "\n",
- " for line in node.split('\\n'):\n",
- " # Ignore blank lines if multiple spaces between lines\n",
- " if len(line) == 0:\n",
- " continue\n",
- "\n",
- " parsed_line = mcf_line.match(line)\n",
- "\n",
- " if parsed_line is None:\n",
- " errors.append((PrecheckError.CRITICAL, \"MalformedLine\", f\"Malformed MCF Line '{line}'\"))\n",
- " else:\n",
- " # Property = Constraint\n",
- " current_mcf_node[parsed_line.group(1)] = parsed_line.group(2)\n",
- " \n",
- " if len(current_mcf_node) > 0:\n",
- " mcf_nodes.append(current_mcf_node)\n",
- "\n",
- " # Add to cache\n",
- " cache[hash_of_contents] = (mcf_nodes, errors)\n",
- "\n",
- " return mcf_nodes, errors\n",
- " \n",
- "def get_nodes_with_property(mcf_contents, prop, constraint):\n",
- " mcf_nodes, errors = validateNodeStructure(mcf_contents)\n",
- "\n",
- " matching_nodes = []\n",
- " for node in mcf_nodes:\n",
- " if prop in node and node[prop] == constraint:\n",
- " matching_nodes.append(node)\n",
- "\n",
- " return matching_nodes\n",
- "\n",
- "def remove_prefix(s):\n",
- " \"\"\"Removes prefixes 'dcs:', 'dcid:' and 'schema:' to ease node comparison.\"\"\"\n",
- " s = s.strip()\n",
- " if s.startswith('dcs:'):\n",
- " return s[4:]\n",
- " if s.startswith('dcid:'):\n",
- " return s[5:]\n",
- " if s.startswith('schema:'):\n",
- " return s[7:]\n",
- " return s\n",
- "\n",
- "def cmp_nodes(n1, n2):\n",
- " \"\"\"Compares two nodes, ignoring prefixes such as in remove_prefix()\"\"\"\n",
- " return remove_prefix(n1) == remove_prefix(n2)\n",
- "\n",
- "def get_newly_defined_nodes(mcf_contents, typeOf = \"\"):\n",
- " mcf_nodes, errors = validateNodeStructure(mcf_contents)\n",
- "\n",
- " new_nodes = []\n",
- " for node in mcf_nodes:\n",
- " if \"Node\" in node and \"typeOf\" in node and \\\n",
- " (typeOf == \"\" or typeOf == remove_prefix(node['typeOf'])):\n",
- " new_nodes.append(node['Node'].replace(\"dcs:\",\"\").replace(\"dcid:\",\"\"))\n",
- "\n",
- " return new_nodes\n"
- ],
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "FwyIS2U_iQUc",
- "colab_type": "text"
- },
- "source": [
- "## Definition of Tests"
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "08YdJ9R14-Pu",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- "class TriplesChecks():\n",
- " \"\"\"Defines the various tests that run on the combined contents of TMCF,\n",
- " uploaded csv, and statistical variable files. \n",
- " \n",
- " To add a test: Make a new method with the following args.\n",
- " Args:\n",
- " df -> Dataframe of uploaded CSV\n",
- " tmcf_contents -> String of TMCF text content\n",
- " stat_vars -> String of Statistical Variables file\n",
- " Yields:\n",
- " Yields tuple of the precheck error level enum, error name, and an error message\n",
- " \"\"\"\n",
- "\n",
- " def ensure_ascii(_, tmcf_contents, stat_vars_content):\n",
- " \"\"\"Checks to ensure that files contents are solely ascii characters.\"\"\"\n",
- " ascii_character_match = re.compile(r\"^[\\x00-\\x7F]+$\")\n",
- "\n",
- " for file_name, contents in \\\n",
- " [(\"TMCF\", tmcf_contents), (\"Statistical Variables\", stat_vars_content)]:\n",
- "\n",
- " if ascii_character_match.match(contents) == None:\n",
- " yield (PrecheckError.CRITICAL, \"NonAsciiInFile\",\n",
- " f\"{file_name} file contains non-ascii characters.\")\n",
- "\n",
- " def tmcf_csv_column_checks(df, tmcf_contents, stat_vars_content):\n",
- " \"\"\"Handles column inconsistencies between tmcf and csv.\"\"\"\n",
- " column_matches = re.compile(r\"C:\\w+->(\\w+)\")\n",
- " tmcf_columns = column_matches.findall(tmcf_contents)\n",
- " csv_columns = df.columns\n",
- " for column in tmcf_columns:\n",
- " if column not in csv_columns:\n",
- " yield (PrecheckError.CRITICAL, \"ColInTMCFMissingFromCSV\",\n",
- " f\"Referenced column {column} in TMCF not found in CSV.\")\n",
- " for column in csv_columns:\n",
- " if column not in tmcf_columns:\n",
- " yield (PrecheckError.WARN, \"UnusedColumnPresent\",\n",
- " f\"Unused column {column} present in CSV.\")\n",
- " \n",
- " def ensure_mcf_not_malformed(_, tmcf_contents, stat_vars_content):\n",
- " \"\"\"Ensures lines of MCF files are property defined.\n",
- " Passes: Node: E:WorldBank->E0\n",
- " Fails: Node E:WorldBank->E0 \n",
- " \"\"\"\n",
- " # Grab error field of tuple\n",
- " for error in validateNodeStructure(tmcf_contents)[1]:\n",
- " yield error\n",
- "\n",
- " for error in validateNodeStructure(stat_vars_content)[1]:\n",
- " yield error\n",
- "\n",
- " def ensure_nodes_properly_referenced(_, tmcf_contents, stat_vars_content):\n",
- " \"\"\"Ensures that constraint field of mcf files are references or constants.\"\"\"\n",
- " tmcf_nodes, _ = validateNodeStructure(tmcf_contents)\n",
- " stat_var_nodes, _ = validateNodeStructure(stat_vars_content)\n",
- "\n",
- " # Ensure that each property is a string, integer, boolean, tmcf reference, or schema reference \n",
- " tmcf_match = re.compile(r\"^(\\\"[^\\\"]+\\\")|(E:\\w+->E\\d+)|(C:\\w+->\\w+)|(\\d+)|(True)|(False)|(((dcs)|(dcid)|(schema)):\\w+)$\")\n",
- "\n",
- " # Ensure that each property is a string, integer, boolean, tmcf reference, schema reference, or quantity range\n",
- " stvr_match = re.compile(r\"^(\\\"[^\\\"]+\\\")|(\\d+)|(True)|(False)|((((dcs)|(dcid)|(schema)):[A-Za-z0-9_\\-\\/]+,? ?)+)|(\\[\\S+ ((\\d+)|(\\d+ \\d+)|(\\d+ \\+)|(\\- \\d+))\\])$\")\n",
- " tmcf_node_match = re.compile(\"^E:\\S+->E\\d+$\")\n",
- " stvr_node_match = re.compile(\"^dcid:\\S+$\")\n",
- "\n",
- " for node_list, node_prop_regex, property_regex in \\\n",
- " [(tmcf_nodes, tmcf_node_match, tmcf_match),\n",
- " (stat_var_nodes, stvr_node_match, stvr_match)]:\n",
- " for node in node_list:\n",
- " for prop, constraint in node.items():\n",
- " if prop == \"Node\":\n",
- " if node_prop_regex.match(constraint) == None:\n",
- " yield (PrecheckError.CRITICAL, \"MalformedNode\",\n",
- " f\"Malformed Node Property '{prop}: {constraint}'\")\n",
- " \n",
- " # Validate properties of TMCF\n",
- " elif prop[0].islower():\n",
- " if property_regex.match(constraint) == None:\n",
- " yield (PrecheckError.WARN, \"MisformedReference\",\n",
- " f\"Misformed Reference: '{prop}: {constraint}'\")\n",
- "\n",
- " # All properties besides Node should be lower case\n",
- " else:\n",
- " yield (PrecheckError.WARN, \"LowerProperties\",\n",
- " f\"All MCF Properties besides Node should be lowercase. Triggered for '{prop}'.\")\n",
- " \n",
- " def spell_check_descriptions(_, tmcf_contents, stat_vars_content):\n",
- " \"\"\"Provides spell checking on all description fields.\"\"\"\n",
- " description_field_parser = re.compile(\"description: \\\"([^\\\"]*)\\\"\")\n",
- " spell = SpellChecker()\n",
- " sets_to_check = [(\"TMCF\", tmcf_contents), \n",
- " (\"Statistical Variables\", stat_vars_content)]\n",
- "\n",
- " for set_name, text in sets_to_check:\n",
- " potential_mispellings = set()\n",
- " for description in description_field_parser.findall(text):\n",
- " potential_mispellings = potential_mispellings.union(\n",
- " spell.unknown(spell.split_words(description))\n",
- " )\n",
- "\n",
- " if len(potential_mispellings) != 0:\n",
- " yield (PrecheckError.WARN, \"Misspelling\",\n",
- " f\"Potential Misspelling(s) in {set_name}: {list(potential_mispellings)})\")\n",
- "\n",
- " def ensure_all_references_exist(df, tmcf_contents, stat_vars_content):\n",
- " if not bq_access:\n",
- " return\n",
- " \n",
- " # Get locally defined instances.\n",
- " new_references = get_newly_defined_nodes(stat_vars_content)\n",
- "\n",
- " # Get all references instances in stat vars\n",
- " ref_finder = re.compile(r\"(?:(?:dcs)|(?:dcid)):(\\S+)\")\n",
- " references = list(set(ref_finder.findall(stat_vars_content)))\n",
- "\n",
- " # Get all stat vars that are not locally defined\n",
- " global_references = []\n",
- " for ref in references:\n",
- " if len(ref) != 0 and ref not in new_references:\n",
- " global_references.append(ref)\n",
- "\n",
- " # Query database \n",
- " instance_query = \"\"\"\n",
- " SELECT distinct id\n",
- " FROM `google.com:datcom-store-dev.dc_v3_clustered.Instance` \n",
- " WHERE id IN ({str})\n",
- " \"\"\"\n",
- " obj_instances = client.query(instance_query.replace(\"{str}\",\n",
- " str(global_references).lstrip(\"[\").rstrip(\"]\"))).to_dataframe()['id'].values\n",
- "\n",
- " missing_references = []\n",
- " for ref in global_references:\n",
- " if ref not in obj_instances:\n",
- " missing_references.append(ref)\n",
- "\n",
- " if len(missing_references) != 0:\n",
- " yield (PrecheckError.WARN, \"UndefinedReference\",\n",
- " f\"Potential Undefined References: {missing_references}\")\n",
- " \n",
- " def ensure_all_statvars_defined(df, tmcf_contents, stat_vars_content):\n",
- " direct_ref = re.compile(r'variableMeasured:\\s(\\w+)$')\n",
- " indirect_ref = re.compile(r'variableMeasured:\\sC:\\w+->(\\w+)')\n",
- " defined_nodes = get_newly_defined_nodes(stat_vars_content)\n",
- " for ref in direct_ref.findall(tmcf_contents):\n",
- " if not any([cmp_nodes(ref, n) for n in defined_nodes]):\n",
- " yield (PrecheckError.CRITICAL, \"TMCFNodeRefNotInMCF\",\n",
- " f\"Node '{ref}' referenced in TMCF, undefined in MCF.\")\n",
- " \n",
- " for col_ref in indirect_ref.findall(tmcf_contents):\n",
- " if col_ref not in df.columns:\n",
- " yield (PrecheckError.CRITICAL, \"ColInTMCFNotInCSV\",\n",
- " f\"Column '{col_ref}' referenced in TMCF, not in CSV.\")\n",
- " else:\n",
- " for ref in df[col_ref].unique():\n",
- " if not any([cmp_nodes(ref, n) for n in defined_nodes]):\n",
- " yield (PrecheckError.CRITICAL, \"ReferencedFieldNotInMcf\",\n",
- " f\"Node '{ref}' referenced in TMCF through '{col_ref}' column in CSV, undefined in MCF.\")\n",
- " \n",
- " def ensure_dcid_not_too_long(_, __, stat_vars_content):\n",
- " for line in stat_vars_content.split(\"\\n\"):\n",
- " if 'Node:' in line and len(line) - len('Node: ') > 256:\n",
- " yield (PrecheckError.CRITICAL, \"MalformedNode\",\n",
- " f\"The following node is too long: '{line.strip()}'\\nMax dcid length is 256.\")\n",
- "\n"
- ],
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "vbJptJFUBnKv",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- ""
- ],
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "8G8pWgrhAVen",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- ""
- ],
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "C7Tbwl3CrKN-",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- "from optparse import OptionParser\n",
- "import inspect\n",
- "\n",
- "def validate_prechecks(df, tmcf_contents, stat_vars_content, demo=False):\n",
- " \"\"\"Runs validation checks on provided triples.\n",
- "\n",
- " Args:\n",
- " df -> Dataframe of uploaded CSV\n",
- " tmcf_contents -> String of TMCF text content\n",
- " stat_vars_content -> String of Statistical Variables file\n",
- " \"\"\"\n",
- " # Log usage to find common errors\n",
- " # process = subprocess.Popen(\"gcloud config get-value account\", shell=True, stdout=subprocess.PIPE)\n",
- " # username = process.stdout.read().decode(\"utf-8\")\n",
- " # gc = gspread.authorize(GoogleCredentials.get_application_default())\n",
- " # workbook = gc.open_by_url('https://docs.google.com/spreadsheets/d/1l4YqvkhzRBKtab5lVCuoR0guGf2qIARK-A995DFZ6Nw/edit?usp=sharing')\n",
- " # sheet = workbook.worksheet('Usage')\n",
- " # if demo:\n",
- " # sheet.append_row([username, \"RanDemo\"])\n",
- "\n",
- " for function_name, function in inspect.getmembers(TriplesChecks, predicate=inspect.isfunction):\n",
- " errors = list(function(df, tmcf_contents, stat_vars_content))\n",
- " if len(errors) != 0:\n",
- " print(f\"Error In Test {function_name}\")\n",
- " for error in errors: \n",
- " # if not demo:\n",
- " # sheet.append_row([username, str(error[0].value), error[1], error[2]])\n",
- " print(f\"{error[0].value} - {error[2]}\")\n",
- " print(\"\")"
- ],
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "6zqZfVVAo9bQ",
- "colab_type": "text"
- },
- "source": [
- "# Sample Upload Demonstrating Common Errors"
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "unseBz2v7ziU",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- "# Sample Input\n",
- "stat_vars_content = \\\n",
- "\"\"\"\n",
- "Node: dcid:Tourism\n",
- "name: “Tourism“\n",
- "typeOf: dcid:TravelPurposeEnum\n",
- "description: \"Ptential mispeling in my description.\" \n",
- "\n",
- "Node: dcid:ThisIsAnExtremelyLongStatVarName__Count_MortalityEvent_From75To79Years_MalignantImmunoproliferativeDiseasesAndCertainOtherB-CellLymphomas_MultipleMyelomaAndMalignantPlasmaCellNeoplasms_OtherAndUnspecifiedMalignantNeoplasmsOfLymphoid_HematopoieticAndRelatedTissue_Female_AsFractionOf_Count_Person\n",
- "name: “Tourism“\n",
- "typeOf: dcid:TravelPurposeEnum\n",
- "description: \"Ptential mispeling in my description.\" \n",
- "\"\"\"\n",
- "\n",
- "tmcf_contents = \\\n",
- "\"\"\"\n",
- "Node E:WorldBank->E0\n",
- "typeOf: dcs:StatVarObservation\n",
- "variableMeasured: C:WorldBank->StatisticalVariable\n",
- "observationDate: C:WorldBank->Year\n",
- "observationPeriod: \"P1Y\"\n",
- "observationAbout: E:WorldBank->E1\n",
- "value: C:WorldBank->Value\n",
- "\n",
- "Node: E:WorldBank->E1\n",
- "typeOf: dcs:Country\n",
- "countryAlpha3Code: C:WorldBank->IsoCode\n",
- "BadProperty: someFieldThatShouldBeAReferenceButIsInterpretedAsAString\n",
- "\"\"\"\n",
- "\n",
- "df = pd.DataFrame.from_dict({\"Value\": [4], \"IsoCode\": [\"USA\"], \"Year\":[2018], \"Foo\": ['bar']})\n",
- "df"
- ],
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "g9s8-wZNrXKE",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- "validate_prechecks(df, tmcf_contents, stat_vars_content, demo=True)"
- ],
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "2TYYTYGJoJEZ",
- "colab_type": "text"
- },
- "source": [
- "# Real Validation Code\n",
- "\n",
- "Upload three files.\n",
- "\n",
- "- StatisticalVariable -> Needs to end in .mcf\n",
- "- Template MCF -> Needs to end in .tmcf\n",
- "- CSV File -> Needs to end in .csv"
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "OMTKha2y8PQ3",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- "from google.colab import files\n",
- "from io import StringIO\n",
- "\n",
- "LARGE_FILE_FLAG = False # Set this to true for large files\n",
- "selected_files = files.upload()\n",
- "\n",
- "# Parse out files\n",
- "df, tmcf_text, stat_var_text = None, None, None\n",
- "\n",
- "for file, contents in selected_files.items():\n",
- " if \".csv\" in file:\n",
- " if not LARGE_FILE_FLAG:\n",
- " df = pd.read_csv(StringIO(contents.decode()))\n",
- " else:\n",
- " df = pd.read_csv(StringIO(contents.decode()), nrows=100)\n",
- " elif \".tmcf\" in file:\n",
- " tmcf_text = contents.decode()\n",
- " elif \".mcf\" in file:\n",
- " stat_var_text = contents.decode()"
- ],
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "e4_63BXZsxtx",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- "validate_prechecks(df, tmcf_text, stat_var_text)"
- ],
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "sfEfIQMNtiIK",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- ""
- ],
- "execution_count": null,
- "outputs": []
- }
- ]
-}
\ No newline at end of file
diff --git a/import-validation-helper/README.md b/import-validation-helper/README.md
deleted file mode 100644
index b637a518..00000000
--- a/import-validation-helper/README.md
+++ /dev/null
@@ -1,15 +0,0 @@
-# Import Validation Helper Colab Notebook
-
-This directory contains an iPython notebook that introduces a few tools to check
-template MCF, statistical variable MCF, and CSV files for common errors.
-Access the
-[`Colab Notebook`](https://colab.research.google.com/drive/11YaPJA_g7O8_zm1iim3vNBrlW8O5SO98?usp=sharing)
-directly or run the file locally as a Jupyter Notebook.
-
-## Maintenance
-
-To maintain an up to date version of this notebook, developers can save a copy
-of the above notebook to a GitHub repository and PR this repository. Navigate
-to `File > Save a copy in GitHub...`
-
-
diff --git a/imputation/README.md b/imputation/README.md
deleted file mode 100644
index 7a286778..00000000
--- a/imputation/README.md
+++ /dev/null
@@ -1 +0,0 @@
-Go module for adding missing values to a time series.
diff --git a/imputation/go.mod b/imputation/go.mod
deleted file mode 100644
index 04512e4f..00000000
--- a/imputation/go.mod
+++ /dev/null
@@ -1,3 +0,0 @@
-module imputation
-
-go 1.15
diff --git a/imputation/ts_impute.go b/imputation/ts_impute.go
deleted file mode 100644
index 5dc2fdf5..00000000
--- a/imputation/ts_impute.go
+++ /dev/null
@@ -1,322 +0,0 @@
-// Copyright 2024 Google LLC
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// https://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-/*
-Package imputation implements a library for time series imputation.
-The dates in the time series are assumed to be either of the forms of (all have the same format):
-"YYYY-MM-DD" or
-"YYYY-MM" or
-"YYYY"
-
-The imputation library first finds the GCD(gap) of existing points and then adds new data points with values according to the selected method.
-*/
-package imputation
-
-import (
- "errors"
- "fmt"
- "log"
- "sort"
- "strings"
- "time"
-)
-
-// TimeSeries stores a time series. Dates are expressed in ISO 8061 format as illustrated in the constants below.
-type TimeSeries map[string]float64
-
-// Assume 3 formatting for the dates: daily, monthly, and yearly.
-const (
- dayfmt = "2006-01-02"
- monthfmt = "2006-01"
- yearfmt = "2006"
-)
-
-// Greatest common divisor (GCD) of two integer values.
-func gcd(a, b int) int {
- for b != 0 {
- t := b
- b = a % b
- a = t
- }
- return a
-}
-
-// dateGapFinder returns the greatest common divisor-like time gap for the given times.
-//
-// Args:
-//
-// keys: A sorted list of times all with the same format, one of those listed above.
-//
-// Returns:
-// 1. The time format of the keys.
-// 2. The GCD (gap between the keys) in terms of years, months, and days.
-// 3. OK is True iff the function successfully computed the frequency of data collection.
-// 4. Error != nil in case input has multiple date formats or non ISO 8601.
-func dateGapFinder(keys []string) (string, int, int, int, bool, error) {
- var parseFormat string
- year, month, day := 0, 0, 0
-
- switch len(keys[0]) {
- case 4:
- parseFormat = yearfmt
- case 7:
- parseFormat = monthfmt
- case 10:
- parseFormat = dayfmt
- default:
- return parseFormat, year, month, day, false, errors.New("The date format is not ISO 8601.")
- }
-
- equalMonths := true
- equalDays := true
-
- if parseFormat != yearfmt {
- for i := 0; i < len(keys)-1; i++ {
- // Return error in case the dates do not have equal formats.
- if len(keys[i+1]) != len(parseFormat) {
- return parseFormat, year, month,
- day, false, errors.New("The dates have multiple formats.")
- }
-
- md1 := strings.Split(keys[i], "-")
- md2 := strings.Split(keys[i+1], "-")
-
- if len(md1) != len(md2) {
- return parseFormat, year, month,
- day, false, errors.New("The dates have multiple formats.")
- }
- // Check for unequal months.
- if md1[1] != md2[1] {
- equalMonths = false
- }
- // Check for unequal days (if exists).
- if len(md1) == 3 {
- if md1[2] != md2[2] {
- equalDays = false
- }
- }
- }
- }
-
- if parseFormat == yearfmt || (equalMonths && equalDays) {
- for i := 0; i < len(keys)-1; i++ {
- if year == 1 {
- // One year is the minimum possible gap in this case.
- break
- }
- start, err1 := time.Parse(parseFormat, keys[i])
- end, err2 := time.Parse(parseFormat, keys[i+1])
- if err1 != nil || err2 != nil {
- return parseFormat, year, month, day, false, errors.New("The date format is not ISO 8601.")
- }
-
- y1, _, _ := start.Date()
- y2, _, _ := end.Date()
- yGap := int(y2 - y1)
-
- if i == 0 {
- // Initialize year with the value of the first gap.
- year = yGap
- } else {
- year = gcd(yGap, year)
- }
- }
- } else if parseFormat == monthfmt || equalDays {
- // One month is the minimum possible gap in this case.
- for i := 0; i < len(keys)-1; i++ {
- if month == 1 {
- break
- }
- start, err1 := time.Parse(parseFormat, keys[i])
- end, err2 := time.Parse(parseFormat, keys[i+1])
- if err1 != nil || err2 != nil {
- return parseFormat, year, month, day, false, errors.New("The date format is not ISO 8601.")
- }
-
- y1, M1, _ := start.Date()
- y2, M2, _ := end.Date()
- yGap := int(y2 - y1)
- mGap := yGap*12 + int(M2-M1)
-
- if i == 0 {
- // Initialize month with the value of the first gap.
- month = mGap
- } else {
- log.Println(mGap)
- month = gcd(mGap, month)
- }
- }
- } else {
- for i := 0; i < len(keys)-1; i++ {
- // One day is the minimum possible gap in this case.
- if day == 1 {
- break
- }
- start, err1 := time.Parse(parseFormat, keys[i])
- end, err2 := time.Parse(parseFormat, keys[i+1])
- if err1 != nil || err2 != nil {
- return parseFormat, year, month, day, false, errors.New("The date format is not ISO 8601.")
- }
-
- dGap := int(end.Sub(start).Hours() / 24)
- if i == 0 {
- // Initialize day with the value of the first gap.
- day = dGap
- } else {
- day = gcd(dGap, day)
- }
- }
- }
- if year == 0 && month == 0 && day == 0 {
- return parseFormat, year, month, day, false, nil
- }
- return parseFormat, year, month, day, true, nil
-}
-
-func getKeysAndValues(ts TimeSeries) ([]string, []float64) {
- keys := make([]string, 0, len(ts))
- values := make([]float64, 0, len(ts))
- for k, v := range ts {
- keys = append(keys, k)
- values = append(values, v)
- }
- return keys, values
-}
-
-// FillNA returns TimeSeries with missing datapoints with assigned values according to the selected method.
-// Method can be "mean", "median", and "zero".
-func FillNA(ts TimeSeries, method string) (TimeSeries, error) {
- if len(ts) < 3 {
- log.Printf("There is not enough data to impute.")
- return ts, nil
- }
- keys, values := getKeysAndValues(ts)
- sort.Strings(keys)
-
- parseFormat, yStep, mStep, dStep, ok, err := dateGapFinder(keys)
- if err != nil {
- return ts, err
- }
- if !ok {
- return ts, errors.New("Dates are not annual, monthly, or daily.")
- }
-
- log.Printf("Step is equal to : %v years, %v months, %v days \n", yStep, mStep, dStep)
-
- // Computing the filling value.
- var fillV float64
- switch method {
- case "mean":
- for _, v := range values {
- fillV += v
- }
- fillV = fillV / float64(len(keys))
- case "zero":
- fillV = 0
- case "median":
- sort.Float64s(values)
- if len(values)%2 == 0 {
- // In case of even length: median = average of two middle values.
- fillV = (values[len(values)/2] + values[len(values)/2-1]) / 2
- log.Printf("Median is %v: for even case", fillV)
- } else {
- fillV = values[len(values)/2]
- log.Printf("Median is %v: for odd case", fillV)
- }
- default:
- return ts, errors.New("The method is unknown.")
- }
-
- startDate, err1 := time.Parse(parseFormat, keys[0])
- endDate, err2 := time.Parse(parseFormat, keys[len(keys)-1])
- if err1 != nil || err2 != nil {
- return ts, errors.New("The date format is not ISO 8601.")
- }
-
- for d := startDate; d.After(endDate) == false; d = d.AddDate(yStep, mStep, dStep) {
- if _, ok := ts[fmt.Sprint(d.Format(parseFormat))]; !ok {
- ts[fmt.Sprint(d.Format(parseFormat))] = fillV
- }
- }
- return ts, nil
-}
-
-// Interpolate function's input is a TimeSeries with possible missing datapoints.
-// The function returns a Timeseries with no missing dates. Interpolation method is used to fill the values.
-// Methods can vary by degree: d = {1} is implemented.
-func Interpolate(ts TimeSeries, degree int) (TimeSeries, error) {
- if len(ts) < 3+(degree-1) {
- log.Printf("There is not enough data to impute.")
- return ts, nil
- }
-
- switch degree {
- case 1:
- return linear(ts)
- default:
- return ts, errors.New("Not implemented yet.")
- }
-}
-
-// Linear interpolation; this is eqivalent to Spline degree 1.
-func linear(ts TimeSeries) (TimeSeries, error) {
- keys, _ := getKeysAndValues(ts)
- sort.Strings(keys)
-
- parseFormat, yStep, mStep, dStep, ok, err := dateGapFinder(keys)
- if err != nil {
- return ts, err
- }
- if !ok {
- return ts, errors.New("Dates are not annual, monthly, or daily.")
- }
- log.Printf("Step is equal to : %v years, %v months, %v days \n", yStep, mStep, dStep)
-
- startDate, err1 := time.Parse(parseFormat, keys[0])
- endDate, err2 := time.Parse(parseFormat, keys[len(keys)-1])
- if err1 != nil || err2 != nil {
- return ts, errors.New("The date format is not ISO 8601.")
- }
-
- // Assuming the time series is the form of {date1 : y1, date2: y2 ,date3 : y3} and y2
- // is the missing value. d21 = date2-date1 and d31 = date3-date1
- var d21, d31, y1, y2, y3 float64
- nextDate := startDate
-
- for d := startDate; d.After(endDate) == false; d = d.AddDate(yStep, mStep, dStep) {
- if _, ok := ts[fmt.Sprint(d.Format(parseFormat))]; !ok {
- d21 += 1
- // Looking for the next available date (date3) and y3. Store gap in d31.
- if nextDate.Before(d) {
- d31 = d21
- nextDate = d
- for !ok {
- nextDate = nextDate.AddDate(yStep, mStep, dStep)
- _, ok = ts[fmt.Sprint(nextDate.Format(parseFormat))]
- d31 += 1
- }
- y3, _ = ts[fmt.Sprint(nextDate.Format(parseFormat))]
- }
-
- y2 = ((d21) * (y3 - y1) / (d31)) + y1
- ts[fmt.Sprint(d.Format(parseFormat))] = y2
- } else {
- d21 = 0
- y1, _ = ts[fmt.Sprint(d.Format(parseFormat))]
- // Reset next available date.
- nextDate = startDate
- }
- }
- return ts, nil
-}
diff --git a/imputation/ts_impute_test.go b/imputation/ts_impute_test.go
deleted file mode 100644
index e2ddce7e..00000000
--- a/imputation/ts_impute_test.go
+++ /dev/null
@@ -1,354 +0,0 @@
-// Copyright 2024 Google LLC
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// https://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package imputation
-
-import (
- "math"
- "sort"
- "testing"
-)
-
-// Data structure for storing precomputed values of each interpolation method.
-type InterpolateValues struct {
- linear float64
-}
-
-// computeMeans computes the mean of the given non-empty time series's data.
-func computeMean(s TimeSeries) float64 {
- var mean float64
- for _, v := range s {
- mean += v
- }
- mean = mean / float64(len(s))
- return mean
-}
-
-// computeMedian computes the median value of the given non-empty time series's data.
-func computeMedian(s TimeSeries) float64 {
- values := make([]float64, 0)
- for _, v := range s {
- values = append(values, v)
- }
- sort.Float64s(values)
- if len(values)%2 == 0 {
- return (values[len(values)/2] + values[len(values)/2-1]) / 2
- } else {
- return values[len(values)/2]
- }
-}
-
-// createExpectedTimeSeriesFillNA generate a new time series by adding desiredValues as data points and replace
-// their value by using the specified algorithm.
-func createExpectedTimeSeriesFillNA(ts TimeSeries, desiredValues []string, method string) TimeSeries {
- out := make(TimeSeries, len(ts)+len(desiredValues))
- for k, v := range ts {
- out[k] = v
- }
- switch method {
- case "mean":
- if len(ts) > 0 {
- m := computeMean(ts)
- for _, k := range desiredValues {
- out[k] = m
- }
- }
- case "zero":
- for _, k := range desiredValues {
- out[k] = 0
- }
- case "median":
- if len(ts) > 0 {
- median := computeMedian(ts)
- for _, k := range desiredValues {
- out[k] = median
- }
- }
- }
- return (out)
-}
-
-// createExpectedTimeSeriesInterpolate generate a new time series by adding desiredValues as data points and replace
-// their value using given precomputed values for each method.
-func createExpectedTimeSeriesInterpolate(ts TimeSeries, desiredValues map[string]InterpolateValues, method string) TimeSeries {
- out := make(TimeSeries, len(ts)+len(desiredValues))
- for k, v := range ts {
- out[k] = v
- }
- switch method {
- case "linear":
- if len(ts) > 2 {
- for k, v := range desiredValues {
- out[k] = v.linear
- }
- }
- }
- return (out)
-}
-
-// Test function for FillNA; possible methods = {mean, median, zero}
-func TestFillNA(t *testing.T) {
- methods := []string{"mean", "zero", "median"}
- for _, m := range methods {
- tests := map[string]struct {
- series TimeSeries
- additionalKeys []string
- }{
- "empty": {
- series: TimeSeries{},
- additionalKeys: []string{},
- },
- "length 1": {
- series: TimeSeries{"2003": 42.5},
- additionalKeys: []string{},
- },
- "length 3, years": {
- series: TimeSeries{
- "2003": 42.5,
- "2002": 18.9,
- "1998": 28,
- "1996": 25,
- },
- additionalKeys: []string{"1999", "2000", "2001", "1997"},
- },
- "length 3, 1-month gap": {
- series: TimeSeries{
- "2003-02": 42.5,
- "2002-03": 18.9,
- "2003-04": 28,
- },
- additionalKeys: []string{"2002-04", "2002-05", "2002-06",
- "2002-07", "2002-08", "2002-09", "2002-10", "2002-11",
- "2002-12", "2003-01", "2003-03"},
- },
- "days": {
- series: TimeSeries{
- "2002-03-01": 42.5,
- "2002-02-01": 18.9,
- "2002-02-02": 28,
- },
- additionalKeys: []string{
- "2002-02-03",
- "2002-02-04",
- "2002-02-05",
- "2002-02-06",
- "2002-02-07",
- "2002-02-08",
- "2002-02-09",
- "2002-02-10",
- "2002-02-11",
- "2002-02-12",
- "2002-02-13",
- "2002-02-14",
- "2002-02-15",
- "2002-02-16",
- "2002-02-17",
- "2002-02-18",
- "2002-02-19",
- "2002-02-20",
- "2002-02-21",
- "2002-02-22",
- "2002-02-23",
- "2002-02-24",
- "2002-02-25",
- "2002-02-26",
- "2002-02-27",
- "2002-02-28",
- },
- },
- "35days": {
- series: TimeSeries{
- "2020-02-09": 42.5,
- "2020-03-15": 18.9,
- "2020-05-24": 28,
- },
- additionalKeys: []string{
- "2020-04-19",
- },
- },
- "28days": {
- series: TimeSeries{
- "2020-01-02": 42.5,
- "2020-05-21": 18.9,
- "2020-01-30": 28,
- },
- additionalKeys: []string{
- "2020-03-26",
- "2020-02-27",
- "2020-04-23",
- },
- },
- "3months": {
- series: TimeSeries{
- "2020-10-15": 42.5,
- "2020-07-15": 18.9,
- "2020-01-15": 28,
- },
- additionalKeys: []string{
- "2020-04-15",
- },
- },
- "annually": {
- series: TimeSeries{
- "2020-03-05": 42.5,
- "2016-03-05": 18.9,
- "2017-03-05": 28,
- },
- additionalKeys: []string{
- "2019-03-05",
- "2018-03-05",
- },
- },
- "non_ISO8601_format": {
- series: TimeSeries{
- "2020-03-05": 42.5,
- "2016-03-05": 18.9,
- "2017-0305": 28,
- },
- additionalKeys: []string{},
- },
- "multiple_date_formats": {
- series: TimeSeries{
- "2020-03-05": 42.5,
- "2016-03-05": 18.9,
- "2017-03": 28,
- },
- additionalKeys: []string{},
- },
- }
- for label, test := range tests {
- want := createExpectedTimeSeriesFillNA(test.series, test.additionalKeys, m)
- got, err := FillNA(test.series, m)
-
- if err != nil {
- t.Log(label, err)
- }
- if len(got) != len(want) {
- t.Errorf("%s: Size mismatch: Fill%v (%v) = %v, want %v",
- label, m, test.series, got, want)
- }
-
- for k, v := range got {
- if vw, ok := want[k]; !ok || vw != v {
- t.Errorf("%s: Value mismatch: Fill%v (%v) = %v, want %v",
- label, m, test.series, got, want)
- }
- }
- }
- }
-}
-
-// Test function for Interpolate; possible methods : {linear}
-// Values are produced using Python, Pandas linear interpolation method:
-func TestInterpolate(t *testing.T) {
- methods := []string{"linear"}
- for _, m := range methods {
- tests := map[string]struct {
- series TimeSeries
- additionalKeys map[string]InterpolateValues
- }{
- "empty": {
- series: TimeSeries{},
- additionalKeys: map[string]InterpolateValues{},
- },
- "length 1": {
- series: TimeSeries{"2003": 42.5},
- additionalKeys: map[string]InterpolateValues{},
- },
- "length 3, years": {
- series: TimeSeries{
- "2003": 42.5,
- "2002": 18.9,
- "1998": 28,
- "1996": 25,
- },
- additionalKeys: map[string]InterpolateValues{
- "1997": {linear: 26.500},
- "1999": {linear: 25.725},
- "2000": {linear: 23.450},
- "2001": {linear: 21.175},
- },
- },
- "length 3, 2-month gap": {
- series: TimeSeries{
- "2003-02": 42.5,
- "2002-02": 18.9,
- "2003-04": 28,
- },
- additionalKeys: map[string]InterpolateValues{
- "2002-04": {linear: 22.833},
- "2002-06": {linear: 26.766},
- "2002-08": {linear: 30.700},
- "2002-10": {linear: 34.633},
- "2002-12": {linear: 38.566},
- },
- },
- "days": {
- series: TimeSeries{
- "2002-03-01": 42.5,
- "2002-02-01": 18.9,
- "2002-02-02": 28,
- },
- additionalKeys: map[string]InterpolateValues{
- "2002-02-03": {linear: 28.537},
- "2002-02-04": {linear: 29.074},
- "2002-02-05": {linear: 29.611},
- "2002-02-06": {linear: 30.148},
- "2002-02-07": {linear: 30.685},
- "2002-02-08": {linear: 31.222},
- "2002-02-09": {linear: 31.759},
- "2002-02-10": {linear: 32.296},
- "2002-02-11": {linear: 32.833},
- "2002-02-12": {linear: 33.370},
- "2002-02-13": {linear: 33.907},
- "2002-02-14": {linear: 34.444},
- "2002-02-15": {linear: 34.981},
- "2002-02-16": {linear: 35.518},
- "2002-02-17": {linear: 36.055},
- "2002-02-18": {linear: 36.592},
- "2002-02-19": {linear: 37.129},
- "2002-02-20": {linear: 37.666},
- "2002-02-21": {linear: 38.203},
- "2002-02-22": {linear: 38.740},
- "2002-02-23": {linear: 39.277},
- "2002-02-24": {linear: 39.814},
- "2002-02-25": {linear: 40.351},
- "2002-02-26": {linear: 40.888},
- "2002-02-27": {linear: 41.425},
- "2002-02-28": {linear: 41.962},
- },
- },
- }
- for label, test := range tests {
- want := createExpectedTimeSeriesInterpolate(test.series, test.additionalKeys, m)
- got, err := Interpolate(test.series, 1)
-
- if err != nil {
- t.Log(label, err)
- }
-
- if len(got) != len(want) {
- t.Errorf("%s: Size mismatch: Fill%v (%v) = %v, want %v",
- label, m, test.series, got, want)
- }
-
- for k, v := range got {
- if vw, ok := want[k]; !ok || math.Abs(vw-v) > 0.001 {
- t.Errorf("%s: Value mismatch: Fill%v (%v) = %v, want %v",
- label, m, test.series, got, want)
- }
- }
- }
- }
-}
diff --git a/llm/README.md b/llm/README.md
deleted file mode 100644
index a4089a88..00000000
--- a/llm/README.md
+++ /dev/null
@@ -1,17 +0,0 @@
-# Data LLM
-
-This directory contains code to inference on LLMs integrated with Data Commons.
-
-It includes a python package called `data_gemma` that can be used for doing
-inference with the Gemma 2 (or other) LLMs fine-tuned for integration with Data
-Commons.
-
-## Install `data_gemma`
-
-```
-pip install git+https://github.com/datacommonsorg/tools#subdirectory=llm
-```
-
-## Examples
-
-TODO: Add links
diff --git a/llm/data_gemma/__init__.py b/llm/data_gemma/__init__.py
deleted file mode 100644
index 780aab07..00000000
--- a/llm/data_gemma/__init__.py
+++ /dev/null
@@ -1,41 +0,0 @@
-# Copyright 2024 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from data_gemma import base
-from data_gemma import baseline
-from data_gemma import datacommons
-from data_gemma import google_api
-from data_gemma import huggingface_api
-from data_gemma import openai_api
-from data_gemma import rag
-from data_gemma import rig
-
-# LLM related classes.
-LLM = base.LLM
-LLMCall = base.LLMCall
-GoogleAIStudio = google_api.GoogleAIStudio
-HFBasic = huggingface_api.HFBasic
-HFPipeline = huggingface_api.HFPipeline
-OpenAI = openai_api.OpenAI
-
-# Data Commons related classes.
-DataCommons = datacommons.DataCommons
-DataCommonsCall = base.DataCommonsCall
-
-# Flow related classes.
-Flow = base.Flow
-FlowResponse = base.FlowResponse
-BaselineFlow = baseline.BaselineFlow
-RAGFlow = rag.RAGFlow
-RIGFlow = rig.RIGFlow
diff --git a/llm/data_gemma/base.py b/llm/data_gemma/base.py
deleted file mode 100644
index e3aa5f98..00000000
--- a/llm/data_gemma/base.py
+++ /dev/null
@@ -1,180 +0,0 @@
-# Copyright 2024 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Base Types."""
-
-import dataclasses
-from typing import Any, Protocol
-
-
-DC = '__DC__'
-
-
-@dataclasses.dataclass(frozen=True)
-class Options:
- """Common options for all APIs."""
- # Print messages to stdout.
- verbose: bool = True
-
- def vlog(self, msg: str) -> None:
- if self.verbose:
- print(msg)
-
-
-@dataclasses.dataclass(frozen=True)
-class LLMCall:
- prompt: str
- response: str
- duration_secs: float
- error: str | None = None
-
- def debug(self, i: int = 0) -> str:
- return (
- f'\n### Prompt {i} ###\n{self.prompt}\n'
- f'### Response {i} ###\n{self.response}\n'
- f'### LLM Duration {i} {self.duration_secs}s ###\n'
- )
-
-
-@dataclasses.dataclass
-class DataCommonsCall:
- """A single request and response from Data Commons."""
-
- id: int = 0
- query: str = ''
-
- # For point: val and date is set
- val: str = ''
- date: str = ''
- # For table: table is set
- table: str = ''
-
- unit: str = ''
- title: str = ''
- src: str = ''
- url: str = ''
- var: str = ''
- score: float = 0.0
-
- # The original LLM Value in case of RIG.
- llm_val: str = ''
-
- def footnote(self) -> str:
- return (
- f'Per {self.src}, value was {self.val}{self._dunit()} in {self.date}.'
- f' See more at {self.url}'
- )
-
- def debug(self) -> str:
- if not self.title:
- return ''
- if self.table:
- return self.answer()
- return (
- f'"{self.title}" was {self.val}{self._dunit()} in'
- f' {self.date} per {self.src} ({self.var}:{self.score})'
- )
-
- def answer(self) -> str:
- if self.table:
- return f'{self.header()}\n{self.table}'
- else:
- return (
- f'According to {self.src}, "{self.title}" was'
- f' {self.val}{self._dunit()} in {self.date}.'
- )
-
- def header(self) -> str:
- if self.table:
- if self.unit:
- header = f'{self.title} (unit: {self.unit})'
- else:
- header = f'{self.title}'
- return f'{header}, according to {self.src}'
-
- return self.title
-
- def val_and_unit(self) -> str:
- return f'{self.val}{self._dunit()}'
-
- def _dunit(self) -> str:
- return ' ' + self.unit if self.unit else ''
-
-
-@dataclasses.dataclass(frozen=True)
-class FlowResponse:
- """A response from Flow."""
-
- main_text: str = ''
- footnotes: str = ''
- tables_str: str = ''
-
- llm_calls: list[LLMCall] = dataclasses.field(default_factory=list)
- dc_calls: list[DataCommonsCall] = dataclasses.field(default_factory=list)
- dc_duration_secs: float = 0.0
-
- def duration_secs(self) -> float:
- return (
- sum([r.duration_secs for r in self.llm_calls]) + self.dc_duration_secs
- )
-
- def answer(self, include_aux: bool = True) -> str:
- """Returns a string representation of the response."""
-
- lines = []
- lines.append(self.main_text)
-
- if include_aux and self.footnotes:
- lines.append('\n#### FOOTNOTES ####')
- lines.append(self.footnotes)
-
- if include_aux and self.tables_str:
- lines.append('\n#### TABLES ####')
- lines.append(self.tables_str)
-
- return '\n'.join(lines)
-
- def debug(self) -> str:
- """Returns a string representation of the response."""
-
- lines = []
- lines.append('\n\n## LLM INTERACTIONS ##\n')
- for i, llm_response in enumerate(self.llm_calls):
- lines.append(llm_response.debug(i))
-
- lines.append('\n\n## DC INTERACTIONS ##\n')
- for dc_response in self.dc_calls:
- dbg = dc_response.debug()
- if dbg:
- lines.append(dbg)
- lines.append(f'\n\n## DC Duration {self.dc_duration_secs} ##')
- lines.append(f'\n\n## Total Duration {self.duration_secs()} ##')
-
- return '\n'.join(lines)
-
- def json(self) -> dict[str, Any]:
- return dataclasses.asdict(self)
-
-
-class LLM(Protocol):
-
- def query(self, prompt: str) -> LLMCall:
- ...
-
-
-class Flow(Protocol):
- """A Flow integrates LLMs with DC in a certain way."""
-
- def query(self, query: str) -> FlowResponse:
- ...
diff --git a/llm/data_gemma/baseline.py b/llm/data_gemma/baseline.py
deleted file mode 100644
index 5cdf6bae..00000000
--- a/llm/data_gemma/baseline.py
+++ /dev/null
@@ -1,45 +0,0 @@
-# Copyright 2024 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Basic Flow."""
-
-from data_gemma import base
-
-
-class BaselineFlow(base.Flow):
- """Baseline Flow."""
-
- def __init__(
- self,
- llm: base.LLM,
- verbose: bool = True,
- ):
- self.llm = llm
- self.options = base.Options(verbose=verbose)
-
- def query(
- self,
- query: str,
- in_context: bool = False,
- prompt1: str = '',
- prompt2: str = '',
- ) -> base.FlowResponse:
- self.options.vlog('... [DEFAULT] Calling BASE model')
- resp = self.llm.query(query)
- return base.FlowResponse(
- main_text=resp.response,
- llm_calls=[resp],
- dc_duration_secs=0,
- dc_calls=[],
- )
diff --git a/llm/data_gemma/datacommons.py b/llm/data_gemma/datacommons.py
deleted file mode 100644
index 5b36cf80..00000000
--- a/llm/data_gemma/datacommons.py
+++ /dev/null
@@ -1,178 +0,0 @@
-# Copyright 2024 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""Data Commons."""
-
-import concurrent.futures
-import csv
-import io
-from typing import Any, Callable
-
-import requests
-
-from data_gemma import base
-from data_gemma import utils
-
-_BASE_URL = 'https://{env}.datacommons.org/nodejs/query'
-
-_POINT_MODE = 'toolformer_rig'
-_TABLE_MODE = 'toolformer_rag'
-
-# Do not allow topics, use higher threshold (0.8).
-_POINT_PARAMS = f'allCharts=1&mode={_POINT_MODE}&idx=base_uae_mem'
-
-# Allow topics, use lower threshold (0.7).
-_TABLE_PARAMS = f'mode={_TABLE_MODE}&client=table&idx=base_uae_mem'
-
-
-class DataCommons:
- """Data Commons."""
-
- def __init__(
- self,
- api_key: str,
- verbose: bool = True,
- num_threads: int = 10,
- env: str = 'datagemma',
- session: requests.Session | None = None,
- ):
- self.options = base.Options(verbose=verbose)
- self.num_threads = num_threads
- self.env = env
- self.api_key = api_key
- if not session:
- session = requests.Session()
- self.session = session
-
- def point(self, query: str) -> base.DataCommonsCall:
- """Calls Data Commons API."""
-
- self.options.vlog(f'... calling DC with "{query}"')
- response = self._call_api(query, _POINT_PARAMS)
- # Get the first LINE chart.
- chart = None
- for c in response.get('charts', []):
- ctype = c.get('type')
- if ctype == 'LINE' or ctype == 'HIGHLIGHT':
- chart = c
- break
- if not chart:
- return base.DataCommonsCall(query=query)
-
- v = str(chart.get('highlight', {}).get('value', ''))
- v = utils.round_float(v)
- if not v:
- return base.DataCommonsCall(query=query)
-
- u = chart.get('unit', '')
- d = chart.get('highlight', {}).get('date')
- s = _src(chart)
- t = chart.get('title', '')
-
- svm = response.get('debug', {}).get('debug', {}).get('sv_matching', {})
- score = svm.get('CosineScore', [-1])[0]
- var = svm.get('SV', [''])[0]
- url = chart.get('dcUrl', '')
- if url:
- url += f'&mode={_POINT_MODE}'
- return base.DataCommonsCall(
- query=query,
- val=v,
- unit=u,
- title=t,
- date=d,
- src=s,
- url=url,
- var=var,
- score=score,
- )
-
- def table(self, query: str) -> base.DataCommonsCall:
- """Calls Data Commons API."""
-
- self.options.vlog(f'... calling DC for table with "{query}"')
- response = self._call_api(query, _TABLE_PARAMS)
- # Get the first chart.
- charts = response.get('charts')
- if not charts:
- return base.DataCommonsCall(query=query)
- chart = charts[0]
-
- data_csv = chart.get('data_csv', '')
- rows = list(csv.reader(io.StringIO(data_csv)))
- if not data_csv or not rows:
- return base.DataCommonsCall(query=query)
-
- u = chart.get('unit', '')
- s = _src(chart)
- t = chart.get('title', '')
-
- parts = []
- parts.append(' | '.join(rows[0]))
- parts.append('-' * len(parts[-1]))
- for row in rows[1:]:
- row = [utils.round_float(v) for v in row]
- parts.append(' | '.join(row))
- parts.append('\n')
- table_str = '\n'.join(parts)
-
- svm = response.get('debug', {}).get('debug', {}).get('sv_matching', {})
- score = svm.get('CosineScore', [-1])[0]
- var = svm.get('SV', [''])[0]
- url = chart.get('dcUrl', '')
- if url:
- url += f'&mode={_TABLE_MODE}'
- return base.DataCommonsCall(
- query=query,
- unit=u,
- title=t,
- src=s,
- table=table_str,
- url=url,
- var=var,
- score=score,
- )
-
- def calln(
- self, queries: list[str], func: Callable[[str], base.DataCommonsCall]
- ) -> dict[str, base.DataCommonsCall]:
- """Calls Data Commons API in parallel if needed."""
-
- if self.num_threads == 1:
- results = [func(q) for q in queries]
- else:
- # TODO: Check why this ~breaks in Colab Borg runtime
- with concurrent.futures.ThreadPoolExecutor(self.num_threads) as executor:
- futures = [executor.submit(func, query) for query in queries]
- results = [f.result() for f in futures]
-
- q2resp: dict[str, base.DataCommonsCall] = {}
- for i, (q, r) in enumerate(zip(queries, results)):
- r.id = i + 1
- q2resp[q] = r
- return q2resp
-
- def _call_api(self, query: str, extra_params: str) -> Any:
- query = query.strip().replace(' ', '+')
- url = _BASE_URL.format(env=self.env) + f'?&q={query}&{extra_params}'
- if self.api_key:
- url = f'{url}&apikey={self.api_key}'
- # print(f'DC: Calling {url}')
- return self.session.get(url).json()
-
-
-def _src(chart: dict[str, Any]) -> str:
- srcs = chart.get('srcs', [{}])
- if not srcs:
- return ''
- return srcs[0].get('name', '')
diff --git a/llm/data_gemma/google_api.py b/llm/data_gemma/google_api.py
deleted file mode 100644
index 33acbdc1..00000000
--- a/llm/data_gemma/google_api.py
+++ /dev/null
@@ -1,125 +0,0 @@
-# Copyright 2024 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""LLM Interface."""
-
-import json
-import logging
-import time
-from typing import Any
-
-import requests
-
-from data_gemma import base
-
-
-_REQ_DATA = {
- 'contents': [{
- 'parts': [{
- 'text': '',
- }],
- }],
- 'generationConfig': {
- 'temperature': 0.1,
- },
- 'safetySettings': [
- {'category': 'HARM_CATEGORY_HARASSMENT', 'threshold': 'BLOCK_NONE'},
- {'category': 'HARM_CATEGORY_HATE_SPEECH', 'threshold': 'BLOCK_NONE'},
- {
- 'category': 'HARM_CATEGORY_SEXUALLY_EXPLICIT',
- 'threshold': 'BLOCK_NONE',
- },
- {
- 'category': 'HARM_CATEGORY_DANGEROUS_CONTENT',
- 'threshold': 'BLOCK_NONE',
- },
- ],
-}
-
-
-class GoogleAIStudio(base.LLM):
- """Google AI Studio."""
-
- def __init__(
- self,
- model: str,
- api_keys: list[str],
- verbose: bool = True,
- session: requests.Session | None = None,
- ):
- self.keys = api_keys
- if not session:
- session = requests.Session()
- self.session: requests.Session = session
- self.next_key_idx = 0
- self.options = base.Options(verbose=verbose)
- self.model = model
-
- def query(self, prompt: str) -> base.LLMCall:
- req_data = _REQ_DATA.copy()
-
- # set the params.
- req_data['generationConfig']['temperature'] = 0.1
- req_data['contents'][0]['parts'][0]['text'] = prompt
-
- # Make API request.
- req = json.dumps(req_data)
-
- start = time.time()
- self.options.vlog(
- f'... calling AIStudio {self.model} "{prompt[:50].strip()}..."'
- )
- resp = _call_api(self.session, self.model, self._get_key(), req)
- t = round(time.time() - start, 3)
- ans = ''
- err = ''
- if (
- 'candidates' in resp
- and resp['candidates']
- and 'content' in resp['candidates'][0]
- and 'parts' in resp['candidates'][0]['content']
- and resp['candidates'][0]['content']['parts']
- and 'text' in resp['candidates'][0]['content']['parts'][0]
- ):
- ans = resp['candidates'][0]['content']['parts'][0]['text']
- elif 'error' not in resp:
- err = 'Got empty response'
- logging.warning(err)
- else:
- err = json.dumps(resp)
- logging.error('%s', err)
-
- return base.LLMCall(prompt=prompt, response=ans, duration_secs=t, error=err)
-
- def _get_key(self):
- key = self.keys[self.next_key_idx]
- self.next_key_idx += 1
- if self.next_key_idx >= len(self.keys):
- self.next_key_idx = 0
- return key
-
-
-_BASE_URL = 'https://generativelanguage.googleapis.com/v1beta/models'
-_API_HEADER = {'content-type': 'application/json'}
-
-
-def _call_api(
- session: requests.Session, model: str, key: str, req_data: str
-) -> Any:
- r = session.post(
- f'{_BASE_URL}/{model}:generateContent?key={key}',
- data=req_data,
- headers=_API_HEADER,
- )
- return r.json()
diff --git a/llm/data_gemma/huggingface_api.py b/llm/data_gemma/huggingface_api.py
deleted file mode 100644
index 13dc3c28..00000000
--- a/llm/data_gemma/huggingface_api.py
+++ /dev/null
@@ -1,96 +0,0 @@
-# Copyright 2024 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""HF Pipeline API based LLM Interface.
-
-For example usage, see: https://huggingface.co/google/gemma-2-27b
-"""
-
-import logging
-import time
-from typing import Any
-
-from data_gemma import base
-
-MAX_NEW_TOKENS = 1024
-
-
-class HFPipeline(base.LLM):
- """HuggingFace Pipeline API."""
-
- def __init__(
- self,
- pipeline: Any,
- verbose: bool = True,
- ):
- self.pipeline = pipeline
- self.options = base.Options(verbose=verbose)
-
- def query(self, prompt: str) -> base.LLMCall:
- self.options.vlog(f'... calling HF Pipeline API "{prompt[:50].strip()}..."')
-
- start = time.time()
- outputs = self.pipeline(prompt, max_new_tokens=MAX_NEW_TOKENS)
- t = round(time.time() - start, 3)
-
- ans = ''
- err = ''
- if not outputs:
- err = 'Empty outputs from pipeline() call!'
- elif 'generated_text' not in outputs[0]:
- err = 'generated_text not found in outputs[0]!'
- else:
- ans = outputs[0]['generated_text']
-
- if err:
- logging.warning(err)
- print(f'WARNING: {err}')
-
- return base.LLMCall(prompt=prompt, response=ans, duration_secs=t, error=err)
-
-
-class HFBasic(base.LLM):
- """HuggingFace Model / Tokenizer API.
-
- Note: Model is assumed to be loaded on a GPU.
- """
-
- def __init__(
- self,
- model: Any,
- tokenizer: Any,
- verbose: bool = True,
- ):
- self.model = model
- self.tokenizer = tokenizer
- self.options = base.Options(verbose=verbose)
-
- def query(self, prompt: str) -> base.LLMCall:
- self.options.vlog(f'... calling HF Pipeline API "{prompt[:50].strip()}..."')
-
- start = time.time()
- input_ids = self.tokenizer(prompt, return_tensors='pt').to('cuda')
- outputs = self.model.generate(**input_ids, max_new_tokens=4096)
-
- ans = ''
- err = ''
- try:
- ans = self.tokenizer.decode(outputs[0])
- except Exception as e:
- err = str(e)
- logging.warning(err)
- print(f'WARNING: {err}')
-
- t = round(time.time() - start, 3)
-
- return base.LLMCall(prompt=prompt, response=ans, duration_secs=t, error=err)
diff --git a/llm/data_gemma/openai_api.py b/llm/data_gemma/openai_api.py
deleted file mode 100644
index a81a12d9..00000000
--- a/llm/data_gemma/openai_api.py
+++ /dev/null
@@ -1,93 +0,0 @@
-# Copyright 2024 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""OpenAI LLM Interface."""
-
-import json
-import logging
-import time
-from typing import Any
-
-import requests
-
-from data_gemma import base
-
-
-class OpenAI(base.LLM):
- """Open AI API."""
-
- def __init__(
- self,
- model: str,
- api_key: str,
- verbose: bool = True,
- session: requests.Session | None = None,
- ):
- self.key = api_key
- if not session:
- session = requests.Session()
- self.session: requests.Session = session
- self.options = base.Options(verbose=verbose)
- self.model = model
-
- def query(self, prompt: str) -> base.LLMCall:
- # set the params.
- req_data = {
- 'temperature': 0.1,
- 'model': self.model,
- 'messages': [{
- 'role': 'user',
- 'content': prompt,
- }],
- }
- # Make API request.
- req = json.dumps(req_data)
-
- start = time.time()
- self.options.vlog(
- f'... calling OpenAI {self.model} "{prompt[:50].strip()}..."'
- )
- resp = self._call_api(req)
- t = round(time.time() - start, 3)
- ans = ''
- err = ''
- if 'error' in resp:
- err = json.dumps(resp)
- logging.error('%s', err)
- print(err)
- elif (
- 'choices' in resp
- and resp['choices']
- and 'message' in resp['choices'][0]
- and 'content' in resp['choices'][0]['message']
- ):
- ans = resp['choices'][0]['message']['content']
- else:
- err = 'Got empty response'
- logging.warning(err)
- print(err)
-
- return base.LLMCall(prompt=prompt, response=ans, duration_secs=t, error=err)
-
- def _call_api(self, req_data: str) -> Any:
- header = {
- 'Content-Type': 'application/json',
- 'Authorization': f'Bearer {self.key}',
- }
- r = self.session.post(
- 'https://api.openai.com/v1/chat/completions',
- data=req_data,
- headers=header,
- )
- return r.json()
diff --git a/llm/data_gemma/prompts.py b/llm/data_gemma/prompts.py
deleted file mode 100644
index e045364f..00000000
--- a/llm/data_gemma/prompts.py
+++ /dev/null
@@ -1,398 +0,0 @@
-# Copyright 2024 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""List of all prompts used in Data Gemma."""
-
-# TODO: Unify this with RIG_IN_CONTEXT_PROMPT. No reason why this should be
-# different, and include the "question"
-RIO_IN_CONTEXT_PROMPT = """
-Your task is to annotate every statistic in the given text with a `__DC__`
-query that can retrieve the statistic. The query should be about metrics
-on topics like demographics, economy, education, health, and so on that are
-associated with geographical places (like USA, California, Miami, etc.).
-
-Concretely, every occurrence of a statistical value for a metric in a place
-should be replaced with `[__DC__("query") --> "stat"]`, where "query"
-must include a metric, a place name and optional date. And "stat" is the
-statistical value that originally occurred in the text. Do not annotate
-values that are dates ("founded in 1760") and ranks ("10th largest by area").
-
-The `__DC__()` calls MUST be in place of the statistical value in the text.
-And DO NOT modify sentences that have no statistical data.
-
-Below is an example of an INPUT and the corresponding annotated OUTPUT.
-
-INPUT:
-
-Question:- Tell me one statistic about California, San Francisco, Alabama and the US.
-Answer:-
-California is 1st as the nation's most populous state, with about 39 million people in 2020.
-In San Francisco, the diabetes rate is 9.2 cases per 10000 people.
-San Francisco and the surrounding San Francisco Bay Area are a global center of economic activity and the arts and sciences.
-In 1861, Alabama seceded from the United States to become part of the Confederate States of America.
-As of 2022, the United States receives approximately 81% of its energy from fossil fuel and the largest source of the country's energy came from petroleum (35.8%), followed by natural gas (33.4%) and renewable sources (13.3%).
-
-OUTPUT:
-
-Question:- Tell me one statistic about California, San Francisco, Alabama and the US.
-Answer:-
-California is 1st as the nation's most populous state, with about [__DC__("what was the population of california in 2020?") --> "39 million"] people.
-In San Francisco, the diabetes rate is [__DC__("what is the prevalence of diabetes in San Francisco?") --> "9.2 cases per 10000 people"].
-San Francisco and the surrounding San Francisco Bay Area are a global center of economic activity and the arts and sciences.
-In 1861, Alabama seceded from the United States to become part of the Confederate States of America.
-As of 2022, the United States receives approximately [__DC__("what percentage of energy in the United States comes from fossil fuels in 2022?") --> "81%"] of its energy from fossil fuel and the largest source of the country's energy came from petroleum ([__DC__("what percentage of energy in the United States comes from petroleum in 2022?") --> "35.8%"]), followed by natural gas ([__DC__("what percentage of energy in the United States comes from natural gas in 2022?") --> "33.4%"]) and renewable sources ([__DC__("what percentage of energy in the United States comes from renewable sources in 2022?") --> "13.3%"]).
-
-INPUT:
-
-Question:- {question}
-Answer:-
-{answer}
-
-OUTPUT:
-
-"""
-
-
-RIG_IN_CONTEXT_PROMPT = """
-Your task is to annotate every statistic in the given text with a `__DC__`
-query that can retrieve the statistic. The query should be about metrics
-on topics like demographics, economy, education, health, and so on that are
-associated with geographical places (like USA, California, Miami, etc.).
-
-Concretely, every occurrence of a statistical value for a metric in a place
-should be replaced with `[__DC__("query") --> "stat"]`, where "query"
-must include a metric, a place name and optional date. And "stat" is the
-statistical value that originally occurred in the text. Do not annotate
-values that are dates ("founded in 1760") and ranks ("10th largest by area").
-
-The `__DC__()` calls MUST be in place of the statistical value in the text.
-And DO NOT modify sentences that have no statistical data.
-
-Below is an example of an INPUT and the corresponding annotated OUTPUT.
-
-INPUT:
-
-California is 1st as the nation's most populous state, with about 39 million people in 2020.
-In San Francisco, the diabetes rate is 9.2 cases per 10000 people.
-San Francisco and the surrounding San Francisco Bay Area are a global center of economic activity and the arts and sciences.
-In 1861, Alabama seceded from the United States to become part of the Confederate States of America.
-As of 2022, the United States receives approximately 81% of its energy from fossil fuel and the largest source of the country's energy came from petroleum (35.8%), followed by natural gas (33.4%) and renewable sources (13.3%).
-
-OUTPUT:
-
-California is 1st as the nation's most populous state, with about [__DC__("what was the population of california in 2020?") --> "39 million"] people.
-In San Francisco, the diabetes rate is [__DC__("what is the prevalence of diabetes in San Francisco?") --> "9.2 cases per 10000 people"].
-San Francisco and the surrounding San Francisco Bay Area are a global center of economic activity and the arts and sciences.
-In 1861, Alabama seceded from the United States to become part of the Confederate States of America.
-As of 2022, the United States receives approximately [__DC__("what percentage of energy in the United States comes from fossil fuels in 2022?") --> "81%"] of its energy from fossil fuel and the largest source of the country's energy came from petroleum ([__DC__("what percentage of energy in the United States comes from petroleum in 2022?") --> "35.8%"]), followed by natural gas ([__DC__("what percentage of energy in the United States comes from natural gas in 2022?") --> "33.4%"]) and renewable sources ([__DC__("what percentage of energy in the United States comes from renewable sources in 2022?") --> "13.3%"]).
-
-INPUT:
-
-{text}
-
-OUTPUT:
-
-"""
-
-
-RAG_IN_CONTEXT_PROMPT = """
-Given a QUERY below, your task is to come up with a maximum of 25
-STATISTICAL QUESTIONS that help in answering QUERY.
-
-Here are the only forms of STATISTICAL QUESTIONS you can generate:
-
-1. "What is $METRIC in $PLACE?"
-2. "What is $METRIC in $PLACE $PLACE_TYPE?"
-3. "How has $METRIC changed over time in $PLACE $PLACE_TYPE?"
-
-where:
-- $METRIC should a publicly accessible metric on societal topics around
- demographics, economy, health, education, environment, etc. Examples are
- unemployment rate, life expectancy, etc.
-- $PLACE is the name of a place like California, World, Chennai, etc.
-- $PLACE_TYPE is an immediate child type within $PLACE, like counties, states,
- districts, etc.
-
-Your response should only include the questions, one per line without any
-numbering or bullet! If you cannot come up with statistical questions to ask,
-return an empty response.
-
-NOTE: Do not repeat questions. Limit the number of questions to 25.
-
-If QUERY asks about multiple concepts (e.g., income and diseases), make sure
-the questions cover all the concepts.
-
-[Start of Examples]
-
-QUERY: Which grades in the middle school have the lowest enrollment in Palo Alto?
-STATISTICAL QUESTIONS:
-What is the number of students enrolled in Grade 6 in Palo Alto schools?
-What is the number of students enrolled in Grade 7 in Palo Alto schools?
-What is the number of students enrolled in Grade 8 in Palo Alto schools?
-
-QUERY: Which industries have grown the most in California?
-STATISTICAL QUESTIONS:
-How have jobs in agriculture changed over time in California?
-How has GDP of agriculture sector changed over time in California?
-How have jobs in information and technology changed over time in California?
-How has GDP of information and technology sector changed over time in California?
-How have jobs in the government changed over time in California?
-How has GDP of the government sector changed over time in California?
-How have jobs in healthcare changed over time in California?
-How has GDP of healthcare sector changed over time in California?
-How have jobs in entertainment changed over time in California?
-How has GDP of entertainment sector changed over time in California?
-How have jobs in retail trade changed over time in California?
-How has GDP of retail trade sector changed over time in California?
-How have jobs in manufacturing changed over time in California?
-How has GDP of manufacturing sector changed over time in California?
-How have jobs in education services changed over time in California?
-How has GDP of education services sector changed over time in California?
-
-QUERY: Which state in the US has the most asian population?
-STATISTICAL QUESTIONS:
-What is the number of asian people in US states?
-
-QUERY: Do specific health conditions affect the richer California counties?
-STATISTICAL QUESTIONS:
-What is the median income among California counties?
-What is the median house price among California counties?
-What is the prevalence of obesity in California counties?
-What is the prevalence of diabetes in California counties?
-What is the prevalence of heart disease in California counties?
-What is the prevalence of arthritis in California counties?
-What is the prevalence of asthma in California counties?
-What is the prevalence of chronic kidney disease in California counties?
-What is the prevalence of chronic obstructive pulmonary disease in California counties?
-What is the prevalence of coronary heart disease in California counties?
-What is the prevalence of high blood pressure in California counties?
-What is the prevalence of high cholesterol in California counties?
-What is the prevalence of stroke in California counties?
-What is the prevalence of poor mental health in California counties?
-What is the prevalence of poor physical health in California counties?
-
-
-[End of Examples]
-
-QUERY: {sentence}
-STATISTICAL QUESTIONS:
-"""
-
-
-RAG_IN_CONTEXT_PROMPT_WITH_VARS = """
-Given a 'Query' below, your task is to come up with a maximum of 25
-'Statistical Questions' that relate to 'Query'.
-
-Here are the only forms of 'Statistical Questions' you can generate:
-
-1. What is $METRIC in $PLACE?
-2. What is $METRIC in $PLACE $PLACE_TYPE?
-3. How has $METRIC changed over time in $PLACE $PLACE_TYPE?
-
-Where:
-- $METRIC should only be from the 'Metrics List' given below.
-- $PLACE is the name of a place like California, World, Chennai, etc.
-- $PLACE_TYPE is first-level child type within $PLACE, like counties or
- districts if $PLACE is a state, states if $PLACE is a country, etc.
-
-Your response should only include the questions, one per line, without any
-numbering or bullets! If you cannot come up with 'Statistical Questions' only
-using the 'Metrics List' below, return an empty response.
-
-NOTE: Do not repeat questions. Limit the number of questions to 25 and
-order the questions from most relevant to least relevant.
-
-If "Query" asks about multiple concepts (e.g., income and diseases), make sure
-the questions cover all the concepts.
-
-[Start of Examples]
-
-Query: Tell me about life expectancy.
-Statistical Questions:
-What is the people life expectancy in the world?
-How has people life expectancy changed over time in the world countries?
-
-Query: Which state in the US has the most asian population?
-Statistical Questions:
-What is the number of asian people in US states?
-How has the number of asian people changed over time in US states?
-
-Query: Which grades in the middle school have the lowest enrollment in Palo Alto?
-Statistical Questions:
-What is the number of students enrolled in Grade 6 in Palo Alto schools?
-What is the number of students enrolled in Grade 7 in Palo Alto schools?
-What is the number of students enrolled in Grade 8 in Palo Alto schools?
-
-QUERY: Do specific health conditions affect the richer California counties?
-STATISTICAL QUESTIONS:
-What is the median income among California counties?
-What is the median house price among California counties?
-What is the prevalence of obesity in California counties?
-What is the prevalence of diabetes in California counties?
-What is the prevalence of heart disease in California counties?
-What is the prevalence of arthritis in California counties?
-What is the prevalence of asthma in California counties?
-What is the prevalence of chronic kidney disease in California counties?
-What is the prevalence of chronic obstructive pulmonary disease in California counties?
-What is the prevalence of coronary heart disease in California counties?
-What is the prevalence of high blood pressure in California counties?
-What is the prevalence of high cholesterol in California counties?
-What is the prevalence of stroke in California counties?
-What is the prevalence of poor mental health in California counties?
-What is the prevalence of poor physical health in California counties?
-
-[End of Examples]
-
-
-[Start of Metrics List]
-
-Here is a list of possible METRIC values:
-
-```
-{metrics_list}
-```
-
-[End of Metrics List]
-
-
-Query: {sentence}
-Statistical Questions:
-"""
-
-
-RAG_FINE_TUNED_PROMPT = """"
-Your role is that of a Question Generator. Given Query below, come up with a
-maximum of 25 Statistical Questions that help in answering Query.
-
-These are the only forms of Statistical Questions you can generate:
-1. What is $METRIC in $PLACE?
-2. What is $METRIC in $PLACE $PLACE_TYPE?
-3. How has $METRIC changed over time in $PLACE $PLACE_TYPE?
-
-where,
-- $METRIC should a metric on societal topics like demographics, economy, health,
- education, environment, etc. Examples are unemployment rate and
- life expectancy.
-- $PLACE is the name of a place like California, World, Chennai, etc.
-- $PLACE_TYPE is an immediate child type within $PLACE, like counties, states,
- districts, etc.
-
-Your response should only have questions, one per line, without any numbering
-or bullet.
-
-If you cannot come up with Statistical Questions to ask for a Query, return an
-empty response.
-
-Query: {sentence}
-Statistical Questions:
-"""
-
-
-RAG_FINAL_ANSWER_PROMPT = """
-Using statistics from the tables below, respond to the query: "{sentence}"
-
-In your response, when using statistics from a table, please cite the table
-by its ID, enclosed in square brackets. For example, for one table reference
-`[Table 1]`, and for multiple tables `[Table 1], [Table 2] and [Table 3]`.
-
-If necessary to answer the query, perform simple calculations on the statistics,
-like adding or subtracting statistics, computing growth rates from statistics
-over time, etc.
-
-If you are unable to answer the query based on the provided tables, start your
-response with `[NO ANSWER]` as the first line.
-
-```
-{table_str}
-```
-
-So now, using statistics from the tables above, respond to the query: "{sentence}"
-"""
-
-
-DC_QA_VALIDATION = """
-You will be provided with a list of up to 20 question-answer pairs, each
-identified by an ID like [[QA1]]. You must return each ID whose answer is
-relevant to its question, one per line. If none of the answers are relevant,
-return `[[EMPTY]]`.
-
-Here is an example INPUT and OUTPUT:
-
-## INPUT ##
-[[QA1]]:
- Question: "What is the average education spending per pupil in New York?"
- Answer: "% Govt Expenditure on Education in United States"
-[[QA2]]
- Question: "What is the Gini coefficient in Chile?"
- Answer: "Gini Index of Economic Activity of a Population in Chile"
-[[QA3]]
- Question: "How many people work in health care jobs in Nevada?"
- Answer: "Population of Health Care Workers in Nevada"
-
-## OUTPUT ##
-[[QA2]]
-[[QA3]]
-
-
-## INPUT ##
-{input}
-
-## OUTPUT ##
-"""
-
-
-LLM_JUDGE_PROMPT = """
-[System]
-
-Please act as an impartial judge and evaluate the quality of the response
-provided by an AI Assistant that annotates statistical numbers with questions.
-
-In the text, you will find patterns of the form: [__DC__("QUERY") --> "ANS"].
-Where, `ANS` is a statistical value, and `QUERY` is a query that can be answered
-with `ANS`.
-
-Your evaluation should consider whether the `__DC__` annotations follow these
-constraints:
-
-(C1) `QUERY` can refer to a very wide variety of measures related to demographics,
- economy, education, health, etc. However, it should accurately describe the
- statistic involved. `QUERY` must include a place name of a city, state,
- country, continent, etc, or words that represent the world (like global).
-
-(C2) `ANS` must not be empty or have the the word "stat". `ANS` must have a
- numeric value, may include percentages, and may additionally have
- non-numeric letters for units, currency symbols, etc.
-
-Provide a classification of the answer quality like "[[- current node is source
-- current node is target
-No nodes to explore
: - null - } -| File Name | -Line Num | -Line | -Error Message | -
|---|---|---|---|
| {errObj['file']} | -{msg[0]} | -{msg[1]} | -{msg[2]} | -
{facet}: {facets[facet]}
: - null - ); - })} - {group.map((seriesObj) => - this.plotSeriesObj(seriesObj, locationMapping))} -- this.props.onNodeClick( - (nodeTarget.localId || nodeTarget.dcid) as string, - )}> - {nodeTarget.getRef()} -
- -{(target as string)}
); - } - - /** - * Returns an html element containing the styled provenance of the triple. The - * provenance if of one of the following formats: - * dc/- this.props.onNodeClick(prov)}>{prov}
); - } - - if (!prov.startsWith('https')) { - // local file(s) as provenance - return ({prov.replace('&', ', ')}
); - } - - if (!prov.includes('&')) { - // single mcf file as provenance - return (- utils.openFile(prov)}>{prov.split('/').pop()}
); - } - - // provenance is one tmcf and one csv - - const fileNames: string[] = []; - const provNames = []; - for (const fileName of prov.split('&')) { - fileNames.push(fileName); - provNames.push(fileName.split('/').pop()); - } - - return ( -- utils.openFile(fileNames[0])}>{provNames[0]}
-,{NON_BREAKING_SPACE}
-- utils.openFile(fileNames[1])}>{provNames[1]}
-