GML-1996 separate createDocIngest and runDocIngest for BDA

README.md

@@ -121,7 +121,7 @@ This line can be changed to support different logging levels.
 > Note: `graphrag` container will be down if TigerGraph service is not ready. Log into the `tigergraph` container, bring up tigergraph services and rerun `docker compose up -d` should resolve the issue.
 
 ## Data Ingestion
-For data ingestion, please follow the ![GraphRAG Demo Notebook](./docs/notebooks/GraphRAGDemo.ipynb)
+For data ingestion, please follow the [GraphRAG Demo Notebook](./docs/notebooks/GraphRAGDemo.ipynb)
 
 ## Detailed Configurations
 
@@ -449,13 +449,15 @@ If you would like to enable openCypher query generation in InquiryAI, you can se
 
 GraphRAG is friendly to both technical and non-technical users. There is a graphical chat interface as well as API access to GraphRAG. Function-wise, GraphRAG can answer your questions by calling existing queries in the database (InquiryAI), build a knowledge graph from your documents (SupportAI), and answer knowledge questions based on your documents (SupportAI).
 
+Please visit [GraphRAG Tutorial](https://github.com/tigergraph/ecosys/blob/master/tutorials/GraphRAG.md) for a detailed demo.
+
 ## Customization and Extensibility
 TigerGraph GraphRAG is designed to be easily extensible. The service can be configured to use different LLM providers, different graph schemas, and different LangChain tools. The service can also be extended to use different embedding services, different LLM generation services, and different LangChain tools. For more information on how to extend the service, see the [Developer Guide](./docs/DeveloperGuide.md).
 
-## Testing
+### Test Code Changes
 A family of tests are included under the `tests` directory. If you would like to add more tests please refer to the [guide here](./docs/DeveloperGuide.md#adding-a-new-test-suite). A shell script `run_tests.sh` is also included in the folder which is the driver for running the tests. The easiest way to use this script is to execute it in the Docker Container for testing.
 
-### Testing with Pytest
+#### Testing with Pytest
 You can run testing for each service by going to the top level of the service's directory and running `python -m pytest`
 
 e.g. (from the top level)
@@ -465,7 +467,7 @@ python -m pytest
 cd ..
 ```
 
-### Test in Docker Container
+#### Test in Docker Container
 
 First, make sure that all your LLM service provider configuration files are working properly. The configs will be mounted for the container to access. Also make sure that all the dependencies such as database are ready. If not, you can run the included docker compose file to create those services.
 ```sh

common/gsql/supportai/SupportAI_InitialLoadJSON.gsql

@@ -1,7 +1,7 @@
 CREATE LOADING JOB load_documents_content_json_@uuid@ {
     DEFINE FILENAME DocumentContent;
     LOAD DocumentContent TO VERTEX Document VALUES(gsql_lower($"doc_id"), gsql_current_time_epoch(0), _, _) USING JSON_FILE="true";
-    LOAD DocumentContent TO VERTEX Content VALUES(gsql_lower($"doc_id"), "doc_type", $"content", gsql_current_time_epoch(0)) USING JSON_FILE="true";
+    LOAD DocumentContent TO VERTEX Content VALUES(gsql_lower($"doc_id"), $"doc_type", $"content", gsql_current_time_epoch(0)) USING JSON_FILE="true";
 
     LOAD DocumentContent TO EDGE HAS_CONTENT VALUES(gsql_lower($"doc_id") Document, gsql_lower($"doc_id") Content) USING JSON_FILE="true";
 }

common/py_schemas/schemas.py

@@ -125,7 +125,7 @@ class CreateIngestConfig(BaseModel):
 
 class LoadingInfo(BaseModel):
     load_job_id: str
-    data_source_id: str
+    data_source_id: Union[str, Dict]
     file_path: str
 
 

graphrag/app/routers/supportai.py

@@ -16,7 +16,7 @@
 import logging
 from typing import Annotated
 
-from fastapi import APIRouter, BackgroundTasks, Depends, Request, Response, status
+from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Request, Response, status
 from fastapi.security.http import HTTPBase
 from supportai import supportai
 from supportai.concept_management.create_concepts import (
@@ -105,44 +105,8 @@ def ingest(
     credentials: Annotated[HTTPBase, Depends(security)],
 ):
     conn = conn.state.conn
-    if loader_info.file_path is None:
-        raise Exception("File path not provided")
-    if loader_info.load_job_id is None:
-        raise Exception("Load job id not provided")
-    if loader_info.data_source_id is None:
-        raise Exception("Data source id not provided")
 
-    try:
-        res = conn.gsql(
-            'USE GRAPH {}\nRUN LOADING JOB -noprint {} USING {}="{}"'.format(
-                graphname,
-                loader_info.load_job_id,
-                "DocumentContent",
-                "$" + loader_info.data_source_id + ":" + loader_info.file_path,
-            )
-        )
-    except Exception as e:
-        if (
-            "Running the following loading job in background with '-noprint' option:"
-            in str(e)
-        ):
-            res = str(e)
-        else:
-            raise e
-
-    log_section = res.split(
-        "Running the following loading job in background with '-noprint' option:"
-    )[1]
-    # Try to extract 'Job name' or 'Log directory'
-    if "Job name: " in log_section:
-        log_location = log_section.split("Job name: ")[1].split("\n")[0]
-    else:
-        log_location = log_section.split("Log directory: ")[1].split("\n")[0]
-    return {
-        "job_name": loader_info.load_job_id,
-        "job_id": log_section.split("Jobid: ")[1].split("\n")[0],
-        "log_location": log_location,
-    }
+    return supportai.ingest(graphname, loader_info, conn)
 
 
 @router.post("/{graphname}/graphrag/search")