FDA AI Search: Making FDA-Authorized AI Devices Searchable

Code accompanying:

Arun Kavishwar and William Lotter. FDA AI Search: Making FDA-Authorized AI Devices Searchable.
Machine Learning for Health (ML4H) Symposium, 2025.

FDA AI Search is a website that enables semantic querying of FDA-authorized AI-enabled devices. The backend includes an embedding-based retrieval system, where LLM-extracted features from authorization summaries are compared to user queries to find relevant matches. The website is hosted at fda-ai-search.com.

Project Workflow

The project follows these main steps:

1. Data Scraping and Preparation: Collects initial device data from the FDA and merges it with an external taxonomy.
2. Content Processing with Gemini: Extracts text from linked PDF documents and uses the Gemini API to generate summaries, keywords, and relevant questions.
3. Embedding Creation and Search: Generates vector embeddings for the processed text data (summaries, keywords, questions) to enable semantic search capabilities.

To update:

Run update_data_and_embeddings.sh, with a GEMINI_API_KEY in the environment.

File Structure

.
├── fda_scraper.py             # Script for scraping FDA data and merging with taxonomy
├── view_fda_data.py           # Script to view and inspect the scraped CSV data
├── process_pdf_summary.py     # Script for processing PDF content with Gemini API
├── create_embeddings.py       # Script for creating and searching text embeddings
├── perform_search.py          # Script dedicated to performing hybrid search (used by frontend and optimization)
├── test_search_performance.py # Script for running a basic test suite against perform_search.py
├── optimize_search_weights.py # Script for optimizing search weights using Optuna
├── fda_ai_records.csv         # Main CSV file storing all processed data
├── data/
│   ├── AI_devices_taxonomy.xlsx # Excel file with device taxonomy information
│   └── ai-ml-enabled-devices-csv.csv # (Potentially another source, or older version)
├── embedding_data/            # Directory for storing FAISS indexes and pickled text data
│   ├── summary_index.faiss
│   ├── summary_texts.pkl
│   ├── keywords_index.faiss
│   ├── keywords_texts.pkl
│   ├── questions_index.faiss
│   ├── questions_texts.pkl
│   ├── concepts_index.faiss
│   ├── concepts_texts.pkl
│   ├── thesis_index.faiss
│   ├── thesis_texts.pkl
│   ├── search_boost_index.faiss
│   ├── search_boost_texts.pkl
│   ├── query_match_1_index.faiss
│   ├── query_match_1_texts.pkl
│   ├── query_match_2_index.faiss
│   ├── query_match_2_texts.pkl
│   ├── query_match_3_index.faiss
│   ├── query_match_3_texts.pkl
│   └── submission_numbers.pkl
├── fda-search-app/            # Next.js frontend application
│   ├── src/
│   │   ├── app/
│   │   │   ├── api/search/route.ts # Backend API route for search
│   │   │   ├── globals.css
│   │   │   ├── layout.tsx          # Root layout for the Next.js app
│   │   │   └── page.tsx            # Main page component for search UI
│   ├── next.config.mjs
│   ├── package.json
│   ├── tailwind.config.ts
│   └── tsconfig.json
└── README.md                  # This documentation file

Detailed Steps and Scripts

1. Data Scraping and Viewing

fda_scraper.py

This script is responsible for the initial data collection and preparation.

Algorithm:

1. Scrape FDA Main List:
   • Fetches the list of AI/ML-enabled medical devices from the FDA's official webpage (https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices).
   • Parses the HTML table to extract device details like 'date_of_final_decision', 'submission_number', 'device_model', 'company', 'panel_lead', and 'primary_product_code'.
2. Scrape PDF Links:
   • For each device, constructs a URL to the FDA device page using its 'submission_number'.
   • Visits this page and attempts to find a link to a summary PDF document.
   • Stores the found PDF link (or "Not Found") in the 'summary_pdf_link' column.
3. Load Taxonomy Data:
   • Reads taxonomy information from an Excel file (data/AI_devices_taxonomy.xlsx).
   • Maps Excel column names to standardized names (e.g., 'lead_panel', 'data_type', 'clinical_function', 'ai_function', 'ai_function_subclass').
4. Merge Data:
   • Merges the scraped FDA data with the taxonomy data based on the 'submission_number'.
5. Output:
   • Saves the combined and processed data into fda_ai_records.csv.
   • If fda_ai_records.csv already exists and seems valid, it can load this existing data to avoid re-scraping the main FDA list, but will still attempt to get PDF links if missing.

Usage:

python fda_scraper.py

view_fda_data.py

This utility script helps in inspecting the contents of fda_ai_records.csv.

Functionality:

• Loads the fda_ai_records.csv into a pandas DataFrame.
• Provides functions to:
  • Display the first N rows (display_head).
  • Show DataFrame information, including data types and non-null counts (display_info).
  • Provide descriptive statistics (display_description).
  • Retrieve and display a specific record by its 'submission_number' (get_record_by_submission_id).
  • List all column names (list_columns).
  • Count total records and unique submission numbers (using 'submission_number').

Usage:

python view_fda_data.py

Or for interactive use:

python -i view_fda_data.py

2. Processing with Gemini

process_pdf_summary.py

This script processes the PDF documents linked in fda_ai_records.csv using the Google Gemini API.

Algorithm:

1. Load Data: Reads fda_ai_records.csv.
2. PDF Text Extraction:
   • For each record with a valid 'summary_pdf_link':
     • Downloads the PDF content from the URL.
     • Uses PyPDF2 to extract text from the PDF. Handles potential encryption and empty PDFs.
3. Gemini API Interaction:
   • If text is successfully extracted:
     • Configures the Gemini API using an environment variable GEMINI_API_KEY.
     • Uses the gemini-2.5-pro-preview-05-06 model.
     • Sends the extracted PDF text to the Gemini API with a prompt requesting:
       • A thorough two-paragraph summary.
       • Exactly 10 salient keywords.
       • Exactly 5 insightful questions a clinician or scientist might ask about the document.
       • A list of 5 concepts.
       • A 2-sentence thesis statement.
     • The API is configured to return a JSON response structured according to a Pydantic model (DocumentAnalysis).
     • Includes retry logic and uses dirtyjson for robust parsing of potentially malformed JSON responses.
4. Update CSV:
   • Updates the DataFrame (and subsequently fda_ai_records.csv) with:
     • summary: The generated summary.
     • summary_keywords: Comma-separated list of keywords.
     • generated_questions: Semicolon-separated list of questions.
     • concepts: Comma-separated list of concepts.
     • thesis: The generated thesis statement.
     • gemini_cost_summary: The calculated cost for the API call.
   • Handles errors during PDF extraction or API calls by logging them into the respective columns.

Setup:

• Ensure the GEMINI_API_KEY environment variable is set.
• Install necessary Python packages: google-generativeai, requests, PyPDF2, pydantic, pandas, dirtyjson.

Usage:

The script accepts a command-line argument -n or --num_records to specify how many records to process.

python process_pdf_summary.py -n 5  # Process up to 5 records

generate_query_match_column.py

This script uses the Google Gemini API to generate three distinct clinician search queries for each record in fda_ai_records.csv. These queries are intended to represent how a clinician might search for a specific device.

Algorithm:

1. Load Data: Reads fda_ai_records.csv.
2. Check for Existing Queries: Iterates through the DataFrame, processing only records where the query_match_1 column is empty, NaN, or explicitly marked "Error".
3. Gemini API Interaction:
   • Configures the Gemini API using the GEMINI_API_KEY environment variable.
   • Uses the gemini-2.5-flash model.
   • Sends a prompt to Gemini with the record's data (excluding the 'summary' column) requesting three distinct clinician search queries.
   • The API is configured to return a JSON response structured according to a Pydantic model (QueryMatch) with keys query_match_1, query_match_2, and query_match_3.
   • Includes retry logic and uses dirtyjson for robust parsing.
4. Update CSV:
   • Updates the DataFrame (and subsequently fda_ai_records.csv) with the generated queries in the query_match_1, query_match_2, and query_match_3 columns.
   • Updates the gemini_cost_summary with the cost of the API call.
   • If query generation fails, the columns are marked "Error".

Setup:

• Ensure the GEMINI_API_KEY environment variable is set.
• Install necessary Python packages: google-generativeai, pandas, pydantic, dirtyjson.

Usage:

The script accepts a command-line argument -n or --num_records to specify how many records to process.

python generate_query_match_column.py -n 10 # Process up to 10 records

3. Creating Embeddings

create_embeddings.py

This script generates vector embeddings for textual data in fda_ai_records.csv and allows for semantic search.

Algorithm:

1. Load Data: Reads fda_ai_records.csv.

2. Load Model: Initializes a Sentence Transformer model (abhinand/MedEmbed-base-v0.1, from Abhinand Balachandran MedEmbed: Medical-Focused Embedding Models).

3. Embedding Generation (if --generate flag is used):

   • Extracts text from:
     • summary column.
     • summary_keywords column.
     • generated_questions column.
     • concepts column.
     • thesis column.
     • search_boost_text column.
     • query_match_1, query_match_2, and query_match_3 columns.
   • For each set of texts:
     • Encodes the texts into numerical embeddings using the loaded model.
     • Normalizes the embeddings (L2 normalization).
     • Creates a FAISS index (IndexFlatIP for cosine similarity).
     • Adds the embeddings to the index.
     • Saves the FAISS index to a .faiss file (e.g., embedding_data/summary_index.faiss).
     • Saves the corresponding original texts and submission numbers to .pkl files (e.g., embedding_data/summary_texts.pkl, embedding_data/submission_numbers.pkl) using pickle.

4. Hybrid Search (if --query argument is provided):

   • Loads the pre-generated FAISS indexes and pickled text data for summaries, keywords, questions, concepts, thesis, search_boost_text, and query_match columns.
   • Encodes the user's query string using the same Sentence Transformer model.
   • Searches each index for the query embedding.
   • Calculates a weighted hybrid similarity score for each document based on scores from the different fields (summary, keywords, questions, concepts, thesis, search_boost_text, query_match_1, query_match_2, query_match_3).
   • Returns the top K results, sorted by the hybrid similarity score, along with the submission number and the text of the summary.

Setup:

• Install necessary Python packages: sentence-transformers, faiss-cpu (or faiss-gpu if you have a compatible GPU), pandas, numpy, pickle.

Usage:

• To generate embeddings:

  python create_embeddings.py --generate

• To perform a hybrid search (after generating embeddings):

  python create_embeddings.py --query "your search query here" --top_k 5

  This script can also accept a --weights_json argument to use a custom set of weights for the search_hybrid function, overriding its internal defaults. Example:

  python create_embeddings.py --query "angiography" --weights_json "{\"summary\": 0.1, \"keywords\": 0.2, \"questions\": 0.1, \"concepts\": 0.3, \"thesis\": 0.3, \"search_boost\": 0.0}"

4. Performing Search (Standalone)

perform_search.py

This script is dedicated to performing the hybrid semantic search. It's used by the Next.js frontend and the weight optimization script. It loads all necessary models, FAISS indexes, and text data to conduct searches.

Functionality:

• Loads the SentenceTransformer model (abhinand/MedEmbed-base-v0.1).
• Loads all pre-generated FAISS indexes and pickled text data from the embedding_data/ directory.
• Accepts a --query (required) and --top_k (optional, default 10) command-line argument.
• Accepts an optional --weights_json argument. If provided (as a JSON string), these weights will be used for the hybrid search scoring. Otherwise, it uses its internal default weights.
• Performs the hybrid search by:
  • Embedding the query.
  • Searching each FAISS index.
  • Calculating a weighted hybrid score for documents found across different text fields.
• Retrieves additional details (Device Name, Applicant, Decision Date, PDF Link, Thesis, Keywords, Concepts) for the top K results from fda_ai_records.csv.
• Outputs the search results as a JSON string to standard output. If an error occurs, it prints a JSON error object to standard error.

Usage:

python perform_search.py --query "cardiac mri" --top_k 5

To use custom weights:

python perform_search.py --query "cardiac mri" --weights_json "{\"summary\": 0.1, \"keywords\": 0.2, \"questions\": 0.1, \"concepts\": 0.3, \"thesis\": 0.3, \"search_boost\": 0.0}"

5. Testing and Optimizing Search Performance

test_search_performance.py

This script provides a basic test suite to evaluate the performance of the perform_search.py script.

Functionality:

Usage:

python test_search_performance.py

You can modify num_records_to_test within the script's if __name__ == "__main__": block to change the number of test cases.

optimize_search_weights.py

This script employs the Optuna hyperparameter optimization framework to systematically find the most effective weights for the different text fields (summary, keywords, questions, concepts, thesis, search_boost) used in the hybrid search performed by perform_search.py. The goal is to maximize the relevance of search results.

Algorithm & Rigorous Rate Evaluation Process:

1. Test Case Generation:

   • The script begins by selecting a configurable number of records (default: NUM_TEST_CASES = 35) from fda_ai_records.csv.
   • For each selected record, it generates a clinically relevant search query. This is done by prompting a local Ollama instance running the gemma3:4b model (user-specified, previously llama3.2:latest). The prompt combines the 'thesis' and 'key concepts' from the FDA record to ask the model for a concise search term a clinician might use.
   • If the Ollama CLI call fails (e.g., command not found, model unavailable, timeout), the script falls back to a simpler heuristic query generation method (based on snippets of thesis, concepts, or keywords).
   • The submission_number of the record is designated as the expected top result for the generated query.

2. Optuna Study Setup:

   • An Optuna study is created or loaded if it already exists. The study progress and results are saved in an SQLite database (e.g., optimization_results/fda-search-weight-opt-TIMESTAMP.db). This allows for resumption of long optimization runs.
   • The optimization direction is set to 'maximize' the success rate.

3. Objective Function (Executed for Each Optuna Trial):

   • Weight Suggestion: Optuna suggests a new set of weights for each of the six text fields (summary, keywords, questions, concepts, thesis, search_boost).
   • Normalization: These raw weights are normalized so that they sum to 1.0, ensuring a consistent basis for comparison.
   • Performance Evaluation:
     • The normalized weights are passed (as a JSON string via the --weights_json argument) to the perform_search.py script for each of the generated test cases.
     • perform_search.py executes a search using the test query and the provided weights.
     • A test case is considered "passed" if its expected_submission_number appears within the top TOP_N_RESULTS_FOR_PASS (default: 3) results returned by perform_search.py.
   • Success Rate Calculation: The objective function returns the overall success rate for the current set of weights, calculated as (number of passed test cases) / (total number of test cases).

4. Optimization Loop:

   • Optuna runs a configurable number of trials (N_OPTUNA_TRIALS, default: 120), iteratively adjusting weights based on past results to find combinations that yield higher success rates.

5. Logging and Results:

   • Comprehensive Logging: All significant events, including initial setup, test case generation details (query source: Ollama or fallback), each trial's weights, detailed results for every test case (query, expected submission, returned submissions with similarity scores, pass/fail status), and final outcomes, are logged to both the console and a timestamped .log file (e.g., optimization_results/optimization_run_YYYYMMDD-HHMMSS.log).
   • Final Output: After all trials are complete, the script reports:
     • The best trial number and its achieved success rate.
     • The raw and normalized weights from the best trial.
     • A Python dictionary representation of the recommended normalized weights, which can be manually used to update the default weights in create_embeddings.py or perform_search.py.
   • The script does not automatically modify any other project files with the optimized weights.

Prerequisites:

• Embeddings Must Be Generated: Ensure python create_embeddings.py --generate has been run to create all necessary FAISS indexes and .pkl files in embedding_data/.
• Install Optuna: pip install optuna
• Ollama and Model: Ollama CLI must be installed and in the system PATH. The gemma3:4b model should be pulled (ollama pull gemma3:4b).

Usage:

python optimize_search_weights.py

The script can be time-consuming due to the large number of search executions. Configuration parameters like NUM_TEST_CASES, N_OPTUNA_TRIALS, and TOP_N_RESULTS_FOR_PASS can be adjusted at the top of the script. The optimization_results/ directory will contain the Optuna database and the detailed log file for the run.

Next.js Web Application (fda-search-app/)

This project includes a Next.js web application that provides a user interface to search the FDA AI/ML device data processed by the Python scripts.

Overview

The frontend allows users to type search queries, view results in a table, and access PDF summaries where available. It aims for a clean, responsive interface.

Key Features

• Interactive Search: Users can type queries into a search bar.
• Debounced Queries: API calls are debounced to avoid excessive requests while typing.
• Results Table: Displays:
  • Device Name
  • Applicant
  • Submission Number (links to PDF if available)
  • Decision Date
  • Similarity Score
  • Thesis Statement (full text, wraps)
  • Keywords & Concepts (full text, wraps, separated by a line)
• Loading Indicator: Shows a loading animation while search results are being fetched.
• Styling: Uses Tailwind CSS for styling and the Merriweather serif font for a professional look.

Setup and Running

1. Navigate to the app directory:

   cd fda-search-app

2. Install dependencies:

   npm install

3. Run the development server:

   npm run dev

   The application will typically be available at http://localhost:3000.

How it Works

1. The main UI is in fda-search-app/src/app/page.tsx. When a user types a query, it triggers a fetch request to the backend API route.
2. The API route fda-search-app/src/app/api/search/route.ts receives the search query.
3. This API route uses Node.js child_process.spawn to execute the perform_search.py script (located in the project root: ../perform_search.py). The query is passed as a command-line argument to the Python script.
4. perform_search.py loads the SentenceTransformer model, the pre-generated FAISS indexes, and the pickled text data from the embedding_data/ directory. It performs the hybrid semantic search based on the query.
5. The Python script prints its results as a JSON string to its standard output.
6. The Next.js API route captures this JSON output and sends it back to the frontend component.
7. The frontend component (page.tsx) then updates its state with the received results, re-rendering the table.

Key Frontend Files

• fda-search-app/src/app/page.tsx: The main React component for the search page, handling user input, API calls, and rendering results.
• fda-search-app/src/app/layout.tsx: The root layout for the Next.js application. It sets up global styles and imports the Merriweather font.
• fda-search-app/src/app/globals.css: Contains global CSS styles, including Tailwind CSS base imports and custom styles.
• fda-search-app/src/app/api/search/route.ts: The backend API endpoint that bridges the Next.js app and the Python search script.
• fda-search-app/tailwind.config.ts: Configuration file for Tailwind CSS.

Data Filtering Note

The create_embeddings.py script filters the data from fda_ai_records.csv before creating embeddings. Specifically, it excludes records where the 'thesis' column is empty, "ERROR", or "none". This means the search indexes only contain data from records with valid thesis statements.

Speeding Up the Next.js Search Application

The current architecture where the Next.js API route (fda-search-app/src/app/api/search/route.ts) calls perform_search.py via child_process.spawn for each query is functional but has performance limitations. The primary bottleneck is that the Python script needs to:

1. Initialize the Python interpreter.
2. Load the SentenceTransformer model (which can be several hundred MB).
3. Load FAISS indexes and associated pickled text data from disk. This entire setup process occurs on every search request, leading to noticeable loading times (e.g., 4-8 seconds or more depending on the system).

Here are strategies to significantly improve search speed:

1. Persistent Python Backend (Highly Recommended for Best Performance)

• Concept: Run the Python search logic as a persistent web server (e.g., using Flask or FastAPI) instead of a script that starts up on each request.
• Pros:
  • The ML model, FAISS indexes, and data are loaded into memory once when the Python server starts.
  • Subsequent search API calls to this server are very fast (typically milliseconds), as they only involve query embedding and the FAISS search itself.
• Cons:
  • Requires managing a separate Python web server process alongside the Next.js application.
• Deployment (e.g., with Vercel for Next.js):
  • The Next.js app is deployed to Vercel.
  • The Python Flask/FastAPI application is deployed to a platform suitable for persistent applications (e.g., Google Cloud Run ($30/mo?), AWS App Runner ($15/mo), Render ($7/mo)).
  • The Next.js API route (/api/search) would then make an HTTP request to the deployed Python API endpoint.

2. Frontend Strategies (Improving Perceived Speed)

• Concept: Provide instant feedback to the user while the full semantic search runs in the background.
• Method:
  1. Client-Side Keyword Filter (Instant): On user input, immediately filter a pre-loaded subset of data (e.g., device names, keywords from fda_ai_records.csv) directly in the browser using simple string matching.
  2. Asynchronous Semantic Search: Simultaneously, send the query to the backend for the full (potentially slower) semantic search.
  3. UI Update: When semantic results arrive, update/re-rank the displayed list.
• Pros: Excellent perceived speed for initial results.
• Cons: Initial results are not semantically ranked. The "true" semantic results still take time to load if the backend isn't optimized (see option 1).

3. Optimizing the Python Script (Minor Gains)

• Smaller Model: Using a smaller, faster SentenceTransformer model could reduce load times, but may trade off search accuracy.
• Caching: OS-level file caching might offer some minor speedups on repeated local script runs, but the primary cost is model and library initialization within the Python process for each spawn call.

Why Not Rewrite Search in JavaScript for Vercel Serverless Functions?

• Model Execution: Efficiently running large NLP models (like SentenceTransformers) in a Node.js serverless environment is challenging due to cold starts and resource limits. The model still needs to be loaded on each (cold) invocation.
• FAISS Availability: A mature, performant JavaScript equivalent to FAISS (for fast vector similarity search on pre-built indexes) is not readily available. This would require a different, likely slower, vector search strategy.
• Overall: This approach is unlikely to be faster for the current semantic search setup and would involve significant reimplementation with potentially less performant tools.

For significant improvements in actual search speed, the Persistent Python Backend (Option 1) is the most robust approach. Combining it with Frontend Strategies (Option 2) can offer the best overall user experience.

Dependencies

This project relies on several Python libraries. You can typically install them using pip:

pip install pandas requests beautifulsoup4 openpyxl google-generativeai PyPDF2 pydantic dirtyjson sentence-transformers faiss-cpu numpy optuna

(Note: faiss-gpu can be used instead of faiss-cpu if you have a CUDA-enabled GPU and the necessary drivers.)

Ensure you also have a GEMINI_API_KEY environment variable set for the process_pdf_summary.py script.

GCR Endpoint:

https://fda-ai-search-224066961651.us-east1.run.app

Evaluation Suite

This project includes a robust evaluation suite located in the py_src/test directory to assess the performance of the semantic search engine.

Components

• test_search_evaluation.py: The main evaluation script that orchestrates the testing process.
• evaluation_dataset.json: A JSON file containing a list of queries to be used for the evaluation.
• relevance_judge.py: A module that uses the Gemini 1.5 Flash model to act as an AI relevance judge, scoring the relevance of documents to a given query.
• bm25_baseline.py: A module that implements the BM25 algorithm, a classic keyword-based search method, to serve as a baseline for comparison.

Workflow

1. AI-Powered Relevance Judgment: For each query in the dataset, the AI judge assesses the relevance of each document's summary, assigning a score of 0 (non-relevant), 1 (partially relevant), or 2 (highly relevant).
2. Comparative Evaluation: The evaluation script runs each query against both the embedding-based semantic search and the BM25 baseline.
3. Metrics Calculation: The script calculates a comprehensive set of industry-standard Information Retrieval (IR) metrics for both systems:
   • Precision@K: The fraction of relevant documents in the top K results.
   • Recall@K: The fraction of all relevant documents that are retrieved in the top K results.
   • Mean Average Precision (MAP): A single-figure measure of quality across all recall levels.
   • Normalized Discounted Cumulative Gain (nDCG): A measure of ranking quality that gives more weight to highly-ranked and more relevant documents.
   • Average LLM Score: The average relevance score assigned by the LLM judge to the retrieved documents.
   • Hit@K: A binary metric (1 or 0) indicating if at least one relevant document is found within the top K results (used when not using LLM judge).
4. Reporting: The script outputs a detailed report comparing the performance of both systems, including an overall score based on the average nDCG and average LLM score.

Usage

To run the evaluation, ensure your GEMINI_API_KEY is set as an environment variable and execute the following command from the project root:

cd py_src && python -m pip install -r requirements.txt && python -m test.test_search_evaluation --output_dir ../embedding_data --use_llm_judge

The --use_llm_judge flag enables the AI relevance judgment. If omitted, the script will use the gold_standard field from evaluation_dataset.json for a simpler, binary relevance assessment (relevant/not relevant).