HPA_Project

VGG-16 CUDA Optimization (HPA Project) This project is part of the High Performance Architectures (HPA) course (CMPE755) at RIT. Our team of six students is optimizing the convolutional layers of the VGG-16 neural network for efficient execution on NVIDIA RTX 3080 GPUs using CUDA, cuDNN, and cuBLAS. The goal is to apply parallel programming techniques from the textbook "Programming Massively Parallel Processors" to achieve significant performance improvements, focusing on data locality, memory bandwidth, and compute throughput.

Project Overview Course: High Performance Architectures (HPA) Timeline: 3-4 weeks Hardware: NVIDIA GeForce RTX 3080 Laptop GPUs (Compute Capability 8.6, 16 GB VRAM) Objective: Optimize VGG-16’s 2D convolutional layers using CUDA C++, leveraging libraries like cuDNN for convolution and cuBLAS for fully connected layers. Sub-Teams: Team A: Convolution Compute & Shared Memory Tiling Team B: Memory Access Patterns & Coalescing Team C: Kernel Fusion / Fully Connected Layer Optimization & Integration This repository contains the source code, build system, scripts, and documentation to achieve these goals, culminating in a final report and presentation.

File Structure Below is the project’s file structure with explanations of what each directory and file is for, and what you’re expected to add as the project progresses.

hpa_project/ ├── CMakeLists.txt # CMake build configuration

├── README.md # Project documentation (this file)

├── .gitignore # Git ignore rules

├── src/ # Source code

│ ├── main.cpp # Main inference application

│ ├── cuda_kernels.cu # CUDA kernels for convolution

│ └── host_utils.cpp # Host-side utilities (e.g., data loading, memory transfers)

├── include/ # Header files

│ ├── cuda_kernels.h # Declarations for CUDA kernels

│ └── host_utils.h # Declarations for host utilities

├── models/ # Exported VGG-16 model

│ └── vgg16.onnx # Exported ONNX model file (add in Phase 1)

├── tests/ # Unit tests for validation

│ ├── test_main.cpp # Test harness entry point

│ └── test_conv.cpp # Tests for convolution correctness

├── external/ # External dependencies (fetched or manually placed)

│ ├── onnxruntime/ # ONNX Runtime (auto-fetched by CMake)

│ └── cudnn/ # cuDNN (manually place here if not system-wide)

├── scripts/ # Utility scripts for automation

│ ├── setup_env.sh # Sets up environment variables (Linux)

│ ├── setup_env.bat # Sets up environment variables (Windows)

│ ├── build_project.sh # Builds the project with CMake and Ninja

│ ├── install_deps.sh # Guides dependency installation

│ ├── profile_kernel.sh # Profiles the application with Nsight tools

│ └── generate_input.py # Generates sample input data

├── build/ # Build directory (ignored by Git)

└── profile/ # Profiling reports (generated by scripts)

Expected Content to Add

• models/vgg16.onnx: Export the VGG-16 model from PyTorch or TensorFlow to ONNX format during Phase 1 (Day 1-2). Place it here.
• src/ and include/: Add new .cu files in src/ for Team A’s tiling optimizations or Team B’s memory access improvements (Phase 2). Update host_utils.cpp with functions for data management (e.g., optimized transfers for Team B).
• tests/: Add more test files (e.g., test_fully_connected.cpp) in Phase 2 to validate sub-team contributions.
• external/cudnn/: Manually place cuDNN files here if not installed system-wide (see dependency installation below).
• profile/: Profiling reports will be generated here by scripts/profile_kernel.sh in Phase 2 and 3.

Understanding CMakeLists.txt

The CMakeLists.txt file is the heart of the build system, using CMake to configure and build the project. It handles dependencies (CUDA, ONNX Runtime, cuDNN), compiles source files, and sets up optional tests. Here’s a breakdown of its key sections:

Project Setup:

cmake_minimum_required(VERSION 3.18)
project(HPA_Project LANGUAGES CXX CUDA)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

**CUDA Configuration:**
find_package(CUDA REQUIRED)
if(CUDA_FOUND)
    enable_language(CUDA)
    set(CMAKE_CUDA_STANDARD 17)
    set(CMAKE_CUDA_ARCHITECTURES 86) # RTX 3080
else()
    message(FATAL_ERROR "CUDA Toolkit not found. Install it manually.")
endif()

Dependency Management (ONNX Runtime):

find_library(ONNX_RUNTIME_LIB onnxruntime HINTS /usr/local/lib ${CMAKE_SOURCE_DIR}/external/onnxruntime/lib)
find_path(ONNX_RUNTIME_INCLUDE onnxruntime/core/session/onnxruntime_cxx_api.h 
          HINTS /usr/local/include ${CMAKE_SOURCE_DIR}/external/onnxruntime/include)
if(ONNX_RUNTIME_LIB AND ONNX_RUNTIME_INCLUDE)
    message(STATUS "ONNX Runtime found system-wide: ${ONNX_RUNTIME_LIB}")
else()
    ExternalProject_Add(
        onnxruntime
        GIT_REPOSITORY https://github.com/microsoft/onnxruntime.git
        GIT_TAG v1.17.1
        PREFIX ${CMAKE_SOURCE_DIR}/external/onnxruntime
        ...
    )
endif()
Checks for ONNX Runtime system-wide; if not found, downloads and builds it into external/onnxruntime/.
Dependency Management (cuDNN):

find_library(CUDNN_LIB cudnn HINTS ${CUDA_TOOLKIT_ROOT_DIR}/lib64)
find_path(CUDNN_INCLUDE cudnn.h HINTS ${CUDA_TOOLKIT_ROOT_DIR}/include)
if(CUDNN_LIB AND CUDNN_INCLUDE)
    message(STATUS "cuDNN found: ${CUDNN_LIB}")
else()
    message(WARNING "cuDNN not found. Auto-fetching not implemented due to NVIDIA login requirement.")
endif()
Looks for cuDNN in the CUDA Toolkit path or external/cudnn/. Requires manual placement if not found.
Executable Definition:

add_executable(hpa_project 
    src/main.cpp 
    src/cuda_kernels.cu 
    src/host_utils.cpp
)

Builds the main executable hpa_project from the listed source files.

Include and Link:

target_include_directories(hpa_project PRIVATE 
    include 
    ${CUDA_INCLUDE_DIRS} 
    ${ONNX_RUNTIME_INCLUDE} 
    ${CUDNN_INCLUDE}
)
target_link_libraries(hpa_project PRIVATE 
    ${CUDA_LIBRARIES} 
    ${CUDA_CUBLAS_LIBRARIES} 
    ${CUDNN_LIB} 
    ${ONNX_RUNTIME_LIB}
)
Specifies header directories and links libraries (CUDA, cuBLAS, cuDNN, ONNX Runtime).
Tests:

option(BUILD_TESTS "Build unit tests" OFF)
if(BUILD_TESTS)
    add_executable(test_hpa_project 
        tests/test_main.cpp 
        tests/test_conv.cpp
    )
endif()
Optionally builds a test executable if -DBUILD_TESTS=ON is passed to CMake.
Model Copy:

add_custom_command(TARGET hpa_project POST_BUILD
    COMMAND ${CMAKE_COMMAND} -E copy_if_different
    ${CMAKE_SOURCE_DIR}/models/vgg16.onnx ${CMAKE_BINARY_DIR}/vgg16.onnx
)

Copies vgg16.onnx to the build directory after compilation.

Building the Project Using Shell Scripts The scripts/ folder contains scripts to automate setup and building. Here’s how to build the project using them.

Prerequisites
CMake: Version 3.18+ (sudo apt install cmake on Linux, or download from cmake.org on Windows).
Ninja: Faster build system (sudo apt install ninja-build on Linux, or download from Ninja releases on Windows and add to PATH).
Git: For fetching ONNX Runtime (sudo apt install git).
C++ Compiler:
Linux: g++ (sudo apt install build-essential).
Windows: Visual Studio 2022 with “Desktop development with C++” workload.
CUDA Toolkit: Version 12.4 recommended. Download from NVIDIA.
cuDNN: Download from NVIDIA and place in external/cudnn/.

Steps
1. Set Up Environment

Linux:

chmod +x scripts/setup_env.sh
source scripts/setup_env.sh
Windows (use Developer Command Prompt for VS 2022):

scripts\setup_env.bat
What it does:
Sets CUDA paths.
Verifies Ninja and CUDA are installed.
Checks for cuDNN in external/cudnn/.
2. Install Dependencies

chmod +x scripts/install_deps.sh
./scripts/install_deps.sh
What it does:
Ensures Git is installed for ONNX Runtime fetching.
Prompts for cuDNN placement in external/cudnn/ if missing.
3. Build the Project

chmod +x scripts/build_project.sh
./scripts/build_project.sh  # Build without tests
or 
./scripts/build_project.sh ON  # Build with tests
What it does:
Creates a build/ directory.
Runs cmake -GNinja -DBUILD_TESTS=[ON/OFF] ...
Builds with ninja.
Outputs build/hpa_project (and build/test_hpa_project if tests are enabled).
4. Run the Executable

cd build ./hpa_project 5. (Optional) Profile the Application

chmod +x scripts/profile_kernel.sh
./scripts/profile_kernel.sh
What it does:
Runs Nsight Systems and Nsight Compute to profile hpa_project.
Saves reports to profile/.
Manual Dependency Installation (if Needed)
CUDA Toolkit:
Download and install CUDA 12.4.
Add to PATH (e.g., /usr/local/cuda-12.4/bin on Linux, C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\bin on Windows).
cuDNN:
Download cuDNN 8.9 (for CUDA 12.x).
Extract to external/cudnn/ (e.g., tar -xzvf cudnn-*.tar.gz -C external/cudnn --strip-components=1).
ONNX Runtime:
Auto-fetched by CMake, but you can manually place a CUDA-enabled build in external/onnxruntime/.

Contributing
Phase 1: Export VGG-16 to models/vgg16.onnx and implement baseline CUDA kernel in src/cuda_kernels.cu.
Phase 2:
Team A: Add tiling optimizations in cuda_kernels.cu.
Team B: Optimize memory transfers in host_utils.cpp.
Team C: Integrate cuBLAS for fully connected layers and profile with scripts/profile_kernel.sh.
Phase 3: Combine optimizations, measure final performance, and document in the final report.

Push changes to GitHub:

git add .
git commit -m "Added tiling optimization for Team A"
git push origin main
Troubleshooting
CMake Errors:
“Ninja not found”: Install Ninja and add to PATH.
“CMAKE_CXX_COMPILER not set”: Ensure Visual Studio or MinGW is installed.
“CMAKE_CUDA_COMPILER not set”: Install CUDA Toolkit and add to PATH.
Build Failures: Check build/CMakeFiles/CMakeOutput.log for details.
Profiling Issues: Ensure Nsight tools are installed with CUDA Toolkit.