From c3c2f404ef17c337de519ab7c52ca774d5b31a61 Mon Sep 17 00:00:00 2001 From: Vivek Date: Sat, 11 Apr 2020 20:02:57 -0400 Subject: [PATCH 1/5] Design documnet for OpenCV Line Detection --- documents/design/opencv_line_detection.md | 51 +++++++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 documents/design/opencv_line_detection.md diff --git a/documents/design/opencv_line_detection.md b/documents/design/opencv_line_detection.md new file mode 100644 index 000000000..ddf003891 --- /dev/null +++ b/documents/design/opencv_line_detection.md @@ -0,0 +1,51 @@ +# OpenCV based Line Detection + +*Issue #Number* +#564 + +**Author:** +- Vivek Mhatre + +## The Problem + +Currently line detection is done using a neural network. We would like to implement line detection with OpenCV as an alternative if the neural network does not work well. Additionally, OpenCV is a useful library to use as it can do useful tasks like thresholding and edge detection. +The color of the line, shape of the line, camera orientation, and position of the image are some challenges that come to mind when trying to detect the line. +The end goal of this project is to have line detection implemented with OpenCV. + +## Proposed Solution + +Depending on the color of the lines I would use one or two masks. The algorithm I would use to detect lines would go as follows. +1. Use BGR to HSV transformation to make lines the same color +2. Filter out pixels using a mask to isolate line color. Possibly a white and yellow mask? +3. Process Gaussian Blur to reduce image noise and help smooth out edges in the image. +4. Process edge detection using Canny +5. Apply a trapezoidal mask over edges to narrow down image. +6. Use Hough Transform (HoughLinesP) to get lines +7. Draw lines on source image + +To solve this problem, I will first implement my algorithm using python and +Steps to solve issue: +1. I will first implement my algorithm using Python. +2. Use the training images on the robojackets cloud to tune my parameters. +3. Then I will implement my algorithm using C++ +4. Test the performance in past simulations using ros bag files. + +## Questions & Research + +Not sure about the line color but I can find that out by looking at the training images. Other than that I do not have many questions. + +## Overall Scope + +### Affected Packages + +I will need the OpenCV package and line_layer.cpp. I will be making changes to the igvc_navigation package. + +### Schedule + +Subtask 1 (April 12th): Implement algorithm in python? + +Subtask 2 (April 19th): Implement algorithm in C++. + +Subtask 3 (April 26th): Test algorithm using ros bag files. + +Code Review (May 29th): Everything should be done now. From 92a629ff2d61473ddf11014b079d2ad69f8606cd Mon Sep 17 00:00:00 2001 From: Vivek Date: Sun, 12 Apr 2020 17:29:22 -0400 Subject: [PATCH 2/5] Additional challenges with OpenCV line detection --- documents/design/opencv_line_detection.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/documents/design/opencv_line_detection.md b/documents/design/opencv_line_detection.md index ddf003891..2f20acc33 100644 --- a/documents/design/opencv_line_detection.md +++ b/documents/design/opencv_line_detection.md @@ -30,9 +30,18 @@ Steps to solve issue: 3. Then I will implement my algorithm using C++ 4. Test the performance in past simulations using ros bag files. +Challenges: +How to handle curves in the track: +- Segment the image and handle in small parts. +How to avoid detecting barrels withh white lines: +- Try to capture lines with a green background (grass) and reject lines with an orange background (barrels) + ## Questions & Research -Not sure about the line color but I can find that out by looking at the training images. Other than that I do not have many questions. +Not sure about the line color but I can find that out by looking at the training images. +How will I handle curves in the course? +How will I handle white lines in the barrel? +How will varying weather conditions affect line detection? ## Overall Scope From 30d5d6f33b7973a2eccc66f44326d045e1fd7984 Mon Sep 17 00:00:00 2001 From: Vivek Date: Wed, 15 Apr 2020 20:21:18 -0400 Subject: [PATCH 3/5] Added design doc as a link in the README of the document folder. Incorporated morphological transform in algorithm to clean up noise. --- documents/README.md | 1 + documents/design/opencv_line_detection.md | 15 ++++++++------- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/documents/README.md b/documents/README.md index 5a9e1d483..ae30e80eb 100644 --- a/documents/README.md +++ b/documents/README.md @@ -22,3 +22,4 @@ This folder contains research and design documents for the repo. * [Slope Detection](design/slope_detection.md) * [Testing Library](design/testing_library.md) * [Waypoint Orientation for TEB](design/waypoint_orientation.md) +* [OpenCV Based Line Detection](design/opencv_line_detection.md) diff --git a/documents/design/opencv_line_detection.md b/documents/design/opencv_line_detection.md index 2f20acc33..3843e03fd 100644 --- a/documents/design/opencv_line_detection.md +++ b/documents/design/opencv_line_detection.md @@ -18,8 +18,8 @@ Depending on the color of the lines I would use one or two masks. The algorithm 1. Use BGR to HSV transformation to make lines the same color 2. Filter out pixels using a mask to isolate line color. Possibly a white and yellow mask? 3. Process Gaussian Blur to reduce image noise and help smooth out edges in the image. -4. Process edge detection using Canny -5. Apply a trapezoidal mask over edges to narrow down image. +4. Use a morphological transformation to further remove noise in the image. Will most likely use opening, which just erosion followed by dilation. +5. Process edge detection using Canny 6. Use Hough Transform (HoughLinesP) to get lines 7. Draw lines on source image @@ -33,7 +33,7 @@ Steps to solve issue: Challenges: How to handle curves in the track: - Segment the image and handle in small parts. -How to avoid detecting barrels withh white lines: +How to avoid detecting barrels with white lines: - Try to capture lines with a green background (grass) and reject lines with an orange background (barrels) ## Questions & Research @@ -42,6 +42,7 @@ Not sure about the line color but I can find that out by looking at the training How will I handle curves in the course? How will I handle white lines in the barrel? How will varying weather conditions affect line detection? +What topic will I need to subscribe to in order to get images from the ros bag files when testing? ## Overall Scope @@ -51,10 +52,10 @@ I will need the OpenCV package and line_layer.cpp. I will be making changes to t ### Schedule -Subtask 1 (April 12th): Implement algorithm in python? +Subtask 1 (April 19th): Finish implementing algorithm in python. -Subtask 2 (April 19th): Implement algorithm in C++. +Subtask 2 (April 26th): Implement algorithm in C++. -Subtask 3 (April 26th): Test algorithm using ros bag files. +Subtask 3 (April 29th): Test algorithm using ros bag files. -Code Review (May 29th): Everything should be done now. +Code Review (May 1st): Everything should be done now. From ac5d8346bdba777e10f9870c105a0db8dda94993 Mon Sep 17 00:00:00 2001 From: Vivek Date: Sat, 18 Apr 2020 16:12:43 -0400 Subject: [PATCH 4/5] Updated README in the design folder --- documents/design/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/documents/design/README.md b/documents/design/README.md index ff8d2e733..5f940f03f 100644 --- a/documents/design/README.md +++ b/documents/design/README.md @@ -14,6 +14,7 @@ * [Barrel Segmentation](actual_barrel_segmentation.md) * [Raycasting Correction](NM_ray_casting.md) * [Slope Detection](slope_detection.md) + * [OpenCV Based Line Detection](design/opencv_line_detection.md) * **Simulation** * [Adding Noise](add_noise_to_simulation.md) From def21ef36cf63aca140eb61770c364333372e74d Mon Sep 17 00:00:00 2001 From: s-hliao <48244293+s-hliao@users.noreply.github.com> Date: Wed, 1 Dec 2021 20:53:58 -0500 Subject: [PATCH 5/5] Update opencv_line_detection.md Design document for OpenCV line detection required updates due to IGVC codebase structure changes. --- documents/design/opencv_line_detection.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/documents/design/opencv_line_detection.md b/documents/design/opencv_line_detection.md index 3843e03fd..bc94a1516 100644 --- a/documents/design/opencv_line_detection.md +++ b/documents/design/opencv_line_detection.md @@ -5,6 +5,7 @@ **Author:** - Vivek Mhatre +- Henry Liao ## The Problem @@ -22,12 +23,13 @@ Depending on the color of the lines I would use one or two masks. The algorithm 5. Process edge detection using Canny 6. Use Hough Transform (HoughLinesP) to get lines 7. Draw lines on source image +8. Return edited copy of source image To solve this problem, I will first implement my algorithm using python and Steps to solve issue: -1. I will first implement my algorithm using Python. -2. Use the training images on the robojackets cloud to tune my parameters. -3. Then I will implement my algorithm using C++ +1. I will set up a new ros node in igvc_perception based on existing cnn detection node setup. +2. I will implement my algorithm using Python. +3. Use the training images on the robojackets cloud to tune my parameters. 4. Test the performance in past simulations using ros bag files. Challenges: @@ -48,14 +50,14 @@ What topic will I need to subscribe to in order to get images from the ros bag f ### Affected Packages -I will need the OpenCV package and line_layer.cpp. I will be making changes to the igvc_navigation package. +I will need the OpenCV package and ros nodes. I will be making changes to the igvc_perception package. ### Schedule -Subtask 1 (April 19th): Finish implementing algorithm in python. +Subtask 1 (December 8th): Finish implementing algorithm in Python. -Subtask 2 (April 26th): Implement algorithm in C++. +Subtask 2 (December 11th): Finish tuning parameters. -Subtask 3 (April 29th): Test algorithm using ros bag files. +Subtask 3 (December 20th): Test algorithm using ros bag files. -Code Review (May 1st): Everything should be done now. +Code Review (By January 1st): Everything should be done now.