From d36dcba7b6191e4831061dbccc82ea3fa876bb5d Mon Sep 17 00:00:00 2001 From: "David P. Steelman" Date: Mon, 15 Jun 2026 08:07:07 -0400 Subject: [PATCH] LIBCIR-498. Updated MDSOAR API "robots.txt" test step In MD-SOAR, the "robots.txt" file returned by the MD-SOAR API backend is supposed to allow crawling (same "robots.txt" as for the Angular front-end), except on the QA site. This change to the "robots.txt" file behavior was initially made in LIBCIR-487. https://umd-dit.atlassian.net/browse/LIBCIR-498 --- dspace/docs/MdsoarTestPlan.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/dspace/docs/MdsoarTestPlan.md b/dspace/docs/MdsoarTestPlan.md index 795a366300..cf9d7686ef 100644 --- a/dspace/docs/MdsoarTestPlan.md +++ b/dspace/docs/MdsoarTestPlan.md @@ -292,7 +292,16 @@ we do not want it included in search results. -and verify that the "robots.txt" file disallows all crawling. +Verify the contents of a "robots.txt" file is displayed, and contains the +following uncommented line (among many others): + +```text +Disallow: /browse/* +``` + +**Note:** On MD-SOAR QA, the “robots.txt” file disallows all crawling, as it is +available on the public internet (to allow access by non-UMD stakeholders), and +we do not want it included in search results. ### 14) sitemap.xml