Fix Javadoc handling in Markdown comments

[shin19991207]

Fixes #3695
Fixes #3696
Fixes #3672
Fixes #3690

Originally, JDT LS used a custom renderer for Markdown documentation comments (///).

The renderer manually walked the Javadoc AST, copied text, created links, and formatted a limited set of block tags from https://help.eclipse.org/latest/index.jsp?topic=%2Forg.eclipse.jdt.doc.isv%2Freference%2Fapi%2Forg%2Feclipse%2Fjdt%2Fcore%2Fdom%2FTagElement.html. This duplicated part of JDT Core's Javadoc rendering logic but did not implement the full semantics of Javadoc tags.
Therefore, tags not explicitly handled by the custom renderer were missing or rendered incorrectly. This included the reported cases for @inheritDoc, @value, @code... etc.

The problem was not specific to the reported cases. Other inline and block tags could also behave differently from traditional Javadoc comments. Adding special cases for individual tags would continue duplicating JDT Core logic and would require every Javadoc tag and its semantics to be reimplemented in JDT LS.

The fix uses JDT Core's Javadoc renderer for Markdown comments that require semantic processing instead of manually rendering their tags in JDT LS.

[shin19991207]

Regarding #3690,

The following case was mentioned in #3699 (review):

/// {@link "some text"}
void method() {}

Eclipse IDE renders "some text" as a link text with empty link. However, this appears to conflict with this existing test case, which expects an DNE {@link ...} reference to be rendered as plain text:

eclipse.jdt.ls/org.eclipse.jdt.ls.tests/src/org/eclipse/jdt/ls/core/internal/handlers/HoverHandlerTest.java

Lines 524 to 541 in 788f0de

	public void testHoverWhenLinkDoesNotExist() throws Exception {
	importProjects("maven/salut");
	project = WorkspaceHelper.getProject("salut");
	handler = new HoverHandler(preferenceManager);
	//given
	String payload = createHoverRequest("src/main/java/java/Foo2.java", 51, 26);
	TextDocumentPositionParams position = getParams(payload);
	// when
	Hover hover = handler.hover(position, monitor);
	assertNotNull(hover, "Hover is null");
	assertEquals(3, hover.getContents().getLeft().size(), "Unexpected hover contents:\n" + hover.getContents());
	Either<String, MarkedString> javadoc = hover.getContents().getLeft().get(1);
	String content = null;
	assertTrue(javadoc != null && javadoc.getLeft() != null && (content = javadoc.getLeft()) != null, "javadoc has null content");
	assertMatches("This link doesnt work LinkToSomethingNotFound", content);
	}

Which behaviour do we prefer?

[datho7561]

I think part of writing a custom solution was to avoid having to convert the Markdown to HTML and then back to Markdown. (VS Code expects Markdown for hover content, whereas Eclipse expects HTML).

Knowing this, do you still think it's worth taking your current approach, or do you think it's worth handing the missing tags in the current implementation?

[datho7561]

I'm cautiously in favour of this. I think the benefit of simplifying the implementation outweighs the potential performance improvement, and I think detecting if there's no tags and doing a simpler conversion then is a good compromise.