enhancement: Replace mismatched license headers instead of appending a new header during auto-fix

[Standing-Man]

Problem

Currently, license header detection is tied to the configured comment style.

For example, a repository may configure Rust files (.rs) to use a line-by-line header style:

[mapping.DOUBLESLASH_STYLE]
extensions = ["rs"]

while existing files still contain a block-style header:

/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.
 */

In this case:

1. hawkeye check reports the file as missing a license header because the existing header does not match the configured style.
2. hawkeye format inserts a new //-style header at the beginning of the file.
3. The original block-style header remains in place.

As a result, the file ends up containing two license headers.

Expected Behavior

Before inserting a new header during auto-fix, Hawkeye could attempt to detect and remove an existing license header that is semantically valid but does not match the configured comment style.

Example:

Before:

/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.
 */

fn main() {}

After:

// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.

fn main() {}

instead of:

// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.

/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.
 */

fn main() {}

Benefits

• Avoids duplicate license headers.
• Makes style migrations easier.
• Produces cleaner auto-fix results.
• Reduces manual cleanup after running hawkeye format.

[hubcio]

just to add, this came up during integration of hawkeye in apache/iggy#3325

@tisonkun and the rest of contributors: big thanks for this great piece of software. much more seamless to use than google addlicense.

[tisonkun]

Thanks for your reporting! I'll check this in this week. Feel free to ping me next week if I miss this item :P

[tisonkun]

This is almost impossible in oneshot - How do you know the previous style?

Typically, you can configure Rust style as SLASHSTAR_STYLE, run hawkeye remove and then reconfig it as DOUBLESLASH_STYLE and run hawkeye format.

[tisonkun]

detect and remove an existing license header

Actually, we have some heuristic rules to detect an existing header, but historically, it can produce false positives (removing unrelated comments) or false negatives (retaining logically existing license headers).

hawkeye/cli/src/subcommand.rs

Lines 158 to 168 in 3c39666

	fn on_not_matched(&mut self, header: &HeaderMatcher, mut doc: Document) -> Result<(), Error> {
	if doc.header_detected() {
	doc.remove_header();
	doc.update_header(header)?;
	self.updated
	.push(format!("{}=replaced", doc.filepath.display()));
	} else {
	doc.update_header(header)?;
	self.updated
	.push(format!("{}=added", doc.filepath.display()));
	}

[hubcio]

@tisonkun what about using something like levenshtein distance?

you might not need to know the old style at all - only whether the leading comment is the license wearing different markers.

read_file_header_on_one_line already does most of this: strips markers, collapses whitespace, compares. problem is it strips the configured style's markers, so a /* */ block under a // config slips right past it. strip the markers generically (//, /* * */, #, <!--) and you can compare regardless of style.

so when structural detection comes up empty (existing_header bails at is_first_header_line, end_pos stays None, format inserts a second header), fall back to a content compare. that's where the distance comes in: levenshtein(stripped_block, stripped_license) divided by the longer length gives a 0..1 ratio, and above some cutoff you call it a header even though the markers differ. contains() can't do that; edit distance can.

few things to watch:

• remove_header only cuts when end_pos is Some, so the fallback has to hand back the real begin/end range, not just a bool.
• a raw ratio is length-sensitive - a one-line SPDX scores low against the long ASF block, a big unrelated banner can score high. no single magic threshold, and since this rewrites files in place i'd keep it add-only and never delete on a weak match.
• the keyword gate sits after the is_first_header_line bail today, so the fallback runs its own keyword pass anyway.

what do u think about that? i think this approach would solve the problem we had in iggy. we could make it experimental/optional, set via licenserc.toml. if you want, i can create a PR.

[tisonkun]

@hubcio Your use case looks like a oneshot work. What about:

configure Rust style as SLASHSTAR_STYLE, run hawkeye remove, and then reconfig it as DOUBLESLASH_STYLE and run hawkeye format.

[hubcio]

@tisonkun yeah this can work also, but it's a dirty workaround. what if someone will insert empty line in valid license? i checked it and hawkeye inserts a new license to the top, while preserving previous invalid one.

[tisonkun]

You can always build bad cases of heuristic rules.

This is not a dirty workaround. The main impossible point here is that you want to "guess" what a mismatched string means and there will always be false positives/false negatives.

A structural way is to define possible_styles_before_removals = [ "SLASHSTAR_STYLE", "DOUBLESLASH_STYLE" ]. This is a real dirty workaround.

If Iggy's case is well-formatted enough, let's avoid hacking in hawkeye because those rules would be even more brittle and subtle.

I can see that in apache/iggy#3325 most license headers are handled properly already.

Except the inter-style's heuristic matching rules, using levenshtein distance within the same style is fine to add.

[tisonkun]

strip the markers generically (//, /* * */, #, <!--) and you can compare regardless of style.

You miss the SQL style comment -- here, and there is no standard for "general comment markers".

I mention the SQL style here to show that there is always missing cases when you want to "DO WHAT I MEAN" when there is no well-defined standards).

[tisonkun]

Another possible way is to generalize the comment header extractor with a structural parser like tree-sitter. That is, capturing the first comment block and seeing.

Then we can bind file extensions with such parser.

Or actually, by extending mapping to allow a file extension to accept multiple styles, removals can work by iterating all possible styles, and when formatting (add a new one), prefer the first style in array (should define what is "the first", or use an extra field to denote it).

This is a possible way to go, while it may introduce a breaking change in config structure.

[tisonkun]

what if someone will insert empty line in valid license

This won't be fixed.

What if:

// <almost correct headers>

// real doc comment
others

Your rule would incorrectly remove the real doc comment based on the levenshtein distance threshold.

[hubcio]

what if someone will insert empty line in valid license

This won't be fixed.

What if:

// <almost correct headers>

// real doc comment
others

Your rule would incorrectly remove the real doc comment based on the levenshtein distance threshold.

@tisonkun you're right and you convienced me.

i think we've been talking past each other though..

please hear me out from user perspective:
i don't need hawkeye to fix a mangled or wrong-style header. i need it to stop making things worse. right now, when it doesn't recognize an existing header, format inserts a second one on top. the duplicate is the only real bug for me.

google addlicense never does that, and it does nothing clever to avoid it. hasLicense just lowercases the first ~1000 bytes and checks for copyright / spdx-license-identifier / mozilla public. if any are there, it skips the file. no style parsing or no marker stripping (so the SQL -- problem never comes up), and it never removes anything. doc-comment case can't happen, because nothing gets deleted - worst case it skips a file it maybe should have touched.

so my propsa: before format inserts a header, a cheap presence check. if the file already looks licensed, skip it and report it instead of stacking a duplicate. no guessing what the old text means, no removal: just "don't make it worse", same contract as addlicense.

the real migrate/replace is a separate, harder feature, and if someone wants it, your multi-style mapping idea is the clean route, not fuzzy matching. lets leave that out entirely.

would that be acceptable form your perspective?

[hubcio]

but on the other hand maybe this is the problem with iggy approach where we call it first with check and then if something is wrong, we use format... hmm. maybe you were right from the beginning.

[hubcio]

@Standing-Man what do you think?