Update to families.csv using a custom ML model

[aaronbell]

This is a v1 of a new ML model that predicts tag data. It is trained on all the existing human-assigned tags and fonts, referencing quantitative font data. Some tags are not well represented in this set due to poor matching or lack of data, but otherwise, this significantly increases the tag data coverage for our fonts.

This also includes entries for all variable instances (at least for fonts available in this repository) and opsz.

The model's performance is somewhat mixed due to the varying availability of data, but some tags are quite accurate. For example:

Metric	Target Style Tag	Active Rate	Accuracy	Precision	Recall
Common/Dense Tags	/Sans/Geometric	18.0%	86.70%	72.33%	41.37%
Structural Classes	/Serif/Transitional	11.9%	94.59%	85.12%	65.71%
Medium/Stylistic	/Theme/Techno	5.6%	97.20%	91.11%	58.48%
Sparse Stylistic	/Script/Handwritten	4.0%	98.63%	100.00%	68.40%
Extremely Sparse	/Theme/Wacky	0.9%	99.01%	0.00%	0.00%

Still, I think that these predictive assessments provide us with a significant amount of useful tag data.

Tagging for attention:
@davelab6
@emmamarichal
@m4rc1e
@bghryct
@EbenSorkin

[davelab6]

Elite

[aaronbell]

It appears that some new fonts haven't been added to the tag data yet:
Akt
Alien Block
Estedad
Finlandica Headline
Finlandica Text
Geist Pixel
Geomini
Hibur Mono
M PLUS U
Montenegrin Gothic One
Pliant
STIX Two Math
Strichpunkt Sans
Yuyu
Yuyu Short

I've asked the model to predict their tags and added them to the PR.

[aaronbell]

Unfortunately, it looks like this PR is blocked due to a failing test (that #10689 seeks to fix).

[aaronbell]

In considering this PR, I think further discussion would be helpful, especially as I'm suggesting a breaking change from previous behavior.

To ensure that the ML model is trained on accurate human data (and thus reducing noise / over-fitting), I've added a "provenance" column to the families.csv file which marks which entries are machine-generated versus the human-tagged data. I've also asked Gemini to analyze which associated scripts which break as a result of this change and resolve them.

[m4rc1e]

Two steps before we can trust this:

1. We need someone to verify how accurate these tags truly are by eye
2. I'd like the model and the supporting code in a repo since we must understand how it works before trusting it.

We need to confirm that the backend team are happy with adding a new column to the csv as well.

Overall, I like this direction a lot.

[aaronbell]

@m4rc1e I have to follow some steps before I can make the repo(s) public. But yes, it would be good to have someone take a look through to see if they make sense.

[felipesanches]

Was "providence" a typo? Did you mean "provenance"?

[bghryct]

looks like tagging work will mostly be checking the ML outputs. I wonder if this could be incorporated into the tagger @m4rc1e ?

or if we need a new visualizer to check @aaronbell ML tags against "human decisions"

[aaronbell]

looks like tagging work will mostly be checking the ML outputs. I wonder if this could be incorporated into the tagger @m4rc1e ?

Maybe. I think that there will be still be quite a bit of tagging work to do, but it will be more focused on things that humans do much better—like analyzing fonts for tags that are poorly represented in the model (like Blackletter, for example)—and any new tags that need to be added. Versus spending hours adding a tag for every single variation instance of a complex variable font, for example.

or if we need a new visualizer to check @aaronbell ML tags against "human decisions"

To note, all human tags have been preserved, even if there's a disagreement between the human and predicted model. Once this thing is more ready, I plan to put together a doc of "conflicts" that would be good to review and determine where the error lies—in the ML prediction or in the Human evaluation.

[m4rc1e]

I think our tagging app should highlight ai genned tags in a diff color so we can see them. this will require either a seperate list or even better an additional "provenance" column.

I know this is bikeshedding but can we just call this new column origin? Happy to stick with provenance though.

It may be the case that we start with ai placeholders and then we hand edit. The numbers are very impressive though, around 25k new tags.

[aaronbell]

I know this is bikeshedding but can we just call this new column origin? Happy to stick with provenance though.

I'm find with whatever makes sense. Currently the column is untitled in the csv so it can be whatever we want to make it! :)