@Yuying-Jin and I discovered that many collation errors are generated by a problem with tokenization, which tags <.+?> are getting mashed in tokens with neighboring texts and spaces between are removed. Adjusting the extract() function in the Python script doesn't help: we can insert spaces as we wish, but the tokens still mash markup and text together.
Example of a "smashed token":
We can add space, but the tokenization is unaffected:
We need to determine how to change the tokenization to definitively split around XML tags.
@Yuying-Jin and I discovered that many collation errors are generated by a problem with tokenization, which tags
<.+?>are getting mashed in tokens with neighboring texts and spaces between are removed. Adjusting the extract() function in the Python script doesn't help: we can insert spaces as we wish, but the tokens still mash markup and text together.Example of a "smashed token":
We can add space, but the tokenization is unaffected:
We need to determine how to change the tokenization to definitively split around XML tags.