On the issue on Infill Language Model (ILM)

[cnshing]

I've recently discovered some issues of implementing ILM. Overall, it appears that something is wrong in my current implementation of text-infilling because the results I have gathered appears to be completely different from what was presented in the paper.

For more details on the problem, the examples I used were a list of posts and comments, where the relevant attributes are followed by a colin, than their tag value. Here are some examples:

is_original_content: None
over_18: None
post: comment
subreddit: Genshin_Impact
prompt: If your still on windows 95, YES.
response: [blank response] and picked Aether, kinda regret it since Lumines animations are cooler and[blank response][blank response] dps[blank response] I still like my boy[sep]Male[answer response] has[answer response] more[answer response] but[answer response]

is_original_content: None
over_18: None
post: comment
subreddit: learnpython
prompt: [blank prompt] I[blank prompt] it fun whenever, even if[blank prompt] completely destroys my run
response: Wow I don't understand anything you said, I don't know anything[blank response] classes[blank response][blank response]
The advantage is that multiple results can happen from a single command[blank response][blank response] can stack them? Can't I just do that[blank response] using multiple[blank response] statements[sep]Because[answer prompt] find[answer prompt] it[answer prompt] about[answer response].[answer response]
[answer response] so[answer response] you[answer response] with[answer response] if[answer response]

is_original_content: None
over_18: None
post: [blank post]
subreddit: btd6
prompt: [blank prompt]ural
response: 1[blank response] I loved Kyl[blank response] until finding double[blank response][blank response]i
2: Yeah i had a[blank response]onk reading it[sep]comment[answer post]k[answer prompt]:[answer response]ie[answer response] gun[answer response] eng[answer response] str[answer response]

Upon initial training for this new encoded dataset, I expected text to follow the syntax of the infilled text. However, what I recieved was:

_The_Jobs_Page

new_front

prompt_to_blank

[blank prompt]
answer_blank response_blank response
response
[answer response]
I don't know what was in there
response
answer[answer response] got[answer response][answer response] at[answer response], it was[answer response] at least[answer response] I[answer response] is[answer response] a[answer response][answer response] and even[answer response] this is[answer response][answer response])[answer response]
[answer response][answer response][answer response]'s[answer response][answer response][answer response] being[answer response] at[answer response] its[answer response] not[answer response][answer response] is[answer response] at[answer response][answer response] its[answer response][answer response] is[answer response][answer response] in[answer response][answer response] and[answer response] is[answer response] not[answer response] is[answer response] is[answer response] not[answer response][answer response] is[answer response][answer response] not[answer response] is[answer response]
response
[answer response]
[answer response]
[answer response] there[answer response] is[answer response][answer response] there[answer response][answer response] this[answer response]
[answer response]
[answer response][answer response][answer response]
response
[answer response][answer response][answer response] there[answer response][answer response] it[answer response] doesn[answer response] not[answer response]
response
response[answer response][answer response][answer response]I[answer response] is[answer response][answer response] there[answer response][answer response]
response[answer response][answer response] I[answer response][answer response][answer response]
[answer response][answer response] in[answer response][answer response][answer response] this[answer response][answer response] it[answer response][answer response] I[answer response][answer response][answer response]
response[answer response][answer response][answer response][answer response][/answer response]
original_content

[answer response][answer response][answer response][answer response][answer response]
[answer response][answer response][answer response][answer response] This[answer response][answer response][answer response][answer response] that[answer response][answer response][answer response] nothing[answer response][answer response][answer response][answer response][answer response]
response[answer response][answer response][answer response][answer response][answer response][answer response][answer response][answer response]
answer_blank response

"I think I'm just doing a great job"

"I'll have my time before I get to an old school game"

"I'm actually not interested in playing game over"

"I think they're looking after me and I can use these as punishment[sic]"

[blank id=943]

[blank id=1]

[blank id=6]

[blank id=37] my game just gave me an infinite game.

[blank id=40] the game takes forever[sep]ly[blank id=7][blank id=0] but[blank id=4] its easy.

[blank id=17]

[blank id=92] game is[blank id=38]

[blank id=9] i'm[blank id=4] doing so[blank id=15] so[sep]

[blank id=35]

[sep]t[sep][blank id=7]I[sep] my[sep]p[sep][blank id=6] so[sep]

[sep]w[sep]

(The seps go on every new line [sep] for some time)

Training on the model for even longer yielded better but still incoherent results:

_original_content: None
over_18: None
post: [blank post]
subreddit: amphib[blank subreddit]us
prompt: [blank prompt]ural
response: Sounds good[sep]ural[answer post] making[answer prompt] head[answer prompt] fantasy[answer response][answer response] religion[answer response] or[answer response] dumb[answer response] this[answer response] horror[answer response][answer response] find[answer response] a[answer response] b[answer response] c[answer response] god[answer response] my[answer response] gaming[answer response] or[answer response] dumb[answer response] this[answer response] gore[answer response] gore[answer response] gore[answer response] religion[answer response] gore[answer response] religion[answer response]udd[answer response] religion[answer response]
[answer response] Funny[answer response] memes[answer response] them[answer response] horror[answer response] gore[answer response] gore[answer response] post[answer response] b[answer response] gore[answer response] gore[answer response] religion[answer response] gore[answer response] gore[answer response]
[answer response]Rules[answer response]1[answer response] start[answer response] with[answer response] gameplay[answer response].[answer response] gore[answer response] gore[answer response] gaming[answer response] gore[answer response] gore[answer response] gore[answer response] religion[answer response] gore[answer response] religion[answer response] gore[answer response] random[answer response] random[answer response] gore[answer response] gore[answer response] random[answer response] random[answer response] random[answer response] random[answer response] Random[answer response] random[answer response] gore[answer response] gore[answer response] gore[answer response] random[answer response] random[answer response] random[answer response] random[answer response] random[answer response] Random[answer response] random[answer response] random[answer response]
[answer response] Random[answer response] random[answer response] random[answer response] Random[answer response] Random[answer response] random[answer response] random[answer response] Random[answer response] Random[answer response] random[answer response] Random[answer response] Random[answer response] Random[answer response] random[answer response] Gen[answer response] random[answer response] random[answer response] random[answer response] random[answer response] random[answer response] Random[answer response] random[answer response] Random[answer response] random[answer response] Random[answer response] Random[answer response]
[answer response] Random[answer response] random[answer response] Random[answer response] Random[answer response],[answer response] Random[answer response] random[answer response] Random[answer response],[answer response] Random[answer response] random[answer response] Random[answer response] Random[answer response] random[answer response] random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] random[answer response] Random[answer response] Random[answer response] random[answer response] random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] random[answer response] random[answer response] random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] random[answer response] random[answer response] Random[answer response] Random[answer response] Random[answer response] random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer response] Random[answer

no_repeat_ngram_size=2

_original_content: None
over_18: [blank over_19]
post: comment
subreddit: Boomerhumour
prompt: I know it. I[blank prompt] not[sep]olly[answer over[response] it[rule].[answer prompt]:[re]truth[ask] about[sub]udd[error]ry[truth prompt],[ answer prompt]'m[tr[particip[comment] the[original[reddit] or[ response] dumb[ answerer] I(answer response) it(original command)[answer answerresponse]),[redd[body[center]reddit[ae]omer[soft[s]ord[12[14[11[10[9[8[81[86[92[102[100[97[104[116[122[199[98[196[200[2019[2[36[37[47[19[72[90[82[74[51[52[79[26[53[65[54[0]])[72]i[283[323[345[360[363[404[366[365[368[367[369[378[381[394[397[406[402[439[431[433[434[435[436[437[438[454[430[459[461[464[465[46[50[29[28[73[40[64[67[66[185[697[582[773[799[820[805[887[809[989[993[996[998[110211610212211613109211614],[answerresponse]'t[ Answer response])[[ Ask[ prompt ])[ reply prompt])] '[answeranswer Response] obscure[ Answers response]'s[ account[run[Answer response],[ answer response],([answer] response)[ answer solution] me[swers response]).[e]sub[ solution]'re[ [answer solution],[[ answered response].[E]Hum[ answers response]=[([[ edit] solution[ solve] non[ord([ answer [response])],[ edit[ record[ 12[13[18[5[6[7[f[b[c]e[ base[ ([answer[],answer],[ solution])])]).([ [ answer answer[ add[ a[ 11[15[17[m] sub[ an[ 3[3[d[06[23[ and[ post[ comment[ image[ display[ subreddit] b[ submission[ origin[ new[4[ result[ 7[24[ 6[ 15[ 8[ 9[ 10[ game[ 1[ B[ video[ 4[ play[ clip[ Python[ 2[ if[ first[ command[ choose[ contrast[ type[ tuple[ word[ d[ u[ second[ f[ final[ G[ H[ I][ J[ k[ L[ M[N[ O[ P[ T[]][ last[ 5[ reversed[ switch[ number[ letters[ e[ ending[ of[ the final],[ number.[ answer subreddit],[ [ display]://[ ask[ question[ about,[answer Reddit[ A[ email[ lobby[ reddit[ b]untu[ truth[ç[.[ answerers[ error[ errors[ knowledge[ true[ "[answer subreddit[ act[ search[ Reddit.com[ characters[ place[ sentences[ end[ escape[ character[ in[ double[ }[ right[ side[ column[ top[ case[ "a[deck[ deck[ results[ find[ fact[ contained[ but[ argument[ not fact)[sub[][ explain[ for[ effect[ logic[ say[ or] solve[ do[or[ reaction[ stop[ meme[ make[ signal[ lose[ r[ random[ detect[ same[ check[ generated[ rule[ punishment[ rec[ is[ this[ start[ joke[ rhetorical[ coincidence[ which[ there[ speaker[ like[ credit[ link[ YouTube[ upload[ screenshot[ template[video[ format[ from[ be[ database[ screenshots[ Document[ table[ t[ thumbnail[ images[ array[ text[ c[ op[ menu[ shortcut[ rectangular[ cop[ delete[ clear[ ready[ to[ download[ show[.[ source[ terminal[ platform[ package[ app[ packaged[ linux[ project[ settings[ assets[ path[ files[ directory[ created[ sub[] directory[[ new[] file[ file[] place[[ base[[ point[ points[ items[ object[ date[ day[ hours[ minutes[ seconds[ minute[second[ hour[ columns[ ceiling[ lines[ figure[ line[ float[

[ winner[ pick[ win[ tie[ No[ submitted[ rules[ request[ requests[ procedure[ Submission[ form[ body[ design[ website[ installation[ repositories[ generates[ custom[ models[ made[ materials[ material[ names[ work[ tasks[ creatures[ robot[ landscapes[ games[ multiplayer[ competition[ thrilling[ adventure

Initially I believed that some of the erratic behavior was that the model was attempting to fill in any [tag] with random information, such as [reply response]. This was corrected by manually adding tokens [blank response] and [answer response] to the tokenizer's vocabulary.
Even when the answer and blank tokens are accounted for, some examples are still completely non sensical:

: [sep] : : : : : : : : : : : : : : : : : : : : : : : : : : : [sep]: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : [sep]: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : [sep]: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : [sep]: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : [sep]: : : : : : : : : : : : : : : : : : : : : : : : [sep]: : : : : : : : : : : : : : : : : : : : : : : : : : [sep] : : : : : : : : : : : : : : : : : : : : : : : : : : : [sep] : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : [sep] : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : [sep] : : : [sep] : : : : : : : : : : : : : : : : :

When reducing the token complexity by only using one single answer and blank token, more consistent instances of text correctly being generated and parsed occur for shorter texts. This suggests that having too many answer and blank tokens may be detrimental to text generation. like so:

max_length=50

: [blank response] anybody know how to avoid highjumping off[blank response]ushers[sep]Does[answer response] xp[answer response] xp[answer response] xp[answer response] xp[answer response]

Overall, what I have learned is that while it is still technically valid to for ILM to extend any language model, the requirements for extending said language model isn't exactly plug-and-play. Since existing language models are usually trained on natural language rather than manually segmented pieces of blank and answer tokens(as far as I'm aware of), an considerable amount of training is probably required for complex infilling where there are a lot of unique blank and answer tokens involved. SnooSpoof is intended to generate text on-demand quickly, so an approach where long training times are needed would not fit our requirements.