I will uild a character-level language model using the Brown corpus.
To do this, I used LgramModel from nltk model, It is a small modification of the NgramModel: NgramModel builds n-grams of words, while LgramModel builds n-grams of characters (letters).
In addition to the Brown corpus, I used a small snapshot of data generated by users of Twitter in a form of tweets – messages that are at most 280 characters, shared publicly by the users of the platform. All of the tweets that are in the data set we use are from 28th of January, 2010 (20100128.txt file in the twitter directory).
Lastly, I applied the character-level language model trained on the Brown corpus to each word in a tweet in order to analyse whether it is written in English or not.