Language Models

Language models assign probability to sequences of words. They have many applications, including machine translation, smartphone typing, information retrieval, though I’m familiar with them through speech recognition.

For many years, the probabilities of N-Grams – that’s words or sequences of words – have been estimated by counting occurrences.

Screen Shot 2014-06-02 at 22.30.29


One of the key problems for speech recognition is obtaining text that represents the way we speak. The web and other archived resources contain a large amount of written text, but the probabilities estimated from these do not match the way that people speak ungrammatically, and with hesitation, correction, um’s, ah’s and er’s etc. It is much more expensive and labour intensive to obtain transcribed spontaneous speech.

More recently, neural network models have had some success for language modelling, there’s a publicly released toolkit available. The amount of data available for language modelling has increased, and Google have recently released a 1 billion word language model project.

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s