Lakhasly

By looking at spoken and written corpora collected from a wide range of users and everyday contexts, we can make fairly reliable statements about how many words are 'in circulation' in everyday communication among native speakers.Conversely, the software often fails to perceive obvious similarities such as young/younger/youngest, so, without considerable manual reprocessing of tens of thousands of words, lemmatised counts can be unreliable: the form bit as in a little bit / a bit small, etc.professional, technical and scientific words, colloquialisms and dialect words), but it will certainly enable us to list the vocabulary of common usage which users are likely to encounter in their daily conversations and in their routine reading of newspapers, magazines, novels, internet texts, etc.This is not to say that a corpus of, say, ten million words can capture all the words of a language, and some users will always know obscure and rare words, for all sorts of reasons, but it will certainly enable us to list the vocabulary of common usage which users are likely to encounter in their daily conversations and in their routine reading of newspapers, magazines, novels, internet texts, etc.However, the computer software also allows us to bring together semi-automatically the inflected forms of words and treat the combined totals as 'lemmas' (i.e. LOOK would be one lemma composed of the total occurrences of look(s), looked and looking).Firstly, the process of lemmatisation tends to bundle together all the forms that the computer judges to look similar, for example the quite different nouns man and mane, or the noun bit, which will be conflated with the past-tense form of the verb to bite.The frequency curve does not decline at a regular rate across the whole of the vocabulary; there is a continental shelf of high-frequency, core items, after which the curve takes a nose-dive into the vast depths of tens of thousands of (relatively) low-frequency words.There are, however, reasons for hesitating in taking lemmas as our benchmark.

By looking at spoken and written corpora collected from a wide range of users and everyday contexts, we can make fairly reliable statements about how many words are 'in circulation' in everyday communication among native speakers. This is not to say that a corpus of, say, ten million words can capture all the words of a language, and some users will always know obscure and rare words, for all sorts of reasons, but it will certainly enable us to list the vocabulary of common usage which users are likely to encounter in their daily conversations and in their routine reading of newspapers, magazines, novels, internet texts, etc. professional, technical and scientific words, colloquialisms and dialect words), but it will certainly enable us to list the vocabulary of common usage which users are likely to encounter in their daily conversations and in their routine reading of newspapers, magazines, novels, internet texts, etc. If we examine the frequency of words in a large corpus of English, a picture emerges where the first 2000 or so word-forms do most of the work, accounting for more than 80% of all of the words in spoken and written texts. As we progress down the frequency list, each successive band of 2000 words covers a progressively smaller proportion of all the words in the texts in the corpus, with many words occurring only a small number of times or, indeed, only once. Figure 1 shows the power of the first 2000 most frequent word-forms in a mixed corpus of ten million words of English (made up of five million words of spoken data, from CANCODE, and five million words of written data taken from the Cambridge International Corpus
Figure1 shows the coverage achieved by word-forms, that is to say, the computer considers look(s), looking and looked as different ‘words’. However, the computer software also allows us to bring together semi-automatically the inflected forms of words and treat the combined totals as ‘lemmas’ (i.e. LOOK would be one lemma composed of the total occurrences of look(s), looked and looking). There are, however, reasons for hesitating in taking lemmas as our benchmark. Firstly, the process of lemmatisation tends to bundle together all the forms that the computer judges to look similar, for example the quite different nouns man and mane, or the noun bit, which will be conflated with the past-tense form of the verb to bite. Conversely, the software often fails to perceive obvious similarities such as young/younger/youngest, so, without considerable manual reprocessing of tens of thousands of words, lemmatised counts can be unreliable: the form bit as in a little bit / a bit small, etc. is vastly more frequent than the past tense of the verb bite, but a lemmatised count might suggest the lemma BITE to be very frequent because it conflates all the occurrences of bit. Secondly, there is no reason to suppose learners do always make the necessary connections between forms of the same lemma in listening or reading; for example, does the learner necessarily associate the first encounter with stuck with the lemma STICK? Will flown be associated with FLY, and so on? These are technical questions and, in the last analysis, neither solution is entirely satisfactory. In most cases learners will be able to extrapolate that look(s), looking and looked are different forms of the same item, and if we do choose to consider lemmas rather than individual word-forms, then the total word-learning burden is considerably less, and the first2000 lemmas will typically cover up to 90% of the items in an everyday text. Whichever is chosen as the benchmark, however, whether word-form or lemma, the picture is strikingly similar, with a hard-working core vocabulary separated from the low frequency, massive bulk of most of the words of the language. The frequency curve does not decline at a regular rate across the whole of the vocabulary; there is a continental shelf of high-frequency, core items, after which the curve takes a nose-dive into the vast depths of tens of thousands of (relatively) low-frequency words.