Have Geek Brain

How would you design a frequency-sorted list of the ten thousand most common words in English?

I was once asked by a current employee how I came up with a frequency-sorted list of ten thousand most used words in English. Suggest a solution in your language of choice, although I prefer C #.

Please provide not only an implementation but also an explanation.

thanks

+1


a source to share


4 answers 


IEnumerable<string> inputList; // input words.
var mostFrequentlyUsed = inputList.GroupBy(word => word)
  .Select(wordGroup => new { Word = wordGroup.Key, Frequency = wordGroup.Count() })
  .OrderByDescending(word => word.Frequency);

Explanation: I really don't know if this requires further explanation, but I'll try. inputList

is an array or any other collection providing the source words. The function GroupBy

will group the collection of inputs using some similar property (i.e., in my code, the object itself, as noted by the lambda word => word

). The output (which is a set of groups by a given key, word) will be converted into an object with properties Word

and Frequency

and sorted by property Frequency

in descending order. You can use this .Take(10000)

to get the first 10000. The whole thing can be easily parallelized with the help .AsParallel()

provided by PLINQ. The syntax for the query statement might look clearer:



var mostFrequentlyUsed = 
     (from word in inputList
      group word by word into wordGroup
      select new { Word = wordGroup.Key, Frequency = wordGroup.Count() })
     .OrderByDescending(word => word.Frequency).Take(10000);
+3


a source


There is no further definition of the problem as the first cut (what do you mean by the most used words in English?) - I would buy Google N-grammar data , cross 1 gram with an English dictionary and transfer to sort -rn -k 2 | head -10000

.



+2


a source


I would use map-reduce. This is a canonical example of a task well suited to him. You can use Hadoop from C # with streaming protocol . There are other approaches as well. See Is there a .NET equivalent for Apache Hadoop? and https://stackoverflow.com/questions/436686/-net-mapreduce-implementation .

+1


a source


First thing to put into my head (not syntax checked and verbose (for perl) for demo purposes)

#!/usr/bin/perl

my %wordFreq
foreach ( my $word in @words)
{
   $wordFreq{$word}++;
}

my @mostPopularWords = sort{$wordFreq{$a} <=> $wordFreq{$b} } keys %wordFreq;
for (my $i=0; $i < 10000; ++$i)
{
   print "$i: $mostPopularWords[$i] ($wordFreq{$mostPopularWords[$i]} hits)\n"
}
+1


a source






More articles:


All Articles