Comments on: N-gram data from Project Gutenberg http://blog.prashanthellina.com/2008/05/04/n-gram-data-from-project-gutenberg/ ( to ) ? be : ! be; Tue, 06 Jan 2009 23:09:14 +0000 http://wordpress.org/?v=2.5 By: prashanthellina http://blog.prashanthellina.com/2008/05/04/n-gram-data-from-project-gutenberg/#comment-3814 prashanthellina Mon, 05 May 2008 06:05:57 +0000 http://blog.prashanthellina.com/?p=63#comment-3814 I am glad you enjoy my blog! I did not attempt doing Ngram generation on Wikipedia data, so I don't have any hard numbers to support my argument. However, I'll explain my reasoning. I need the Ngram data for a feature of Wordza where the user gets to see the most frequently used phrases containing a given word. Eg: "abysmal corruption", "abysmal conditions" for the word abysmal. Wikipedia being an encyclopedia will tend to have a sanitized usage of English where many of the "non-frequent" words in english don't even occur. In comparison, Gutenberg data is better suited because it is English literature. When I can I'll try to generate Ngram data for Wikipedia. It will be interesting to compare the results. I am glad you enjoy my blog! I did not attempt doing Ngram generation on Wikipedia data, so I don’t have any hard numbers to support my argument. However, I’ll explain my reasoning. I need the Ngram data for a feature of Wordza where the user gets to see the most frequently used phrases containing a given word. Eg: “abysmal corruption”, “abysmal conditions” for the word abysmal. Wikipedia being an encyclopedia will tend to have a sanitized usage of English where many of the “non-frequent” words in english don’t even occur. In comparison, Gutenberg data is better suited because it is English literature. When I can I’ll try to generate Ngram data for Wikipedia. It will be interesting to compare the results.

]]> By: Som http://blog.prashanthellina.com/2008/05/04/n-gram-data-from-project-gutenberg/#comment-3812 Som Mon, 05 May 2008 03:50:05 +0000 http://blog.prashanthellina.com/?p=63#comment-3812 Hi Prashant, Thanks for the wonderful blog you do. I was curious to know what kind of problem you faced with Wikipedia and that is not there in the Gutenberg data. May be the related question is how you use the N-gram statistics in Wordza and why you think the statistics obtained from Gutenberg data is better. Thanks, Som Hi Prashant,

Thanks for the wonderful blog you do. I was curious to know what kind of problem you faced with Wikipedia and that is not there in the Gutenberg data. May be the related question is how you use the N-gram statistics in Wordza and why you think the statistics obtained from Gutenberg data is better.

Thanks,
Som

]]>