Comments on: Ways to process and use Wikipedia dumps http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/ ( to ) ? be : ! be; Tue, 06 Jan 2009 23:43:21 +0000 http://wordpress.org/?v=2.5 By: b.b.goyal, barnala http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-10138 b.b.goyal, barnala Fri, 07 Nov 2008 07:13:56 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-10138 sir, i want to display wikipedia contents on my site. is it possible? if yes, how? regrds. sir, i want to display wikipedia contents on my site. is it possible? if yes, how? regrds.

]]> By: Andy Bailey http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-7924 Andy Bailey Mon, 29 Sep 2008 11:00:41 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-7924 Prashanth, I managed to load the sql imports into the table 'text'. It now holds just over 10,000,000 records but i can't do anything with it. Everytime i try a select count(*) or a Select * limit 0,1 i get no return from MySQL. I may have to abandon all and use a mediaWiki method instead. Did you have any trouble accessing the data? One thing i've noticed is that the primary key index (old_id) is taking up no diskspace which seems a little weird. Prashanth,
I managed to load the sql imports into the table ‘text’. It now holds just over 10,000,000 records but i can’t do anything with it. Everytime i try a select count(*) or a Select * limit 0,1 i get no return from MySQL. I may have to abandon all and use a mediaWiki method instead. Did you have any trouble accessing the data? One thing i’ve noticed is that the primary key index (old_id) is taking up no diskspace which seems a little weird.

]]> By: Andy Bailey http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-5875 Andy Bailey Mon, 25 Aug 2008 00:11:55 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-5875 Prashanth, Thanks for a great post. I found i had to make some small alterations for mysql 5.0.21 CREATE DATABASE wiki; use wiki; create table text { old_int int UNSIGNED NOT NULL, AUTO_INCREMENT, old_text mediumblob NOT NULL, old_flags tinyblob NOT NULL, PRIMARY KEY (old_id) } MAX_ROWS 10000000 AVG_ROW_LENGTH=10240; Prashanth,
Thanks for a great post. I found i had to make some small alterations for mysql 5.0.21

CREATE DATABASE wiki;
use wiki;
create table text {
old_int int UNSIGNED NOT NULL, AUTO_INCREMENT,
old_text mediumblob NOT NULL,
old_flags tinyblob NOT NULL,
PRIMARY KEY (old_id)
} MAX_ROWS 10000000 AVG_ROW_LENGTH=10240;

]]> By: prashanthellina http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-5263 prashanthellina Mon, 04 Aug 2008 15:57:32 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-5263 Jared, the intra-wiki links are internal automatically. Jared, the intra-wiki links are internal automatically.

]]> By: Jared http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-5202 Jared Fri, 01 Aug 2008 20:52:04 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-5202 Does anyone know how to make all the links internal to my domain after we do the dump? Does anyone know how to make all the links internal to my domain after we do the dump?

]]> By: prashanthellina http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4769 prashanthellina Thu, 17 Jul 2008 16:09:11 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4769 Indrajeet, I did not understand your question. What are you trying to do? Indrajeet, I did not understand your question. What are you trying to do?

]]> By: prashanthellina http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4768 prashanthellina Thu, 17 Jul 2008 16:08:25 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4768 Rich, way to go! That's a wonderful way to learn. Am glad I was of help. Enjoy! RK, will get back to you by mail shortly. Rich, way to go! That’s a wonderful way to learn. Am glad I was of help. Enjoy!
RK, will get back to you by mail shortly.

]]> By: RK http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4567 RK Tue, 08 Jul 2008 17:47:52 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4567 Mailed you but couldnt get a revert. You are the only person i know who can get this done :) RKs last blog post..<a href="http://www.managementparadise.com/forums/introduce-yourself/34682-naishadh86-intro.html" rel="nofollow">naishadh86 Intro</a> Mailed you but couldnt get a revert. You are the only person i know who can get this done :)
RKs last blog post..naishadh86 Intro

]]> By: rich http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4562 rich Tue, 08 Jul 2008 13:13:13 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4562 THANK YOU!! This is extremely helpful. I downloaded one of the dumps and had no clue what to do with it. I figured I would just start and tinker with it along the way (that's how I learn everything for computers, treat it like a puzzle to solve and I end up teaching myself just about anything) but when I saw the dump would decompress into massively large files and take a while to do it, I decided to take a step back and slow down before I blow up my computer in the process. This helped me more than you can know, thanks so much!!!! THANK YOU!! This is extremely helpful. I downloaded one of the dumps and had no clue what to do with it. I figured I would just start and tinker with it along the way (that’s how I learn everything for computers, treat it like a puzzle to solve and I end up teaching myself just about anything) but when I saw the dump would decompress into massively large files and take a while to do it, I decided to take a step back and slow down before I blow up my computer in the process. This helped me more than you can know, thanks so much!!!!

]]> By: indrajeet http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4538 indrajeet Mon, 07 Jul 2008 11:32:34 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4538 tell me, how to dump the database after installing mediawiki please tell me i am waiting for ur reply. thanks and regards Indrajeet Dhanjode tell me, how to dump the database after installing mediawiki
please tell me i am waiting for ur reply.

thanks and regards
Indrajeet Dhanjode

]]> By: prashanthellina http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4339 prashanthellina Sun, 15 Jun 2008 05:12:29 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4339 RK, I prefer communicating via email. Please mail me at the same address and we can have a discussion. RK, I prefer communicating via email. Please mail me at the same address and we can have a discussion.

]]> By: RK http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4334 RK Sat, 14 Jun 2008 15:35:38 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4334 Hello, I've sent you an add request on gmail for a site we need done integrated with mediawiki and wikipedia database dump. PLease accept the add request so that we can discuss it in detail. Hello,

I’ve sent you an add request on gmail for a site we need done integrated with mediawiki and wikipedia database dump. PLease accept the add request so that we can discuss it in detail.

]]> By: prashanthellina http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4146 prashanthellina Fri, 23 May 2008 14:26:12 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4146 Thanks for the info, Totic. Thanks for the info, Totic.

]]> By: totic http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4076 totic Sun, 18 May 2008 07:06:25 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-4076 I also had the problem with wget, it seems to always fail when the files are incredible large, my solution was to just use curl example: curl http://download.wikimedia.org/enwiki/20070206/enwiki-20070206-pages-articles.xml.bz2 -o enwiki-20070206-pages-articles.xml.bz2 I also had the problem with wget, it seems to always fail when the files are incredible large, my solution was to just
use curl

example:

curl http://download.wikimedia.org/enwiki/20070206/enwiki-20070206-pages-articles.xml.bz2 -o enwiki-20070206-pages-articles.xml.bz2

]]> By: prashanthellina http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-3654 prashanthellina Sun, 27 Apr 2008 06:28:39 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-3654 Great! It worked for me so I assume it is correct. I have not run this against the latest xml though. Great! It worked for me so I assume it is correct. I have not run this against the latest xml though.

]]> By: someone http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-3556 someone Fri, 25 Apr 2008 13:23:31 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-3556 Hi I used your split script.._thanks_ a lot for it! but are you sure that it is accurate? does it need to be updated? Hi

I used your split script.._thanks_ a lot for it!
but are you sure that it is accurate? does it need to be updated?

]]> By: prashanthellina http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-198 prashanthellina Wed, 24 Oct 2007 14:55:19 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-198 Hi Satheesh, My email is prashanthBLAHellina AT gmail DOTT com (remove BLAH). Hi Satheesh,

My email is prashanthBLAHellina AT gmail DOTT com (remove BLAH).

]]> By: satheesh nair http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-197 satheesh nair Wed, 24 Oct 2007 14:24:12 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-197 Prashanth, Can we discuss a deal to develop a wikipedia mirror for a project of mine please. Please give me your contact details, I am in bangalore Prashanth, Can we discuss a deal to develop a wikipedia mirror for a project of mine please. Please give me your contact details, I am in bangalore

]]> By: senddesks » Ways to process and use Wikipedia dumps http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-195 senddesks » Ways to process and use Wikipedia dumps Wed, 24 Oct 2007 02:59:58 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-195 [...] here for full [...] […] here for full […]

]]> By: prashanthellina http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-138 prashanthellina Thu, 18 Oct 2007 02:25:20 +0000 http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comment-138 Interesting, I tried wget from my local machine and It was downloading alright. I want to get the processed data to be accessible through some kind of API so anybody can "query" the database over HTTP. Instead of getting 5GB to my disk and then uploading back to Dreamhost, I felt it better to do it there itself. I have a 128kbps "broadband" connection, so you can imagine the upload rate. Interesting, I tried wget from my local machine and It was downloading alright.

I want to get the processed data to be accessible through some kind of API so anybody can “query” the database over HTTP. Instead of getting 5GB to my disk and then uploading back to Dreamhost, I felt it better to do it there itself. I have a 128kbps “broadband” connection, so you can imagine the upload rate.

]]>