Comments on: Ways to process and use Wikipedia dumps

By: bhupinder

bhupinder — Wed, 03 Jun 2009 19:27:21 +0000

Hi everybody there . Could just help me for the structure for tables revision and page also. i am able to get the three files named as text, page and revision using xmltosql tool. i have been able to get the “text” table structure here but not for rest of the two.if any body can help me out for them also. one can jsut help me by responding to my query here.

i will be thankful to that person. i know i can figure out some thing from data itself, but then also if probably some body has there something near to their structure, let me know .

i am still waiting for some kind of help

thanks,
bhupinder

By: b.b.goyal, barnala

b.b.goyal, barnala — Fri, 07 Nov 2008 07:13:56 +0000

sir, i want to display wikipedia contents on my site. is it possible? if yes, how? regrds.

By: Andy Bailey

Andy Bailey — Mon, 29 Sep 2008 11:00:41 +0000

Prashanth,
I managed to load the sql imports into the table ‘text’. It now holds just over 10,000,000 records but i can’t do anything with it. Everytime i try a select count(*) or a Select * limit 0,1 i get no return from MySQL. I may have to abandon all and use a mediaWiki method instead. Did you have any trouble accessing the data? One thing i’ve noticed is that the primary key index (old_id) is taking up no diskspace which seems a little weird.

By: Andy Bailey

Andy Bailey — Mon, 25 Aug 2008 00:11:55 +0000

Prashanth,
Thanks for a great post. I found i had to make some small alterations for mysql 5.0.21

CREATE DATABASE wiki;
use wiki;
create table text {
old_int int UNSIGNED NOT NULL, AUTO_INCREMENT,
old_text mediumblob NOT NULL,
old_flags tinyblob NOT NULL,
PRIMARY KEY (old_id)
} MAX_ROWS 10000000 AVG_ROW_LENGTH=10240;

By: prashanthellina

prashanthellina — Mon, 04 Aug 2008 15:57:32 +0000

Jared, the intra-wiki links are internal automatically.

By: Jared

Jared — Fri, 01 Aug 2008 20:52:04 +0000

Does anyone know how to make all the links internal to my domain after we do the dump?

By: prashanthellina

prashanthellina — Thu, 17 Jul 2008 16:09:11 +0000

Indrajeet, I did not understand your question. What are you trying to do?

By: prashanthellina

prashanthellina — Thu, 17 Jul 2008 16:08:25 +0000

Rich, way to go! That’s a wonderful way to learn. Am glad I was of help. Enjoy!
RK, will get back to you by mail shortly.

By: RK

RK — Tue, 08 Jul 2008 17:47:52 +0000

Mailed you but couldnt get a revert. You are the only person i know who can get this done

RKs last blog post..naishadh86 Intro

By: rich

rich — Tue, 08 Jul 2008 13:13:13 +0000

THANK YOU!! This is extremely helpful. I downloaded one of the dumps and had no clue what to do with it. I figured I would just start and tinker with it along the way (that’s how I learn everything for computers, treat it like a puzzle to solve and I end up teaching myself just about anything) but when I saw the dump would decompress into massively large files and take a while to do it, I decided to take a step back and slow down before I blow up my computer in the process. This helped me more than you can know, thanks so much!!!!

By: indrajeet

indrajeet — Mon, 07 Jul 2008 11:32:34 +0000

tell me, how to dump the database after installing mediawiki
please tell me i am waiting for ur reply.

thanks and regards
Indrajeet Dhanjode

By: prashanthellina

prashanthellina — Sun, 15 Jun 2008 05:12:29 +0000

RK, I prefer communicating via email. Please mail me at the same address and we can have a discussion.

By: RK

RK — Sat, 14 Jun 2008 15:35:38 +0000

Hello,

I’ve sent you an add request on gmail for a site we need done integrated with mediawiki and wikipedia database dump. PLease accept the add request so that we can discuss it in detail.

By: prashanthellina

prashanthellina — Fri, 23 May 2008 14:26:12 +0000

Thanks for the info, Totic.

By: totic

totic — Sun, 18 May 2008 07:06:25 +0000

I also had the problem with wget, it seems to always fail when the files are incredible large, my solution was to just
use curl

example:

curl http://download.wikimedia.org/enwiki/20070206/enwiki-20070206-pages-articles.xml.bz2 -o enwiki-20070206-pages-articles.xml.bz2

By: prashanthellina

prashanthellina — Sun, 27 Apr 2008 06:28:39 +0000

Great! It worked for me so I assume it is correct. I have not run this against the latest xml though.

By: someone

someone — Fri, 25 Apr 2008 13:23:31 +0000

I used your split script.._thanks_ a lot for it!
but are you sure that it is accurate? does it need to be updated?

By: prashanthellina

prashanthellina — Wed, 24 Oct 2007 14:55:19 +0000

Hi Satheesh,

My email is prashanthBLAHellina AT gmail DOTT com (remove BLAH).

By: satheesh nair

satheesh nair — Wed, 24 Oct 2007 14:24:12 +0000

Prashanth, Can we discuss a deal to develop a wikipedia mirror for a project of mine please. Please give me your contact details, I am in bangalore

By: senddesks » Ways to process and use Wikipedia dumps

senddesks » Ways to process and use Wikipedia dumps — Wed, 24 Oct 2007 02:59:58 +0000

[...] here for full [...]