Aliaser works by monitoring your bash history. It analyses command frequency and suggests candidates for aliasing. It manages aliases so created. The feature I like most in Aliaser is that it reminds you to use the aliases you created by showing tips on opening a new terminal session.

Download Aliaser from http://aliaser.googlecode.com.

Usage

prashanth@prashanth-desktop:~$ translate bonjour
hello
prashanth@prashanth-desktop:~$ translate guten morgen
Good morning

Code

#!/usr/bin/env python
'''
Translates text into english using Google Translate.
Usage: python translate.py 
        (or)
       echo  | python translate.py

For convenience, make a symlink to this file from /usr/bin/translate.
'''
# derived from : http://code.google.com/p/py-gtranslate/source/browse/trunk/gtrans.py

import sys
import urllib2
import urllib
import simplejson as json

FROM_LANGUAGE = ''
TO_LANGUAGE = 'en'
BASE_URL = 'http://ajax.googleapis.com/ajax/services/language/translate'

def translate(from_language, to_language, text):
    langpair = '%s|%s' % (from_language, to_language)
    params = {'v': '1.0', 'langpair': langpair, 'q': urllib.quote_plus(text)}

    params = '%s' % ('&'.join(['%s=%s' % (k,v) for (k,v) in params.items()]))

    url = '%s?%s' % (BASE_URL, params)
    resp = json.load(urllib2.urlopen(url))
    try:
        return resp['responseData']['translatedText']
    except:
        return text

def main(text):
    if text:
        print translate(FROM_LANGUAGE, TO_LANGUAGE, text)

    else:
        lines = [l.strip() for l in sys.stdin.readlines()]
        for line in lines:
            if line:
                text = translate(FROM_LANGUAGE, TO_LANGUAGE, line)
                print '[%s]' % line
                print text
            else:
                print

if __name__ == '__main__':
    args = sys.argv[1:]
    text = ' '.join(sys.argv[1:])
    main(text)

This algorithm has the following steps:

• make doc from html data (clean html)
• identify content nodes (nodes having substantial content)
• prune xml tree to remove irrelevant nodes
• get the most linked node from pruned tree (subtree contains relevant text)
• make the dot graph

I’ve pasted the relevant python module below for easy reading. However, if you want to download the code and hack it, you can get all the files from here.

Code files

• content_extract.py – actual work gets done here (file pasted below)
• cextract.py – cgi front-end which fetched url content and feeds to above script.
• cextract_config.py – cgi script configuration file. You have to adjust this to your environment.

Try it right here and right now

Some samples

• http://news.bbc.co.uk/sport2/hi/motorsport/formula_one/8169436.stm
• http://www.prashanthellina.com/cextract_data/640ca0b7c6e818b2b1bf952a206a6388.html
• http://www.telegraph.co.uk/news/worldnews/europe/france/5913494/Nicolas-Sarkozy-to-slow-down-after-collapsing-while-jogging.html

Please let me know if you find cases for which the algorithm does not work. Even better would be to download the code and hack it up and post back. I am eager to see what you can come up with.

#!/usr/bin/env python

import sys
from cStringIO import StringIO

from lxml import etree #http://codespeak.net/lxml/

IGNORABLE_TAGS = set(['script', 'a'])
MIN_TEXT_LEN = 50

def get_text(node):
    '''
    Given a XML node, extract all the text it contains.
    (does not recurse into children)
    '''
    text = [node.text or '']
    for cnode in node.getchildren():
        tail = cnode.tail
        if tail is not None:
            text.append(cnode.tail)

    text = '\n'.join(text).strip()
    return text

def get_xml(node):
    '''
    Convert the sub-tree from node downwards
    into string XML representation.
    '''
    return etree.tostring(node)

def create_doc(data):
    '''
    Construct XML tree datastructure from xml string representation.
    '''
    parser = etree.HTMLParser()
    doc = etree.parse(StringIO(data), parser)
    return doc

def get_content_nodes(doc):
    '''
    Identify nodes in the XML document that
    have substantial text.
    '''
    nodes = []

    for n in doc.xpath('//*'):
        tag = n.tag

        if tag.lower() in IGNORABLE_TAGS:
            continue

        text = get_text(n)
        if not text:
            continue

        if len(text) < MIN_TEXT_LEN:
            continue

        nodes.append(n)

    return nodes

def make_pruned_tree(content_nodes):
    '''
    Prune the whole XML tree by remnoving nodes
    other than content nodes and their ancestors.
    '''
    nodes = {}
    links = {}

    for node in content_nodes:

        nodes[id(node)] = node

        parent = node.getparent()
        if parent is not None:
            links[id(node)] = id(parent)

        for anode in node.iterancestors():
            _id = id(anode)
            parent = anode.getparent()
            if parent is not None:
                links[_id] = id(parent)

            if _id not in nodes:
                nodes[_id] = anode

    return nodes, links

def get_inlink_counts(links):
    '''
    Given the inter-node links, find out which
    node has maximum number of links coming into it.
    '''
    counts = {}

    for from_id, to_id in links.iteritems():
        count = counts.setdefault(to_id, 0)
        counts[to_id] = count + 1

    return counts

def get_most_linked_node(nodes, links):
    '''
    Identify the node which is most linked.
    (i,e) has most number of inlinks.
    '''
    inlink_counts = get_inlink_counts(links)

    mcount, mid = max([(count, _id) for _id, count in inlink_counts.iteritems()])
    node = nodes[mid]
    return node

def make_dot_graph(nodes, links, chosen_node, stream):
    '''
    Construct the dot format graph representation
    so that graphviz can render the tree for visualization.
    '''
    o = stream

    print >> o, "digraph G {"

    for _id, node in nodes.iteritems():

        tlen = len(get_text(node))
        tag = node.tag

        if tlen:
            text = '%s (%d)' % (tag, tlen)
        else:
            text = tag

        if _id == chosen_node:
            attrs = 'style=filled color=lightblue'
        else:
            attrs = ''

        print >> o, "%s [label=\"%s\" %s];" % (_id, text, attrs)

    for fid, tid in links.iteritems():
        print >> o, "%d -> %d;" % (fid, tid)

    print >> o, "}"

def main():
    # make doc from html data (cleans html)
    doc = create_doc(sys.stdin.read())

    # identify content nodes
    content_nodes = get_content_nodes(doc)

    # prune xml tree to remove irrelevant nodes
    nodes, links = make_pruned_tree(content_nodes)

    # get the most linked node from pruned tree
    mnode = get_most_linked_node(nodes, links)

    # make the dot graph
    make_dot_graph(nodes, links, id(mnode), sys.stdout)

if __name__ == '__main__':
    #Eg: wget "http://blog.prashanthellina.com" -O - | python thisscript.py | dot -Tpng -o /tmp/test.png ; eog /tmp/test.png
    main()

Code

import pprint
from difflib import SequenceMatcher

# http://python-cluster.sourceforge.net/
from cluster import HierarchicalClustering

# input urls to be clustered
urls = [
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814385',
    '#articles',
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814335',
    'http://yro.slashdot.org/~drDugan/',
    'http://web.sourceforge.com/privacy.php',
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28815123',
    'http://slashdot.org//slashdot.org/~Darkness404',
    'http://slashdot.org//radio.slashdot.org',
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814429',
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814457',
    'http://slashdot.org//slashdot.org/article.pl?sid=09/07/24/1545238',
    'http://slashdot.org//slashdot.org/comments.pl?sid=09/07/24/1545238&cid=28810581',
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28815269',
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814657',
    'http://web.sourceforge.com/terms.php'
    'http://slashdot.org//it.slashdot.org/search',
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814581',
    'http://xkcd.com/612/',
    'http://web.sourceforge.com/advertising',
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814785',
]

# distance function compares two urls and finds the distance
# uses SequenceMatcher from python standard module difflib
def distance(url1, url2):
    ratio = SequenceMatcher(None, url1, url2).ratio()
    return 1.0 - ratio

# Perform clustering
hc = HierarchicalClustering(urls, distance)
clusters = hc.getlevel(0.2)

pprint.pprint(clusters)

Output

[['#articles'],
 ['http://xkcd.com/612/'],
 ['http://web.sourceforge.com/privacy.php'],
 ['http://web.sourceforge.com/advertising'],
 ['http://web.sourceforge.com/terms.phphttp://slashdot.org//it.slashdot.org/search'],
 ['http://yro.slashdot.org/~drDugan/'],
 ['http://slashdot.org//slashdot.org/~Darkness404'],
 ['http://slashdot.org//radio.slashdot.org'],
 ['http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814785',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814429',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814457'],
 ['http://slashdot.org//slashdot.org/article.pl?sid=09/07/24/1545238',
  'http://slashdot.org//slashdot.org/comments.pl?sid=09/07/24/1545238&cid=28810581',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28815123',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28815269',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814385',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814335',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814657',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814581']]

Note that you will need to have Python installed to try out the script and the examples in this article. I believe that the code is simple enough to be understood with minimal or no understanding of the Python language. If any part of the article is not clear, feel free to point it out to me. Thank you.

Estimating difficulty

Let us take a look at the following problems.

Problem 1

5 +
3

Problem 2

99999 +
12345

Now, which problem is more difficult? It is quite obvious that Problem 2 is more difficult. But why? One answer would be – “because Problem 2 involves the addition of two large numbers”. Fine. Consider this now …

Problem 3

1000000 +
1000001

Is Problem 3 more difficult than Problem 2 because the numbers undergoing addition are bigger? The intuitive answer is “No”. But why? Because the time it takes to compute the result is lesser. Why? Because the number of operations performed during the course of solving Problem 3 are lesser than those of Problem 2? No.

Problem 3 is easier than Problem 2 because the nature of the operations performed in Problem 3. The operations there are simpler.

0 + 1,
0 + 0,
1 + 1

are arguably simpler operations than

9 + 5,
9 + 2,
4 + 9

I’ve attempted to capture the basic operations we perform while doing arithmetic and assign a level of difficulty to each. I’ve then emulated the algorithms we follow to compute solutions to such problems. By doing the following, it becomes possible to estimate the “difficulty” of an arithmetic problem (involving the basic operations).

Operation difficulties

Addition

addition_difficulties = {
    'digit_zero' : 1,   # any digit added to zero
    'even_even'  : 2,   # sum of even digits
    'odd_odd'    : 2,   # sum of odd digits
    'even_odd'   : 3,   # sum of even and odd digits
    'carry'      : 2    # difficulty of carry (for remembering and then adding)
}

Here is the function which computes the addition difficulty. It is a recursive function that can compute the addition difficulty for n numbers. It computes the sum and difficulty of the first two numbers and then inserts the sum at the head of the list of n numbers in place of the two numbers just summed. It then calls itself with the modified list.

def compute_addition_difficulty(*numbers):
    '''
    Generates a difficulty number and sum for given
    list of numbers for the addition operation
    *args - integers
    returns: (sum, difficulty)

    >>> compute_addition_difficulty(0, 0)
    (0, 1)

    >>> compute_addition_difficulty(0, 1)
    (1, 1)

    >>> compute_addition_difficulty(1, 1)
    (2, 2)

    >>> compute_addition_difficulty(1, 2)
    (3, 3)

    >>> compute_addition_difficulty(2, 2)
    (4, 2)

    >>> compute_addition_difficulty(19, 9)
    (28, 5)

    >>> compute_addition_difficulty(99999, 12345)
    (112344, 22)
    '''

    numbers = list(numbers)
    if len(numbers) == 0: return 0, 0

    elif len(numbers) == 1: return numbers[0], 0

    num1, num2 = numbers[:2]
    num1 = str(num1)
    num2 = str(num2)

    max_length = max([len(num1), len(num2)])
    num1 = num1.rjust(max_length, '0')
    num2 = num2.rjust(max_length, '0')

    difficulty = 0
    carry = 0
    r = reversed
    ad = addition_difficulties
    sum = []

    for index, (d1, d2) in enumerate( izip( r(num1), r(num2) ) ):
        d1 = int(d1)
        d2 = int(d2)

        d1_is_even = is_even(d1)
        d2_is_even = is_even(d2)

        if not d1 or not d2: difficulty += ad['digit_zero']
        elif d1_is_even and d2_is_even: difficulty += ad['even_even']
        elif not d1_is_even and not d2_is_even: difficulty += ad['odd_odd']
        elif d1_is_even != d2_is_even: difficulty += ad['even_odd']

        dsum = d1 + d2 + carry

        if dsum > 9:
            carry = 1
            difficulty += ad['carry']
        else:
            carry = 0

        sum.append( str(dsum % 10) )

    if carry:
        sum.append( str(carry) )

    sum.reverse()
    sum = ''.join(sum)
    sum = int (sum)

    numbers = [sum] + numbers[2:]

    sum, sub_difficulty = compute_addition_difficulty(*numbers)

    difficulty += sub_difficulty

    return sum, difficulty

Subtraction

subtraction_difficulties = {
    'digit_zero' : 1,    # difference of zero and any digit
    'same_digits': 1,    # difference of same values digits
    'even_even'  : 2,    # difference of even digits
    'odd_odd'    : 2,    # difference of odd digits
    'even_odd'   : 3,    # difference of even and odd digits
    'borrow'     : 2,    # doing a borrow
    'twodigit_digit' : 4 # difference of two-digit number and digit
}

The subtraction function is also recursive and operates similar to addition in that respect. Before looking at the subtraction algorithm, take a look at how “borrowing” is implemented. Let us say we are trying to perform the following,

200005 -
     6

After borrowing the number would be, 19999(1)5. We would then subtract 6 from 15 and proceed. The do_borrow() function captures this aspect of substraction.

def do_borrow(num, index):
    if index == len(num)-1: raise Exception('cannot perform borrow')

    num_borrows = 1
    next_digit = int(num[index+1])

    if next_digit > 0:
        num[index+1] = str(next_digit-1)

    else:
        num[index+1] = str(9)
        num_borrows += do_borrow(num, index+1)

    return num_borrows

This implementation is a bit confusing so check out this example (do_borrow() applied to the problem mentioned above).

>>> x = list('200005')
>>> x.reverse()
>>> x
['5', '0', '0', '0', '0', '2']
>>> do_borrow(x, 1)
4
>>> ''.join(x)
'509991'
>>> x.reverse()
>>> x
['1', '9', '9', '9', '0', '5']

Note that do_borrow returned 4, which indicates that the difficulty of performing this borrow operation is 4. Also note that do_borrow takes a list of digits in reversed order. It takes a list because the result is provided in-place (remember that Python strings are immutable). Now that you have seen how the borrowing works, all introduce the subtraction function.

def compute_subtraction_difficulty(*numbers):
    '''
    Generates a difficulty number and result for given
    list of numbers for the subtraction operation
    *args (tuple) - integers
    returns: (difference, difficulty)

    >>> compute_subtraction_difficulty(0,0)
    (0, 1)

    >>> compute_subtraction_difficulty(1,0)
    (1, 1)

    >>> compute_subtraction_difficulty(1,1)
    (0, 1)

    >>> compute_subtraction_difficulty(2,1)
    (1, 3)

    >>> compute_subtraction_difficulty(4,2)
    (2, 2)

    >>> compute_subtraction_difficulty(4,3)
    (1, 3)

    >>> compute_subtraction_difficulty(100,1)
    (99, 10)

    >>> compute_subtraction_difficulty(5000007,9)
    (4999998, 22)
    '''

    numbers = list(numbers)
    if len(numbers) == 0: return 0, 0
    elif len(numbers) == 1: return numbers[0], 0

    num1, num2 = numbers[:2]
    if num1 < num2: num1, num2 = num2, num1

    num1 = str(num1)
    num2 = str(num2)

    max_length = max([len(num1), len(num2)])
    num1 = list(num1.rjust(max_length, '0'))
    num2 = list(num2.rjust(max_length, '0'))

    difficulty = 0
    borrow = 0
    sd = subtraction_difficulties
    difference = []
    num1.reverse()
    num2.reverse()

    for index, (d1, d2) in enumerate( izip( num1, num2 ) ):
        d1 = int(d1)
        d2 = int(d2)

        d1_is_even = is_even(d1)
        d2_is_even = is_even(d2)

        if d1 > d2:
            if not d1 or not d2: difficulty += sd['digit_zero']
            elif d1_is_even and d2_is_even: difficulty += sd['even_even']
            elif not d1_is_even and not d1_is_even: difficulty += sd['odd_odd']
            elif d1_is_even != d2_is_even: difficulty += sd['even_odd']
            ddiff = d1 - d2

        elif d1 < d2:
            num_borrows = do_borrow(num1, index)
            difficulty += sd['borrow']*num_borrows
            ddiff = 10 + d1 - d2
            difficulty += sd['twodigit_digit']

        elif d1 == d2:
            difficulty += sd['same_digits']
            ddiff = 0

        difference.append( str(ddiff) )

    difference.reverse()
    difference = ''.join(difference)
    difference = int (difference)

    numbers = [difference] + numbers[2:]

    difference, sub_difficulty = compute_subtraction_difficulty(*numbers)

    difficulty += sub_difficulty

    return difference, difficulty

Multiplication

multiplication_difficulties = {
    'carry'  : 1,   # carry operation, summation difficulty is seperate
    'offset' : 1,
}

The basic operation of multiplying big numbers is one digit by one digit multiplication. I've tried to capture the difficulty of this operation as follows

def get_single_digit_multiplication_difficulty(d1, d2):
    '''
    Max value of multiplication of two digits is 81 (9*9).
    Difficulty of d1*d2 increases as the resulting value
    of the multiplication increases. thereby 9*9=81 is most difficult
    and 0*0 is least difficult.

    We will represent this difficult with value from 1-4 (inclusive)

    Note that the difficulty will be incremented by 1 for presence
    of odd digit in the operands (except for 1 and 5 because it is arguably
    easier to multiply)

    >>> get_single_digit_multiplication_difficulty(0, 1)
    (0, 1)
    >>> get_single_digit_multiplication_difficulty(1, 1)
    (1, 1)
    >>> get_single_digit_multiplication_difficulty(2, 5)
    (10, 1)
    >>> get_single_digit_multiplication_difficulty(2, 7)
    (14, 2)
    >>> get_single_digit_multiplication_difficulty(2, 6)
    (12, 1)
    >>> get_single_digit_multiplication_difficulty(8, 9)
    (72, 5)
    >>> get_single_digit_multiplication_difficulty(8, 
    (64, 4)
    '''

    if not d1 or not d2: return 0, 1 

    res = d1 * d2
    difficulty = math.ceil((4/81.) * res)

    # odd numbers are harder to multiply except 1 and 5
    if d1 not in [1,5] and d2 not in [1,5] and (is_odd(d1) or is_odd(d2)):
        difficulty += 1

    return res, int(difficulty)

At the next level, the operation would be multiplying a multi-digit number by a single digit, which I call simple multiplication. Over here I take into account the difficulty of carry. I hope the following code is self explanatory.

def compute_simple_multiplication_difficulty(num, digit):
    '''
    >>> compute_simple_multiplication_difficulty(1, 0)
    (0, 1)
    >>> compute_simple_multiplication_difficulty(1, 1)
    (1, 1)
    >>> compute_simple_multiplication_difficulty(2, 1)
    (2, 1)
    >>> compute_simple_multiplication_difficulty(10, 1)
    (10, 2)
    >>> compute_simple_multiplication_difficulty(15, 1)
    (15, 2)
    >>> compute_simple_multiplication_difficulty(15, 5)
    (75, 7)
    >>> compute_simple_multiplication_difficulty(999, 7)
    (6993, 25)
    '''
    num = str(num)

    result = []
    md = multiplication_difficulties
    carry = 0
    difficulty = 0

    for index, d in enumerate(reversed(num)):
        d = int(d)
        res, s_diff = get_single_digit_multiplication_difficulty(d, digit)
        difficulty += s_diff

        if (carry):
            res, carry_sum_difficulty = compute_addition_difficulty(res, carry)
            difficulty += carry_sum_difficulty + md['carry']

        carry = res/10
        result_digit = int(str(res)[-1:])
        result.append( str(result_digit) )

    if carry:
        result.append( str(carry) )

    result.reverse()
    result = ''.join(result)
    result = int(result)

    return result, difficulty

Simple multiplication is a basic operation of complex multiplication (multi-digit multiplied by multi-digit). Let us see an example of "complex multiplication" which the multiplication code follows.

    345 x
    123
  -----
   1035
   690x  # result is "offset" or multiplied by 10
  345xx  # result if "offset" or multiplied by 100
  -----
  42435  # result of summation of above numbers
  -----

The first part of the multiplication is to perform simple multiplications - 345x3, 345x2, 345x1 and to offset the numbers accordingly. The second part of the operation is to sum the numbers. I used the addition algorithm to achieve the second part. Check out the multiplication function.

def compute_multiplication_difficulty(*numbers):
    '''
    Generates a difficulty number and result for given
    list of numbers for the multiplication operation
    *args (tuple) - integers
    returns: (product, difficulty)

    >>> compute_multiplication_difficulty(0,0)
    (0, 1)
    >>> compute_multiplication_difficulty(1,0)
    (0, 1)
    >>> compute_multiplication_difficulty(1,1)
    (1, 1)
    >>> compute_multiplication_difficulty(2,1)
    (2, 1)
    >>> compute_multiplication_difficulty(5,1)
    (5, 1)
    >>> compute_multiplication_difficulty(5,2)
    (10, 1)
    >>> compute_multiplication_difficulty(17,2)
    (34, 7)
    >>> compute_multiplication_difficulty(17,29)
    (493, 20)
    >>> compute_multiplication_difficulty(17,29,3)
    (1479, 31)
    >>> compute_multiplication_difficulty(1776,29,3)
    (154512, 77)
    '''

    numbers = list(numbers)
    if len(numbers) == 0: return 1, 0
    elif len(numbers) == 1: return numbers[0], 0

    num1, num2 = numbers[:2]
    if num1 < num2: num1, num2 = num2, num1

    num2 = str(num2)

    difficulty = 0
    borrow = 0
    md = multiplication_difficulties
    m_numbers = []

    for index, d in enumerate(reversed(num2)):

        d = int(d)

        m_number, m_diff = compute_simple_multiplication_difficulty(num1, d)
        difficulty += m_diff

        if index:
            m_number = int(m_number * math.pow(10, index))
            difficulty += md['offset']

        m_numbers.append(m_number)

    m_numbers_sum, m_numbers_diff = compute_addition_difficulty(*m_numbers)
    difficulty += m_numbers_diff

    numbers = [m_numbers_sum] + numbers[2:]
    product, m_diff = compute_multiplication_difficulty(*numbers)

    difficulty += m_diff

    return product, difficulty

Division

division_difficulties = {
    # long division
    'use_digit'       : 1,  # brinding down digit from dividend
    'multiple_lookup' : 1,  # looking up precomputed multiple of divisor
    'quotient_update' : 1,  # updating quotient with digit or period
}

Implementing long division posed a problem because my understanding of the mechanics behind the long division algorithm was minimal if not inexistent. I spent some time trying to "reverse-engineer" it. I came across "Egyptian Division" which made things clearer. With a little help, I managed to implement the following division algorithm. Please let me know, if you come up with a better approach.

def compute_division_difficulty(dividend, divisor, precision):
    '''
    Generates a difficulty number, quotient and remainder
    for dividend / divisor
    @precision (int) -- max required number of digits after decimal point in quotient
    returns: (quotient, remainder, difficulty)

    >>> compute_division_difficulty(54, 5, 0)
    (10.0, 4, 6)
    >>> compute_division_difficulty(50, 5, 0)
    (10.0, 0, 6)
    >>> compute_division_difficulty(575, 6, 0)
    (95.0, 5, 60)
    >>> compute_division_difficulty(575, 6, 1)
    (95.829999999999998, 20, 112)
    >>> compute_division_difficulty(6, 9, 1)
    (0.66000000000000003, 60, 50)
    >>> compute_division_difficulty(410, 2, 1)
    (205.0, 0, 54)
    '''

    if not divisor: raise Exception('Division by Zero')
    if not dividend: return 0, 0, 1

    dividend = str(dividend)

    divisor  = str(divisor)

    difficulty = 0
    previous_multiples_difficulty = 0
    precision_reached = 0
    decimal_point_used = 0
    dd = division_difficulties
    num = []
    quotient = []

    num = dividend[0]
    difficulty += dd['use_digit']
    index = 0

    while 1:
        q_digit = int(num) / int(divisor)

        multiple, multiple_difficulty = compute_multiples_difficulty(int(divisor), q_digit)
        difficulty += int(math.fabs(multiple_difficulty - previous_multiples_difficulty))
        if previous_multiples_difficulty: difficulty += dd['multiple_lookup']
        previous_multiples_difficulty = max(multiple_difficulty, previous_multiples_difficulty)

        quotient.append( str(q_digit) )
        difficulty += dd['quotient_update']

        if decimal_point_used:
            precision_reached += 1

        num, sub_difficulty = compute_subtraction_difficulty(int(num), multiple)
        difficulty += sub_difficulty
        num = str(num)

        index += 1

        if index == len(dividend):
            if precision == 0: break
            quotient.append('.')
            difficulty += dd['quotient_update']
            decimal_point_used = 1

        if not decimal_point_used:
            num += dividend[index]
        else:
            num += '0'

        difficulty += dd['use_digit']

        if (len(num) == 1 and not int(num)) or precision_reached >= precision + 1:
            break

    remainder = int(num)
    quotient = float(''.join(quotient))

    return quotient, remainder, difficulty

You must've noticed the usage of compute_multiples_difficulty(). In long division, at every step, you will try to find the largest multiple of the divisor less than or equal to the number in hand to proceed further. During the process of division, if you've computed the 5th (say) multiple of the divisor with difficulty N. The computation of 6th multiple at a later point in the division is difficulty(lookup of 5th multiple) + difficulty(6-5th multiple).

Results

#Problem 1
>>> compute_addition_difficulty(5, 3)
(8, 2)

#Problem 2
>>> compute_addition_difficulty(99999, 12345)
(112344, 22)

#Problem 3
>>> compute_addition_difficulty(1000000, 1000001)
(2000001, 

Problem 1: 2
Problem 2: 22
Problem 3: 8

The End

I wish I could have written a more detailed explanation instead of sprinkling this article with code. Time however prevents me from doing so . It is some relief that Python is a wonderful language for readability and the code samples above are pretty close to pseudo-code.

I will post back when I complete the program to generate arithmetic problems by order of difficulty. Until then, Adios amigos. Almost forgot! Here is the code.

In between I did some projects using Java, C#, C, C++. None of these impressed me too much. I hated Java’s imposition of stiff rules and it’s dogged adherence to “everything in a class” attitude. C# was better. C++ just turned me off because of the monster it is. I did not like C at all because of its total lack of automated memory handling (like GC). I’ve been doing a lot of coding in C now-a-days as part of my job and I must admit that I like it a lot for its simplicity in primitives and promise of “closeness to hardware” and hence the predictability and performance.

I did a small part of my final year project using Python. However, for some unfathomable reason, Python did not impress me at all then. When I started working at Veveo I used Python for a project and got hooked. It’s simplicity and “readability” got me. The power of wielding this tool got me drunk

Python was designed from the beginning to be a “easy to read” language. Most, if not all, of the syntax is intuitive. The indentation adds to the readability aspect. The policy of “only one way to do a thing” does wonders for readability. Everyone does a certain thing only the “one” way. If you are wondering why the heck I am talking so much about readability, you should consider the fact that an average programmer spends most of his time “reading” code. You have to read your code after you’ve just written it. You’ve to read your code the next day when you resume work. You’ve to read your code the moment a bug is found. You have to read your code when someone asks you how some aspect of it works a couple of months later. You’ve have to read your code when making a teeny-weeny feature addition. I just cannot emphasize enough how much time is spent is just reading. There have been times when I would spend a whole day just reading code and finally making “a single line of code change” at the end of the day!

So there it is. Python makes it possible to write readable code and that does wonders to programmer productivity. Maintaining your code becomes easier. Understanding your collegues code becomes easier and most of all understanding code written by someone across the world becomes easier – so you can start reusing components more quickly and with more confidence than ever before.

I simply love Python.

You must be wondering where I am taking this discussion… because the title says something about a Python being abused. Who is abusing the Python anyway?

I’ve noticed how newbies code in Python and found it particulary intriguing. What is interesting here is that every person comes from a certain programming background and are used to doing things in a certain way. When they are thrown into a situation where they have to learn a new language and write some code in it, they invariably apply the vast experience they have in their language of choice

I’ve had the oppurtunity of observing people from Java, C, Perl, VB (yes, Perl) backgrounds writing code in Python. The Java guys stress test multiple inhertitance in Python and bring the much cherished “everything in a class” practice to the Python. The C guys who are more often than not obsessed with performace and optimization put their brains to work and implement a strcpy using a “for loop” and insist on doing a “shift” instead of “division/multiplication”. The Perl guys just don’t seem to like the alphanumerals. They craft Python code with ingenious application making it look very concise. The more characters in one line the better the code. The more non-alphanumerals the better coder you are. That’s the way of the “Perl’ies”. The VB guys languish for a while complaining constantly about the lack of a proper IDE and after trying out various Python editors, decide to call it quits and go home to comfortable VB. I know, I know ….. I was a VB guy too and I did search for IDE’s too … But then I found VIM and everything was good

I’ve had the good (snigger) fortune of maintaining some of these brilliant artifacts and had my share of nightmares and laugh-outs. I thought I had seen it all, until I saw something today. I told myself — “Never underestimate a brilliant C programmer who has found exec and eval in Python” … Yes, you can quote me on this

    guido = "is speechless"
    larry = "went nuts"
    sergey = "has seen it all now"
    ...
    ...
    for idx in ['guido','larry','sergey']:
        idxv=eval(idx)
        if(not idxv and dd.has_key(idx)):
            idxv='%s="%s"' %(idx,dd[idx])
            exec(idxv)

    ...
    ...

No….. I don’t think I’ve seen it all….

I’ve been reading up on TDD and it has struck me as particularly useful methodology to achieve “clean code that works”. TDD encourages writing unit tests to cover all the code (because by definition, you write a test before a line of code is written). Because all your code is covered you are freed from the fear of breakage due to change and can instantly be more confident and productive. Also, the test cases act as a specification in code – very useful.

Python has standard modules, unittest and doctest to help you write test cases. I simply love doctest. It alleviates much of the pain of writing a test case (setup and all) besides acting as “executable documentation”. The unittest module has a Java legacy and is not to my taste. Also, I wanted to find a solution that would help in automated test enumeration (discovery) in my source directories without having to write any “infrastructure” code. One more thing I was looking for was a way to run both unit tests and doc tests together.

After a bit of searching, I found “Nose“. Nose is a clone of “py.test” which I liked better than the original (subjectively). To get a feel of “Nose”, I set up some python test files.

The following is the directory structure and the contents of the files. I’ve put in both unit tests and doc tests in the files to see how “Nose” handles them. Also, the tests are spread across directories. Note that I had to put an “__init__.py” to allow “Nose” to import tests in a subdirectory.

The setup

The directory structure

prashanth@prashanth-desktop:~/tmp$ tree
.
|-- bingo.py
|-- somedir
|   |-- __init__.py
|   `-- test_another.py
`-- test_prashanth.py

1 directory, 4 files

bingo.py

def boing(a, b):
    '''
    >>> boing(10, 20)
    30
    '''
    return a+b

def boing1(a, b):
    '''
    >>> boing1(10, 20)
    40
    '''
    return a+b

test_prashanth.py

def test_a():
    assert 1

def test_b():
    print "hello"
    assert 0

somedir/test_another.py

def test_bingo():
    raise Exception('hgello')

Installing “Nose”

sudo easy_install nose

If you don’t have easy_install, head over here to get information on installation.

Running the tests

Now that “Nose” is installed, let us run the tests,

nosetests --with-doctest

The output is

..E.F
======================================================================
ERROR: somedir.test_another.test_bingo
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/nose-0.10.2-py2.5.egg/nose/case.py", line 182, in runTest
    self.test(*self.arg)
  File "/home/prashanth/tmp/somedir/test_another.py", line 2, in test_bingo
    raise Exception('hgello')
Exception: hgello

======================================================================
FAIL: test_prashanth.test_b
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/nose-0.10.2-py2.5.egg/nose/case.py", line 182, in runTest
    self.test(*self.arg)
  File "/home/prashanth/tmp/test_prashanth.py", line 7, in test_b
    assert 0
AssertionError:
-------------------- >> begin captured stdout << ---------------------
hello

--------------------- >> end captured stdout << ----------------------

----------------------------------------------------------------------
Ran 5 tests in 0.057s

FAILED (errors=1, failures=1)

The first line in the output is the "test progress" indication (..E.F) . When a test succeeds, a '.' is written. When a test fails, an 'F' is written. When a test throws an Exception, an 'E' is written. Very useful to get a sense of progress as a huge test suite being executed.

"Nose" captures the stdout and stderr when a test case fails to help you debug the issue. To learn more about using "Nose" go here.

Get Project Gutenberg Ngram data

The Ngram data contains bi-grams and tri-grams for now. I plan to generate uni-grams soon. I’ve made the data available here so you can download and use it! This data contains all of the e-books hosted by Project Gutenberg (which means the data contains English, French, German and other languages). If you want an English only dataset, check back in a week or two. I am in the process of generating the same.

The Ngram data containing bi-grams and tri-grams. Each line is prepended with the occurence count.

gutenberg_ngrams.tar.bz2 (624 MB)

This is the compressed tarball of all the txt files in Project Gutenberg (as of a week before this blog post). Note that you don’t need this file unless you want to generate the Ngrams yourself using the scripts provided below.

gutenberg_files.tar.bz2.0,
gutenberg_files.tar.bz2.1,
gutenberg_files.tar.bz2.2 (5.3 GB)

My webserver (Apache) has a problem serving out files bigger than 2GB, so I had to split the file up. After you download the splits, you have to join them like this.

mv gutenberg_files.tar.bz2.0 gutenberg_files.tar.bz2
cat gutenberg_files.tar.bz2.1 >> gutenberg_files.tar.bz2
cat gutenberg_files.tar.bz2.2 >> gutenberg_files.tar.bz2

To decompress the files, you will need bunzip2 on *nix/Cygwin. On Windows, use 7zip.

Generate the data yourself

In case you want to generate the Ngrams yourself by processing the Project Gutenberg data files, follow these instructions. You will have to get the Project gutenberg data files. Use the following command to get all the English language files in txt format.

mkdir gutenberg
cd gutenberg
wget -w 2 -m "http://www.gutenberg.org/robot/harvest?filetypes[]=txt&langs[]=en"

The txt files are compressed and stored in files ending with .zip extension. These zip files are spread across multiple directories. The following command will move the zip files into the “gutenberg” directory you created in the above step.

for i in `find . -name "*.zip"`; do mv $i . ; done;

Now that all the zip files are in the same directory, unzip the zip files.Some zip files may contain files other than .txt’s. The following command extracts only .txt’s in the zip files.

cd ..
mkdir gutenberg_txt
for i in `find gutenberg -name "*.zip"`; do unzip $i \*.txt -d gutenberg_txt/ ; done;
cd gutenberg_txt
for i in `find . -name "*.txt"`; do mv $i . ; done;
cd ..

The gutenberg txt files have gutenberg headers and footers which should be removed lest they skew the frequency of Ngrams. The script “remove_gutenberg_text.py” does exactly this. The “generate_ngrams.py” script creates uni, bi and tri-grams of whatever text is piped into it. The following command pipes all the txt files through both the scripts to create the ngrams file.

for i in `find gutenberg_txt/ -name "*.txt"`; \
do cat $i | python remove_gutenberg_text.py | \
grep -i -v "project gutenberg" |\
 python generate_ngrams.py >> gutenberg_ngrams; done;

Now you have to count the number of times an ngram occurs. The following sequence of commands process the ngrams file generated above and produce a file with the frequency counts of the ngrams. Note that the “512K” option to sort is because I had to run these scripts on my host which kills processes that take too much memory. If you have a machine with a lot of memory, sorting can be significantly faster if you use a higher value, say “1G”.

sort -S 512K -T tmp_sort/ gutenberg_ngrams > gutenberg_ngrams.sorted
uniq -c gutenberg_ngrams.sorted > gutenberg_ngrams.counted
sort -S 512K -T tmp_sort/ gutenberg_ngrams.counted > gutenberg_ngrams.counted.sorted

Gutenberg data processing scripts

• remove_gutenberg_text.py — removes Project Gutenberg header and footer from txt files
• generate_ngrams.py — generate uni, bi and tri-grams for any text

Do get back

If you use this data, I would really appreciate if you get back with details about how you used it in the context of your project

Alexa collects statistics about visits by internet users to websites through the Alexa Toolbar. Based on the collected data, Alexa computes site ranking. By examining the Alexa rank of a site, you can get a rough idea of how popular the site is. Many argue that Alexa rank is misleading but it has its uses.

The Alexa rank script

You can find out the Alexa rank for any site by using this page. However, if you want to programatically get the Alexa rank, you can do it using this script.

Get the Alexa rank script

Using the script

After downloading the script, give it execute permission by doing this. You will need to have Python installed.

chmod +x get_alexa_rank.py

$ ./get_alexa_rank.py google.com
popularity rank = 2
reach_rank = 1

$ ./get_alexa_rank.py wikipedia.com
popularity rank = 7
reach_rank = 6

$ ./get_alexa_rank.py blog.prashanthellina.com
popularity rank = 557287
reach_rank = 482289

$ ./get_alexa_rank.py www.inexistantsite.com
popularity rank = -1
reach_rank = -1

How does the script work?

If you make a http request for the following url,

http://data.alexa.com/data?cli=10&dat=snbamz&url=$URL

after replacing $URL with the url of the site for which you need the Alexa rank, the following XML response is sent out. I tried with “http://blog.prashanthellina.com”.

The script parses the XML response and extracts POPULARITY/@TEXT and REACH/@RANK.

If you are looking for a PHP script for doing the same, check this out.

I have come across more than one instance when I had to select a random record from a table in a MySQL database. Here is how to do it.

The simple but slow method

SELECT * FROM mytable ORDER BY RAND() LIMIT 1;

Although simple, the above query can be very slow on tables which have a large number of records. This happens because MySQL makes a temporary table with a random number assigned to each row. It then sorts the table and returns the one record from the sorted list.

Random record selection by id generation : A better and faster method

The table will need to have an ‘id’ field which is an auto-incrementing integer. The approach is to generate a random number which falls in the range of the values of the ‘id’ field. MySQL MAX() and MIN() grouping functions allow you to choose the maximum and minimum values of a field.

The python approach

Sample python code to do the random record selection

import random
...
cursor.execute('SELECT MIN(id) FROM mytable')
min_value = cursor.fetchone()
cursor.execute('SELECT MAX(id) FROM mytable')
max_value = cursor.fetchone()

random_id = random.randint(min_value, max_value)

cursor.execute('SELECT * FROM mytable WHERE id = %s', (random_id,))
random_record = cursor.fetchone()

Pure SQL approach

MySQL offers a RAND() function which generates a random floating point number between 0.0 and 1.0. We will use this value and scale it to fit into the max – min value range of the id field.

SELECT * FROM mytable WHERE id = (SELECT MIN(id) + FLOOR((MAX(id)+1) * RAND()) FROM mytable ) LIMIT 1;

So next time you have to select a random record, you know what to do

Additional information

• http://www.phptoys.com/e107_plugins/content/content.php?content.28
• http://www.carlj.ca/2007/12/16/selecting-random-records-with-sql/
• http://akinas.com/pages/en/blog/mysql_random_row/

From the beginning, Python’s designers understood this use-case and exposed a “C” API. Using the Python C-API, one can write “modules” in C which can then be imported into Python. This solution is not bad and provided you have enough patience, works great. There is a tool called SWIG which can generate the “glue” code around C code. It automates writing of code using C-API and makes it easier for one to maintain the C “module”. However, since SWIG generates code, when some problem occurs, it is quite painful to debug through the wrapper code. For the lazy developers out there (like me ), this can be quite a deterrent.

I’ve been working on a project for using Wikipedia data to assign “topics” to arbitrary pieces of English text. The code written in pure Python takes about 3 minutes to run. When I profiled this code, I found that most of the time was being spent in disk reads (through the db). I decided, after examining the data I had, to load all of it into memory. I tried doing this in pure Python and saw the memory usage creeping up beyond the 4GB capacity when I decided to save my poor machine from thrashing by killing the hog!

On doing some tests, I found that Python consumes about 32 bytes to store an integer! Hmmm…. time to move the data structures to C for more efficient memory usage. I started looking around for tools to interface C code with Python when I came across “ctypes” and immediately fell in love with it.

ctypes lets you load any “shared object” or “dynamic link library” in a Python program and unobtrusively call functions. You can construct C datatypes required by C functions (pointers, longs, ints, chars, arrays, structs) and even specify callback functions to be passed to C code! I will give you a basic introduction to ctypes to get you started with Python – C interfacing.

Code

test.c

#include 

// you can initialize stuff in this function
// it is called when the so is loaded
void _init()
{
    printf("Initialization of shared object\n");
}

// you can do final clean-up in this function
// it is called when the so is getting unloaded
void _fini()
{
    printf("Clean-up of shared object\n");
}

int add(int a, int b)
{
    return(a+b);
}

int sum_values(int *values, int n_values)
{
    int i;
    int sum = 0;

    for (i=0; i

We have to compile test.c to create the shared object.
gcc -fPIC -c test.c
ld -shared -soname libtest.so.1 -o libtest.so.1.0 -lc test.o


test.py
from ctypes import *

# load the shared object
libtest = cdll.LoadLibrary('./libtest.so.1.0')

# call the function, yes it is as simple as that!
print libtest.add(10, 20)

# call the sum_values() function
# we have to create a c int array for this
array_of_5_ints = c_int * 5
nums = array_of_5_ints()

# fill up array with values
for i in xrange(5): nums[i] = i

# since the function expects an array pointer, we pass is byref (provided by ctypes)
print libtest.sum_values(byref(nums), 5)


python test.py

Output

Initialization of shared object

30

10

Clean-up of shared object




How much simpler can Python - C interfacing become? ctypes is a standard module from Python 2.5.
Resources

ctypes
Shared objects for the object disoriented! - IBM Developer Works
swig

]]>
			http://blog.prashanthellina.com/2008/01/07/interfacing-python-with-c-using-ctypes/feed/
		10
		
		
		
		http://blog.prashanthellina.com/2007/12/21/topic-extraction-using-wikipedia-data/
		http://blog.prashanthellina.com/2007/12/21/topic-extraction-using-wikipedia-data/#comments
		Fri, 21 Dec 2007 11:49:14 +0000
		prashanthellina
				
		
		
		
		
		
		
		
		
		

		http://blog.prashanthellina.com/2007/12/21/topic-extraction-using-wikipedia-data/
		
			

    




In an earlier article, I mentioned that I was trying to use Wikipedia data to do news article clustering to make it easy for me follow news feeds. I have made some progress. I’ve written an algorithm to produce a list of Wikipedia articles relevant to the input text. Input text has to be in English. The algorithm will not work well for very short pieces of text. At least a paragraph or two with sizable text are required. The list of Wikipedia articles will represent the “topic” of the input text.
Test run
To test the algorithm, I gave the text from an earlier article from this blog (Accessing your home computer from the internet). The top Wikipedia articles in the output are

Internet
Domain Name System (DNS)
IP Address
Hypertext Transfer Protocol (HTTP)
Modem
World Wide Web (WWW)
Domain Name
Dynamic Host Configuration Protocol (DHCP)
Internet Service Provider
Network Address Translation (NAT)
Firewall

How it works?
The basis of the algorithm is to find Wikipedia article titles occuring in the input text. The “found” set of Wikipedia articles are then used to construct a sub-graph from the Wikipedia graph (formed by linkages between Wikipedia pages). The most interconnected nodes happen to be relevant. However, as I have not applied any filtering on the input text, a lot of “junk” matches happened. For example, the word “let” is picked up and it matches a Wikipedia article by the same title which redirects to Lashkar-e-Toiba. This is totally irrelevant to the input text. To remove such spurious matches, I dropped all the least interconnected nodes and constructed a sub-graph with the remaining nodes. In the sub-graph, I did recomputation for node interconnection.
Below is the output of the first phase. This graph contains all nodes found from matching phrases in the input text. The nodes of darker blue are more relevant than lighter ones. The darker and thicker a link is, the more relevant it is.



    



Download higher resolution image here. 8.2MB
A lot of extracted articles are not relevant to the input text. Some of these spurious nodes are totally disconnected from the main body of the graph.


This is a slightly higher resolution picture of a section of the full graph above.


Below is the output of second phase of the algorithm where relevant nodes are extracted and a sub-graph computed. Node and edge relevances are recomputed within this set.



    



Download the higher resolution image here. 1.9MB
All the graphs above were produced using Graphviz.
What next
I tried applying the logic to some sample input texts and results look very encouraging. The next step towards news article clustering would be apply the topic extraction algorithm to multiple news articles and look for common Wikipedia articles (maybe plain intersection). I still have not given much thought to this stage. Once I do, I will post back.
As I said before, Wikipedia amazes me every time I use it. The wealth of information (both as text and as interconnects) is astounding. As a token of appreciation, I’ve donated a small amount to the current Wikipedia donation round. If you like Wikipedia and have used it, do consider making a donation.
]]>
			http://blog.prashanthellina.com/2007/12/21/topic-extraction-using-wikipedia-data/feed/
		12
		
		
		
		http://blog.prashanthellina.com/2007/11/29/language-people-interesting-picture/
		http://blog.prashanthellina.com/2007/11/29/language-people-interesting-picture/#comments
		Thu, 29 Nov 2007 06:50:37 +0000
		prashanthellina
				
		

		http://blog.prashanthellina.com/2007/11/29/language-people-interesting-picture/
		
			

I like the representation for Logo, Machine Language, Prolog and Ada. Wonder what “N.W” is… (the Modula-2 guy is holding it). I wish python was featured too   but the picture says “’85″. Python did not even exist then!
original from here
]]>
			http://blog.prashanthellina.com/2007/11/29/language-people-interesting-picture/feed/
		3
		
		
		
		http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/
		http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/#comments
		Wed, 17 Oct 2007 16:44:13 +0000
		prashanthellina
				
		
		
		
		
		

		http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/
		
			  

Wikipedia is a superb resource for reference (taken with a pinch of salt of course). I spend hours at a time spidering through its pages and always come away amazed at how much information it hosts. In my opinion this ranks amongst the defining milestones of mankind’s advancement.

Apart from being available through http://www.wikipedia.org, the data is provided for download so that you can create a mirror locally for quicker access. This is very convenient when you are not connected to the internet, say when you are on the move.
Setting up a local copy of Wikipedia
Windows

If you have Windows installed, Webaroo is an easy way to get Wikipedia locally as a “web pack”. Check out Webaroo here. Another way on Windows is to use WikiFilter. I tried WikiFilter and found it a good option (It is open source, so you can tweak it). It takes up around 3 to 3.5GB on your disk.
Linux

This page has instructions to setup on Linux. “Building a (fast) Wikipedia offline reader” is a good option too.
Any operating system

Wikipedia provides static wiki dumps for download which should work fine on any operating system that supports a decent web browser. Although I have not tried it, I have heard that the dumps can take up as much as 80GB of space on your disk.
Windows Mobile, iPhone and Blackberry

To access Wikipedia from your mobile, check out vTap from Veveo. I must tell you that I work for this company but I am being very objective in suggesting this service to you. A Java version is being developed and will be out soon. Since the space on mobile devices is very limited, the data is hosted on vTap servers and network connectivity is required.
Other uses for Wikipedia data dumps
In being such a vast repository of knowledge, Wikipedia is useful in many other ways. I want to use Wikipedia’s data to handle the feeds I read every day. The same news article comes in from different sources and multiple times from the same source and I end up reading all of them. I am going to try and use Wikipedia to help me automatically pull together these news articles and cluster them around topics.
The first step in this experiment would be to get the data dumps from Wikipedia and process them to load them into a Mysql database. Once we get the data into a database, things become more manageable.
Getting the dumps
    Wikipedia is huge and this reflects in the data dumps. It took me about 40 hours to get the articles xml alone on my home connection. Put together all the relevant files of Wikipedia dumps come to 5 GB. I got the dumps from here. You can check for more recent ones here.
Files to get

 pages-articles.xml.bz2 (2.8 GB) – xml containing page texts
 redirect.sql.gz (10.5 MB) – sql for redirected pages info
 page.sql.gz (318.9 MB) – minimal page information (id, page title)
 externallinks.sql.gz (439.6 MB) – links from pages to external sites
 categorylinks.sql.gz (235.5 MB) – categories
 pagelinks.sql.gz (1.2 GB) – inter wiki page links

Since DreamHost (My Web host) offers a lot of disk space and the network connection is way better than the one I have at home, I downloaded the files to my DreamHost space. (read more about DreamHost here)
I was able to download all the files using wget but for “pages-articles.xml.bz2″. For some reason I cannot understand, Wget was bailing out after downloading a few bytes (this seems to be a DreamHost specific issue). To work around this issue, I wrote this python script
get_wiki_file.py
import urllib2, sys

url = "http://download.wikimedia.org/enwiki/20070908/enwiki-20070908-pages-articles.xml.bz2"
outfname = 'enwiki-20070908-pages-articles.xml.bz2'

o = open(outfname, 'wb')
r = urllib2.urlopen(url)
total_bytes = 0
counter = 0

while 1:
        bytes = r.read(10240)
        total_bytes += len(bytes)
        o.write(bytes)
        if len(bytes) < 10240: break
        counter += 1
        if counter % 10 == 0:
                counter = 0
                print "%.2f MB" % (total_bytes/1024.0/1024.0)
o.close()

Preparing dumps
The next step was to extract all the archives. I used bunzip2 for .bz2 files and gunzip for the .gz files. I tried loading one of the large .sql files into the database and the process was killed (I guess this is because DreamHost does not like resource hungry processes running for a long time). To work around this I had to split all the big .sql files into smaller chunks.
for example
split -d -l 50 ../enwiki-20070908-page page.input.

-l option tells split how many lines per split we need and -d tells split to use numerical suffix (which will be useful soon).
However the pages-article file is an xml and not sql. To load it into the database, I had to first convert it to a .sql dump. xml2sql is a handy program for doing this. You can get it here.
xml2sql -v -m pages-articles.xml

This command will produce text.sql, page.sql and revision.sql. However, I ran into a problem here because xml2sql was leaking and slowly rose to 88 MB of resident memory when it got killed by the DreamHost process. I tried running valgrind on it but could not find any leaks (they must be getting freed on exit).
This forced me to split the huge xml into manageable parts so that xml2sql would stay within 88MB. I wrote this python script for splitting the xml.
split_xml_dump.py
import os, os.path

fname = "enwiki-20070908-pages-articles.xml"
total_page_counter = cur_page_counter = 0
o = None
outf_counter = 0
outdir = 'pages_xml_splits/'

for line in open(fname):
        if line.startswith(""):
                continue

        if o is None or cur_page_counter == 250000:
                cur_page_counter = 0
                outf_counter += 1
                outfname = 'pagexml.%d' % outf_counter
                outfname = os.path.join(outdir, outfname)
                print outfname

                if o:
                        o.write('\n')
                        o.close()

                o = open(outfname, 'wb')
                o.write('\n')

        if line == '  \n':
                cur_page_counter += 1
                total_page_counter += 1
                if total_page_counter % 10000 == 0: print total_page_counter

        o.write(line)

if o:
        o.write('\n')
        o.close()

print total_page_counter

Once the file got split not chunks of 250,000 articles each, I used xml2sql on each chunk to get the corresponding text.sql.
Loading the dumps into the database
I had now readied all the input (a bunch of .sql files) and had to load them into the database. Before I started the load, I had to create a database to hold the tables and the "text" table.
mysql -h hostname -u username -ppassword
> create database wiki;
> create table `text` (
  old_id int unsigned NOT NULL auto_increment,
  old_text mediumblob NOT NULL,
  old_flags tinyblob NOT NULL,
  PRIMARY KEY old_id (old_id)
) MAX_ROWS=10000000 AVG_ROW_LENGTH=10240;

It took about 90 minutes for the splitting to get over. This script will load the .sql files into the database one after the other.
load_splits.py
import os
import os.path
import glob
import shutil

while 1:
        fnames = [os.path.basename(f) for f in glob.glob('splits/*.input.*')]
        fnames = [(int(f.split('.')[2]), f) for f in fnames]
        fnames.sort()

        if len(fnames) == 0: break
        print "found %d files" % len(fnames)

        fname = os.path.join('splits/', fnames[0][1])
        to_fname = os.path.join('processed_splits/', fnames[0][1])
        error_fname = os.path.join('processed_splits/', fnames[0][1] + '.error')

        print "processing %s" % fname

        cmd = 'mysql -h hostname -ppassword -u username wiki < "%s"' % fname
        result = os.system(cmd)
        if result != 0:
                shutil.move(fname, error_fname)
        else:
                shutil.move(fname, to_fname)

        print "processed %s" % fname

Loading the .sql files into the database will take a long long time. I started it yesterday morning and it is still running! As the data is loading, you can check out this Wikipedia database schema diagram.
In a continuation to this article, I will write about how I will use the Wikipedia database to streamline my news feeds.
]]>
			http://blog.prashanthellina.com/2007/10/17/ways-to-process-and-use-wikipedia-dumps/feed/
		22
		
		
		
		http://blog.prashanthellina.com/2007/09/29/vtap-windows-mobile-source-code/
		http://blog.prashanthellina.com/2007/09/29/vtap-windows-mobile-source-code/#comments
		Sat, 29 Sep 2007 14:52:35 +0000
		prashanthellina
				
		

		http://blog.prashanthellina.com/2007/09/29/vtap-windows-mobile-source-code/
		
			Veveo has released the source code for the windows mobile client application. This is great because it gives you a way to fine tune our app to suit your needs. You can sign up for the developer program here to receive updates from Veveo. Get the source here.

]]>
			http://blog.prashanthellina.com/2007/09/29/vtap-windows-mobile-source-code/feed/
		0
		
		
		
		http://blog.prashanthellina.com/2007/08/26/word-arithmetic-puzzle-generator/
		http://blog.prashanthellina.com/2007/08/26/word-arithmetic-puzzle-generator/#comments
		Sun, 26 Aug 2007 13:21:34 +0000
		prashanthellina
				
		
		
		

		http://blog.prashanthellina.com/2007/08/26/word-arithmetic-puzzle-generator/
		
			I was digging through some old code of mine when I came across this script. To me, this script was a demonstration of the beauty and elegance of the python language.

You might have come across a puzzle like this:

AID + ICED = IDEA. What digits would you assign to the characters A,C,D,E,I so that the arithmetic will work?

The answer is A=0, C=4, D=5, E=1, I=2. 
The “word arithmetic” script can generate such puzzles given a word list and characters.

Here is the code and a sample word list.
Running the script:
python word_arithmetic.py --words-file words --letters AEIBCDFGHI

In the command above you are asking the script to generate word combinations where A=0, E=1, I=2, B=3 and so on.
Sample output
letters = A=0, E=1, I=2, B=3, C=4, D=5, F=6, G=7, H=8, I=9
FEE + AHA = FIE
DACCA + BEEBE = HEDGE
ACHED + AHA = CHID
BIG + BAH = GAD
BIB + ACED = HAH
DADA + HIE = DICE
BIG + CAF = AHAB
ABB + CHE = DEC
AID + ICED = IDEA
...
...

]]>
			http://blog.prashanthellina.com/2007/08/26/word-arithmetic-puzzle-generator/feed/
		2
		
		
		
		http://blog.prashanthellina.com/2007/08/22/matrix-desktop/
		http://blog.prashanthellina.com/2007/08/22/matrix-desktop/#comments
		Wed, 22 Aug 2007 16:32:26 +0000
		prashanthellina
				
		
		
		
		

		http://blog.prashanthellina.com/2007/08/22/matrix-desktop/
		
			

What you see above is how my desktop looks now. You need to be using gnome to get this working.
Nautilus draws the desktop (including the icons) for you in gnome by default. We have to tell it to stop doing that, so we can do the matrix animation in its place. Nautilus can be configured using gconf.
gconftool-2 --type bool --set /apps/nautilus/preferences/show_desktop false
Now that we have the desktop to ourselves, let us ask xscreensaver ‘glmatrix’ to start drawing itself in the desktop window (‘root’ window).
/usr/lib/xscreensaver/glmatrix -root
If you want the animation every time you log in, open “~/.config/autostart/glmatrix.desktop” in your text editor and paste the following.
[Desktop Entry]
Version=1.0
Encoding=UTF-8
Name=No name
Name[en_IN]=Desktop matrix
Exec=/usr/lib/xscreensaver/glmatrix -root
X-GNOME-Autostart-enabled=true

(suggested by ElecBoy) After playing around, if you want to get back to your default desktop, do 
gconftool-2 --type bool --set /apps/nautilus/preferences/show_desktop true && nautilus
]]>
			http://blog.prashanthellina.com/2007/08/22/matrix-desktop/feed/
		41