Prashanth Ellina

Tutorials TODO

2017-07-02T11:58:00-07:00

Basics

How does a browser work?
- URL, Server
- Web Server
- HTTP
- TCP
Telnet
Ping
Tracepath / Traceroute
Structure of an IP address
netstat

All your aliases are belong to you

2009-08-28T08:05:00-07:00

I like setting up shortcuts to frequently used commands whether I used Windows or Linux. I use the terminal often and create shortcuts to frequently used commands using “alias” feature of BASH. This has saved me considerable time in the past. However, I recently felt that if I could have a helper tool to monitor my usage of commands and automatically suggest candidates for aliasing, that would be useful. The output of that is Aliaser.

Aliaser works by monitoring your bash history. It analyses command frequency and suggests candidates for aliasing. It manages aliases so created. The feature I like most in Aliaser is that it reminds you to use the aliases you created by showing tips on opening a new terminal session.

Download Aliaser from http://aliaser.googlecode.com.

Query Wikipedia from your terminal

2009-08-23T10:52:00-07:00

I refer Wikipedia frequently. I use this BASH function to help me do that from the terminal. For explanation of how this works head over here.

BASH function

# wiki
# eg: wiki India
#     wiki Apple_Inc
#     wiki Anglo_Saxon
wiki()
{
    dig +short txt $1.wp.dg.cx
}

Example usage

prashanth@prashanth-desktop:~$ wiki India
"India, officially the Republic of India ( '\; see also other Indian languages), is a country in South Asia.
It is the seventh-largest country by geographical area, the second-most populous country, and the most
populous democracy in the world. Bounded by t" "he Indian Ocean on the south, the Arabian Sea on
the west, and the Bay of Bengal on the east, India has a coastline of ... http://a.vu/w:India"

prashanth@prashanth-desktop:~$ wiki Anglo_Saxon
"Anglo-Saxons (or Anglo-Saxon) is the term usually used to describe the invading tribes in the south
and east of Great Britain starting from the early 5th century AD, and their creation of the English
nation, lasting until the Norman conquest of 1066. The " "Benedictine monk, Bede, identified
them as the descendants of three Germanic tribes: http://a.vu/w:Anglo-Saxons"

Command-line language translation

2009-08-18T19:33:00-07:00

Here is a simple utility created using Python for translating text from various languages into English. It uses the Google AJAX API to do this.

Usage

prashanth@prashanth-desktop:~$ translate bonjour
hello
prashanth@prashanth-desktop:~$ translate guten morgen
Good morning

Code

#!/usr/bin/env python
'''
Translates text into english using Google Translate.
Usage: python translate.py
        (or)
       echo  | python translate.py

For convenience, make a symlink to this file from /usr/bin/translate.
'''
# derived from : http://code.google.com/p/py-gtranslate/source/browse/trunk/gtrans.py

import sys
import urllib2
import urllib
import simplejson as json

FROM_LANGUAGE = ''
TO_LANGUAGE = 'en'
BASE_URL = 'http://ajax.googleapis.com/ajax/services/language/translate'

def translate(from_language, to_language, text):
    langpair = '%s|%s' % (from_language, to_language)
    params = {'v': '1.0', 'langpair': langpair, 'q': urllib.quote_plus(text)}

    params = '%s' % ('&'.join(['%s=%s' % (k,v) for (k,v) in params.items()]))

    url = '%s?%s' % (BASE_URL, params)
    resp = json.load(urllib2.urlopen(url))
    try:
        return resp['responseData']['translatedText']
    except:
        return text

def main(text):
    if text:
        print translate(FROM_LANGUAGE, TO_LANGUAGE, text)

    else:
        lines = [l.strip() for l in sys.stdin.readlines()]
        for line in lines:
            if line:
                text = translate(FROM_LANGUAGE, TO_LANGUAGE, line)
                print '[%s]' % line
                print text
            else:
                print

if __name__ == '__main__':
    args = sys.argv[1:]
    text = ' '.join(sys.argv[1:])
    main(text)

On setting up USB RAID

2009-08-07T17:27:00-07:00

I bought two Dane-Elec 8GB USB drives recently. Flash memory (as opposed to Hard disk storage) has faster “seek” capability. This is inherent in the design as flash memory is solid state whereas hard disks are electro-mechanical with a “head” that needs to be moved around using a “drive” mechanism. Since seek times are better on flash drives, they are faster when you are reading or writing a lot of small files.

However flash drives do not have sustained data transfer rates that hard disks have (i.e throughput). My thought process what that the throughput can be made up by slapping together two or more USB drives and applying software RAID 0 over them. Below are some performance results and they look encouraging.

Note that the timings are in seconds.

Single 8GB Dane-Elec USB drive 131.11 for 683MB (write) 44.62 for 683MB (read)

Single 2GB Transcend USB drive 204.50 for 683MB (write) 63.26 for 683MB (read)

Single 8GB Sandisk drive 197.61 for 683MB (write) 29.73 for 683MB (read)

RAID0 (two Dane-Elec 8GB USB drives) 61.177 for 683MB (write) 17.9 for 683MB (read)

I created a test file with 683 MB of data by funnelling /dev/urandom into it and then copied it to the USB drive(s) to measure write performance. Then I unmounted the USB drive(s) to make sure the buffer cache is emptied. On remounting I measured read performance by copying the test file back to the harddisk. The hard disk used was a Seagate Barracuda 320GB (7200rpm).

If four USB drives (say Sandisk 8GB) can be RAID’ed together on level 0 we would have a cheap SSD. One thing that prevents me from trying this out is the question of reliability. I have not used this setup long enough or stressed it hard enough to be comfortable using this for normal use.

To rig something like this yourself, read this article.

Extracting relevant text from HTML pages

2009-07-27T16:46:00-07:00

Some time back I had done some work on extracting topics from an arbitrary piece of text using Wikipedia data. Recently I thought of a concept to put that algorithm to work. As a part of this project, I need to extract relevant text from an arbitrary HTML page. By relevant I mean the “meat” of the page devoid of navigation links and side-content.

This algorithm has the following steps:

make doc from html data (clean html)
identify content nodes (nodes having substantial content)
prune xml tree to remove irrelevant nodes
get the most linked node from pruned tree (subtree contains relevant text)
make the dot graph

I’ve pasted the relevant python module below for easy reading. However, if you want to download the code and hack it, you can get all the files from here.

Code files

content_extract.py - actual work gets done here (file pasted below)
cextract.py - cgi front-end which fetched url content and feeds to above script.
cextract_config.py - cgi script configuration file. You have to adjust this to your environment.

Try it right here and right now

url:

Some samples

http://news.bbc.co.uk/sport2/hi/motorsport/formula_one/8169436.stm
<{filename}/cextract_data/640ca0b7c6e818b2b1bf952a206a6388.html>
http://www.telegraph.co.uk/news/worldnews/europe/france/5913494/Nicolas-Sarkozy-to-slow-down-after-collapsing-while-jogging.html

Please let me know if you find cases for which the algorithm does not work. Even better would be to download the code and hack it up and post back. I am eager to see what you can come up with.

#!/usr/bin/env python

import sys
from cStringIO import StringIO

from lxml import etree #http://codespeak.net/lxml/

IGNORABLE_TAGS = set(['script', 'a'])
MIN_TEXT_LEN = 50

def get_text(node):
    '''
    Given a XML node, extract all the text it contains.
    (does not recurse into children)
    '''
    text = [node.text or '']
    for cnode in node.getchildren():
        tail = cnode.tail
        if tail is not None:
            text.append(cnode.tail)

    text = '\n'.join(text).strip()
    return text

def get_xml(node):
    '''
    Convert the sub-tree from node downwards
    into string XML representation.
    '''
    return etree.tostring(node)

def create_doc(data):
    '''
    Construct XML tree datastructure from xml string representation.
    '''
    parser = etree.HTMLParser()
    doc = etree.parse(StringIO(data), parser)
    return doc

def get_content_nodes(doc):
    '''
    Identify nodes in the XML document that
    have substantial text.
    '''
    nodes = []

    for n in doc.xpath('//*'):
        tag = n.tag

        if tag.lower() in IGNORABLE_TAGS:
            continue

        text = get_text(n)
        if not text:
            continue

        if len(text) < MIN_TEXT_LEN:
            continue

        nodes.append(n)

    return nodes

def make_pruned_tree(content_nodes):
    '''
    Prune the whole XML tree by remnoving nodes
    other than content nodes and their ancestors.
    '''
    nodes = {}
    links = {}

    for node in content_nodes:

        nodes[id(node)] = node

        parent = node.getparent()
        if parent is not None:
            links[id(node)] = id(parent)

        for anode in node.iterancestors():
            _id = id(anode)
            parent = anode.getparent()
            if parent is not None:
                links[_id] = id(parent)

            if _id not in nodes:
                nodes[_id] = anode

    return nodes, links

def get_inlink_counts(links):
    '''
    Given the inter-node links, find out which
    node has maximum number of links coming into it.
    '''
    counts = {}

    for from_id, to_id in links.iteritems():
        count = counts.setdefault(to_id, 0)
        counts[to_id] = count + 1

    return counts

def get_most_linked_node(nodes, links):
    '''
    Identify the node which is most linked.
    (i,e) has most number of inlinks.
    '''
    inlink_counts = get_inlink_counts(links)

    mcount, mid = max([(count, _id) for _id, count in inlink_counts.iteritems()])
    node = nodes[mid]
    return node

def make_dot_graph(nodes, links, chosen_node, stream):
    '''
    Construct the dot format graph representation
    so that graphviz can render the tree for visualization.
    '''
    o = stream

    print >> o, "digraph G {"

    for _id, node in nodes.iteritems():

        tlen = len(get_text(node))
        tag = node.tag

        if tlen:
            text = '%s (%d)' % (tag, tlen)
        else:
            text = tag

        if _id == chosen_node:
            attrs = 'style=filled color=lightblue'
        else:
            attrs = ''

        print >> o, "%s [label=\"%s\" %s];" % (_id, text, attrs)

    for fid, tid in links.iteritems():
        print >> o, "%d -> %d;" % (fid, tid)

    print >> o, "}"

def main():
    # make doc from html data (cleans html)
    doc = create_doc(sys.stdin.read())

    # identify content nodes
    content_nodes = get_content_nodes(doc)

    # prune xml tree to remove irrelevant nodes
    nodes, links = make_pruned_tree(content_nodes)

    # get the most linked node from pruned tree
    mnode = get_most_linked_node(nodes, links)

    # make the dot graph
    make_dot_graph(nodes, links, id(mnode), sys.stdout)

if __name__ == '__main__':
    #Eg: wget "http://blog.prashanthellina.com" -O - | python thisscript.py | dot -Tpng -o /tmp/test.png ; eog /tmp/test.png
    main()

Clustering Data using Python

2009-07-25T09:24:00-07:00

As a part of a project I am working on, I had to cluster urls on a page. After some light googling I found, python-cluster. You can find below a simple python script to illustrate the usage of python-cluster library.

Code

import pprint
from difflib import SequenceMatcher

# http://python-cluster.sourceforge.net/
from cluster import HierarchicalClustering

# input urls to be clustered
urls = [
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814385',
    '#articles',
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814335',
    'http://yro.slashdot.org/~drDugan/',
    'http://web.sourceforge.com/privacy.php',
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28815123',
    'http://slashdot.org//slashdot.org/~Darkness404',
    'http://slashdot.org//radio.slashdot.org',
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814429',
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814457',
    'http://slashdot.org//slashdot.org/article.pl?sid=09/07/24/1545238',
    'http://slashdot.org//slashdot.org/comments.pl?sid=09/07/24/1545238&cid=28810581',
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28815269',
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814657',
    'http://web.sourceforge.com/terms.php'
    'http://slashdot.org//it.slashdot.org/search',
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814581',
    'http://xkcd.com/612/',
    'http://web.sourceforge.com/advertising',
    'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814785',
]

# distance function compares two urls and finds the distance
# uses SequenceMatcher from python standard module difflib
def distance(url1, url2):
    ratio = SequenceMatcher(None, url1, url2).ratio()
    return 1.0 - ratio

# Perform clustering
hc = HierarchicalClustering(urls, distance)
clusters = hc.getlevel(0.2)

pprint.pprint(clusters)

Output

[['#articles'],
 ['http://xkcd.com/612/'],
 ['http://web.sourceforge.com/privacy.php'],
 ['http://web.sourceforge.com/advertising'],
 ['http://web.sourceforge.com/terms.phphttp://slashdot.org//it.slashdot.org/search'],
 ['http://yro.slashdot.org/~drDugan/'],
 ['http://slashdot.org//slashdot.org/~Darkness404'],
 ['http://slashdot.org//radio.slashdot.org'],
 ['http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814785',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814429',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814457'],
 ['http://slashdot.org//slashdot.org/article.pl?sid=09/07/24/1545238',
  'http://slashdot.org//slashdot.org/comments.pl?sid=09/07/24/1545238&cid=28810581',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28815123',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28815269',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814385',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814335',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814657',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814581']]

XMonad: A Window Manager for “real” people :)

2009-04-26T21:34:00-07:00

I have been a happy Gnome user for many years now and only recently started thinking about switching to KDE 4.2 when Ubuntu 9.04 (Jaunty Jackalope) comes out. However, it so happened that I bought two new widescreen monitors and setup a dual-monitor environment. This is when I started realizing the Gnome was clumsy at best when it comes to managing windows across monitors.

The reason I bought multiple monitors is to maximize my work area so I do not have to keep switching between overlapping windows. Gnome it seems is ill-suited to effectively and effortlessly managing space.

When I maximize a window on a widescreen monitor, the window takes up the entire space on the screen. Although this is the expected behavior, I could not rest looking at the huge amount of space being wasted. What I would like better is having two windows occupying the height of the screen but sitting right next to each other (i.e. tiled). This way less space is wasted and you get to see both windows at the same time. I could do this in gnome, but with considerable effort, as closing a window and re-opening another requires another round of frantic window arrangement.

In comes a tiling window manager … Tiling window managers do not allow overlapping windows (except for dialogs). They are a good solution when you need to tile windows to occupy all available screen space. Also, most tiling window managers emphasize keyboard control over mouse control. If you are productivity freak you will love the keyboard shortcuts that do away with a painful context switch required in navigating with the mouse.

There are many Tiling Window Managers out there. Rat Poison, Stump WM, Ion, dwm, wmii and XMonad are a few. I chose XMonad because I was able to figure how to integrate XMonad into the Gnome desktop environment. Most of these window managers don’t play well with Gnome. I use a number of Gnome apps and I would hate to not be able to use them.

Okay, here is how you Install XMonad on Ubuntu to work within Gnome …

mkdir ~/.xmonad
cd ~/.xmonad
vim xmonad.hs

Then paste the following into the xmonad.hs file and save it.

import XMonad
import XMonad.Config.Gnome
import XMonad.ManageHook

myManageHook :: [ManageHook]
myManageHook =
    [ resource  =? "Do"   --> doIgnore ]

main = xmonad gnomeConfig
    { manageHook = manageHook gnomeConfig <+> composeAll myManageHook}

sudo apt-get install xmonad libghc6-xmonad-dev

Note that libghc6-xmonad-dev is a 200MB dependency!

vim ~/.gnomerc

Add this line .gnomerc file and save.

export WINDOW_MANAGER=xmonad

The installation is done. Now to experience your new Window Manager, log out and then log back in. You will see your usual desktop with Gnome panels.

To learn more about XMonad and how to use it, follow these links:

http://ubuntu-snippets.blogspot.com/2008/08/xmonad-tiling-window-manager.html

http://tombuntu.com/index.php/2009/03/17/introduction-to-the-xmonad-tiling-window-manager/ https://www.haskell.org/haskellwiki/Xmonad/Using_xmonad_in_Gnome/0.5

Microsoft Surface: Some Videos

2009-04-22T20:33:00-07:00

I had written earlier about my experience with Microsoft Surface. I’ve captured some videos of me using it. Here they are …

My Dual Monitor Setup

2009-04-21T22:48:00-07:00

Greetings webizen, I tried hard to get back to my blogging schedule but my laziness got the better of me. I am back truly with a new batch of posts which I will publish over the next few days.

Recently, I went to Veveo’s main office near Boston, USA and had the privilege of experiencing a finger freezing winter! (Not to mention a three day power-cut which I spent under multiple layers of blankets).

A few weeks before I was going to return to India, I started shopping. One of things I bought was a Refurbished NVidia GeForce 9800 GTX+ by EVGA. It was the best deal I could find on NewEgg in terms of value for money. I bought it primarily to have a betterr experience when flying over Paris … in Google Earth.

This is the first graphics card I bought ever and was very excited when I received it. I was surprised to see how big graphics cards have become nowadays (compared to S3 cards of the decade gone past). While oggling at the card I noticed that it had two video outputs. I had heard of dual-head setups but always considered them a distant reality considering my cash-strappedness as a college student. This sparked off more info

an urge to finally setup my own Dual monitor setup back in Bangalore.

I got back to India and promptly headed to SP Road in Bangalore. After some asking around, I figured that the best choice at the moment was two Samsung SyncMaster 2233SW’s each capable of 1920x1080 (Yes, HD). I bought both for about 22,000 Rupees (about USD 420).

I set them up in Linux and upgraded to the latest (at the time) Nvidia drivers (180.x). I spent a few hours trying to hide a satisfied smile lest people thing I had gone nuts.

Dual monitors give you more screen space to work with. They will help you reduce Window-Switching and Workspace-Switching which does save time as I have observed. Now that I have experience the dual-head setup, I am never going back to using a single monitor. It is painful ;) So much did I feel this that I bought a low-end Nvidia GeForce card (7200) with dual-head support and stuck it into my computer at work :)

If you do not already have a dual-head setup, definitely consider an upgrade. You are worth it:)

Go to flickr set for this

Microsoft Surface Unboxing

2008-12-30T05:16:00-08:00

Today, we received the shipment from Microsoft at Veveo. If you have not heard of Microsoft Surface before, It is a touch screen based computer embedded in a table. The surface of table is illuminated from underneath by a projector (rear-projection) and touch input is implemented by reflecting IR radiation off the fingers and then being captured by five IR camera hidden inside the unit.

To learn more about Microsoft Surface head over to:

Unboxing Pictures

flickr set on microsoft surface unboxing

Some observations:

It is very heavy!
and expensive (around $15,000)
The power socket is hidden underneath and is very difficult to access. The power button is equally well hidden and difficult to find.
Installation was non-trivial. The touch input did not start working out of the box. We had to use the bundled mouse to initial installation steps.
The “surface shell” with the ripples in the water is a great way to understand the potential of this device. It feels like you are touching water! and your brain expects that water will drip when you lift your fingers up. I think it is more realistic (compared to devices with smaller touch screens) because of the size of the display and the fact that it is aligned horizontally making it more natural.
Since rear-projection is used for the display, the viewing angle is very wide (nearly 180 degrees)
The matte finish on the touch surface as a good feel (almost like paper).

Determining the difficulty of Arithmetic Operations

2008-07-27T22:09:00-07:00

I am trying to write a program to test my arithmetic skills. The program should pose arithmetic problems involving the four basic operations - addition, subtraction, multiplication and division. When the testing session starts, the program should issue problems of less difficulty and the difficulty should be ramped up gradually. A score should be computed based on number of questions and the difficulty of questions. The hope is that if I keep using this program for a little time every day, I’ll be able to improve my abysmal arithmetic performance :)

Note that you will need to have Python installed to try out the script and the examples in this article. I believe that the code is simple enough to be understood with minimal or no understanding of the Python language. If any part of the article is not clear, feel free to point it out to me. Thank you.

Estimating difficulty

Let us take a look at the following problems.

Problem 1

5 +
3

Problem 2

99999 +
12345

Now, which problem is more difficult? It is quite obvious that Problem 2 is more difficult. But why? One answer would be - “because Problem 2 involves the addition of two large numbers”. Fine. Consider this now …

Problem 3

1000000 +
1000001

Is Problem 3 more difficult than Problem 2 because the numbers undergoing addition are bigger? The intuitive answer is “No”. But why? Because the time it takes to compute the result is lesser. Why? Because the number of operations performed during the course of solving Problem 3 are lesser than those of Problem 2? No.

Problem 3 is easier than Problem 2 because the nature of the operations performed in Problem 3. The operations there are simpler.

0 + 1,
0 + 0,
1 + 1

are arguably simpler operations than

9 + 5,
9 + 2,
4 + 9

I’ve attempted to capture the basic operations we perform while doing arithmetic and assign a level of difficulty to each. I’ve then emulated the algorithms we follow to compute solutions to such problems. By doing the following, it becomes possible to estimate the “difficulty” of an arithmetic problem (involving the basic operations).

Operation difficulties

Addition

addition_difficulties = {
    'digit_zero' : 1,   # any digit added to zero
    'even_even'  : 2,   # sum of even digits
    'odd_odd'    : 2,   # sum of odd digits
    'even_odd'   : 3,   # sum of even and odd digits
    'carry'      : 2    # difficulty of carry (for remembering and then adding)
}

Here is the function which computes the addition difficulty. It is a recursive function that can compute the addition difficulty for n numbers. It computes the sum and difficulty of the first two numbers and then inserts the sum at the head of the list of n numbers in place of the two numbers just summed. It then calls itself with the modified list.

def compute_addition_difficulty(*numbers):
    '''
    Generates a difficulty number and sum for given
    list of numbers for the addition operation
    *args - integers
    returns: (sum, difficulty)

    >>> compute_addition_difficulty(0, 0)
    (0, 1)

    >>> compute_addition_difficulty(0, 1)
    (1, 1)

    >>> compute_addition_difficulty(1, 1)
    (2, 2)

    >>> compute_addition_difficulty(1, 2)
    (3, 3)

    >>> compute_addition_difficulty(2, 2)
    (4, 2)

    >>> compute_addition_difficulty(19, 9)
    (28, 5)

    >>> compute_addition_difficulty(99999, 12345)
    (112344, 22)
    '''

    numbers = list(numbers)
    if len(numbers) == 0: return 0, 0


    elif len(numbers) == 1: return numbers[0], 0

    num1, num2 = numbers[:2]
    num1 = str(num1)
    num2 = str(num2)

    max_length = max([len(num1), len(num2)])
    num1 = num1.rjust(max_length, '0')
    num2 = num2.rjust(max_length, '0')

    difficulty = 0
    carry = 0
    r = reversed
    ad = addition_difficulties
    sum = []

    for index, (d1, d2) in enumerate( izip( r(num1), r(num2) ) ):
        d1 = int(d1)
        d2 = int(d2)

        d1_is_even = is_even(d1)
        d2_is_even = is_even(d2)

        if not d1 or not d2: difficulty += ad['digit_zero']
        elif d1_is_even and d2_is_even: difficulty += ad['even_even']
        elif not d1_is_even and not d2_is_even: difficulty += ad['odd_odd']
        elif d1_is_even != d2_is_even: difficulty += ad['even_odd']

        dsum = d1 + d2 + carry

        if dsum > 9:
            carry = 1
            difficulty += ad['carry']
        else:
            carry = 0

        sum.append( str(dsum % 10) )

    if carry:
        sum.append( str(carry) )

    sum.reverse()
    sum = ''.join(sum)
    sum = int (sum)

    numbers = [sum] + numbers[2:]

    sum, sub_difficulty = compute_addition_difficulty(*numbers)

    difficulty += sub_difficulty

    return sum, difficulty

Subtraction

subtraction_difficulties = {
    'digit_zero' : 1,    # difference of zero and any digit
    'same_digits': 1,    # difference of same values digits
    'even_even'  : 2,    # difference of even digits
    'odd_odd'    : 2,    # difference of odd digits
    'even_odd'   : 3,    # difference of even and odd digits
    'borrow'     : 2,    # doing a borrow
    'twodigit_digit' : 4 # difference of two-digit number and digit
}

The subtraction function is also recursive and operates similar to addition in that respect. Before looking at the subtraction algorithm, take a look at how “borrowing” is implemented. Let us say we are trying to perform the following,

200005 -
     6

After borrowing the number would be, 19999(1)5. We would then subtract 6 from 15 and proceed. The do_borrow() function captures this aspect of substraction.

def do_borrow(num, index):
    if index == len(num)-1: raise Exception('cannot perform borrow')

    num_borrows = 1
    next_digit = int(num[index+1])

    if next_digit > 0:
        num[index+1] = str(next_digit-1)

    else:
        num[index+1] = str(9)
        num_borrows += do_borrow(num, index+1)

    return num_borrows

This implementation is a bit confusing so check out this example (do_borrow() applied to the problem mentioned above).

>>> x = list('200005')
>>> x.reverse()
>>> x
['5', '0', '0', '0', '0', '2']
>>> do_borrow(x, 1)
4
>>> ''.join(x)
'509991'
>>> x.reverse()
>>> x
['1', '9', '9', '9', '0', '5']

Note that do_borrow returned 4, which indicates that the difficulty of performing this borrow operation is 4. Also note that do_borrow takes a list of digits in reversed order. It takes a list because the result is provided in-place (remember that Python strings are immutable). Now that you have seen how the borrowing works, all introduce the subtraction function.

def compute_subtraction_difficulty(*numbers):
    '''
    Generates a difficulty number and result for given
    list of numbers for the subtraction operation
    *args (tuple) - integers
    returns: (difference, difficulty)

    >>> compute_subtraction_difficulty(0,0)
    (0, 1)

    >>> compute_subtraction_difficulty(1,0)
    (1, 1)

    >>> compute_subtraction_difficulty(1,1)
    (0, 1)

    >>> compute_subtraction_difficulty(2,1)
    (1, 3)

    >>> compute_subtraction_difficulty(4,2)
    (2, 2)

    >>> compute_subtraction_difficulty(4,3)
    (1, 3)

    >>> compute_subtraction_difficulty(100,1)
    (99, 10)

    >>> compute_subtraction_difficulty(5000007,9)
    (4999998, 22)
    '''

    numbers = list(numbers)
    if len(numbers) == 0: return 0, 0
    elif len(numbers) == 1: return numbers[0], 0

    num1, num2 = numbers[:2]
    if num1 < num2: num1, num2 = num2, num1

    num1 = str(num1)
    num2 = str(num2)

    max_length = max([len(num1), len(num2)])
    num1 = list(num1.rjust(max_length, '0'))
    num2 = list(num2.rjust(max_length, '0'))

    difficulty = 0
    borrow = 0
    sd = subtraction_difficulties
    difference = []
    num1.reverse()
    num2.reverse()

    for index, (d1, d2) in enumerate( izip( num1, num2 ) ):
        d1 = int(d1)
        d2 = int(d2)

        d1_is_even = is_even(d1)
        d2_is_even = is_even(d2)

        if d1 > d2:
            if not d1 or not d2: difficulty += sd['digit_zero']
            elif d1_is_even and d2_is_even: difficulty += sd['even_even']
            elif not d1_is_even and not d1_is_even: difficulty += sd['odd_odd']
            elif d1_is_even != d2_is_even: difficulty += sd['even_odd']
            ddiff = d1 - d2

        elif d1 < d2:
            num_borrows = do_borrow(num1, index)
            difficulty += sd['borrow']*num_borrows
            ddiff = 10 + d1 - d2
            difficulty += sd['twodigit_digit']

        elif d1 == d2:
            difficulty += sd['same_digits']
            ddiff = 0

        difference.append( str(ddiff) )

    difference.reverse()
    difference = ''.join(difference)
    difference = int (difference)

    numbers = [difference] + numbers[2:]

    difference, sub_difficulty = compute_subtraction_difficulty(*numbers)

    difficulty += sub_difficulty

    return difference, difficulty

Multiplication

multiplication_difficulties = {
    'carry'  : 1,   # carry operation, summation difficulty is seperate
    'offset' : 1,
}

The basic operation of multiplying big numbers is one digit by one digit multiplication. I’ve tried to capture the difficulty of this operation as follows

def get_single_digit_multiplication_difficulty(d1, d2):
    '''
    Max value of multiplication of two digits is 81 (9*9).
    Difficulty of d1*d2 increases as the resulting value
    of the multiplication increases. thereby 9*9=81 is most difficult
    and 0*0 is least difficult.

    We will represent this difficult with value from 1-4 (inclusive)

    Note that the difficulty will be incremented by 1 for presence
    of odd digit in the operands (except for 1 and 5 because it is arguably
    easier to multiply)

    >>> get_single_digit_multiplication_difficulty(0, 1)
    (0, 1)
    >>> get_single_digit_multiplication_difficulty(1, 1)
    (1, 1)
    >>> get_single_digit_multiplication_difficulty(2, 5)
    (10, 1)
    >>> get_single_digit_multiplication_difficulty(2, 7)
    (14, 2)
    >>> get_single_digit_multiplication_difficulty(2, 6)
    (12, 1)
    >>> get_single_digit_multiplication_difficulty(8, 9)
    (72, 5)
    >>> get_single_digit_multiplication_difficulty(8, 8)
    (64, 4)
    '''

    if not d1 or not d2: return 0, 1

    res = d1 * d2
    difficulty = math.ceil((4/81.) * res)

    # odd numbers are harder to multiply except 1 and 5
    if d1 not in [1,5] and d2 not in [1,5] and (is_odd(d1) or is_odd(d2)):
        difficulty += 1

    return res, int(difficulty)

At the next level, the operation would be multiplying a multi-digit number by a single digit, which I call simple multiplication. Over here I take into account the difficulty of carry. I hope the following code is self explanatory.

def compute_simple_multiplication_difficulty(num, digit):
    '''
    >>> compute_simple_multiplication_difficulty(1, 0)
    (0, 1)
    >>> compute_simple_multiplication_difficulty(1, 1)
    (1, 1)
    >>> compute_simple_multiplication_difficulty(2, 1)
    (2, 1)
    >>> compute_simple_multiplication_difficulty(10, 1)
    (10, 2)
    >>> compute_simple_multiplication_difficulty(15, 1)
    (15, 2)
    >>> compute_simple_multiplication_difficulty(15, 5)
    (75, 7)
    >>> compute_simple_multiplication_difficulty(999, 7)
    (6993, 25)
    '''
    num = str(num)

    result = []
    md = multiplication_difficulties
    carry = 0
    difficulty = 0

    for index, d in enumerate(reversed(num)):
        d = int(d)
        res, s_diff = get_single_digit_multiplication_difficulty(d, digit)
        difficulty += s_diff

        if (carry):
            res, carry_sum_difficulty = compute_addition_difficulty(res, carry)
            difficulty += carry_sum_difficulty + md['carry']

        carry = res/10
        result_digit = int(str(res)[-1:])
        result.append( str(result_digit) )

    if carry:
        result.append( str(carry) )

    result.reverse()
    result = ''.join(result)
    result = int(result)

    return result, difficulty

Simple multiplication is a basic operation of complex multiplication (multi-digit multiplied by multi-digit). Let us see an example of “complex multiplication” which the multiplication code follows.

    345 x
    123
  -----
   1035
   690x  # result is "offset" or multiplied by 10
  345xx  # result if "offset" or multiplied by 100
  -----
  42435  # result of summation of above numbers
  -----

The first part of the multiplication is to perform simple multiplications - 345x3, 345x2, 345x1 and to offset the numbers accordingly. The second part of the operation is to sum the numbers. I used the addition algorithm to achieve the second part. Check out the multiplication function.

def compute_multiplication_difficulty(*numbers):
    '''
    Generates a difficulty number and result for given
    list of numbers for the multiplication operation
    *args (tuple) - integers
    returns: (product, difficulty)

    >>> compute_multiplication_difficulty(0,0)
    (0, 1)
    >>> compute_multiplication_difficulty(1,0)
    (0, 1)
    >>> compute_multiplication_difficulty(1,1)
    (1, 1)
    >>> compute_multiplication_difficulty(2,1)
    (2, 1)
    >>> compute_multiplication_difficulty(5,1)
    (5, 1)
    >>> compute_multiplication_difficulty(5,2)
    (10, 1)
    >>> compute_multiplication_difficulty(17,2)
    (34, 7)
    >>> compute_multiplication_difficulty(17,29)
    (493, 20)
    >>> compute_multiplication_difficulty(17,29,3)
    (1479, 31)
    >>> compute_multiplication_difficulty(1776,29,3)
    (154512, 77)
    '''

    numbers = list(numbers)
    if len(numbers) == 0: return 1, 0
    elif len(numbers) == 1: return numbers[0], 0

    num1, num2 = numbers[:2]
    if num1 < num2: num1, num2 = num2, num1

    num2 = str(num2)

    difficulty = 0
    borrow = 0
    md = multiplication_difficulties
    m_numbers = []

    for index, d in enumerate(reversed(num2)):

        d = int(d)

        m_number, m_diff = compute_simple_multiplication_difficulty(num1, d)
        difficulty += m_diff

        if index:
            m_number = int(m_number * math.pow(10, index))
            difficulty += md['offset']

        m_numbers.append(m_number)

    m_numbers_sum, m_numbers_diff = compute_addition_difficulty(*m_numbers)
    difficulty += m_numbers_diff

    numbers = [m_numbers_sum] + numbers[2:]
    product, m_diff = compute_multiplication_difficulty(*numbers)

    difficulty += m_diff

    return product, difficulty

Division

division_difficulties = {
    # long division
    'use_digit'       : 1,  # brinding down digit from dividend
    'multiple_lookup' : 1,  # looking up precomputed multiple of divisor
    'quotient_update' : 1,  # updating quotient with digit or period
}

Implementing long division posed a problem because my understanding of the mechanics behind the long division algorithm was minimal if not inexistent. I spent some time trying to “reverse-engineer” it. I came across “Egyptian Division” which made things clearer. With a little help, I managed to implement the following division algorithm. Please let me know, if you come up with a better approach.

def compute_division_difficulty(dividend, divisor, precision):
    '''
    Generates a difficulty number, quotient and remainder
    for dividend / divisor
    @precision (int) -- max required number of digits after decimal point in quotient
    returns: (quotient, remainder, difficulty)

    >>> compute_division_difficulty(54, 5, 0)
    (10.0, 4, 6)
    >>> compute_division_difficulty(50, 5, 0)
    (10.0, 0, 6)
    >>> compute_division_difficulty(575, 6, 0)
    (95.0, 5, 60)
    >>> compute_division_difficulty(575, 6, 1)
    (95.829999999999998, 20, 112)
    >>> compute_division_difficulty(6, 9, 1)
    (0.66000000000000003, 60, 50)
    >>> compute_division_difficulty(410, 2, 1)
    (205.0, 0, 54)
    '''

    if not divisor: raise Exception('Division by Zero')
    if not dividend: return 0, 0, 1

    dividend = str(dividend)

    divisor  = str(divisor)

    difficulty = 0
    previous_multiples_difficulty = 0
    precision_reached = 0
    decimal_point_used = 0
    dd = division_difficulties
    num = []
    quotient = []

    num = dividend[0]
    difficulty += dd['use_digit']
    index = 0

    while 1:
        q_digit = int(num) / int(divisor)

        multiple, multiple_difficulty = compute_multiples_difficulty(int(divisor), q_digit)
        difficulty += int(math.fabs(multiple_difficulty - previous_multiples_difficulty))
        if previous_multiples_difficulty: difficulty += dd['multiple_lookup']
        previous_multiples_difficulty = max(multiple_difficulty, previous_multiples_difficulty)

        quotient.append( str(q_digit) )
        difficulty += dd['quotient_update']

        if decimal_point_used:
            precision_reached += 1

        num, sub_difficulty = compute_subtraction_difficulty(int(num), multiple)
        difficulty += sub_difficulty
        num = str(num)

        index += 1

        if index == len(dividend):
            if precision == 0: break
            quotient.append('.')
            difficulty += dd['quotient_update']
            decimal_point_used = 1

        if not decimal_point_used:
            num += dividend[index]
        else:
            num += '0'

        difficulty += dd['use_digit']

        if (len(num) == 1 and not int(num)) or precision_reached >= precision + 1:
            break

    remainder = int(num)
    quotient = float(''.join(quotient))

    return quotient, remainder, difficulty

You must’ve noticed the usage of compute_multiples_difficulty(). In long division, at every step, you will try to find the largest multiple of the divisor less than or equal to the number in hand to proceed further. During the process of division, if you’ve computed the 5th (say) multiple of the divisor with difficulty N. The computation of 6th multiple at a later point in the division is difficulty(lookup of 5th multiple) + difficulty(6-5th multiple).

Results

#Problem 1
>>> compute_addition_difficulty(5, 3)
(8, 2)

#Problem 2
>>> compute_addition_difficulty(99999, 12345)
(112344, 22)

#Problem 3
>>> compute_addition_difficulty(1000000, 1000001)
(2000001, 8)

Problem 1: 2
Problem 2: 22
Problem 3: 8

The End

I wish I could have written a more detailed explanation instead of sprinkling this article with code. Time however prevents me from doing so :(. It is some relief that Python is a wonderful language for readability and the code samples above are pretty close to pseudo-code.

I will post back when I complete the program to generate arithmetic problems by order of difficulty. Until then, Adios amigos. Almost forgot! Here is the code.

Even a python can be abused

2008-07-11T08:34:00-07:00

The first programming language I coded in is QuickBasic. I loved the simplicity and especially the IDE. It made things simple for a starter. Later I discovered Visual Basic which extended the same simplicity and added the “Visual” element with a splendid editor for GUI.

In between I did some projects using Java, C#, C, C++. None of these impressed me too much. I hated Java’s imposition of stiff rules and it’s dogged adherence to “everything in a class” attitude. C# was better. C++ just turned me off because of the monster it is. I did not like C at all because of its total lack of automated memory handling (like GC). I’ve been doing a lot of coding in C now-a-days as part of my job and I must admit that I like it a lot for its simplicity in primitives and promise of “closeness to hardware” and hence the predictability and performance.

I did a small part of my final year project using Python. However, for some unfathomable reason, Python did not impress me at all then. When I started working at Veveo I used Python for a project and got hooked. It’s simplicity and “readability” got me. The power of wielding this tool got me drunk :)

Python was designed from the beginning to be a “easy to read” language. Most, if not all, of the syntax is intuitive. The indentation adds to the readability aspect. The policy of “only one way to do a thing” does wonders for readability. Everyone does a certain thing only the “one” way. If you are wondering why the heck I am talking so much about readability, you should consider the fact that an average programmer spends most of his time “reading” code. You have to read your code after you’ve just written it. You’ve to read your code the next day when you resume work. You’ve to read your code the moment a bug is found. You have to read your code when someone asks you how some aspect of it works a couple of months later. You’ve have to read your code when making a teeny-weeny feature addition. I just cannot emphasize enough how much time is spent is just reading. There have been times when I would spend a whole day just reading code and finally making “a single line of code change” at the end of the day!

So there it is. Python makes it possible to write readable code and that does wonders to programmer productivity. Maintaining your code becomes easier. Understanding your collegues code becomes easier and most of all understanding code written by someone across the world becomes easier - so you can start reusing components more quickly and with more confidence than ever before.

I simply love Python.

You must be wondering where I am taking this discussion… because the title says something about a Python being abused. Who is abusing the Python anyway?

I’ve noticed how newbies code in Python and found it particulary intriguing. What is interesting here is that every person comes from a certain programming background and are used to doing things in a certain way. When they are thrown into a situation where they have to learn a new language and write some code in it, they invariably apply the vast experience they have in their language of choice :)

I’ve had the oppurtunity of observing people from Java, C, Perl, VB (yes, Perl) backgrounds writing code in Python. The Java guys stress test multiple inhertitance in Python and bring the much cherished “everything in a class” practice to the Python. The C guys who are more often than not obsessed with performace and optimization put their brains to work and implement a strcpy using a “for loop” and insist on doing a “shift” instead of “division/multiplication”. The Perl guys just don’t seem to like the alphanumerals. They craft Python code with ingenious application making it look very concise. The more characters in one line the better the code. The more non-alphanumerals the better coder you are. That’s the way of the “Perl’ies”. The VB guys languish for a while complaining constantly about the lack of a proper IDE and after trying out various Python editors, decide to call it quits and go home to comfortable VB. I know, I know ….. I was a VB guy too and I did search for IDE’s too … But then I found VIM and everything was good :)

I’ve had the good (snigger) fortune of maintaining some of these brilliant artifacts and had my share of nightmares and laugh-outs. I thought I had seen it all, until I saw something today. I told myself —- “Never underestimate a brilliant C programmer who has found exec and eval in Python” … Yes, you can quote me on this :)

    guido = "is speechless"
    larry = "went nuts"
    sergey = "has seen it all now"
    ...
    ...
    for idx in ['guido','larry','sergey']:
        idxv=eval(idx)
        if(not idxv and dd.has_key(idx)):
            idxv='%s="%s"' %(idx,dd[idx])
            exec(idxv)

    ...
    ...

No….. I don’t think I’ve seen it all…. ;)

Nose - TDD - Python

2008-05-22T20:20:00-07:00

What, why

I’ve been reading up on TDD and it has struck me as particularly useful methodology to achieve “clean code that works”. TDD encourages writing unit tests to cover all the code (because by definition, you write a test before a line of code is written). Because all your code is covered you are freed from the fear of breakage due to change and can instantly be more confident and productive. Also, the test cases act as a specification in code - very useful.

Python has standard modules, unittest and doctest to help you write test cases. I simply love doctest. It alleviates much of the pain of writing a test case (setup and all) besides acting as “executable documentation”. The unittest module has a Java legacy and is not to my taste. Also, I wanted to find a solution that would help in automated test enumeration (discovery) in my source directories without having to write any “infrastructure” code. One more thing I was looking for was a way to run both unit tests and doc tests together.

After a bit of searching, I found “Nose“. Nose is a clone of “py.test” which I liked better than the original (subjectively). To get a feel of “Nose”, I set up some python test files.

The following is the directory structure and the contents of the files. I’ve put in both unit tests and doc tests in the files to see how “Nose” handles them. Also, the tests are spread across directories. Note that I had to put an “__init__.py” to allow “Nose” to import tests in a subdirectory.

The setup

The directory structure

prashanth@prashanth-desktop:~/tmp$ tree
.
|-- bingo.py
|-- somedir
|   |-- __init__.py
|   `-- test_another.py
`-- test_prashanth.py

1 directory, 4 files

bingo.py

def boing(a, b):
    '''
    >>> boing(10, 20)
    30
    '''
    return a+b

def boing1(a, b):
    '''
    >>> boing1(10, 20)
    40
    '''
    return a+b

test_prashanth.py

def test_a():
    assert 1

def test_b():
    print "hello"
    assert 0

somedir/test_another.py

def test_bingo():
    raise Exception('hgello')

Installing “Nose”

sudo easy_install nose

If you don’t have easy_install, head over here to get information on installation.

Running the tests

Now that “Nose” is installed, let us run the tests,

nosetests --with-doctest

The output is

..E.F
======================================================================
ERROR: somedir.test_another.test_bingo
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/nose-0.10.2-py2.5.egg/nose/case.py", line 182, in runTest
    self.test(*self.arg)
  File "/home/prashanth/tmp/somedir/test_another.py", line 2, in test_bingo
    raise Exception('hgello')
Exception: hgello

======================================================================
FAIL: test_prashanth.test_b
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/nose-0.10.2-py2.5.egg/nose/case.py", line 182, in runTest
    self.test(*self.arg)
  File "/home/prashanth/tmp/test_prashanth.py", line 7, in test_b
    assert 0
AssertionError:
-------------------- >> begin captured stdout << ---------------------
hello

--------------------- >> end captured stdout << ----------------------

----------------------------------------------------------------------
Ran 5 tests in 0.057s

FAILED (errors=1, failures=1)

The first line in the output is the “test progress” indication (..E.F) . When a test succeeds, a ‘.’ is written. When a test fails, an ‘F’ is written. When a test throws an Exception, an ‘E’ is written. Very useful to get a sense of progress as a huge test suite being executed.

“Nose” captures the stdout and stderr when a test case fails to help you debug the issue. To learn more about using “Nose” go here.

Wordza - A Smart Word Quizzer

2008-04-27T13:54:00-07:00

I’d thought of making a word quizzer as a web application to improve my vocabulary when I took the GRE test a couple of years back. I’d written one in Visual Basic 6 when I wrote SAT :), but desktop applications are boring!

I got inspired to bring my long standing idea to fruition and the outcome is Wordza.

Wordza is a smart word quizzing service designed to help you rapidly learn new words. It adapts to your level and continually challenges you to get better. Preparing for GRE/SAT/CAT? Use Wordza 15 minutes a day.

Questions have difficulty ratings ranging between 1 and 100. The ratings are derived from user activity. A question is easy if a lot of people get it right and difficult when very few do!

You start at difficulty level 50. If you get three questions right in a row, you drive up the difficulty level a notch. On the other hand, answering a question wrong will reduce the difficulty level by one.

If you are interested in the implementation … I used Python, MySQL behind Apache to get this up. When I started making it, I wanted to keep it simple and make it work fast. I hope I’ve met both these requirements. Do give it a try and let me know what you think.

Selecting a random row from a table in mysql

2008-04-08T21:30:00-07:00

I have come across more than one instance when I had to select a random record from a table in a MySQL database. Here is how to do it.

The simple but slow method

SELECT * FROM mytable ORDER BY RAND() LIMIT 1;

Although simple, the above query can be very slow on tables which have a large number of records. This happens because MySQL makes a temporary table with a random number assigned to each row. It then sorts the table and returns the one record from the sorted list.

Random record selection by id generation : A better and faster method

The table will need to have an ‘id’ field which is an auto-incrementing integer. The approach is to generate a random number which falls in the range of the values of the ‘id’ field. MySQL MAX() and MIN() grouping functions allow you to choose the maximum and minimum values of a field.

The python approach

Sample python code to do the random record selection

import random
...
cursor.execute('SELECT MIN(id) FROM mytable')
min_value = cursor.fetchone()
cursor.execute('SELECT MAX(id) FROM mytable')
max_value = cursor.fetchone()

random_id = random.randint(min_value, max_value)

cursor.execute('SELECT * FROM mytable WHERE id = %s', (random_id,))
random_record = cursor.fetchone()

Pure SQL approach

MySQL offers a RAND() function which generates a random floating point number between 0.0 and 1.0. We will use this value and scale it to fit into the max - min value range of the id field.

SELECT * FROM mytable WHERE id = (SELECT MIN(id) + FLOOR((MAX(id)+1) * RAND()) FROM mytable ) LIMIT 1;

So next time you have to select a random record, you know what to do :)

Additional information

How to generate domain names, place names … product names ?

2008-04-03T22:35:00-07:00

One of the trickiest and enjoyable parts of starting something new (be it a website, project, band) is naming it! Sometimes a good name can be quite elusive and cause more than the deserved share of brain ache. Here is a list of automated services around the internet that will help you get name suggestions.

Let us name a domain!

http://www.domaintools.com/

DomainTools takes a concept as input and comes up with domain name suggestions. Let us say you are starting a website about “Vacations in Mexico”. Go to their website and type in “Mexico Vacations” in the text box and click on the button to get suggestions.

Domain name suggestions for “Mexico Vacations”

MexicoFreeTours.com
GreatMexicoTrips.com
MexicoGreatBreaks.com
MyMexTrips.com
FunMexVacations.com

Not bad at all. I like GreatMexicoTrips.com.

Another tool out there is Noemata Domain name generator. This one gives random domain name suggestions. Not very impressive.

kawlant.com
phobolapa.com
jupidox.com
nitrurth.com

Doing a web2.0 start-up? Don’t sweat over the name

Flickr, tumblr, zoho - all taken up? Find more!

The Web2.0 name generator

Gedel
Eavu
Plamba
Trilith
Zooveo

Another Web2.0 name generator

65pages
inclicks
type17
bubbledesks

Another one! - This one checks domain availability too. sweet.

Skiplounge
Mivee

Web2.0 buzz generator

2.0 is the new New. It’s all changing. Label what defies categorization. “ASL” is geezer speak. Podcasts. The words aren’t what they were. Clear that. Hack it. Faster. Faster! Float this. Roll your own roll-your-own. MSM just doesn’t get it. Always be launching. The buzz is loud and clear. You need someone who gets it. On-demand streams. Splog is an aggregate noun. We are on the brink of a new age. This will change everything. We’re about what Web 2.0 is about. Single. Word. Sentences! News clouds. An AJAX-driven GUI. Cry out, blogosphere! This is newer media. The new is old. Tag me. We shall transcend borders. Clustering. It’s all about community. Social is the new push. Folksonomy.

Starting a band? Check out the band name generator

B.A.N.G - Band automatic name generator!
Subtle Alphabet
Uncommon Elasticity
Notorious Sand
Crass Ink

Play Civ3? Want a name for your new city?

The place name generator

Agea
Byzernova
Genesini
Mercentia
Myruaror

The fantasy place name generator

Aldmeadow
Blacklake
Southshore
Wildecoast
Witchmead

The British town name generator

Cuddleballabridge, Cambridgeshire
Buttercanter Gate up the Snotty, Wiltshire
Old Chippingbeck, Shropshire
Dorkstrathsack in the Gloaming, Somerset

Game name generators

RPG Name Generator

Gilder’s Airship
Ruby Law V
Sad Guardian
Star: Ghost of the Interstellar Air
Unnamed Peace XI

The Ultimate game name generator

Aqua Badger
Nuclear Dazzler

Castlevania - game name generator

Surf Saxophone of Dog Poo
Minuet of Skunky Beer
Bathroom Break of Stubbed Toes

The Video game name generator

Intelligent Kangaroo Park
Spectacular Dentist All-Stars
Master Chief’s Fantasy Rocket

Character name generator

Ageroth
Cheon
Hagiti
Roddareld

Name your dream gadget!

Wacky gadget name generator

Atomic Chronostinker
Computerized Golfing Zapperion
Configurable Barfer
Configurable Electromagnetic Lithowhiner
Dimension Whiner

Name a corporation or government

Corporation name generator

Boyd Science Logistics Holdings
Cosmotechnics
Estrada Medical Transportation
Genetics Cosmosystems
Gold Informational Solutions

Government generator

Ancient Imperial Unity
Constitutional Brotherhood
Dominion of States
Provincial Authority of Territories

Name the pub

Pub name generator

The tenrec & indian
The boa constrictor & bar
The kudu & deal
The serval & guide

Cook up a research topic for your thesis :) !

A research topic generator!

A virtual high-speed agent embedded in a collaborative digital interface
An active mobile display embedded in a meta-level programmable solution
A high-level logical preprocessor related to an interactive concurrent theorem prover
A responsive high-speed preprocessor applied to an active proxy network

Throw some insults around!

An insult generator

Sneezefinger
Muckfoot
Bumface
Wadface

People names!

Greek names

Turatia
Dodius
Struinaus
Kymius

Japanese names

Omira
Ayuro
Ekochiyu
Hetamuri

Cook up some colors

Color name generator

Cobalt Mesa
French Latte
Slate
Southern Maroon
Swirl Purple

I am sure there are many more around the web that I missed. If you find anything interesting, comment about it!

Visualizing mpeg4 motion compensation vectors using mplayer

2008-03-30T00:17:00-07:00

The MPEG4 video encoding process makes use of block motion compensation to achieve compression. The motion compensation process serves to produce the intra frames which are the frames between keyframes. I’ve always been fascinated by this process and was delighted to find out that my favorite video player, mplayer, allows one to visualize this process. I tried it and it is wonderful!

This is the video I tried the process on: download

To try this out yourself, you will first need to have mplayer installed. If you are running Debian/Ubuntu, do this,

sudo apt-get install mplayer

After this, download the video using the link provided above. Run this command,

mplayer -lavdopts vismv=1 -vo x11 motion_vector_video.avi

To learn more about motion compensation, go here. Enjoy!

Creating video thumbnails using ffmpeg

2008-03-29T00:28:00-07:00

Generating thumbnails/screenshots of a video is useful in many ways. Youtube and many other video sites use this to show a preview of the video as a small thumbnail. Google video captures a series of thumbnails from a video at various time intervals to show a better video preview.

There is a simple way to generate thumbnails in Linux using ffmpeg, a very very useful tool for processing videos. Let us generate a thumbnail for this video. Download the video and rename it to test.avi.

Then, run this command,

ffmpeg  -itsoffset -4  -i test.avi -vcodec mjpeg -vframes 1 -an -f rawvideo -s 320x240 test.jpg

This command generates a 320x240 sized PNG thumbnail at the 4th second in the video. The output looks like this.

If you do not have ffmpeg installed and are using Debian/Ubuntu, do,

sudo apt-get install ffmpeg

A series of thumbnails, at 4, 8, 12 and 16 seconds.

ffmpeg  -itsoffset -4  -i test.avi -vcodec mjpeg -vframes 1 -an -f rawvideo -s 320x240 test.jpg
ffmpeg  -itsoffset -8  -i test.avi -vcodec mjpeg -vframes 1 -an -f rawvideo -s 320x240 test.jpg
ffmpeg  -itsoffset -12  -i test.avi -vcodec mjpeg -vframes 1 -an -f rawvideo -s 320x240 test.jpg
ffmpeg  -itsoffset -16  -i test.avi -vcodec mjpeg -vframes 1 -an -f rawvideo -s 320x240 test.jpg

useful links

Happy thumbnailing!

Songza - music search engine and jukebox

2008-03-27T23:17:00-07:00

Every once in a while, someone comes up with a way of doing things in an extremely obvious and simple way. When this happens, a zillion others say, “of course that’s the way to do it!”. Songza is a music search engine and jukebox that is dead simple to use. You should try it to really grok how simple the interface is.

Songza is the single best place to search for and listen to music on the Web. Find a song and listen immediately, in one place.

Unlike KaZaa or Bit Torrent, Songza users can only listen to songs, not download them. And unlike Last.fm or Rhapsody, Songza permits users to choose exactly the song or artist they want to hear, and does not require them to subscribe or pay for its services.

Songza is the brainchild of 24-year-old Aza Raskin — the son of Apple Macintosh founder Jef Raskin. He and Scott Robbin worked feverishly to bring the idea to life. The site launched on November 8, 2007, and instantly generated significant buzz around the world. Hundreds of bloggers and news writers have praised its elegant user interface, beautiful design, and all-around utility. Like all good ideas, both the concept and design came to Aza while he was in the shower.

The thing I liked most about songza is the “pie-menu”.

useful links

http://ajaxian.com/archives/songza-would-you-like-a-habituatable-pie-menus-with-your-social-music
http://www.downloadsquad.com/2008/03/04/songza-adds-new-features/
http://humanized.com/weblog/2007/11/13/songza-launch/
http://blog.wired.com/music/2007/11/songza-turns-yo.html
http://lifehacker.com/software/music/listen-to-music-on+demand-with-songza-321638.php

Rendezvous with Rama - Goodbye Mr. Clarke

2008-03-25T22:36:00-07:00

I am a huge fan of the science fiction genre. Arthur C Clarke is one of my favorite science fiction writers after Isaac Asimov. It saddens me to have learnt that he has passed away. Most people get reminded of “2001: A Space Odyssey” when they hear the name Arthur C Clarke. I get reminded of “Rendezvouz with Rama“, a brilliantly conceived novel that set my imagination on fire. For all the Clarke fans out there, “Rendezvouz with Rama“.

Rendezvous with Rama

Rama interior - somewhat inaccurate

A short film

links

The ramans always do things in threes!

Watching Television on Linux: setting up a TV Tuner card

2008-03-23T18:37:00-07:00

A couple of weeks back, I went shopping looking for a TV tuner card that is compatible with Linux. Googling had told me that “Hauppauge” card was known to be compatible. However, I could not find it anywhere in the market (SP Road, Bangalore, India). On going to one of the shops, I found a “Pinnacle PCTV 50i” card. I had heard from many people before that Pinnacle was a good card for Windows both in terms of quality of decoding and software provided. I checked in google to ensure that Pinnacle card would work on Linux. I found that the card uses Philips’ SAA7134 chipset for which drivers are available in Linux. I went on and bought the card for Rs 2000 (\$50).

The card is available through Amazon.com. Pinnacle PCTV Analog PCI 50i - TV / radio tuner / video input adapter - PCI - SECAM, PAL

Setting up the card in Linux

I have Ubuntu Gutsy installed (7.10) and followed this guide to setup the card and the instructions worked like a charm. A gist of the procedure.

Ubuntu Gutsy recognizes the card automatically, the following commands help to check if the recognition happened or not. Both commands should give atleast one line of output each.

sudo update-pciids
lspci | grep -i saa713
dmesg | grep saa713

Watching TV - tvtime

tvtime is a simple to use application for watching TV on Linux. Here is how you configure it.

sudo apt-get install tvtime
tvtime &

Right click to open a setup menu. You can also hit [F1] or [tab] key to show configuration menu: Linux Tvtime configuration hit [F1] or [tab] key to show configuration menu (Fig. 01: Linux tvtime application configuration menu) First, you need to setup video source, frequencies and channels etc.

Setup video source

Visit Input Configuration > Change Video Source > Setup source as per your setup.

Setup frequencies table

You need to setup the channels you are receiving with the tuner. Here we use europe frequencies table. Visit Channel management > Change Frequency table > Select new table Scan Channel

Finally, you need to scan for channels by visiting Channel management > Change Frequency table > Scan channels for signal > Back > Exit

Sound configuration

You need to attach internal cables to get sound out of TV tuner card. Use driver saa7134-oss or saa7134-alsa to get sound out of card. This driver get loaded automatically. Open terminal and type the following command to hear the audio through ALSA using tvtime (or other programs that don’t support it directly):

$ sox -c 2 -s -w -r 32000 -t ossdsp /dev/dsp1 -t ossdsp -w -r 32000 /dev/dsp

Recording TV using mencoder

if you do not already have mencoder, you can install it (on Debian/Ubuntu) by doing

sudo apt-get install mencoder mplayer

Note that the above command will also install mplayer which is useful for viewing video files and tv channels. If you are the command line kind of person, you can do away with tvtime and use mplayer!

To record a TV channel, you will first need to figure out the frequency of the channel. For this I found scantv useful. This command will scan channels in the specified range and print out the found channels. After figuring out the frequency of the channel to record, use this mencoder command to record the video in a compressed format.

mencoder tv:// -tv driver=v4l2:input=0:norm=pal:width=640:height=480:device=/dev/video0:\
freq=775.25:adevice=/dev/dsp1:forceaudio:audiorate=32000 buffersize=64\
 -ovc lavc -lavcopts vcodec=mpeg4:vbitrate=400:keyint=30 -oac mp3lame\
-lameopts br=32:cbr:mode=3 -ffourcc divx -o "test.avi"

To use mplayer to view TV directly (without using tvtime),do this.

mplayer tv:// -tv driver=v4l2:input=0:norm=pal:width=640:height=480:device=/dev/video0:
freq=775.25:adevice=/dev/dsp1:forceaudio:audiorate=32000

I’ve been using this card without a glitch and recommend for HTPC attempts on Linux.

All NCERT text books!

2008-02-06T21:01:00-08:00

I recently came to know that NCERT was providing all the text books (class 1 - 12) for download. However, I found their interface hard to use for browsing through. So I wrote a crawler in python to get all the books to my webserver.

Along with storing the books, I generated thumbnails for every chapter. Navigation pages have also been generated to help browse through the books easily. You can access this dump here.

ncertbooks.prashanthellina.com

The crawler code is here.

These are the steps to run it yourself.

# scrape ncert site to get details about book pdf urls
python get_ncert_books_data.py > ncert_books_data

# create a directory to store all downloaded files
mkdir ncert_books

# download all the pdfs and cover page images
cat ncert_books_data | python download_ncert_books.py ncert_books

# generate thumbnails for each pdf (requires ImageMagick)
find ncert_books/ -iname "*.pdf" > all_pdf_files
sed -i "s/^/0\t/" all_pdf_files
python generate_thumbnails.py

# resize the book cover images
python resize_image_thumbnails.py ncert_books

# generate the navigation pages
python generate_navigation_pages.py ncert_books

For the thumbnail generation to work, you will need ImageMagick. Read this for more information.

FOSSConf08 - A disappointment

2008-02-03T14:22:00-08:00

I gave a talk at FOSSConf08 yesterday and came away feeling very disappointed and let down! It was a poorly organized event where total chaos prevailed. My friend Venkat and I found ourselves unable to show our presentation slides because of an overheating projector whose brightness could not out-match the sunlight coming in from the open windows!

Venkat’s talk got delayed by nearly 30 minutes because the machine provided to him was ancient running a version of Fedora which could not automount the USB drive he had.

The only thing we were happy about was the audience who patiently listened to chalk screeching on the blackboard!

To any of you out there organizing a conference, here is a piece of advice,

The contributors spend their time and money visiting such events to interact with fellow enthusiasts. The least, you, as an organizer can do is to make sure everything is smooth and taken care of. I am not saying, you should bear the travel expenses etc. Just make sure something as basic as a projector works!

Here is my poor presentation which did not see the light of the day :)

Create PDF thumbnails using ImageMagick on Linux

2008-02-03T13:59:00-08:00

I have a bunch of PDF files for which I wanted to generate thumbnails. On looking around a bit, I found “ImageMagick“. Since I have Ubuntu installed, I did

sudo apt-get install imagemagick

It is not too big a download at around 740KB.

To create a thumbnails for all pages in the PDF document (say test.pdf which has 3 pages), do

convert -thumbnail x300 test.pdf test.png
ls
test.pdf test-0.png test-1.png test-2.png

The “x300” tells the “convert” tool to keep the height at 300 pixels and modify the width to suit the height. If you have absolute dimensions in mind (say 400x300), do

convert -thumbnail 400x300 test.pdf test.png

If you want to control the width and have the height suited, do

convert -thumbnail 400 test.pdf test.png

What do you do if you want only the first page’s thumbnail? do,

convert -thumbnail x300 test.pdf[0] test.png

A neat feature of the “convert” tool is creation of animated gif’s having all the pages from the PDF document. do,

convert -thumbnail x300 test.pdf test.gif

“convert” is smart enough to figure out the output format from the file extension you provide in the arguments.

Nifty tools like “convert” from the ImageMagick suite are one of the reasons I love Linux! (ImageMagick is cross-platform, I’m referring the to utility of command-line tools and the UNIX philosophy of specialized tools that do one task really well)