Comments on: Clustering Data using Python

By: prashanthellina

prashanthellina — Mon, 27 Jul 2009 02:48:02 +0000

Ashish, I am reposting your distance function code with indentation preserved.

#Distance function compares two urls and finds the distance ,also considering the structure of url
#into account and having different weights for different parts of url, as so to give precedence of
#domain over params and so on
# uses SequenceMatcher from python standard module difflib
def distance(url1, url2):
    url1_frag = urlparse(url1)
    url2_frag = urlparse(url2)
    ratio = 0.0
    #  weights for now are [6,5,4,3,2,1]  for each of the tuples of urlparse
    #ratio sums (user assigned weight of the urlparse part) * (ratio returned by Sequence matcher )
    for index in xrange(0,6):
        ratio += (6-index) * (1.0 - SequenceMatcher(None, url1_frag[index],url2_frag[index] ).ratio())
    return ratio / 21.0

People, note that you will have to add this line to the top of the script to use this function.

from urlparse import urlparse

By: prashanthellina

prashanthellina — Mon, 27 Jul 2009 02:46:22 +0000

Wonderful suggestion Ashish. I was planning something similar to enhance the clustering. You have saved me some work.

Output of the above script after having replaced with your distance function.

[['#articles'],
 ['http://xkcd.com/612/',
 ['http://web.sourceforge.com/terms.phphttp://slashdot.org//it.slashdot.org/search',
  'http://web.sourceforge.com/privacy.php',
  'http://web.sourceforge.com/advertising',
 ['http://yro.slashdot.org/~drDugan/',
  'http://slashdot.org//slashdot.org/article.pl?sid=09/07/24/1545238',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814785',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814429',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814457',
  'http://slashdot.org//slashdot.org/comments.pl?sid=09/07/24/1545238&cid=28810581',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28815123',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28815269',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814385',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814335',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814657',
  'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814581',
  'http://slashdot.org//slashdot.org/~Darkness404',
  'http://slashdot.org//radio.slashdot.org'

By: sriram

sriram — Sun, 26 Jul 2009 02:39:02 +0000

Like the previous comment says, the distance function can be improved. python rocks!

By: Ashish Yadav

Ashish Yadav — Sat, 25 Jul 2009 09:47:58 +0000

As indentation is got all messed up, you can see the code snippet at http://python.pastebin.com/fceb4031

By: Ashish Yadav

Ashish Yadav — Sat, 25 Jul 2009 09:44:39 +0000

Hey prashant,
Blog came after a long time indeed.
I think that taking into account the structure of url, can make clustering more accurate.
Like for example two urls from same domain should be closer to each other than anything else, etc,
so try using this function for calculating distance between two urls.

# distance function compares two urls and finds the distance ,also considering the structure of url
#into account and having different weights for different parts of url, as so to give precedence of
#domain over params and so on
# uses SequenceMatcher from python standard module difflib
def customDistance(url1, url2):
url1_frag = urlparse(url1)
url2_frag = urlparse(url2)
ratio = 0.0
# weights for now are [6,5,4,3,2,1] for each of the elements of urlparse tuple, maximum 6 for domain and then decreasing.
#ratio sums (user assigned weight of the urlparse part) * (ratio returned by Sequence matcher )
#denominator is sum of the user assigned weights,
for index in xrange(0,6):
ratio += (6-index) * (1 – SequenceMatcher(None, url1_frag[index],url2_frag[index] ).ratio())
return ratio / 21