Clustering Data using Python (1,148 views)

Comments on "Clustering Data using Python": 5 Comments

1. Ashish Yadav wrote,

   Hey prashant,
   Blog came after a long time indeed.
   I think that taking into account the structure of url, can make clustering more accurate.
   Like for example two urls from same domain should be closer to each other than anything else, etc,
   so try using this function for calculating distance between two urls.

   # distance function compares two urls and finds the distance ,also considering the structure of url
   #into account and having different weights for different parts of url, as so to give precedence of
   #domain over params and so on
   # uses SequenceMatcher from python standard module difflib
   def customDistance(url1, url2):
   url1_frag = urlparse(url1)
   url2_frag = urlparse(url2)
   ratio = 0.0
   # weights for now are [6,5,4,3,2,1] for each of the elements of urlparse tuple, maximum 6 for domain and then decreasing.
   #ratio sums (user assigned weight of the urlparse part) * (ratio returned by Sequence matcher )
   #denominator is sum of the user assigned weights,
   for index in xrange(0,6):
   ratio += (6-index) * (1 – SequenceMatcher(None, url1_frag[index],url2_frag[index] ).ratio())
   return ratio / 21

   Link | July 25th, 2009 at 3:02 pm

2. Ashish Yadav wrote,

   As indentation is got all messed up, you can see the code snippet at http://python.pastebin.com/fceb4031

   Link | July 25th, 2009 at 3:05 pm

3. sriram wrote,

   Like the previous comment says, the distance function can be improved. python rocks!

   Link | July 26th, 2009 at 7:57 am

4. prashanthellina wrote,

   Wonderful suggestion Ashish. I was planning something similar to enhance the clustering. You have saved me some work.

   Output of the above script after having replaced with your distance function.

   [['#articles'],
    ['http://xkcd.com/612/'],
    ['http://web.sourceforge.com/terms.phphttp://slashdot.org//it.slashdot.org/search',
     'http://web.sourceforge.com/privacy.php',
     'http://web.sourceforge.com/advertising'],
    ['http://yro.slashdot.org/~drDugan/',
     'http://slashdot.org//slashdot.org/article.pl?sid=09/07/24/1545238',
     'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814785',
     'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814429',
     'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&op=Reply&threshold=1&commentsort=0&mode=thread&pid=28814457',
     'http://slashdot.org//slashdot.org/comments.pl?sid=09/07/24/1545238&cid=28810581',
     'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28815123',
     'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28815269',
     'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814385',
     'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814335',
     'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814657',
     'http://slashdot.org//it.slashdot.org/comments.pl?sid=1314601&cid=28814581',
     'http://slashdot.org//slashdot.org/~Darkness404',
     'http://slashdot.org//radio.slashdot.org']]
   

   Link | July 27th, 2009 at 8:04 am

5. prashanthellina wrote,

   Ashish,

   I am reposting your distance function code with indentation preserved.

   #Distance function compares two urls and finds the distance ,also considering the structure of url
   #into account and having different weights for different parts of url, as so to give precedence of
   #domain over params and so on
   # uses SequenceMatcher from python standard module difflib
   def distance(url1, url2):
       url1_frag = urlparse(url1)
       url2_frag = urlparse(url2)
       ratio = 0.0
       #  weights for now are [6,5,4,3,2,1]  for each of the tuples of urlparse
       #ratio sums (user assigned weight of the urlparse part) * (ratio returned by Sequence matcher )
       for index in xrange(0,6):
           ratio += (6-index) * (1.0 - SequenceMatcher(None, url1_frag[index],url2_frag[index] ).ratio())
       return ratio / 21.0
   

   People, note that you will have to add this line to the top of the script to use this function.

   from urlparse import urlparse
   

   Link | July 27th, 2009 at 8:06 am