Geek Answers Handbook

Grouping Related Strings

I am trying to analyze a lot of search terms, so many of them individually do not say anything. However, I would like to group the terms, because I think that similar terms should have similar effectiveness. For instance,

Term               Group
NBA Basketball     1
Basketball NBA     1
Basketball         1
Baseball           2

This is a contrived example, but hopefully it explains what I'm trying to do. So what is the best way to do what I described? I thought I nltkmight have something like that, but I only know little about him.

thank

+3
source share
1 answer

, Dice . , (term1 = "NB", "BA", "A", "B", "Ba"...).

nltk, , nltk.metrics.association.BigramAssocMeasures.dice(), . , , .

import sys, operator

def tokenize(s, glen):
  g2 = set()
  for i in xrange(len(s)-(glen-1)):
    g2.add(s[i:i+glen])
  return g2

def dice_grams(g1, g2): return (2.0*len(g1 & g2)) / (len(g1)+len(g2))

def dice(n, s1, s2): return dice_grams(tokenize(s1, n), tokenize(s2, n))

def main():
  GRAM_LEN = 4
  scores = {}
  for i in xrange(1,len(sys.argv)):
    for j in xrange(i+1, len(sys.argv)):
      s1 = sys.argv[i]
      s2 = sys.argv[j]
      score = dice(GRAM_LEN, s1, s2)
      scores[s1+":"+s2] = score
  for item in sorted(scores.iteritems(), key=operator.itemgetter(1)):
    print item

, :

./dice.py "NBA Basketball" "Basketball NBA" "Basketball" "Baseball"

('NBA Basketball:Baseball', 0.125)
('Basketball NBA:Baseball', 0.125)
('Basketball:Baseball', 0.16666666666666666)
('NBA Basketball:Basketball NBA', 0.63636363636363635)
('NBA Basketball:Basketball', 0.77777777777777779)
('Basketball NBA:Basketball', 0.77777777777777779)

, , . .

+7

More articles:


All Articles