How to count the number of unique characters in a file?

Question

Given a file in UTF-8 containing characters in different languages, how can I get a counter of the number of unique characters that it contains, excluding the selected number of characters (for example: "!", "@", "#", ".") From this account?

Answer 1

Using Single Line Perl:

echo -e "aba\ncfg!ഡ.#g" | perl -C7 -ne 'for(split(//)){if ($_ !~ /[!@#.]/) { print $_."\n"}}' | sort | uniq | wc -l

OUTPUT 7

If you want to ignore the new line:

echo -e "aba\ncfg!ഡ.#g" | perl -C7 -ne 'for(split(//)){if ($_ !~ /[!@#.\n]/) { print $_."\n"}}' | sort | uniq | wc -l

OUTPUT 6

Answer 2

Here is the bash solution. :)

bash$ perl -CSD -ne 'BEGIN { $s{$_}++ for split //, q(!@#.) }
                     $s{$_}++ || $c++ for split //;
                     END { print "$c\n" }' *.utf8

Answer 3

Python:

import itertools, codecs

predicate = set('!@#.').__contains__
unique_char_count = len(set(itertools.ifilterfalse(
                      predicate, itertools.chain.from_iterable(codecs.open(filename, encoding="UTF-8")))))

, . chain , , , . ifilterfalse , , , .

itertools:

import codecs
disallowed = set('!@#.')
unique_char_count = len(set(char for line in codecs.open(filename, encoding="UTF-8") for char in line 
                              if char not in disallowed))

set:

import codecs
unique = set()
any(unique.update(line) for line in codecs.open(filename, encoding="UTF-8"))
unique.difference_update('!@#.')
unique_char_count = len(unique)

Answer 4

Ruby, :

require 'set'
string = 'ababbbababbabcdcccdbbaaba'
ignore = 'c'
(Set.new(string.chars) - Set.new(ignore.chars)).count
# => 3 

• string -
• ignore -
• string.chars -
• Set.new
• -
• count -

Answer 5

Another ruby:

#encoding: utf-8
string = '@étude#@étude ฒณ!'
ignore = '!@#.'
p string.chars.to_a.uniq.join.delete(ignore).size #=>8

Answer 6

I will just put my version, which does not require a language, for a good assessment:

sed 's/[!@#.]//g' /path/to/file | sed 's/./\0\n/g' | sort -u | wc -l

Answer 7

I did it in python after 3 hours of research, but I did it

fname = "temp.txt"
num_lines = 0
num_words = 0
num_chars = 0
num_uniq  = 0
a = []
exclude = ",.@#$"
with open(fname, 'r') as f:
    for line in f:
        words = line.split()
        for word in words:
                char = list(word)
                a = a + char
        num_lines += 1
        num_words += len(words)
        num_chars += len(line)
print "Lines:%s\nWords:%s\nChars:%s" % (num_lines, num_words, num_chars)
num_uniq =  len(set(a)-set(exclude))
print "Unique Characters:%d" % (num_uniq)

displayed here

Lines:6
Words:74
Chars:385
Unique Characters:26

Answer 8

Use sets in python. let's say you want to know unique characters in url.txt

f=open('url.txt')
a=''
for x in f:
    x=x.split(' ')
    for y in x:
     a+=y
unique=set(a)-set('@!#.')  #add the characters that you wanna neglect in the second set
print(unique)
print('unique characters : ',len(unique))

let's say url.txt contains:

Google --!  google.com  --!  coolest search engine 

facebook --!  facebook.com  --!  biggest social network 

yahoo --!  yahoo.com  --!  biggest web portal 

the output will be:

{'a', 'G', 'm', '\n', 'n', 'c', 'b', 'e', 'g', 'f', 'i', 'h', 'k', '-', 'l', 'o', 'p', 's', 'r', 't', 'w', 'y'}
unique characters :  22

Answer 9

One alternative:

filename='/somewhere/my-file-in-utf8'

iconv -f UTF8 -t UTF16 $filename | tail -c +3 |  \
perl -pi -e "s/\x00\@//g; s/\x00\!//g; s/\x00\#//g; s/\x00\.//g;" | \
od | cut -b 8- | xargs -n 1 | sort | uniq | wc -l