Shannon's Entropy for Mutual Information

Question

I have statistics on some properties, for example:

1st iter : p1:10 p2:0 p3:12 p4:33 p5:0.17 p6:ok p8:133 p9:89 2nd iter : p1:43 p2:1 p6:ok p8:12 p9:33 3rd iter : p1:14 p2:0 p3:33 p5:0.13 p9:2 ... (p1 -> number of tries, p2 -> try done well, p3..pN -> properties of try). 

I need to calculate the amount of information of each property. After some quantization procedures (for example, up to 10 levels), to make all input numbers on one level, the input file starts to look like this:

 p0: 4 3 2 4 5 5 6 7 p3: 4 5 3 3 p4: 5 3 3 2 1 2 3 ... 

Where p(0) = funct(p1,p2) .

Not every input line received every pK , so len(pk) <= len(p0) .

Now I know how to calculate the entropy of each property from the Shannon entropy for each row. I need to calculate mutual information from here.

The calculation of the joint entropy for the mutual information I(p0,pK) stuck due to different lengths.

I calculate the entropy for a single element as follows:

 def entropy(x): probs = [np.mean(x == c) for c in set(x)] return np.sum(-p * np.log2(p) for p in probs) 

So, for the connection, do I need to use product to create the input array x and use zip(p0,pk) instead of set(x) ?

Answer 1

I assume that you want to calculate the mutual information between each p1 and each of p2 , p3 , ... afterwards.

1) Calculate H(X) as the entropy from p1 with:

each x is the next element of p1 .

2) Calculate H(Y) as the entropy of pK with the same equation, with each x being the next element of p1

3) Create a new pair of p1 and pK :

 pairs = zip(p1, pK) 

Note that if the values ​​in the columns of your data have different meanings, you should probably fill in the missing data (for example, using 0 or the values ​​from the previous iteration).

4) Calculate the entropy of the joints H(X,Y) using:

Note that you cannot just use the first equation and treat each pair as one element - you must iterate over the entire Cartesian product between p1 and pK in this equation, calculating the probabilities using the pairs collection. So, to iterate over the entire Cartesian product, use for xy in itertools.product(p1, pK): ...

5) Then you can have mutual information between p1 and pK as:

Using the capabilities of numpy, you can calculate the total entropy presented here here :

 def entropy(X, Y): probs = [] for c1 in set(X): for c2 in set(Y): probs.append(np.mean(np.logical_and(X == c1, Y == c2))) return np.sum(-p * np.log2(p) for p in probs if p > 0) 

where if p > 0 is consistent with the definition of entropy :

In the case p (x _i ) = 0 for some i, the value of the corresponding term 0 log _b (0) is taken equal to 0

If you do not want to use numpy , then the version without it might look something like this:

 def entropyPart(p): if not p: return 0 return -p * math.log(p) def entropy(X, Y): pairs = zip(X, Y) probs = [] for pair in itertools.product(X,Y): probs.append(1.0 * sum([p == pair for p in pairs]) / len(pairs)) return sum([entropyPart(p) for p in probs]) 

Answer 2

Take the formula from the formal definition section of this Wikipedia article. They call it information amplification, but it's the same as mutual information. To calculate the entropy of the sample contained in this formula, take the formula from the definition section of this Wikipedia article.

So, you first calculate the entropy of your entire dataset and subtract from it the entropy remaining when you know the value of the attribute in question.

A multi-dimensional histogram can be computed in Python using numpy.histogramdd() .