What does the value of ValueError mean: cannot be reindexed from the duplicated axis?

Question

I get ValueError: cannot reindex from a duplicate axis when I try to set the index to a specific value. I tried to reproduce this with a simple example, but I could not do it.

Here is my session inside ipdb trace. I have a DataFrame with a string index and integer columns, float values. However, when I try to create an index sum for the sum of all columns, I get a ValueError: cannot reindex from a duplicate axis error. I created a small DataFrame with the same characteristics, but could not reproduce the problem, what could I lose?

I really don't understand what ValueError: cannot reindex from a duplicate axis means ValueError: cannot reindex from a duplicate axis , what does this error message mean? Perhaps this will help me diagnose the problem, and this is the most crucial part of my question.

 ipdb> type(affinity_matrix) <class 'pandas.core.frame.DataFrame'> ipdb> affinity_matrix.shape (333, 10) ipdb> affinity_matrix.columns Int64Index([9315684, 9315597, 9316591, 9320520, 9321163, 9320615, 9321187, 9319487, 9319467, 9320484], dtype='int64') ipdb> affinity_matrix.index Index([u'001', u'002', u'003', u'004', u'005', u'008', u'009', u'010', u'011', u'014', u'015', u'016', u'018', u'020', u'021', u'022', u'024', u'025', u'026', u'027', u'028', u'029', u'030', u'032', u'033', u'034', u'035', u'036', u'039', u'040', u'041', u'042', u'043', u'044', u'045', u'047', u'047', u'048', u'050', u'053', u'054', u'055', u'056', u'057', u'058', u'059', u'060', u'061', u'062', u'063', u'065', u'067', u'068', u'069', u'070', u'071', u'072', u'073', u'074', u'075', u'076', u'077', u'078', u'080', u'082', u'083', u'084', u'085', u'086', u'089', u'090', u'091', u'092', u'093', u'094', u'095', u'096', u'097', u'098', u'100', u'101', u'103', u'104', u'105', u'106', u'107', u'108', u'109', u'110', u'111', u'112', u'113', u'114', u'115', u'116', u'117', u'118', u'119', u'121', u'122', ...], dtype='object') ipdb> affinity_matrix.values.dtype dtype('float64') ipdb> 'sums' in affinity_matrix.index False 

Here is the error:

 ipdb> affinity_matrix.loc['sums'] = affinity_matrix.sum(axis=0) *** ValueError: cannot reindex from a duplicate axis 

I tried to reproduce this with a simple example, but I failed

 In [32]: import pandas as pd In [33]: import numpy as np In [34]: a = np.arange(35).reshape(5,7) In [35]: df = pd.DataFrame(a, ['x', 'y', 'u', 'z', 'w'], range(10, 17)) In [36]: df.values.dtype Out[36]: dtype('int64') In [37]: df.loc['sums'] = df.sum(axis=0) In [38]: df Out[38]: 10 11 12 13 14 15 16 x 0 1 2 3 4 5 6 y 7 8 9 10 11 12 13 u 14 15 16 17 18 19 20 z 21 22 23 24 25 26 27 w 28 29 30 31 32 33 34 sums 70 75 80 85 90 95 100 

Answer 1

This error usually occurs when attaching / assigning to a column when the index has duplicate values. Since you are assigning a row, I suspect that affinity_matrix.columns has a duplicate value, maybe not shown in your question.

Answer 2

As others have said, you probably have double values ​​in your source index. To find them, follow these steps:

df[df.index.duplicated()]

Answer 3

Duplicate indexes often occur if you create a DataFrame by combining other DataFrames. IF you do not care about storing the values ​​of your index and want them to be unique values ​​when you combine the data, set ignore_index=False .

Alternatively, to overwrite the current index with a new one, instead of using df.reindex() set:

 df.index = new_index 

Answer 4

In my case, I had a duplicate column with the same name

Answer 5

I ran into this error today when I wanted to add a new column like this

 df_temp['REMARK_TYPE'] = df.REMARK.apply(lambda v: 1 if str(v)!='nan' else 0) 

I wanted to process the REMARK df_temp column to return 1 or 0. However, I typed the wrong variable using df . And he answered the error as follows:

 ----> 1 df_temp['REMARK_TYPE'] = df.REMARK.apply(lambda v: 1 if str(v)!='nan' else 0) /usr/lib64/python2.7/site-packages/pandas/core/frame.pyc in __setitem__(self, key, value) 2417 else: 2418 # set column -> 2419 self._set_item(key, value) 2420 2421 def _setitem_slice(self, key, value): /usr/lib64/python2.7/site-packages/pandas/core/frame.pyc in _set_item(self, key, value) 2483 2484 self._ensure_valid_index(value) -> 2485 value = self._sanitize_column(key, value) 2486 NDFrame._set_item(self, key, value) 2487 /usr/lib64/python2.7/site-packages/pandas/core/frame.pyc in _sanitize_column(self, key, value, broadcast) 2633 2634 if isinstance(value, Series): -> 2635 value = reindexer(value) 2636 2637 elif isinstance(value, DataFrame): /usr/lib64/python2.7/site-packages/pandas/core/frame.pyc in reindexer(value) 2625 # duplicate axis 2626 if not value.index.is_unique: -> 2627 raise e 2628 2629 # other ValueError: cannot reindex from a duplicate axis 

As you can see, the correct code should be

 df_temp['REMARK_TYPE'] = df_temp.REMARK.apply(lambda v: 1 if str(v)!='nan' else 0) 

Because df and df_temp have a different number of lines. Therefore, he returned ValueError: cannot reindex from a duplicate axis .

Hope you can figure it out, and my answer can help other people debug their code.