Reading csv in python pandas and handling bad values

Question

I am using pandas to read a csv file. Data is numbers, but stored in a CSV file as text. Some of the values ​​are non-numeric when they are bad or missing. How to filter these values ​​and convert the remaining data to integers.

I assume there is a better / faster way than isdigit() through all the values ​​and using isdigit() to check that they are numeric.

Does pandas or numpy have a way to recognize bad values ​​in the reader? If not, what is the easiest way to do this? Do I have to specify dtypes types to make this work?

Answer 1

pandas.read_csv has the na_values parameter:

 na_values : list-like, default None List of additional strings to recognize as NA/NaN 

where you can identify these bad values.

Answer 2

You can pass a custom list of values ​​to be considered missing using pandas.read_csv . Alternatively, you can pass functions to the converters argument.

Answer 3

NumPy provides the genfromtxt() function specifically for this purpose. First sentence from related documentation:

Loading data from a text file with missing values ​​specified as indicated.