Geek Answers Handbook

This is a good way to open a "complex" txt file in python

I have a txt file with the following format (simplified):

date                 this that other
2007-05-25 11:00:00  10   20   30
2007-05-25 11:10:00  15   18   30
2007-05-25 11:20:00  10   27   30
2007-05-25 11:30:00  20   35   30
2007-05-25 11:50:00  30   20   
2007-05-25 12:00:00  30   13   
2007-05-25 12:10:00  30   13

The first raw one is the row defining what the column above them is. The first column makes it clear that this is the time. You may also notice that some values ​​are missing. I do not want to erase lines that are missing in some values. Since I want to do some calculations with this data later, I thought of using numpy to import this data with numpy.loadtxt:

data = numpy.loadtxt('data.txt')

This gives an error ValueError: could not convert string to float: b'date'due to the first raw. Using:

data = numpy.genfromtxt('data.txt')

gives an error Line #51028 (got 38 columns instead of 37)for many lines because some values ​​are missing. What should i try?

+4
source share
2

Pandas - NumPy. , , .

pandas :

$ pip install pandas

http://pastebin.com/NuNaTW9n .

>>> import pandas as pd
>>> from urllib import urlopen
>>> df = pd.read_csv(urlopen("http://pastebin.com/raw.php?i=NuNaTW9n"), sep='\t')
>>> df
                  date  this  that  other
0  2007-05-25 11:00:00    10    20     30
1  2007-05-25 11:10:00    15    18     30
2  2007-05-25 11:20:00    10    27     30
3  2007-05-25 11:30:00    20    30    NaN
4  2007-05-25 11:50:00    30    20    NaN
5  2007-05-25 12:00:00    30    13    NaN
6  2007-05-25 12:10:00    30    13    NaN

, :

>>> df["this"].sum()
145

>>> df["that"].mean()
20.142857142857142

>>> df[df["that"] < 20]["date"]
1    2007-05-25 11:10:00
5    2007-05-25 12:00:00
6    2007-05-25 12:10:00

pandas (, , df["that"] int64), , dtype read_csv.

+5

, , genfromtxt, delimiter , :

In [2]: a = np.genfromtxt('test.txt', delimiter=[19,4,5,5], skip_header=1)

. dtype . :

In [3]: a = np.genfromtxt('test.txt', delimiter=[19,4,5,5], skip_header=1,
                          dtype=np.dtype([('date', 'datetime64[s]'),
                                          ('this', int), ('that', int),
                                          ('other', int)])
                         )

In [4]: a
Out[4]: array([(datetime.datetime(2007, 5, 25, 15, 0), 10, 20, 30),
   (datetime.datetime(2007, 5, 25, 15, 10), 15, 18, 30),
   (datetime.datetime(2007, 5, 25, 15, 20), 10, 27, 30),
   (datetime.datetime(2007, 5, 25, 15, 30), 20, 35, 30),
   (datetime.datetime(2007, 5, 25, 15, 50), 30, 20, -1),
   (datetime.datetime(2007, 5, 25, 16, 0), 30, 13, -1),
   (datetime.datetime(2007, 5, 25, 16, 10), 30, 13, -1)], 
  dtype=[('date', '<M8[s]'), ('this', '<i8'), ('that', '<i8'), ('other', '<i8')])
+3

More articles:


All Articles