Geek Answers Handbook

How to speed up the re-election process in Pandas?

Let's say you have a time series of data with a 1 minute time series next to the index, 4 columns and 4 million rows. When you try to do something like:

 conversion = {'Open': 'first', 'High': 'max', 'Low': 'min', 'Close': 'last'}
 df1 = df.resample('5Min', how=conversion)

An absurd amount of time is required (20-30 minutes). How to speed up this process?

Pandas 18, Python 2.7

+4
source share
1 answer

Resample works pretty fast in a data set of size (4,000,000, 4):

idx = pd.date_range('1/1/2010', periods=4000000, freq='T')
df = pd.DataFrame(np.random.rand(4000000, 4), columns = ["Open", "High", "Low", "Close"], index = idx)
%timeit df.resample("5Min").agg(conversion)
1 loop, best of 3: 253 ms per loop

With an irregular index and some nanons:

idx1 = pd.date_range('1/1/1900', periods=10000000, freq='Min')
idx2 = pd.date_range('1/1/1940', periods=10000000, freq='Min')
idx3 = pd.date_range('1/1/1980', periods=10000000, freq='Min')
idx4 = pd.date_range('1/1/2020', periods=10000000, freq='Min')
idx = np.array([np.random.choice(idx1, 1000000), np.random.choice(idx2, 1000000), np.random.choice(idx3, 1000000), 
                np.random.choice(idx4, 1000000)]).flatten()
np.random.shuffle(idx)
df = pd.DataFrame(np.random.randint(100, size=(4000000, 4)), columns = ["Open", "High", "Low", "Close"], index = idx)
df.loc[np.random.choice(idx, 100000), "Open"] = np.nan
df.loc[np.random.choice(idx, 50000), "High"] = np.nan
df.loc[np.random.choice(idx, 500000), "Low"] = np.nan
df.loc[np.random.choice(idx, 20000), "Close"] = np.nan
%timeit df.resample("5Min").agg(conversion)
1 loop, best of 3: 5.06 s per loop

So, it seems that something other than re-fetching takes time for your business.

+3
source

More articles:


All Articles