Continuous-Time Event Calculation

I have pandas data with a 1 minute time step. This data is not recorded continuously, now I would like to split all my data into a separate event based on the following condition: If there is continuous data recorded for 5 minutes or more, then this is only considered an event, and for such events, data must be extracted separately . Is there any way to implement this in a pandas dataframe.

My data looks like this (with the Event column):

Date                  X    Event
2017-06-06 01:08:00 0.019   1
2017-06-06 01:09:00 0.005   1
2017-06-06 01:10:00 0.03    1
2017-06-06 01:11:00 0.005   1
2017-06-06 01:12:00 0.003   1
2017-06-06 01:13:00 0.001   1
2017-06-06 01:14:00 0.039   1
2017-06-06 01:15:00 0.003   1
2017-06-06 01:17:00 0.001   nan
2017-06-06 01:25:00 0.006   nan
2017-06-06 01:26:00 0.006   nan
2017-06-06 01:27:00 0.032   nan
2017-06-06 01:29:00 0.013   2
2017-06-06 01:30:00 0.065   2
2017-06-06 01:31:00 0.013   2
2017-06-06 01:32:00 0.001   2
2017-06-06 01:33:00 0.02    2
2017-06-06 01:38:00 0.05    nan
2017-06-06 01:40:00 0.025   3
2017-06-06 01:41:00 0.01    3
2017-06-06 01:42:00 0.008   3
2017-06-06 01:43:00 0.009   3
2017-06-06 01:44:00 0.038   3
2017-06-06 01:45:00 0.038   3

Your suggestion is much appreciated.

Using the solution provided by nnnmmm, the result looks like this:

2015-01-01 03:24:00       NaN
2015-01-01 04:59:00       NaN
2015-01-01 05:01:00       NaN
2015-01-01 05:02:00       NaN
2015-01-01 05:03:00       NaN
2015-01-13 01:12:00       1.0
2015-01-13 01:13:00       1.0
2015-01-13 01:14:00       1.0
2015-01-13 01:15:00       1.0
2015-01-13 01:16:00       1.0
2015-01-13 01:49:00       1.0
2015-01-13 01:50:00       1.0
2015-01-13 01:51:00       1.0
2015-01-13 01:52:00       1.0
2015-01-13 01:53:00       1.0
2015-01-13 01:54:00       1.0
2015-01-13 01:55:00       1.0

In this case, the time changes between 01:16:00 and 01:49:00, this should not be considered as the same event, rather, 01:49:00 should be the second event.

+6
3

( ), - (, , , , , ).

df['new'] = (df.reset_index().Date.diff() == pd.Timedelta('1min')).astype(int).values
df['grp'] = (df.new != 1).cumsum()
df['cnt'] = df.groupby('grp')['new'].transform(size)

df['event'] = df['cnt'] > 4
df['Event'] = ((df.event) & (df.new != 1)).cumsum()
df['Event'] = np.where( df.event, df.Event, np.nan )

                         X  new  grp  cnt  event  Event
Date                                                   
2017-06-06 01:08:00  0.019    0    1    8   True    1.0
2017-06-06 01:09:00  0.005    1    1    8   True    1.0
2017-06-06 01:10:00  0.030    1    1    8   True    1.0
2017-06-06 01:11:00  0.005    1    1    8   True    1.0
2017-06-06 01:12:00  0.003    1    1    8   True    1.0
2017-06-06 01:13:00  0.001    1    1    8   True    1.0
2017-06-06 01:14:00  0.039    1    1    8   True    1.0
2017-06-06 01:15:00  0.003    1    1    8   True    1.0
2017-06-06 01:17:00  0.001    0    2    1  False    NaN
2017-06-06 01:25:00  0.006    0    3    3  False    NaN
2017-06-06 01:26:00  0.006    1    3    3  False    NaN
2017-06-06 01:27:00  0.032    1    3    3  False    NaN
2017-06-06 01:29:00  0.013    0    4    5   True    2.0
2017-06-06 01:30:00  0.065    1    4    5   True    2.0
2017-06-06 01:31:00  0.013    1    4    5   True    2.0
2017-06-06 01:32:00  0.001    1    4    5   True    2.0
2017-06-06 01:33:00  0.020    1    4    5   True    2.0
2017-06-06 01:38:00  0.050    0    5    1  False    NaN
2017-06-06 01:40:00  0.025    0    6    6   True    3.0
2017-06-06 01:41:00  0.010    1    6    6   True    3.0
2017-06-06 01:42:00  0.008    1    6    6   True    3.0
2017-06-06 01:43:00  0.009    1    6    6   True    3.0
2017-06-06 01:44:00  0.038    1    6    6   True    3.0
2017-06-06 01:45:00  0.038    1    6    6   True    3.0
+5

, , :

# True where the previous timestamp is one minute away
prev_ok = pd.Series(df['Date'].diff().values == np.timedelta64(1, 'm'))
# True where the previous four rows of prev_ok are True
a = prev_ok.rolling(4).sum() == 4
# extend True back down the previous four rows, this could be done with a loop
b = a | a.shift(-1) | a.shift(-2) | a.shift(-3) | a.shift(-4)
# calculate edges from False to True to get the event indices
c = (~a.shift(-3).fillna(False) & a.shift(-4)).cumsum()
# only display event indices where b is 
df['Event'] = c.mask(~b)

                  Date      X  Event
0  2017-06-06 01:08:00  0.019    1.0
1  2017-06-06 01:09:00  0.005    1.0
2  2017-06-06 01:10:00  0.030    1.0
3  2017-06-06 01:11:00  0.005    1.0
4  2017-06-06 01:12:00  0.003    1.0
5  2017-06-06 01:13:00  0.001    1.0
6  2017-06-06 01:14:00  0.039    1.0
7  2017-06-06 01:15:00  0.003    1.0
8  2017-06-06 01:17:00  0.001    NaN
9  2017-06-06 01:25:00  0.006    NaN
10 2017-06-06 01:26:00  0.006    NaN
11 2017-06-06 01:27:00  0.032    NaN
12 2017-06-06 01:29:00  0.013    2.0
13 2017-06-06 01:30:00  0.065    2.0
14 2017-06-06 01:31:00  0.013    2.0
15 2017-06-06 01:32:00  0.001    2.0
16 2017-06-06 01:33:00  0.020    2.0
17 2017-06-06 01:38:00  0.050    NaN
18 2017-06-06 01:40:00  0.025    NaN
19 2017-06-06 01:42:00  0.010    NaN
20 2017-06-06 01:43:00  0.008    NaN
21 2017-06-06 01:44:00  0.009    NaN
22 2017-06-06 01:45:00  0.038    NaN

: a erosion 4, b dilation . , b prev_ok opening, True prev_ok True b, True True s.

+3

, , - .

,

x = [1,2,3,4,6,7,8,9,10,11,12,14,15,17,18,19,20,21,22,23,24,25,28,30,32,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50]

We will denote data Trueif they are “continuous” for at least 5 members.

To do this, first get a list of differences and assign 1if the difference is 1, or assign 0, if not. This is done using y = [int(round(1/(x[i]-x[i-1]))) for i in range(1, len(x))].

After that, we get an index where the difference is assigned 0, and use it to check your status.

Full code:

import copy

x = [1,2,3,4,6,7,8,9,10,11,12,14,15,17,18,19,20,21,22,23,24,25,28,30,32,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50]
y = [int(round(1/(x[i]-x[i-1]))) for i in range(1, len(x))]
z = copy.deepcopy(y);


zeros_check = [abs(y[i]-1)*i for i in range(0,len(y))];
zeros_id = list(set(zeros_check));
zeros_id.remove(0);
zeros_id.append(len(y));


idx = 0;
for i in zeros_id:
   if sum(y[idx+1:i])>=5:
      z[idx+1:i] = [True for i in range(idx,idx+i-1)];
   else:
      z[idx:i+1] = [False for i in range(idx,idx+i+1)];
   idx = i;

for i,j,k in zip(x,y,z):
     print(i,j,k)

Exit:

1 1 False
2 1 False
3 1 False
4 0 False
6 1 True
7 1 True
8 1 True
9 1 True
10 1 True
11 1 True
12 0 False
14 1 False
15 0 False
17 1 True
18 1 True
19 1 True
20 1 True
21 1 True
22 1 True
23 1 True
24 1 True
25 0 False
28 0 False
30 0 False
32 0 False
34 1 True
35 1 True
36 1 True
37 1 True
38 1 True
39 1 True
40 1 True
41 1 True
42 1 True
43 1 True
44 1 True
45 1 True
46 1 True
47 1 True
48 1 True
49 1 True
+1
source

All Articles