Geek Answers Handbook

Alternative strptime for comparing dates?

Is there a way to compare two dates without calling strptime every time in python? I am sure that I have no problem, there is no other way, but I want to make sure that I checked all the parameters.

I am looking at a very large log file, each line has a date that I need to compare to see if this date is in the range of two other dates. I need to convert every date for every line with strptime, which causes a big bottleneck;

Fri Sep 2 15:12:43 2016 output2.file 63518075 function calls (63517618 primitive calls) in 171.409 seconds Ordered by: cumulative time List reduced from 571 to 10 due to restriction <10> ncalls tottime percall cumtime percall filename:lineno(function) 1 0.003 0.003 171.410 171.410 script.py:3(<module>) 1 0.429 0.429 171.367 171.367 scipt.py:1074(main) 1 3.357 3.357 162.009 162.009 script.py:695(get_data) 1569898 14.088 0.000 141.175 0.000 script.py:648(check_line) 1569902 6.899 0.000 71.706 0.000 {built-in method strptime} 1569902 31.198 0.000 64.805 0.000 /usr/lib64/python2.7/_strptime.py:295(_strptime) 1569876 15.324 0.000 43.170 0.000 script.py:626(dict_add) 4709757 23.370 0.000 23.370 0.000 {method 'strftime' of 'datetime.date' objects} 1569904 1.655 0.000 18.799 0.000 /usr/lib64/python2.7/_strptime.py:27(_getlang) 1569899 2.103 0.000 17.452 0.000 script.py:592(reverse)

Dates are formatted as follows:

 current_date = 01/Aug/1995:23:59:53

And I compare them like this:

 with open(logfile) as file: for line in file: current_date = strptime_method(line) if current_date => end_date: break

Is there an alternative when it comes to comparing dates?

Edit: Thanks to everyone, in particular user2539738. Here are the results based on his / her assumption, a big difference in speed ;

 Fri Sep 2 16:14:59 2016 output3.file 24270567 function calls (24270110 primitive calls) in 105.466 seconds Ordered by: cumulative time List reduced from 571 to 10 due to restriction <10> ncalls tottime percall cumtime percall filename:lineno(function) 1 0.002 0.002 105.466 105.466 script.py:3(<module>) 1 0.487 0.487 105.433 105.433 script.py:1082(main) 1 3.159 3.159 95.861 95.861 script.py:702(get_data) 1569898 21.663 0.000 77.138 0.000 script.py:648(check_line) 1569876 14.979 0.000 43.408 0.000 script.py:626(dict_add) 4709757 23.865 0.000 23.865 0.000 {method 'strftime' of 'datetime.date' objects} 1569899 1.943 0.000 15.556 0.000 script.py:592(reverse) 1 0.000 0.000 9.078 9.078 script.py:1066(print_data) 1 0.021 0.021 9.044 9.044 script.py:1005(print_ip) 10 0.001 0.000 7.067 0.707 script.py:778(ip_api)
+5
source share
2 answers

I assume current_date is a string

Make a dictionary first

 moDict = {"Aug":8, "Jan":1} #etc

Then find the year / month / day, etc.

 current_date = "01/Aug/1995:23:59:53" Yr = int(current_date[7:11]) Mo = moDict[(current_date[3:6])] Day = int(current_date[0:2]) m_date = datetime.datetime(Yr,Mo,Day)

And you can use this to make comparisons

+1
source

Since your dates look in fixed-length format, it is trivially easy to parse, and you don't need strptime to do this. You can rearrange them to the ISO 8601 date / time format and compare them directly as strings!

 mos = {'Jan': '01', 'Feb': '02', 'Mar': '03', 'Apr': '04', 'May': '05', 'Jun': '06', 'Jul': '07', 'Aug': '08', 'Sep': '09', 'Oct': '10', 'Nov': '11', 'Dec': '12'} def custom_to_8601(dt): return dt[7:11] + '-' + mos[dt[3:6]] + '-' + dt[0:2] + 'T' + dt[12:] >>> custom_to_8601('01/Aug/1995:23:59:53') '1995-08-01T23:59:53'

Perhaps faster use of join instead of string concatenation and punctuation elimination:

 def comparable_date(dt): return ''.join([dt[7:11], mos[dt[3:6]], dt[0:2], dt[12:]]) >>> comparable_date('01/Aug/1995:23:59:53') '1995080123:59:53'

Running cProfile for 1,000,000 reps for me produces these timings:

  • custom_to_8601 : 0.978 seconds
  • comparable_date : 0.937 seconds
  • your source code with strptime : 15.492 seconds
  • earlier answer using datetime constructor: 1.134 seconds
+1
source

More articles:


All Articles