Geek Answers Handbook

Parsing a growing log file

I want to make a log parser using ruby, this parser should parse the log file while it grows. It should be sorted out in turn to the end, and then wait (somehow?) For more lines, so my question is how best to handle it?

edit: Prefer a portable way to do this, although my log file is on Windows (for now).

+4
source share
3 answers

On Windows, you can use Directory Change Notifications . You will tell Windows (with FindFirstChangeNotification ) to control the c: / foo / logs directory, then Windows will update your handle when something happens in that directory. At this point, you are checking to see if the change is related to the file you care about.

Ruby has bindings for the Win32 API, and an example of receiving these notifications.

+1
source

At http://www.biterscripting.com/SS_WebLogParser.html . This is a sample script written for web server logs, but can be used as a starting point for writing your own log analyzer for any type of logs. To use it in continuous mode while the log file continues to grow, here is the script.

# Script LogParser.txt # Go in a continuous loop, sleeping 1 hr each time. while (true) do # The number of lines in the log file the last time we checked is in following # variable. Initially, it will be 0. var int lines_old # Read the log file into a str variable. var str log ; cat "file.log" > $log # Get the number of lines found this time. var str lines_new ; set $lines_new = { len -e $log } # Strip off the first $lines lines. lex -e (makestr(int($lines))+"]") $log > null # The new lines are now available in $log. Process them with something similar to # SS_WebLogParser script. # Update $lines_old, then, sleep. set $lines_old = $lines_new sleep 3600 # 3600 seconds = 1 hour done

To try

  • Save this script in, say, C: \ LogParser.txt (since you are on a window).
  • Download biterscripting. Google it.

  • Call our script by entering the following command.

    script "\ LogParser.txt"

If you need to use any of your sample scripts, install them using the following command.

 script "http://www.biterscripting.com/Download/SS_AllSamples.txt"

Patrick

0
source

For this task, you can use IO.popen to get the file stream in the pipe of the growing command line result. Then use the readline function during the loop. Here is an example with the adb logcat command, which receives a live log on an Android device:

 #! /usr/bin/env ruby IO.popen("adb logcat") do |io| while line = io.readline line.strip! # Process here print "#{line}\n" end end

EDIT

For a file, this is slightly different. I would have a "readline" in a poll in a file stream.

 #! /usr/bin/env ruby File.open("test.log") do |io| loop do begin line = io.readline line.strip! rescue sleep 0.2 retry end # Process here print "#{line}\n" end end
0
source

More articles:


All Articles