Geek Answers Handbook

Grepping IP Logs

Am I pretty bad at using "core?" Unix teams, and this question tests my knowledge even more. What I would like to do is grep all the IP addresses from the log (e.g. access.log from apache) and count how often they happen. Can I do this with one command or do I need to write a script for this?

+6
source share
7 answers

You will need at least a short conveyor.

sed -e 's/\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+\).*$/\1/' -e t -e d access.log | sort | uniq -c

What each IP will print (will only work with ipv4, though), sorted with the count prefix.

I tested it with apache2 access.log (it is configurable, so you will need to check) and it worked for me. The IP address is assumed to be the first on each line.

IP- ( 4 , ) . -e t , , -e d ( IP-). sort sorts..:) uniq -c (, , ).

+13

( datafile - )

egrep '[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}' datafile | sort | uniq -c

edit: , ​​

+2

egrep '[[: digit:]] {1,3} (. [[: digit:]] {1,3}) {3}' | awk '{print $1}' | sort | uniq -c

0

script, . apache. Ubuntu 11.10 (oneiric) 3.0.0-32-generi# 51-Ubuntu SMP Thu Mar 21 15:51:26 UTC 2013 i686 i686 i386 GNU/Linux . Gvim Vim , unique_visits, ips . , grep. IP-. IPV4. , . script, Slackware, : http://www.perpetualpc.net/srtd_bkmrk.html

#!/bin/sh
#eliminate search engine referals and zombie hunters. combined_log is the original file
egrep '(google)|(yahoo)|(mamma)|(query)|(msn)|(ask.com)|(search)|(altavista)|(images.google)|(xb1)|(cmd.exe)|(trexmod)|(robots.txt)|(copernic.com)|(POST)' combined_log > search
#now sort them to eliminate duplicates and put them in order
sort -un search > search_sort
#do the same with original file
sort -un combined_log > combined_log_sort
#now get all the ip addresses. only the numbers
grep -o '[0-9][0-9]*[.][0-9][0-9]*[.][0-9][0-9]*[.][0-9][0-9]*' search_sort > search_sort_ip
grep -o '[0-9][0-9]*[.][0-9][0-9]*[.][0-9][0-9]*[.][0-9][0-9]*' combined_log_sort > combined_log_sort_ip
sdiff -s combined_log_sort_ip search_sort_ip > final_result_ip
#get rid of the extra column
grep -o '^\|[0-9][0-9]*[.][0-9][0-9]*[.][0-9][0-9]*[.][0-9][0-9]*' final_result_ip > bookmarked_ip
#remove stuff like browser versions and system versions
egrep -v '(4.4.2.0)|(1.6.3.1)|(0.9.2.1)|(4.0.0.42)|(4.1.8.0)|(1.305.2.109)|(1.305.2.12)|(0.0.43.45)|(5.0.0.0)|(1.6.2.0)|(4.4.5.0)|(1.305.2.137)|(4.3.5.0)|(1.2.0.7)|(4.1.5.0)|(5.0.2.6)|(4.4.9.0)|(6.1.0.1)|(4.4.9.0)|(5.0.8.6)|(5.0.2.4)|(4.4.8.0)|(4.4.6.0)' bookmarked_ip > unique_visits

exit 0
0

None of the answers presented here worked for me, so here is a working one:

cat yourlogs.txt | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" | sort | uniq -c | sort

it uses grep to isolate all ips. then sorts them, counts and sorts the result again.

0
source

Using sed:

$ sed 's/.*\(<regex_for_ip_address>\).*/\1/' <filename> | sort | uniq -c

You can search and find the regular expression for the ip address on Inernet and replace it with <regex_for_ip_address>. e.g. fooobar.com/questions/19794 / ...

-1
source 
cat access.log |egrep -o '[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}' |uniq -c|sort
-1
source

More articles:


All Articles