Nginx disables logging for specific user agents

Question

Basically, I'm trying to remove search bots such as Google, Bing, and that are not from my access logs. They really grow over time, eventually adding hundreds of thousands of useless log access log entries, which is especially painful if you ever have to look for them. The problem I am facing is that I define an access log in my blocks, so Nginx just looks at it and ignores my second one, which I define in the location / block. If I comment out my access log for my site (and not the crawler block), then it works fine. Here is the configuration:

server { listen 80; server_name example.com; access_log /home/domains/example.com/logs/access; error_log /home/domains/example.com/logs/error; root /home/domains/example.com/forums; location / { index index.html index.htm; if ($http_user_agent ~* ("googlebot") ) { access_log off; } } 

I deleted everything except this when posting (php include and what not), although I checked that nothing was stopping him by commenting on everything except the one above. Therefore, to summarize, I have a log defined in my virtual block to record all traffic (I defined it for each block in order to make it more accurate and what not). I am trying to turn off logging for specific user agents, if I do not turn off the main log for the site, it will continue to log what I am telling it not to user agents.

I have been on this for several hours, any help would be greatly appreciated.

Answer 1

You should not use if in nginx - if it is evil

Use conditional logging :

 http { map $http_user_agent $excluded_ua { ~Googlebot 0; default 1; } ....... } server { access_log /home/domains/example.com/logs/access combined if=$excluded_ua; } 

However, be careful to exclude googlebot , as some offensive bots disguise themselves.

Answer 2

Well, actually regex ("googlebot") will match the user agent with double quotes, clearly not what you want. Put brackets and quotation marks if you want, and everything will be all right.

Answer 3

You need to add return 200; after access access_log off;

so that it looks like this:

 location / { if ($http_user_agent ~* "(googlebot)" ) { access_log off; return 200; } }