We have a strange problem with one seeker. Sometimes it raises a Rails FATAL error on some request, but the trace is very limited and looks something like this:
[2014-07-01 18:16:37] FATAL Rails : ArgumentError (invalid %-encoding (c ^ FK+ 9u$_ t Kl Ξ₯E! =k \ Μ* ί>c+<O Ϋo Κ> CR! 2 D (5 xq
Crawler User Agent
Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
We can ask him to stop scanning through robots.txt , but it would be better to deal with the root cause and not fail with 500 for these requests, if possible.
We also cannot reproduce such a request, so any suggestions on how to generate such a request will be very useful.
We use Rails 3.2.19, Unicorn on Ubuntu 12.04. Here locale_middleware.rb
ruby-on-rails encoding web-crawler fatal-error
gingerlime Jul 07 '14 at 13:33 2014-07-07 13:33
source share