How to prevent offensive scanners from scanning a rail application deployed to Heroku?

Question

I want to restrict crawler access to my rails device running on Heroku. This would be a straightforward task if I were using Apache OR nginX. Since the application is deployed on Heroku, I'm not sure how to restrict access at the HTTP server level.

I tried using the robots.txt file, but distorting scanners do not comply with robot.txt.

These are the solutions that I am considering:

1) A before_filter in the rails layer to restrict access.

2) Rack-based solution to restrict access

I am wondering if there are any more efficient ways to solve this problem.

Answer 1

I read about honeypot solutions: you have one URI that cannot be bypassed (put in a robots.txt file). If any IP address calls this URI, block it. I would use it as a Rack middleware, so hitting does not fall into the full Rails stack.

Sorry, I googled around, but could not find the original article.