How to limit the size of a Net :: HTTP request?

Question

I am creating an API service that allows people to provide an image URL to an API call, and the service loads the image for processing.

How can I guarantee that someone does NOT give me a URL, for example, a 5 MB image? Is there a way to limit the request?

This is what I still have, which basically captures everything.

req = Net::HTTP::Get.new(url.path) res = Net::HTTP.start(url.host, url.port) { |http| http.request(req) } 

Thanks Conrad

Answer 1

Try this first:

 Net::HTTP.start(url.host, url.port) { |http| response = http.request_head(url.path) raise "File too big." if response['content-length'].to_i > 5*1024*1024 } 

You still have a race condition (someone can change the file after the HEAD request), but in the simple case this will ask the server for headers, which it will send back from the GET request.

Answer 2

cwninja , unfortunately, has given you an answer that will only work for random attacks. A smart attacker will have no problem defeating this test. There are two main reasons why his method should not be used. Firstly, nothing guarantees that the information in the HEAD response will correspond to the corresponding GET response. A properly functioning server will certainly do this, but a malicious actor should not follow the specification. An attacker can simply send a HEAD response that says it has a Content-Length that is less than your threshold, but then pass you a huge file in the GET response. But this does not even cover the server’s ability to send a response using the Transfer-Encoding: chunked header set. The reaction with a series may well end. A few people pointing to your server in endless answers can perform a trivial resource-exhaustion attack, even if your HTTP client uses a timeout.

To do this correctly, you need to use the HTTP library, which allows you to count bytes as they are received and abort if it crosses the threshold. I would rather recommend Curb for this, and not for Net :: HTTP. (Can you even do this with Net :: HTTP?) If you use the on_body and / or on_progress callbacks, you can count incoming bytes and abort the average response if you get a file that is too large. Obviously, as cwninja already noted, if you get a Content-Length header that exceeds your threshold, you also want to abort it. Curb is also noticeably faster than Net :: HTTP .

Answer 3

Combining the other two answers, I would like to: 1) check the size of the header, 2) look at the size of the pieces, and 3) support https and 4) aggressively apply a timeout. Here is the helper I came up with:

 require "net/http" require 'uri' module FetchUtil # Fetch a URL, with a given max bytes, and a given timeout def self.fetch_url url, timeout_sec=5, max_bytes=5*1024*1024 uri = URI.parse(url) t0 = Time.now.to_f body = '' Net::HTTP.start(uri.host, uri.port, :use_ssl => (uri.scheme == 'https'), :open_timeout => timeout_sec, :read_timeout => timeout_sec) { |http| # First make a HEAD request and check the content-length check_res = http.request_head(uri.path) raise "File too big" if check_res['content-length'].to_i > max_bytes # Then fetch in chunks and bail on either timeout or max_bytes # (Note: timeout won't work unless bytes are streaming in...) http.request_get(uri.path) do |res| res.read_body do |chunk| raise "Timeout error" if (Time.now().to_f-t0 > timeout_sec) raise "Filesize exceeded" if (body.length+chunk.length > max_bytes) body += chunk end end } return body end end 

Answer 4

Another way to limit download size (full code should check response status, exception handling, etc. This is just an example)

 Net::HTTP.start(uri.host, uri.port) do |http| request = Net::HTTP::Get.new uri.request_uri http.request request do |response| # check response codes here body='' response.read_body do |chunk| body += chunk break if body.size > MY_SAFE_SIZE_LIMIT end break end end