Problem with urlfetch for 1M in the application engine

Question

for fetch () over 1M in the application engine, I use the range header and then combine those parts. and my codes:

int startpos=0; int endpos; int seg=1; int len=1; while(len>0){ endpos=startpos+seg; httpConn = (HttpURLConnection) u.openConnection(); httpConn.setRequestMethod("GET"); con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14"); con.setRequestProperty("Range", "bytes=" + startpos + "-" + endpos); con.connect(); InputStream in=con.getInputStream(); len=con.getContentLength(); byte[] b=new byte[len]; in.read(b, 0, len); startpos+=len; 

} but when it goes to “InputStream in = con.getInputStream ();”, its debugging is “Error fetching URLs too big problems”, so I don’t know what is wrong with these codes. and are there other ways to extract () through 1M?

Answer 1

Not all HTTP servers support range requests, especially when it comes to infrastructures serving dynamic content — they simply ignore the Range header and send you the entire response.

The recent version 1.4.0 has increased the URLFetch response limit to 32 MB, so you no longer need to do this.

Answer 2

I had the same problem and hacked a small class to simulate an input stream in Appengine using an HTTP range parameter. This allows you to read files that exceed the limits in a linear orientation. I am attaching it below, although you may need to adapt it for your purposes:

 package com.theodorebook.AEStreamer; import java.io.InputStream; import java.net.URL; import java.net.URLConnection; import java.util.Arrays; import java.util.logging.Logger; /** * A class to simulate a stream in appengine, which insists on downloading * an entire URL before letting you do anything with it. This enables one * to read files larger than the size limits. * * @author Theodore Book (theodorebook at gmail dot com) * */ public class AEStreamer { private static final int BITE_SIZE = 0x10000; //How big a chunk to grab at a time private static final byte TERMINATOR = '\n'; //String terminator private int mCurrentPosition = 0; //The current position in the file private int mOffset = -1; //The offset of the current block private long mValidBytes = 0; //The number of valid bytes in the chunk private byte[] mChunk = new byte[BITE_SIZE]; private boolean mComplete = false; private String mURL; private static final Logger log = Logger.getLogger(AEStreamer.class.getName()); public AEStreamer(String url) { mURL = url; } /** * Returns the next line from the source, or null on empty * @return */ public String readLine() { String line = ""; //See if we have something to read if (mCurrentPosition >= mOffset + mValidBytes) { if (mComplete) return null; readChunk(); } if (mValidBytes == 0) return null; //Read until we reach a terminator int endPtr = mCurrentPosition - mOffset; while (mChunk[endPtr] != TERMINATOR) { endPtr++; //If we reach the end of the block if (endPtr == mValidBytes) { line += new String(Arrays.copyOfRange(mChunk, mCurrentPosition - mOffset, endPtr)); mCurrentPosition += (endPtr - mCurrentPosition + mOffset); if (mComplete) { return line; } else { readChunk(); endPtr = mCurrentPosition - mOffset; } } } line += new String(Arrays.copyOfRange(mChunk, mCurrentPosition - mOffset, endPtr)); mCurrentPosition += (endPtr - mCurrentPosition + mOffset); mCurrentPosition++; return line; } /** * Reads the next chunk from the server */ private void readChunk() { if (mOffset < 0) mOffset = 0; else mOffset += BITE_SIZE; try { URL url = new URL(mURL); URLConnection request = url.openConnection(); request.setRequestProperty("Range", "bytes=" + (mOffset + 1) + "-" + (mOffset + BITE_SIZE)); InputStream inStream = request.getInputStream(); mValidBytes = inStream.read(mChunk); inStream.close(); } catch (Exception e) { log.severe("Unable to read " + mURL + ": " + e.getLocalizedMessage()); mComplete = true; mValidBytes = 0; return; } if (mValidBytes < BITE_SIZE) mComplete = true; //log.info("Read " + mValidBytes + " bytes"); } }