How to get a direct link to a remote video from an embedded URL in an Android URL using JSoup?

Question

Earlier, I asked a question about how to extract the embedded URL for a video file and did it successfully. Now I have a different problem. The json response for the WUnderground API webcam response gives the following URL:

https://www.wunderground.com/webcams/cadot1/902/show.html

Using JSoup and for answering my original problem, I was able to get this inline link:

https://www.wunderground.com/webcams/cadot1/902/video.html?month=11&year=2016&filename=current.mp4

When I tried to "stream" video from this URL in VideoView, I continued to receive the error message "cannot play video." Looking at the source of this link, I noticed that the video file to be played is not mentioned in html, but rather in javascript. How can I get a direct link to the video file that I want to play? Using JSoup or another process?

The source for the URL https://www.wunderground.com/webcams/cadot1/902/video.html?month=11&year=2016&filename=current.mp4 shows the following for the required video in the <script> bracket:

url: "//icons.wunderground.com/webcamcurrent/c/a/cadot1/902/current.mp4?e=1480377508"

I use JSoup to get the embedded URL for the video from the response URL:

  private class VideoLink extends AsyncTask<Void, Void, Void> { String title; @Override protected void onPreExecute() { super.onPreExecute(); mProgressDialog.setTitle("JSOUP Test"); mProgressDialog.setMessage("Loading..."); mProgressDialog.setIndeterminate(false); mProgressDialog.show(); } @Override protected Void doInBackground(Void... params) { try { // for avoiding javax.net.ssl.SSLProtocolException: handshake alert: unrecognized_name System.setProperty("jsse.enableSNIExtension", "false"); // WARNING: do it only if security isn't important, otherwise you have // to follow this advices: http://stackoverflow.com/a/7745706/1363265 // Create a trust manager that does not validate certificate chains TrustManager[] trustAllCerts = new TrustManager[]{new X509TrustManager(){ public X509Certificate[] getAcceptedIssuers(){return null;} public void checkClientTrusted(X509Certificate[] certs, String authType){} public void checkServerTrusted(X509Certificate[] certs, String authType){} }}; // Install the all-trusting trust manager try { SSLContext sc = SSLContext.getInstance("TLS"); sc.init(null, trustAllCerts, new SecureRandom()); HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory()); } catch (Exception e) { ; } // Connect to the web site Document doc = Jsoup.connect(TEST_URL).get(); Elements elements = doc.getElementsByClass("videoText"); // Get the html document title for (Element link : elements) { String linkHref = link.attr("href"); // linkHref contains something like video.html?month=11&year=2016&filename=current.mp4 // TODO check if linkHref ends with current.mp4 title = linkHref; } } catch (IOException e) { e.printStackTrace(); mProgressDialog.dismiss(); } return null; } @Override protected void onPostExecute(Void result) { // Set title into TextView resultTxt.setText(title); String resVid = TEST_URL; Log.d(TAG, "URL: " + resVid); Uri resUri = Uri.parse(resVid); try { // Start the MediaController MediaController mediacontroller = new MediaController( MainActivity.this); mediacontroller.setAnchorView(resultVidVw); // Get the URL from String VideoURL Uri video = Uri.parse(resVid); resultVidVw.setMediaController(mediacontroller); resultVidVw.setVideoURI(video); } catch (Exception e) { Log.e("Error", e.getMessage()); e.printStackTrace(); } resultVidVw.requestFocus(); resultVidVw.setOnPreparedListener(new MediaPlayer.OnPreparedListener() { // Close the progress bar and play the video public void onPrepared(MediaPlayer mp) { mProgressDialog.dismiss(); resultVidVw.start(); } }); } } 

Note that I need to do this on every JSONObject in the response array.

Answer 1

Here's how you can GET the file:

( Notice : the extraction part only works with the current html of the site, and if this happens, it may not be so!)

 String url = "https://www.wunderground.com/webcams/cadot1/902/video.html"; int timeout = 100 * 1000; // Extract video URL Document doc = Jsoup.connect(url).timeout(timeout).get(); Element script = doc.getElementById("inner-content") .getElementsByTag("script").last(); String content = script.data(); int indexOfUrl = content.indexOf("url"); int indexOfComma = content.indexOf(',', indexOfUrl); String videoUrl = "https:" + content.substring(indexOfUrl + 6, indexOfComma - 1); System.out.println(videoUrl); 

[Exit: https://icons.wunderground.com/webcamcurrent/c/a/cadot1/902/current.mp4?e=1481246112 ]

Now you can retrieve the file by specifying .ignoreContentType(true) to avoid org.jsoup.UnsupportedMimeTypeException and .maxBodySize(0) to remove the file size limit.

 // Get video file byte[] video = Jsoup.connect(videoUrl) .ignoreContentType(true).timeout(timeout).maxBodySize(0) .execute().bodyAsBytes(); 

I don't know if you can play Android or not, but I think you can save it using org.apache.commons.io.FileUtils (I tested it in Java SE, but not in the Android development environment.)

 // Save video file org.apache.commons.io.FileUtils.writeByteArrayToFile(new File("test.mp4"), video);