How to search YouTube using HtmlUnit

Question

I wonder if you can search YouTube using HtmlUnit . I started writing code, here it is:

import java.io.IOException; import java.net.MalformedURLException; import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException; import com.gargoylesoftware.htmlunit.WebClient; import com.gargoylesoftware.htmlunit.html.HtmlForm; import com.gargoylesoftware.htmlunit.html.HtmlPage; import com.gargoylesoftware.htmlunit.html.HtmlSubmitInput; public class HtmlUnitExampleTestBase { private static final String YOUTUBE = "http://www.youtube.com"; public static void main(String[] args) throws FailingHttpStatusCodeException, MalformedURLException, IOException { WebClient webClient = new WebClient(); webClient.setThrowExceptionOnScriptError(false); //This is equivalent to typing youtube.com to the adress bar of browser HtmlPage currentPage = webClient.getPage("http://www.youtube.com"); //Get form where submit button is located HtmlForm searchForm = (HtmlForm) currentPage.getElementById("masthead-search"); //Printing result form System.out.println(searchForm.asText()); final List<HtmlAnchor> listLinks = (List<HtmlAnchor>) newPage.getByXPath("//a[@class='ux-thumb-wrap result-item-thumb']"); for (int i=0; i<listLinks.size(); i++){ System.out.println(YOUTUBE + listLinks.get(i).getAttribute("href")); } } } 

Now I do not know how to enter text in the search field and click the "Search" button.

I saw tutorials about HtmlUnit, but I have a problem because they use a method called: getElementByName , but the YouTube search button does not have a name, but only an identifier. Can someone help me?

EDIT: I edited the code above the code and now I get YouTube links from the first page. But before that, I need to sort by download date, and then grab the links. Can someone help me sort?

Answer 1

I am not an HtmlUnit expert, but there is a workaround. You can add your own button to the form and use it to submit the form.

Here is a sample code with comments:

 import java.io.IOException; import java.net.MalformedURLException; import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException; import com.gargoylesoftware.htmlunit.WebClient; import com.gargoylesoftware.htmlunit.html.HtmlButton; import com.gargoylesoftware.htmlunit.html.HtmlForm; import com.gargoylesoftware.htmlunit.html.HtmlPage; import com.gargoylesoftware.htmlunit.html.HtmlTextInput; public class HtmlUnitExampleTestBase { public static void main(String[] args) throws FailingHttpStatusCodeException, MalformedURLException, IOException { WebClient webClient = new WebClient(); webClient.setThrowExceptionOnScriptError(false); // This is equivalent to typing youtube.com to the adress bar of browser HtmlPage currentPage = webClient.getPage("http://www.youtube.com"); // Get form where submit button is located HtmlForm searchForm = (HtmlForm) currentPage.getElementById("masthead-search"); // Get the input field. HtmlTextInput searchInput = (HtmlTextInput) currentPage.getElementById("masthead-search-term"); // Insert the search term. searchInput.setText("Nyan Cat"); // Workaround: create a 'fake' button and add it to the form. HtmlButton submitButton = (HtmlButton) currentPage.createElement("button"); submitButton.setAttribute("type", "submit"); searchForm.appendChild(submitButton); // Workaround: use the reference to the button to submit the form. HtmlPage newPage = submitButton.click(); System.out.println(newPage.asText()); } } 

Answer 2

HtmlUnit is fine, but I really prefer Watir or Selenium for web automation.

One of the weaknesses of HtmlUnit is the lack of selection methods for accessing DOM elements using jQuery. Check out the css-selector project that will add HtmlUnit to help you do what you need very easily. It introduced the Gooder Code .

Once you get this, the selector for the YouTube search form will be ".search-term", and the submit button selector will be ".search-button"