Geek Answers Handbook

Using Jsoup to login and scan data

I want to use Jsoupto crawl a page that is accessible only at login. I guess this means that I need to log in to one page and send cookies to another page.
I read an earlier entry here and write the following code:

public static void main(String[] args) throws IOException {
    Connection.Response res = Jsoup.connect("login.yahoo.com")
        .data("login", "myusername", "passwd", "mypassword")
        .method(Method.POST)
        .execute();

Document doc=res.parse();
String sessionId = res.cookie("SESSIONID");

Document doc2 = Jsoup.connect("http://health.groups.yahoo.com/group/asthma/messages")
        .cookie("SESSIONID", sessionId)
        .get();

Elements Eles=doc2.getElementsByClass("message");

String content=Eles.first().text();

System.out.println(content);

My question is, how can I find out the name of my cookie (ie "SESSIONID") here to send my account? I used the method .cookies()to get all cookies from the login page:

B
DK
YM
T
PH
Y
F

, . sessionId , , , . - ? !

+3
2

Ive jsoup.

, , webdriver jsoup.

Webdriver , .

, . "" webdriver: HtmlUnitDriver. :

HtmlUnitDriver driver = new HtmlUnitDriver(true); // true meaning javascript support (Using rhino i be leave)

:

String baseUrl = "http://www.thesite.com";

driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);

driver.get(baseUrl);

driver.findElement(By.id("TextBoxUser")).clear();
driver.findElement(By.id("TextBoxUser")).sendKeys("username");
driver.findElement(By.id("TextBoxPass")).clear();
driver.findElement(By.id("TextBoxPass")).sendKeys("password");
driver.findElement(By.id("Button1")).click();

:

String htmlContent = driver.getPageSource();

jsoup:

Document document = Jsoup.parse(htmlContent);

.

+7

- :

Connection.Response res = Jsoup.connect("https://login.yahoo.com/config/login?")
    .data("login", "myusername", "passwd", "mypassword")
    .method(Method.POST)
    .execute();

 Map<String, String> cookies = res.cookies();

 Connection connection = Jsoup.connect("http://health.groups.yahoo.com/group/asthma/messages");

 for (Map.Entry<String, String> cookie : cookies.entrySet()) {
     connection.cookie(cookie.getKey(), cookie.getValue());     
 }

 Document doc=  connection.get();
 // #code selector
 // Example
 // Element e=doc.select(".ygrp-grdescr").first();
 // System.out.println(e.text()); // Print => This list will be for asthmatics, and anyone whose       life is affected by it. Discussions include causes, problems, and treatment

, .

+2

More articles:


All Articles