SOLVED: I fixed it by installing HTML :: HeadParser. But I still don't know why this just stopped working.
I noticed that some of the LWP requests that worked before this stopped working, so I made a little script to check why. For some reason, I don't get anything as content from most sites (only two that have worked so far are icanhazip.com and gdata.youtube.com api). I get status 200 back and I get the correct Content-Lenght for the title. I tried to get the page in both curl and wget without any problems. And if I check requests with tcpdump, I really get a response with all the necessary information. I tried with LWP :: UserAgent and LWP :: Simple with the same results.
#!/usr/bin/perl use warnings; use strict; use LWP::UserAgent; my $agent = LWP::UserAgent->new; $agent->agent("Mozilla/5.0"); my $req = HTTP::Request->new(GET => 'http://stackoverflow.com'); my $res = $agent->request($req); print $res->status_line, "\n"; print $res->header("Content-Length"), "\n"; print $res->content, "\n";
Dumping $ res with DumpLex gives the following:
$HTTP_Response1 = bless( { _content => '', _headers => bless( { "cache-control" => 'public, max-age=15', "client-aborted" => 'die', "client-date" => 'Sat, 23 Jun 2012 15:07:23 GMT', "client-peer" => '64.34.119.12:80', "client-response-num" => 1, connection => 'close', "content-length" => 211684, "content-type" => 'text/html; charset=utf-8', date => 'Sat, 23 Jun 2012 15:07:22 ', expires => 'Sat, 23 Jun 2012 15:07:38 ', "last-modified" => 'Sat, 23 Jun 2012 15:06:38 GMT ', vary => '*', "x-died" => 'Can\'t locate HTML/HeadParser.pm in @INC (@INC contains: /usr/lib/perl5/site_perl /usr/share/perl5/site_perl /usr/lib/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/core_perl /usr/share/perl5/core_perl .) at /usr/share/perl5/site_perl/LWP/UserAgent.pm line 663.' }, 'HTTP::Headers' ), _msg => 'OK', _protocol => 'HTTP/1.1', _rc => 200, _request => bless( { _content => '', _headers => bless( { "content-type" => 'application/html', "user-agent" => 'Mozilla/5.0' }, 'HTTP::Headers' ), _method => 'GET', _uri => \do { my $v = 'http://stackoverflow.com/' }, _uri_canonical => 'V: $HTTP_Response1->{_request}{_uri}' }, 'HTTP::Request' ), default_add_content => 1 }, 'HTTP::Response' ); $HTTP_Response1->{_request}{_uri_canonical} = $HTTP_Response1->{_request}{_uri}; bless( $HTTP_Response1->{_request}{_uri}, 'URI::http' );