I use Watin in a C # console application to crawl websites, five console applications are running at the same time. The reason I partially use Watin as a crawler is because several websites use javascript (or ajax) to set the content of the page.
The following is an example code for counting the number of pages:
Settings.Instance.MakeNewIeInstanceVisible = false;
using (var browser = new IE(commentLink, true))
{
browser.Link(Find.ByUrl(commentLink)).WaitUntilExists(20);
Span commentSpan = browser.Span("COUNT_TOTAL");
if (commentSpan.Exists)
{
int commentCount;
if (Int32.TryParse(commentSpan.InnerHtml, out commentCount))
{
return commentCount;
}
}
}
My problem is that after starting these 5 console applications for some time (90 minutes) many IE instances remain open (due to a timeout or error or IE is busy), so the system is quite slow and requires a reboot.
How can I change my code so that this does not happen, and so that my applications remain efficient?