Which web host allows you to run scanners?

Question

I am working on a graduation project for one of my university courses, and I need to find a place to run several scanners that I wrote in C #. Without the experience of web hosting, I got a little lost. Is this what any site allows? Do I need a special host that gives more access to the server? A scanner is a simple application that does its job and then periodically writes information to a remote database.

Answer 1

A web crawler is a simulation of a regular user. These are acess sites, as browsers do, receiving the html code (javascript, etc.) returned from the server (so there is no internal access to the server code). Thus, any site can be crawled.

Check out some guidelines for a web crawler ethic . There are pages that you should not index or follow your links. And web developers create some files and instructions for web crawlers, saying that you can index or follow.

Answer 2

This is not like web hosting. You just need a machine with an Internet connection and a database server.

I would look at your university if I were you. At least, at one time, much could be organized in the house when it came to graduation projects.

Otherwise, you could explore a simple VPS ( Virtual Private Server ) account. If you are not sure if your application is running Mono , you will need Windows. Resource limits are usually much lower than you could get from a dedicated server, but they are relatively affordable. Some of them offer an MS SQL Server database, which you can use next to the VPS account (on another machine). Installing SQL Server on the VPS itself can be a problem license.

Make sure you check the terms of use before opening an account, as well as the (virtual) system specifications. Also check if there is any minimum contract period. Sometimes it can be more than one month, especially if there is no installation fee.

If at all possible, find a host that is geographically close to you. A server on the other side of the world can be a little annoying to access Remote Desktop.

Answer 3

If you cannot start it from the desktop for some reason, you will need a host that allows you to execute arbitrary C # code. The cheapest web servers do not because of potential security implications, as there will be several other users on the same server.

This means that you need to be on a server where you have your own OS. Or VPS - Virtual Private Server, where virtualization is used to provide you with your own OS, but to share hardware or your own dedicated server where you have both hardware and software.

Please note that if you are running on any server, you need to make sure that you are throttling yourself so as not to create problems for your neighbors; your main problem will not use too much CPU or bandwidth. This is not just for courtesy - most web hosts suspend your hosting if you cause problems on your network, for example, abandoning other users of the hardware you work on, consuming them all yourself. You can usually hack higher levels of use, but they will cut you back if you maintain them for a considerable period of time.

Answer 4

80legs allows you to use your crawlers to process millions of web pages with your own program.

Rates:

• $ 2.00 per million pages
• $ 0.03 per processor hour

They claim they will crawl 2 billion web pages per day.

Answer 5

You will need a VPS (virtual private server) or a full dedicated server. Scanners are nothing more than applications that scan the Internet. Although you can configure the website as a crawler, this is not practical because you need access to a search engine to access the web page. You will need to read the ToS (Terms of Service) for the host to find out what conditions are intended for use. Some of the lower priced hosts will reduce your connection for the purpose of "negative impact on the network" if you try to use more bandwidth, even if they give you a lot of opportunities to use.

VPS are around $ 30-80 for a Linux server and $ 60 + for a Windows server. Dedicated services run $ 100 + for linux and windows servers.

Answer 6

You do not need web hosting to run your spider. Just ask a computer with a web connection, which can act as a dedicated server, configure the database and run the scanner there.