How to block 100,000+ individual IP addresses

Question

Introduction

How do you block a large number of IP address from your web application / server. Obviously, this is easy to do in PHP or any programming language.

 $ipList = []; // array list or from database if (in_array(getIP(), $ipList)) { // Log IP & Access information header("https://www.google.com.ng/search?q=fool"); // redirect exit(); // exit } 

Or Using htaccess

 order allow,deny deny from 123.45.6.7 deny from 012.34.5. # .... the list continues allow from all 

Problems

• I'm trying to block the whole 100k plus individual IPs not subnets
• I try to avoid user access to PHP before blocking such an IP
• 100000+ - more than 1.5 MB, and this is a lot if the information that needs to be uploaded to htaccess all the time
• The IP database is still growing ... and it would be convenient for them to dynamically add more values
• To set bans in iptables for 100000+ is just ridiculous (may be wrong)

Stupid performance

 order allow,deny deny from database <-------- Not sure if this is possible allow from all 

Question

• Is it possible that htaccess will get the list from the database (Redis, Crunchbase, Mongo, MySQL or even Sqlite) ... any
• Is there a visible solution to managing such a problem in production.
• I know the best solution is Block the IPs at the firewall level , is there a way to pragmatically add / remove an IP address in the firewall

Finally

My approach may be completely wrong ... all I want is a visible solution as spammers and botnets grow ...

Please, this has nothing to do with DOS attacking it simple ... get lost response

Update

• Firewall: Cisco PIX 515UR

Answer 1

Something you can try is to save the list of IPs you want to block in a text file or convert it to a dbm hash file , then use mod_rewrite RewriteMap . You must install this in your server / vhost configuration. You cannot initialize the card in the htaccess file .

 RewriteEngine On RewriteMap deny_ips txt:/path/to/deny_ips.txt RewriteCond ${deny_ips:%{REMOTE_ADDR}|0} !=0 RewriteRule ^ - [L,F] 

The file /path/to/deny_ips.txt will look something like this:

 12.34.56.78 1 11.22.33.44 1 etc. 

Essentially, the IP address you want to reject, and a space, then "1". Any IP address in this text file will cause the server to return 403 Forbidden. To speed httxt2dbm up a bit, you can use httxt2dbm to generate the dbm hash, and then you define the mapping as follows:

 RewriteMap deny_ips dbm:/path/to/deny_ips.dbm 

I'm not sure that mod_rewrite is used for performance, like this, with a lot of IP addresses, but a quick test test on apache 2.2 running on 3Ghz i686 under Linux, the difference between the 5 IP addresses in the list is not significant compared to 102418. According to the conclusion of ab , they are almost identical.

---

Solving specific issues:

Is it possible that htaccess will get the list from the database (Redis, Crunchbase, Mongo, MySQL or even Sqlite) ... any

Using the rewrite map, you can use the prg map type to run an external program for the map type. Then you can write perl, php, etc. script to talk to the database to find the IP address. Also note that reservations are listed in the Caution section. Then you would use this card like any other card ( RewriteCond ${deny_ips:%{REMOTE_ADDR}|0} !=0 ). This would essentially create a bottleneck for all requests. Not the best solution for talking to a database.

However, in apache 2.4 there is a dbd / fastdbd card type that allows you to create requests through mod_dbd . This is a much better option, and the mod_dbd module manages database connections, pool connections, etc. Thus, the map definition will look something like this:

 RewriteMap deny_ips "fastdbd:SELECT active FROM deny_ips WHERE source = %s" 

Suppose you have a table “ deny_ips ” with 2 columns “ source ” (IP address) and “ active ” (1 for active, 0 for inactive).

Is there a visible solution to managing such a problem in production

If you store all blocked IP addresses in a database, it is a matter of managing the contents of your database table. If you are using the dbm card type, I know that perl has a DBI for managing dbm files, so you can use this to add / remove IP entries from the ban list. I have never used it before, so I can’t talk much about it. Managing a flat text file will be much more difficult, especially if you plan on deleting entries, rather than just adding to it. Outside of using the database and apache 2.4 mod_dbd, I don’t think any of these solutions are ready or ready. He will need special work.

I know that the best solution is to block IP addresses at the firewall level, is there a way to pragmatically add / remove an IP address in the firewall

There is a perl interface for IPtables that is marked as beta, but I have never used it before. There is libiptc , but according to netfilter faq :

Is there a C / C ++ API for adding / removing rules?

The answer, unfortunately, is: No.

Now you might be thinking, "what about libiptc?" As repeatedly noted on the mailing list, libiptc NEVER was intended to be used as an open interface. We do not guarantee a stable interface, and it is planned to remove it in the next embodiment of linux package filtering. libiptc is too low level to use it anyway.

We are well aware that there is a fundamental flaw for such an API, and we are working to improve this situation. Until then, it is recommended that you either use system () or open a pipe in stdin iptables-restore. The latter will give you better performance.

Therefore, I do not know how viable the libiptc solution is the lack of API stability.

Answer 2

OTHER PROSPECTS

Hello. You can check if the address is locked or not by accessing two bytes in two 8 KB data blocks. Yes, I'm serious ... Please be patient because it takes a little time to explain this.

THEORY

An IP address is an address, in fact a 4-byte number.

The question is, what if we do this to access bit positions ?.

Answer: Well, we will have

  2^32 = 4 Giga Bits 

addressing space and it will take

  4Gb/8 = 512 Mega Bytes 

distribution. Ouch! But don’t worry, we are not going to block everything in ipverse, and 512MB is an exaggeration.

This may open the door to a solution.

Liliput case

Think of a Lilliputian world in which there are only IP addresses from 0 to 65535. Thus, the addresses have a value of 0.1 or 42.42 to 255.255.

Now the king of this world wants to block several L-IP addresses (lilliput ip).

First, he creates a virtual two-dimensional bitmap with a length of 256 * 256 bits, which takes up:

  64 K Bits = 8 K Bytes. 

He decides to block this unpleasant “revolution” site, which he hates because he is king, for example, the address 56.28.

 Address = (56 * 256) + 28 = 14364.(bit position in whole map) Byte in map = floor(14364 / 8) = 1795. Bit position= 14364 % 8 = 4.(modulus) 

He opens the map file, accesses the 1795th byte and sets bit 4 (to | 16), then writes it back to mark the site as blocked.

When his script sees 56.28, it performs the same calculation and looks at the bit, and if it is set, it blocks the address.

Now, what is the moral of this story? Well, we can use this lilliputic structure.

PRACTICE

Real world thing

We can apply the Lilliput affair to the real world with “use it when you need it,” because allocating a 512 MB file is not a good choice.

Think of a database table called BLOCKS with these elements:

 IpHead(key): unsigned 16 bit integer, Map : 8KB BLOB(fixed size), EntryCount : unsigned 16 bit integer. 

And another table with only one record with the structure below called BASE

 Map : 8KB BLOB(fixed size). 

Now let's say you have an incoming address 56.28.10.2

Script accesses the BASE table and receives a map.

He is looking for IP numbers :

 Address = (56 * 256) + 28 = 14364.(bit position in whole map) Byte in map = floor(14364 / 8) = 1795. Bit position= 14364 % 8 = 4.(modulus) 

Look at the byte of 1795 bit 4 on the map.

If the bit is not set, further operation is not required, since there is no blocked IP address in the range 56.28.0.0 - 56.28.255.255.

If the bit is set, the script accesses the BLOCKS table.

Higher order IP numbers were 56.28, which gives 14364, so the script queries the BLOCKS table with index IpHead = 14364. Selects the entry. The record must exist, as it is marked on BASE.

Script calculates for a lower IP address

 Address = (10 * 256) + 2 = 2562.(bit position in whole map) Byte in map = floor(2562 / 8) = 320. Bit position= 2562 % 8 = 2.(modulus) 

He then checks to see if the address is blocked by looking at bit 2 of byte 320 of the field map.

The task is completed!

Q1: Why do we use BASE at all, we can directly request BLOCKS from 14364.

A1: Yes, we could, but a search in a BASE card will be faster than a BTREE search for any database server.

Q2: What is the EntryCount field in the BLOCKS table for?

A2:. The number of IP addresses is blocked in the map field in the same entry. So, if we unlock ip and EntryCount reaches 0, then the BLOCKS record will become unnecessary. It can be erased, and the corresponding bit on the BASE card will be canceled.

IMHO, this approach will be lightning fast. Also for highlighting 8K blob for recording. Since db servers store drops in separate files, file systems with 4K, 8K or multiples of 4K paging will respond quickly.

In case blocked addresses are too scattered

Well, this is a problem that will make the database table BLOCK unnecessarily.

But for such cases, an alternative is to use a 256 * 256 * 256-bit cube with a length of 16777216 bits, equal to 2097152 bytes = 2 MB.

In our previous example, a higher resolution Ip:

 (56 * 65536)+(28 * 256)+10 

So, the BASE will become a 2MB file instead of writing the db table to be opened (fopen, etc.), and the bit will be processed by searching (for example, fseek, never read the contents of the entire file, it is not necessary), then enter the BLOCKS table with structure below:

 IpHead(key): unsigned 32 bit integer, (only 24 bit is used) Map : 32 unsigned 8 bit integers(char maybe),(256 bit fixed) EntryCount : unsigned 8 bit integer. 

Here is a sample PHP code for checking the blocks of the bitplane-bitplane (8K 8K) version:

Side note: this script can be optimized by eliminating multiple calls, etc. But it is written so that it is easy to understand.

 <? define('BLOCK_ON_ERROR', true); // WARNING if true errors block everyone $shost = 'hosturl'; $suser = 'username'; $spass = 'password'; $sdbip = 'database'; $slink = null; $slink = mysqli_connect($shost, $suser, $spass, $sdbip); if (! $slink) { $blocked = BLOCK_ON_ERROR; } else { $blocked = isBlocked(); mysqli_close($slink); // clean, tidy... } if ($blocked) { // do what ever you want when blocked } else { // do what ever you want when not blocked } exit(0); function getUserIp() { $st = array( 'HTTP_CLIENT_IP', 'REMOTE_ADDR', 'HTTP_X_FORWARDED_FOR' ); foreach ( $st as $v ) if (! empty($_SERVER[$v])) return ($_SERVER[$v]); return (""); } function ipToArray($ip) { $ip = explode('.', $ip); foreach ( $ip as $k => $v ) $ip[$k] = intval($v); return ($ip); } function calculateBitPos($IpH, $IpL) { $BitAdr = ($IpH * 256) + $IpL; $BytAdr = floor($BitAdr / 8); $BitOfs = $BitAdr % 8; $BitMask = 1; $BitMask = $BitMask << $BitOfs; return (array( 'bytePos' => $BytAdr, 'bitMask' => $BitMask )); } function getBaseMap($link) { $q = 'SELECT * FROM BASE WHERE id = 0'; $r = mysqli_query($link, $q); if (! $r) return (null); $m = mysqli_fetch_assoc($r); mysqli_free_result($r); return ($m['map']); } function getBlocksMap($link, $IpHead) { $q = "SELECT * FROM BLOCKS WHERE IpHead = $IpHead"; $r = mysqli_query($link, $q); if (! $r) return (null); $m = mysqli_fetch_assoc($r); mysqli_free_result($r); return ($m['map']); } function isBlocked() { global $slink; $ip = getUserIp(); if($ip == "") return (BLOCK_ON_ERROR); $ip = ipToArray($ip); // here you can embed preliminary checks like ip[0] = 10 exit(0) // for unblocking or blocking address range 10 or 192 or 127 etc.... // Look at base table base record. // map is a php string, which in fact is a good byte array $map = getBaseMap($slink); if (! $map) return (BLOCK_ON_ERROR); $p = calculateBitPos($ip[0], $ip[1]); $c = ord($map[$p['bytePos']]); if (($c & $p['bitMask']) == 0) return (false); // No address blocked // Look at blocks table related record $map = getBlocksMap($slink, $p[0]); if (! $map) return (BLOCK_ON_ERROR); $p = calculateBitPos($ip[2], $ip[3]); $c = ord($map[$p['bytePos']]); return (($c & $p['bitMask']) != 0); } ?> 

Hope this helps.

If you have questions about the details, I will be happy to answer.

Answer 3

You need to do this using an external firewall, not PHP. I recommend pfSense or PF . I have used it before and it is very easy to use, very intuitive and extremely effective. This is the choice of the best system administrators. I run it on FreeBSD, but it works fine on OpenBSD as well. I'm a Linux guy, so it hurts me to say this, but don't try to run it on Linux. BSD is easy and you can quickly figure it out.

The amazing pfSense feature is the ability to configure scripts and restrict access to the configuration by a single network interface (so that only settings on the local network can be configured). It also has several ID10T level features so that you cannot accidentally disable your access.

You should also be aware that many spammers can quickly switch to IP addresses using Tor . To fix this, you must include addresses that are known output nodes (this list is accessible from different places) in your list of blocks.

Answer 4

Block traffic before it reaches the www server using iptables and ipset.

Catch the black IP traffic in the filter table of the INPUT chain, assuming that your web server is on the same computer. If you are blocking IP addresses on the router, you will need the FORWARD chain.

First create an ipset:

 ipset create ip_blacklist hash:ip 

IP addresses can be added via:

 ipset add ip_blacklist xxx.xxx.xxx.xxx 

Add the ipset matching rule to your iptables (DROP all packages match ipset):

 iptables --table filter --insert INPUT --match set --match-set ip_blacklist src -j DROP 

This will stop the blacklist in front of the www server.

Edit: I had the opportunity to see the default maximum size, and this is 65536, so you will need to configure it to support 100,000+ entries:

 ipset create ip_blacklist hash:ip maxelem 120000 

You can also adjust the hash size:

ipset create ip_blacklist hash:ip maxelem 120000 hashsize 16384 (must be 2)

My experience - searching for ipset has little effect on my system (~ 45,000 entries). There are several tests on the net. Memory for dialing is a limiting factor.

Answer 5

If you need a way to add / remove using code, check out denyhosts . You can either save the IP list through code, or fix the source to read from any location you want.

Answer 6

There is a project with netfilter for what is called ipset, so you can add or remove ip to the list, and you just need to create a rule against this list

http://ipset.netfilter.org/

Answer 7

If you block IP addresses, you really have to do it at the firewall level (you do not want users from unwanted IP addresses to get very far into your system). Thus, I suggest writing a bash script that queries the database and modifies your firewall configuration file accordingly (it is assumed that you want a solution using the IP addresses stored in your web database to be a very good place for storing such information).

EDIT: if you want to add IP addresses to the blacklist at the PHP level, as @Populus suggested, here is a guide to using system calls in PHP: http://php.net/manual/en/function.system.php

And here are the commands you need to use to add an IP address to the blacklist if you use iptables: http://www.cyberciti.biz/faq/linux-iptables-drop/

Answer 8

I know a way
him by php. as you mentioned at the very beginning of this question.

 $ipList = []; // array list or from database if (in_array(getIP(), $ipList)) { // Log IP & Access information header("https://www.google.com.ng/search?q=fool"); // redirect exit(); // exit } 

I read what you wrote. but wait a bit.
Why would you want to do that. It does not matter. But why in this situation.
I mean, why do you want to avoid access to php because of its speed or simply because of prevention and because it is so difficult to call a function on all pages? if the only reason to avoid using some ip from access to php is to search for a topic to view the content.
so I got an idea and I missed you like that.

using one entry point.

I worked with this solution.

first, using simple htaccess, you send all requests to a single page called an entry point. (e.g. index.php)
with a simple rewrite rule I will give it to you. so when the user asks

 mysite.com/some/path/page.php or anything 

htaccess will do something like the following without changing the url. therefore, the user does not feel anything.

 mysite.com/index.php?r=some/path/page.php 

therefore, each request became one request with different parameters $ _GET ['r']. therefore, an index.php will be executed for the evey request. and now we can do something like this in index.php

 $ipList = []; // array list or from database if (in_array(getIP(), $ipList)) { // Log IP & Access information header("https://www.google.com.ng/search?q=fool"); // redirect exit(); // exit } //if after this execute means the IP is not banned //now we can include file that $_GET['r'] points to include $_GET['r']; 

Its that simple. And its real is so complicated. But the basic idea is the same. what do you think?

Answer 9

Most of us seem to agree at the firewall level .

You may have a program that listens on your site for ips to block and generates a script:

 ip = getNextIpToBlock() an = increment_unique_alphanum_generator() script = generate_script(ip, an) 

the script will look something like this (where [an] is the alphanumeric value, and [ip] is the ip lock):

 en [enter] *password* [enter] conf t [enter] access-list [an] deny ip [ip] 0.0.0.0 any [enter] access-group [an] in interface outside [enter] 

Then you load this script into another program that makes remote telnet or ssh calls to your FW CLI.

Remember to log out and perhaps every 100 ips you copy the current configuration to start the configuration.

I don’t know, but now you can find out what are the restrictions for your firewall.

Best

Answer 10

Do a geo-search for IP addresses in your list. My own experience has shown that the most malicious (i.e. Spam) connections originated from China. If you find the same thing for you, and you don’t have a specific need to service China, see if you can effectively block the whole country at the firewall level.

Answer 11

IMHO, there are several angles from which this question can be considered

• You have a very large set of unique IP addresses to block. the sooner you block them on your server, the less computing power you will spend on them. You can add / remove IP addresses in your firewall using the appropriate system calls with PHP.

Given the above option, the only relevant issues are:

• How often do they visit? => If yes, then solution (1) is your best choice.
• Can your firewall handle it efficiently? => If not, you may consider another solution.

.htaccess will be my second choice.