Handling large arrays with array_diff

Question

I am trying to compare two arrays. Using array_intersect is not a problem. When using array_diff and arrays with ~ 5000 values, it works. When I get ~ 10,000 values, the script dies when I get the diff array. Enabling error_reporting did nothing.

I tried to create my own array_diff function:

function manual_array_diff($arraya, $arrayb) { foreach ($arraya as $keya => $valuea) { if (in_array($valuea, $arrayb)) { unset($arraya[$keya]); } } return $arraya; } 

source: How does array_diff work?

I would expect it to be less efficient than the official array_diff, but it can handle arrays of size ~ 10,000. Unfortunately, both array_diffs fail when I get to ~ 15,000.

I tried the same code on another machine and it works fine, so this is not a problem with the code or PHP. There must be some limit on this particular server. Any idea how I can get around this restriction or change it or just find out what it is?

Answer 1

Having encountered the same problem, I really hoped for an answer here.

So, I had to find my own path around it and came up with the following ugly shred that works for me with arrays of around 50,000 elements. This is based on your observation that array_intersect works, but array_diff does not.

Sooner or later, this will also overflow resource limits, in which case it will be necessary to trim arrays and process smaller bits. We will cross this bridge when we get to it.

 function new_array_diff($arraya, $arrayb) { $intersection = array_intersect($arraya, $arrayb); foreach ($arraya as $keya => $valuea) { if (!isset($intersection[$keya])) { $diff[$keya] = $valuea; } } return $diff; } 

Answer 2

In my php.ini:

 max_execution_time = 60 ; Maximum execution time of each script, in seconds memory_limit = 32M ; Maximum amount of memory a script may consume 

Can differences in these settings or, alternatively, in machine performance cause problems? Have you checked the web server error logs (if you run it through one)?

Answer 3

You mentioned that this works in a browser. Try running the script through the command line and see if the result is different.