Finding and removing outliers in PHP

Question

Suppose I take a selection of database records that return the following numbers:

20.50, 80.30, 70.95, 15.25, 99.97, 85.56, 69.77 

Is there an algorithm that can be effectively implemented in PHP to find outliers (if any) from an array of floats, depending on how much they deviate from the average?

Answer 1

Ok let's assume that you have your data points in an array like this:

 <?php $dataset = array(20.50, 80.30, 70.95, 15.25, 99.97, 85.56, 69.77); ?> 

Then you can use the following function (see comments for what happens) to remove all numbers that go beyond the average +/- standard deviation time from the value you set (default is 1):

 <?php function remove_outliers($dataset, $magnitude = 1) { $count = count($dataset); $mean = array_sum($dataset) / $count; // Calculate the mean $deviation = sqrt(array_sum(array_map("sd_square", $dataset, array_fill(0, $count, $mean))) / $count) * $magnitude; // Calculate standard deviation and times by magnitude return array_filter($dataset, function($x) use ($mean, $deviation) { return ($x <= $mean + $deviation && $x >= $mean - $deviation); }); // Return filtered array of values that lie within $mean +- $deviation. } function sd_square($x, $mean) { return pow($x - $mean, 2); } ?> 

For your example, this function returns the following with a value of 1:

 Array ( [1] => 80.3 [2] => 70.95 [5] => 85.56 [6] => 69.77 ) 

Answer 2

For a normally distributed dataset, values ​​greater than 3 standard deviations from the mean are deleted.

 <?php function remove_outliers($array) { if(count($array) == 0) { return $array; } $ret = array(); $mean = array_sum($array)/count($array); $stddev = stats_standard_deviation($array); $outlier = 3 * $stddev; foreach($array as $a) { if(!abs($a - $mean) > $outlier) { $ret[] = $a; } } return $ret; }