Save client / server string filtering directives for real-time use

Question

I have an application that captures strings from external APIs and sends them to the browser without saving to the database. The database contains only URLs for API requests.

Lines obtained from the API must be filtered for each line, both the server (PHP) and the client side, and filters must be stored in the database. A filter for a string can be, for example, a substring or a replacement.

I assume you never want to have javascript / PHP code in the database, but how else can this be done? Are there any suitable javascript solutions?

In my database table, I want:

String ------------------ API URL ------------ ------- Filter

Line 1 --------------- api.com/get_data----------- FILTER FILTER HERE

Line 2 --------------- api.com/get_more_data---STRING FILTER HERE

If the "Filter" contains instructions on how to filter the string as a result of the API.

Answer 1

I would suggest using the Factory template and storing the json object in the database containing instructions for creating the filter. The json object will be retrieved from the database and passed to Factory, which will create the correct type of Filter object.

Suppose you have two filters that use a regular expression and the other for a range of values. Then you can create each filter as follows:

class Filter{ public function filter($results){ $filteredResults = array(); foreach($results as $item){ if($this->isMatch($item)){ $filteredResults[]=$item; } } return $filteredResults } public function isMatch($row){ return true; } } class RegexFilter extends Filter{ public function __construct($regexString){ $this->regex = $regexString; } public function isMatch($item){ return preg_match($this->regex,$item) ==1; } } class RangeFilter extends Filter{ public function __construct($min,$max){ $this->max = $max; $this->min = $min; } public function isMatch($item){ return $this->min < $item && $item <$this->max; } } class FilterFactory{ public function createFilter($json){ $filterData = json_decode($json); if($filterData->filterType == 'regex'){ return new RegexFilter($filterData->regexStr); } elseif($filterData->filterType == 'range'){ return new RangeFilter($filterData->min,$filterData->max); } } } // Example use // match only results that start with http $regexJsonStr = '{ "filterType":"regex", "regexStr":"/^http/" }'; $filter = FilterFactory::createFilter($regexJson); $filteredResults = $filter->filter($results); // Match results that values between 0 and 100 exclusive $regexJsonStr = '{ "filterType":"range", "min":0, "max":100 }'; $filter = FilterFactory::createFilter($rangeJson); $filteredResults = $filter->filter($results); 

This approach allows you to use best practices to support your code base while storing filtering instructions in your database.

The child classes use the Filter filter method, but their own isMatch method, which makes it easy to add new filters. You simply A) inherit from the Filter class and implement the new isMatch method; B) add a new sentence to FilterFactory->createFilter to create the filter correctly, and C) add Json describing the filter to the database.

Everywhere you can use the same logic to filter results:

 $filter=FilterFactory::createFilter($jsonFromDatabase); $filteredResults = $filter->filter($resultsFromApi); 

Answer 2

Depending on the type of filter you want to use, you may need to store the codes in the database, and I don't think there is a way around it, since you have no filter restrictions. However, you can try the following:

1 / Function names: if there is only a small number of filters, you can write functions for them and save the filter name in the database.

2 / Regular expression: RegExp is enough to execute the main substring, replace and filter. You can even have multiple RegExp for each URL for complex filters.

Answer 3

How to save the filter class name, instantiate the instance, and then run the data through the filter this way?

So, you have filters configured as:

 abstract class FilterAbs { abstract public function filter($to_filter); } class TrimFilter extends FilterAbs { public function filter($to_filter) { return trim($to_filter); } } 

Then your filter column will have a list of filters (for example, "TrimFilter, EmailFilter"). You must create instances and filter like this:

 $to_filter = "blah"; $filter_col = "TrimFilter, EmailFilter"; $filters = explode(",", $filter_col); foreach($filters as $filter_name) { $filter_name = trim($filter_name); $filter_obj = new $filter_name(); if( !is_a($filter_obj, "FilterAbs") ) continue; $to_filter = $filter_obj->filter($to_filter); } 

If you need to have parameters, you can save them as an encoded json array and use ReflectionClass to instantiate using params (cf create an instance of a class from a variable in PHP? ).

Answer 4

The APIs you store already return data in a structured format. The most common data formats here will be JSON or XML.

Using a regular expression as a filter will work, however, if the data is already structured in a certain way, there may be certain ways to deal with it (query languages ​​that were created for this particular format).

For example, if the API returns XML, you can use XPath and XQuery as filters. matches() as well as replace() (some examples can be found here ). Some other good sources of XPath information are these MDN Articles . PHP also comes with XPath support through the built-in DOMXPath or SimpleXMLElement :: XPath class .

If your API returns JSON, there are some filtering methods:

• JSON-Path (the documentation mentioned that this library implements RFC 6901. ). Here you can find an implementation for PHP .
• JsonPath (implementation of this specification )
• jq (very easy to use and convenient utility for filtering / processing JSON, another good option for the syntax of your filters)
• json: select . The PHP implementation is available here .
• jsonpatch.js is an implementation of both RFC 6901 and RFC 6902. There is also a PHP implementation of this here .

Sidenote: RFC 6902 adds add/remove/replace/move/copy/test operations to the aforementioned RFC 6901.

These can be viable options when viewed as languages ​​that describe filters (which you want to save in your database).

Answer 5

// First, it should be clear to you that you must specify a code to filter your data as desired. Since you use several APIs, and their format is not common (as I saw in the previous comments and question), you can create a class (Filter) that has filtering functions.

 // First create a formatted container of your API data from database in below format $filterContainer[0]['URL'] = 'http://www.api.com/get_data'; $filterContainer[0]['Filter'] = 'ALPHA_NUMERIC_ONLY'; $filterContainer[1]['URL'] = 'http://www.api.com/get_more_data'; $filterContainer[1]['Filter'] = 'STRING'; 

String ------------------ API URL ------------------- Filter

Line 1 --------------- api.com/get_data ------------- ALPHA_NUMERIC_ONLY

Line 2 --------------- api.com/get_more_data -------- STRING

 /** * You will need to use these constant values to store in database in filters. * * Class APIFilter */ class APIFilter { const ALPHA_NUMERIC = 'ALPHA_Numeric'; const STRING = 'STRING'; /** * @param string $filterType Filter Constant value * @param string $data Raw data * * @return string|bool */ static public function GetFilteredData($filterType, $data) { if (empty($filterType) || empty($data)) { return false; } switch ($filterType) { case self::ALPHA_NUMERIC: // Call Alpha numeric Filter functions and return filtered data break; case self::STRING: // Call string filter functions and return filtered data break; default: break; } return false; } /** ******** Write related filter functions here ******** * */ }