How to combine the results of a REST call in an Angular application more efficiently

Question

I have an Angular SPA running on a SharePoint 2013 page. In the code, I use $ q to retrieve data from 10 different SharePoint lists using REST, and then merges them into a single JSON object for use in the grid. The code starts and displays the intended combined data, but it leaks, and after a while it crashes from the browser.

Here is the code in the service:

factory.getGridInfo = function() { var deferred = $q.defer(); var list_1a = CRUDFactory.getListItems("ListA", "column1,column2,column3"); var list_1b = CRUDFactory.getListItems("ListB", "column1,column2,column3"); var list_2a = CRUDFactory.getListItems("ListC", "column4"); var list_2b = CRUDFactory.getListItems("ListD", "column4"); var list_3a = CRUDFactory.getListItems("ListE", "column5"); var list_3b = CRUDFactory.getListItems("ListF", "column5"); var list_4a = CRUDFactory.getListItems("ListG", "column6"); var list_4b = CRUDFactory.getListItems("ListH", "column6"); var list_5a = CRUDFactory.getListItems("ListI", "column7"); var list_5b = CRUDFactory.getListItems("ListJ", "column7"); $q.all([list_1a, list_1b, list_2a, list_2b, list_3a, list_3b, list_4a, list_4b, list_5a, list_5b]) .then(function(results){ var results_1a = results[0].data.d.results; var results_1b = results[1].data.d.results; var results_2a = results[2].data.d.results; var results_2b = results[3].data.d.results; var results_3a = results[4].data.d.results; var results_3b = results[5].data.d.results; var results_4a = results[6].data.d.results; var results_4b = results[7].data.d.results; var results_5a = results[8].data.d.results; var results_5b = results[9].data.d.results; var combined_1 = results_1a.concat(results_1b); var combined_2 = results_2a.concat(results_2b); var combined_3 = results_3a.concat(results_3b); var combined_4 = results_4a.concat(results_4b); var combined_5 = results_5a.concat(results_5b); for(var i = 0; i < combined_1.length; i++){ var currObj = combined_1[i]; currObj["column4"] = combined_2[i].column4; currObj["column5"] = combined_3[i].column5; currObj["column6"] = combined_4[i].column6; currObj["column7"] = combined_5[i].column7; factory.newObjectArray[i] = currObj; } deferred.resolve(factory.newObjectArray); }, function (error) { deferred.reject(error); }); return deferred.promise; }; 

Here's the REST call in CRUDFactory:

 factory.getListItems = function (listName, columns){ var webUrl = _spPageContextInfo.webAbsoluteUrl + "/_api/web/lists/getByTitle('"+listName+"')/items?$select="+columns+"&$top=5000"; var options = { headers: { "Accept": "application/json; odata=verbose" }, method: 'GET', url: webUrl }; return $http(options); }; 

And here is the controller bit:

 $scope.refreshGridData = function(){ $scope.hideLoadingGif = false; $scope.GridData = ""; GlobalFactory.getGridInfo() .then(function(){ $scope.GridData = GlobalFactory.newObjectArray; $scope.hideLoadingGif = true; }); }; 

UPDATE 1: For the request here is HTML (just a simple div that we use angular -ui-grid on)

 <div ui-grid="GridOptions" class="grid" ui-grid-selection ui-grid-exporter ui-grid-save-state></div> 

This code begins by declaring some receive calls, and then uses $ q.all to iterate over the calls and get the data. Then it saves the results and combines them up to 5 full arrays. Then, since my list structure is correct and static, I can iterate over one of the joined arrays and extract data from the other arrays into one main array, which I assign factory.newObjectArray, which I declared global in my service and using the mesh as my data source .

The code runs and does not cause any errors, but the problem is related to (I believe) the getGridInfo function. If I do not comment on any REST calls, the browser uses 45 MB of data that the GC will not receive, which are then exacerbated for each click until the session ends or ends. If I comment on all the calls, but one, my page uses only 18.4 MB of memory, which is high, but I can live with it.

So what is the deal? Do I need to destroy something? If so, what and how? Or does this apply to the REST function that I use?

UPDATE 2: The return result that the grid uses (factory.newObjectArray) contains a total of 5450 elements, and each element has about 80 properties after merging. The above code is simplified and shows extruding several columns in a list, but in fact I am extruding 5-10 columns in a list.

Answer 1

At the end of the day, you are dealing with a lot of data, so memory problems will potentially always be a problem, and you should probably consider whether you need to have all the data in memory.

The main goal you should probably be trying to achieve is to limit duplication of arrays and try to keep the memory area as small as possible and free up memory as quickly as possible when you finish processing.

Please pay attention to the following. You indicate that the actual number of columns returned is greater than your example, so I took this into account.

 factory.getGridInfo = function () { var deferred = $q.defer(), // list definitions lists = [ { name: 'ListA', columns: ['column1', 'column2', 'column3'] }, { name: 'ListB', columns: ['column1', 'column2', 'column3'], combineWith: 'ListA' }, { name: 'ListC', columns: ['column4'] }, { name: 'ListD', columns: ['column4'], combineWith: 'ListC' }, { name: 'ListE', columns: ['column5'] }, { name: 'ListF', columns: ['column5'], combineWith: 'ListE' }, { name: 'ListG', columns: ['column6'] }, { name: 'ListH', columns: ['column6'], combineWith: 'ListG' }, { name: 'ListI', columns: ['column7'] }, { name: 'ListJ', columns: ['column7'], combineWith: 'ListI' }, ], // Combines two arrays without creating a new array, mindful of lenth limitations combineArrays = function (a, b) { var len = b.length; for (var i = 0; i < len; i = i + 5000) { a.unshift.apply(a, b.slice(i, i + 5000)); } }; $q.all(lists.map(function (list) { return CRUDFactory.getListItems(list.name, list.columns.join()); })) .then(function (results) { var listResultMap = {}, var baseList = 'ListA'; // map our results to our list names for(var i = 0; i < results.length; i++) { listResultMap[lists[i].name] = results[i].data.d.results; } // loop around our lists for(var i = 0; i < lists.length; i++) { var listName = lists[i].name, combineWith = lists[i].combineWith; if(combineWith) { combineArrays(listResultMap[combineWith], listResultMap[listName]); delete listResultMap[listName]; } } // build result factory.newObjectArray = listResultMap[baseList].map(function(item) { for(var i = 0; i < lists.length; i++) { if(list.name !== baseList) { for(var c = 0; c < lists[i].columns.length; c++) { var columnName = lists[i].columns[c]; item[columnName] = listResultMap[list.name][columnName]; } } } return item; }); // clean up our remaining results for (var i = 0; i < results.length; i++) { delete results[i].data.d.results; delete results[i]; } deferred.resolve(factory.newObjectArray); }, function (error) { deferred.reject(error); }); return deferred.promise; }; 

Answer 2

I would suggest adding some kind of swap option ... It might be a great idea to add all the results to one big list.

Next, I suggest against ng-repeat or add "track by" to the repeat function.

Check out: http://www.alexkras.com/11-tips-to-improve-angularjs-performance/

Squeak your requests, the problem is probably that all the elements in the dom ... Which can be slow (investigate)