How to improve or avoid the find / fetch loop in publishing meteors?

Question

TL; DR:

Chat is one collection. ChatMess is another that has messages that link to Chat _id. How to get the latest messages from the chat list with fewer calculations? Here the find / fetch loop in the loop is too heavy and long.

I have this publication, which is used to return the cursor set to the user:

• Chats in which he takes part (from the chat collection)
• The last message from each chat session referenced by the first cursor (from the ChatMess collection)

Currently, the logic is as follows:

• Get a list of chat sessions from a user profile
• Find chat sessions and pass it through
• In a loop, I find the last message from this chat session and store its _id in an array. In addition, I save all other _ID users.
• Then I find messages that match the idid in my array.

Here is my main problem:

Is there a way to speed up getting the latest messages from each chat session? Using this algorithm, I easily reach 8000 ms response time, which is too complicated calculation time, since most of this time is spent searching / receiving chat messages _id (cf linked screen from Kadira).

Meteor.publish("publishNewChat", function() { this.unblock(); // we get a list of chat _id let chatIdList = _get_all_the_user_chats_ids(this.userId); if (!chatList) return ; // get the chat sessions objects let chats_cursor = Modules.both.queryGet({ type : 'chat', method : 'find', query : { _id: { $in: chatIdList } }, projection : { sort: { _id: 1 }, limit : 1000 } }); let array_of_fetched_chats = chats_cursor.fetch(); let chat_ids = []; // and here we loop through the chat documents in order to get the last message that been attached to each of them array_of_fetched_chats.forEach(function(e) { let lastMess = Modules.both.queryGet({ type : 'chatMess', method : 'findOne', query : { chatId: e._id }, projection : { sort: { date: -1 } } }); if (lastMess) chat_ids.push(lastMess._id); }); return ([ chats_cursor, Modules.both.queryGet({ type : 'chatMess', method : 'find', query : { _id: { $in: chat_ids } }, projection : { sort: { date: -1 }, limit: 1000 } }) ]); }); 

Finally, it also adds latency to all of my next DDP request. I am currently using this.unblock () to avoid this, but I would rather not use it here.

FYI, I have another publication that is updated every time the client changes the current active chat session: on the client, routing to a new chat adds its _id to the reactive array, which updates my getChatMess subscription to get messages from each chat, which the user visited in this since he connected. The goal, obviously, is to save on the server sending every message from every chat session that the user has visited in his life.

Unfortunately, I don't have enough ideas to improve this algorithm without breaking all my chat logic: S. Do you have any ideas? How are you?

Thanks.

EDIT: here is a screen from kadira that clearly shows the problem:

Answer 1

Here is the solution I developed:

 Meteor.publish("publishNewChat", function() { this.unblock(); let user = Modules.both.queryGet({ type : 'users', method : 'findOne', query : { _id: this.userId }, projection : { fields: { "profile.chat": true } } }); let thisUserschats = tryReach(user, "profile", "chat").value; if (!thisUserschats) return ; thisUserschats = thisUserschats.map(function(e) { return (e.chatId); }); let chats = Modules.both.queryGet({ type : 'chat', method : 'find', query : { _id: { $in: thisUserschats } }, projection : { sort : { _id: 1 }, limit : 1000 } }); let chatArray = chats.fetch(), uids = cmid = []; let messages_id_list = [], i = chatArray.length; let _parallelQuery = index => { Meteor.setTimeout(function () { let tmp = Modules.both.queryGet({ type : 'chatMess', method : 'find', query : { chatId: chatArray[index]._id }, projection: { limit: 1, sort: { date: -1 } } }); tmp.forEach(doc => { messages_id_list.push((doc && doc._id) ? doc._id : null); }); }, 1); } while (--i >= 0) _parallelQuery(i); let cursors = { chats : chats, chatMessages : null } let interval = Meteor.setInterval(function () { if (messages_id_list.length === chatArray.length) { Meteor.clearInterval(interval); cursors.chatMessages = Modules.both.queryGet({ type : 'chatMess', method : 'find', query : { _id: { $in: messages_id_list } }, projection : { sort: { date: -1 }, limit: 1000 } }); cursors.chats.observeChanges({ // ... }); cursors.chatMessages.observeChanges({ // ... }); self.ready(); self.onStop(() => subHandle.stop(); ); } }, 10); 

});

I used the async function with Meteor.setTimeout to parallelize queries and save an index that references _id chat for search. Then, when the request is complete, I add the last message to the array. With Meteor.setInterval, I check the length of the array to find out when all requests are complete. Then, since I can no longer return cursors, I use the low-level publishing API to publish documents.

FYI: on the first try, I used 'findOne' in my _parallelQueries, which divided my calculation time by 2/3. But then, thanks to a friend, I tried the cursor.foreach () function, which allowed me to split the calculation time by 2 again!

In production, benchmarks allowed me to switch from a response time of 7/8 seconds to an average response time of 1.6 seconds :)

Hope this will be helpful to you people! :)

Answer 2

Have you considered using reywood / publishComposite package ? With this package, you can publish related data in the same way without performing a bunch of logic to get the correct data published.

In the code below you need to start:

 Meteor.publishComposite("publishNewChat", function() { return [{ find:function(){ return Users.find({ _id: this.userId },{fields:{"profile.chat":1}}); }, children:[{ find:function(user){ //this function is passed each user returned from the cursor above. return UserChats.find({userId:user._id},{fields:{blah:1,blah:1}}); //find the user chats using whatever query }, children:[ //if there are any children of user chats that you need to publish, do so here... { find:function(userchat){ return Chats.find({_id:userchat.chatId}) }, children:[ { find:function(chat){ return ChatMess.find({chatId:chat._id},{ sort: { date: -1 } }); }, children:[ { find:function(chatMess){ var uids = _.without(chatMess.participants, this.userId); return Users.find({_id:{$in:uids}}); } } ] } ] } ] }, ] }] 

This will publish cursors for all documents associated with each of the parent documents. It's pretty fast, I use this package on a high production platform platform and large data sets without any problems. On the client, you can request documents as usual to get the ones you need to display.

Something like:

 Users.findOne({_id:Meteor.userId()}); UserChats.find({userId:Meteor.userId()}); etc...