Publish / subscribe to multiple subsets of the same server collection

Question

EDIT: This question, some answers and some comments contain a lot of misinformation. See how Meteor collections, publications, and subscriptions work for an accurate understanding of publishing and subscribing to multiple subsets of the same server collection.

---

How can I publish different subsets (or “views”) of one collection on the server as multiple collections on the client?

Here is some kind of pseudo code that will help illustrate my question:

items collection on server

Suppose I have a collection of items on a server with millions of records. Let also assume that:

• 50 entries have the enabled property set to true , and;
• 100 records have the processed property set to true .

All others are set to false .

 items: { "_id": "uniqueid1", "title": "item #1", "enabled": false, "processed": false }, { "_id": "uniqueid2", "title": "item #2", "enabled": false, "processed": true }, ... { "_id": "uniqueid458734958", "title": "item #458734958", "enabled": true, "processed": true } 

Server code

Publish two "views" of the same set of servers. One will send a cursor with 50 entries, and the other will send a cursor with 100 entries. This dummy server database contains more than 458 million records, and the client does not need to know about all of these (in fact, sending them all down will probably take several hours in this example):

 var Items = new Meteor.Collection("items"); Meteor.publish("enabled_items", function () { // Only 50 "Items" have enabled set to true return Items.find({enabled: true}); }); Meteor.publish("processed_items", function () { // Only 100 "Items" have processed set to true return Items.find({processed: true}); }); 

Client code

To support the latency compensation method, we are forced to declare a single collection of items on the client. It becomes obvious where this drawback is: how to differentiate between items for enabled_items and items for processed_items ?

 var Items = new Meteor.Collection("items"); Meteor.subscribe("enabled_items", function () { // This will output 50, fine console.log(Items.find().count()); }); Meteor.subscribe("processed_items", function () { // This will also output 50, since we have no choice but to use // the same "Items" collection. console.log(Items.find().count()); }); 

My current solution includes decapitating _publishCursor, allowing you to use a subscription name instead of a collection name. But this will not compensate for latency. Each entry must complete a circuit to the server:

 // On the client: var EnabledItems = new Meteor.Collection("enabled_items"); var ProcessedItems = new Meteor.Collection("processed_items"); 

With a monkey patch, this will work. But go offline, and the changes will not immediately appear on the client - we will need to connect to the server to see the changes.

What is the right approach?

---

EDIT: I just looked at this thread and I understand that since this is so, my questions and answers and a lot of comments contain a lot of misinformation.

What happens is that I misunderstood the publishing-subscribing relationship. I thought that when you publish the cursor, it will land on the client as a separate collection of other published cursors that originated from the same server collection. It just isn't how it works. The idea is that both the client and server have the same collections, but this is what is in the collections that are different. Pub-sub contracts agree on which documents end on the client. Tom's answer is technically correct, but a few details are not enough to turn my assumptions. I answered a similar question to my question in another SO thread based on Tom's explanation, but bearing in mind my initial misunderstanding of Meteor pub-sub: Meteor publishes / signs strategies for unique collections on the client side

I hope this helps those who come across this thread and leave more than confuse something!

Answer 1

Could you just use the same client side of the request if you want to look at the elements?

In the lib directory:

 enabledItems = function() { return Items.find({enabled: true}); } processedItems = function() { return Items.find({processed: true}); } 

On server:

 Meteor.publish('enabled_items', function() { return enabledItems(); }); Meteor.publish('processed_items', function() { return processedItems(); }); 

On the client

 Meteor.subscribe('enabled_items'); Meteor.subscribe('processed_items'); Template.enabledItems.items = function() { return enabledItems(); }; Template.processedItems.items = function() { return processedItems(); }; 

If you think about it, it’s better as if you were inserting (locally) an element that was included and processed, it can appear in both lists (as opposed to two different collections).

Note

I realized that I was obscure, so I expanded it a bit, hope this helps.

Answer 2

you could make two separate publications like this.

Server Publishing

 Meteor.publish("enabled_items", function(){ var self = this; var handle = Items.find({enabled: true}).observe({ added: function(item){ self.set("enabled_items", item._id, item); self.flush(); }, changed: function(item){ self.set("enabled_items", item._id, item); self.flush(); } }); this.onStop(function() { handle.stop(); }); }); Meteor.publish("disabled_items", function(){ var self = this; var handle = Items.find({enabled: false}).observe({ added: function(item){ self.set("disabled_items", item._id, item); self.flush(); }, changed: function(item){ self.set("disabled_items", item._id, item); self.flush(); } }); this.onStop(function() { handle.stop(); }); }); 

Customer Subscriptions

 var EnabledItems = new Meteor.Collection("enabled_items"), DisabledItems = new Meteor.Collection("disabled_items"); Meteor.subscribe("enabled_items"); Meteor.subscribe("disabled_items"); 

Answer 3

I managed to achieve some promising preliminary results by approaching the problem with one publication / subscription to the collection and using $or in the find query.

The idea is to provide a wrapper around Meteor.Collection , which allows you to add “views”, which are mostly called cursors. But what actually happens is that these cursors do not start individually ... their selectors are retrieved, $ or'd together, and run as one request and one pub-sub.

This is not ideal, since the offset / limit will not work with this technique, but minimongo does not currently support it in any way.

But in the end, what it allows you to do is to declare that it looks like different subsets of the same collection, but under the hood they are one and the same subset. There's just a bit of abstraction to make them feel cleanly separated.

Example:

 // Place this code in a file read by both client and server: var Users = new Collection("users"); Users.view("enabledUsers", function (collection) { return collection.find({ enabled: true }, { sort: { name: 1 } }); }); 

Or if you want to pass parameters:

 Users.view("filteredUsers", function (collection) { return collection.find({ enabled: true, name: this.search, { sort: { name: 1 } }); }, function () { return { search: Session.get("searchterms"); }; }); 

The parameters are set as objects, because they are a single publication / subscription $ or'd together, I need a way to get the correct parameters, since they are mixed.

And actually use it in the template:

 Template.main.enabledUsers = function () { return Users.get("enabledUsers"); }; Template.main.filteredUsers = function () { return Users.get("filteredUsers"); }; 

In short, I use the same code as on the server and on the client, and if the server does nothing, the client will be or vice versa.

And most importantly, only those records that interest you are sent to the client. This can be achieved without a layer of abstraction, simply by making $ or by yourself, but this $ or will turn out pretty ugly as more subsets are added. It just helps manage it with minimal code.

I quickly wrote this to check it, apologize for the length and lack of documentation:

test.js

 // Shared (client and server) var Collection = function () { var SimulatedCollection = function () { var collections = {}; return function (name) { var captured = { find: [], findOne: [] }; collections[name] = { find: function () { captured.find.push(([]).slice.call(arguments)); return collections[name]; }, findOne: function () { captured.findOne.push(([]).slice.call(arguments)); return collections[name]; }, captured: function () { return captured; } }; return collections[name]; }; }(); return function (collectionName) { var collection = new Meteor.Collection(collectionName); var views = {}; Meteor.startup(function () { var viewName, view, pubName, viewNames = []; for (viewName in views) { view = views[viewName]; viewNames.push(viewName); } pubName = viewNames.join("__"); if (Meteor.publish) { Meteor.publish(pubName, function (params) { var viewName, view, selectors = [], simulated, captured; for (viewName in views) { view = views[viewName]; // Run the query callback but provide a SimulatedCollection // to capture what is attempted on the collection. Also provide // the parameters we would be passing as the context: if (_.isFunction(view.query)) { simulated = view.query.call(params, SimulatedCollection(collectionName)); } if (simulated) { captured = simulated.captured(); if (captured.find) { selectors.push(captured.find[0][0]); } } } if (selectors.length > 0) { return collection.find({ $or: selectors }); } }); } if (Meteor.subscribe) { Meteor.autosubscribe(function () { var viewName, view, params = {}; for (viewName in views) { view = views[viewName]; params = _.extend(params, view.params.call(this, viewName)); } Meteor.subscribe.call(this, pubName, params); }); } }); collection.view = function (viewName, query, params) { // Store in views object -- we will iterate over it on startup views[viewName] = { collectionName: collectionName, query: query, params: params }; return views[viewName]; }; collection.get = function (viewName, optQuery) { var query = views[viewName].query; var params = views[viewName].params.call(this, viewName); if (_.isFunction(optQuery)) { // Optional alternate query provided, use it instead return optQuery.call(params, collection); } else { if (_.isFunction(query)) { // In most cases, run default query return query.call(params, collection); } } }; return collection; }; }(); var Items = new Collection("items"); if (Meteor.isServer) { // Bootstrap data -- server only Meteor.startup(function () { if (Items.find().count() === 0) { Items.insert({title: "item #01", enabled: true, processed: true}); Items.insert({title: "item #02", enabled: false, processed: false}); Items.insert({title: "item #03", enabled: false, processed: false}); Items.insert({title: "item #04", enabled: false, processed: false}); Items.insert({title: "item #05", enabled: false, processed: true}); Items.insert({title: "item #06", enabled: true, processed: true}); Items.insert({title: "item #07", enabled: false, processed: true}); Items.insert({title: "item #08", enabled: true, processed: false}); Items.insert({title: "item #09", enabled: false, processed: true}); Items.insert({title: "item #10", enabled: true, processed: true}); Items.insert({title: "item #11", enabled: true, processed: true}); Items.insert({title: "item #12", enabled: true, processed: false}); Items.insert({title: "item #13", enabled: false, processed: true}); Items.insert({title: "item #14", enabled: true, processed: true}); Items.insert({title: "item #15", enabled: false, processed: false}); } }); } Items.view("enabledItems", function (collection) { return collection.find({ enabled: true, title: new RegExp(RegExp.escape(this.search1 || ""), "i") }, { sort: { title: 1 } }); }, function () { return { search1: Session.get("search1") }; }); Items.view("processedItems", function (collection) { return collection.find({ processed: true, title: new RegExp(RegExp.escape(this.search2 || ""), "i") }, { sort: { title: 1 } }); }, function () { return { search2: Session.get("search2") }; }); if (Meteor.isClient) { // Client-only templating code Template.main.enabledItems = function () { return Items.get("enabledItems"); }; Template.main.processedItems = function () { return Items.get("processedItems"); }; // Basic search filtering Session.get("search1", ""); Session.get("search2", ""); Template.main.search1 = function () { return Session.get("search1"); }; Template.main.search2 = function () { return Session.get("search2"); }; Template.main.events({ "keyup [name='search1']": function (event, template) { Session.set("search1", $(template.find("[name='search1']")).val()); }, "keyup [name='search2']": function (event, template) { Session.set("search2", $(template.find("[name='search2']")).val()); } }); Template.main.preserve([ "[name='search1']", "[name='search2']" ]); } // Utility, shared across client/server, used for search if (!RegExp.escape) { RegExp.escape = function (text) { return text.replace(/[-[\]{}()*+?.,\\^$|#\s]/g, "\\$&"); }; } 

test.html

 <head> <title>Collection View Test</title> </head> <body> {{> main}} </body> <template name="main"> <h1>Collection View Test</h1> <div style="float: left; border-right: 3px double #000; margin-right: 10px; padding-right: 10px;"> <h2>Enabled Items</h2> <input type="text" name="search1" value="{{search1}}" placeholder="search this column" /> <ul> {{#each enabledItems}} <li>{{title}}</li> {{/each}} </ul> </div> <div style="float: left;"> <h2>Processed Items</h2> <input type="text" name="search2" value="{{search2}}" placeholder="search this column" /> <ul> {{#each processedItems}} <li>{{title}}</li> {{/each}} </ul> </div> </template>