Are there reasons why FTSearch would not be a suitable alternative to DBColumn in Type-Ahead on XPage when trying to improve performance?

Question

I have a general requirement in my current project to make an existing XPage application faster. One thing we were looking at was how to speed up some slower type fields forward, and one solution for this, which seems fast, implements it using FTSearch, and not with the DBColumn that we originally had. I want to get advice on whether this will be an OK approach, or if there are any suggestions on what we need differently.

Background: Although there are a number of factors affecting speed (for example, network latency, server OS, available server memory, etc.), since we use 8.5.3, we optimized the application as much as we can using the IBM Toolkit to find problem areas, and also use the functions that IBM added to help with this in 8.5.3 (for example, partial execution using the optimized JS and CSS option, etc.). Unfortunately, we are stuck with a server running on 32-bit Windows with 3.5Gb Ram for a few more months.

One of the slowest elements to answer is certain types that reference a large number of documents. The worst of them is about 5 or 6 seconds before the proposed list appears for the field with the included type. It uses SSJS to call the java class to make a dbcolumn call (using Ferry Kranenburg XPages Snippet ) to get a unique list from the view, then it loops back into SSJS to check if each record contains the value of the search key, and if it is found , it adds an html (bold) label around the search text in the word, and then returns the formatted list back to the browser. I added a print expression to display the elapsed time needed to run the code, and on average today on our dev server it is about 3250 ms.

I tried several things to see how we can make this process faster:

• Added Java class to perform all processing (therefore, SSJS is not used). This saved an average of 100 ms.

• Using a view-driven Bean, I loaded a unique Lookup list into memory when the page loaded. This gives a very fast forward-type response (16 ms), but I suspect that this is a bad way to do this very much with a large data set - and can really affect the shared server if several users accessed the application. I tried to find information about what would be considered a large object, but could not find any recommendations or recommendations regarding how much is too much to store in memory (I was looking for JSF and XPage sites). Anyone have any suggestions on this?

• Still in the Java class - instead of doing dblookup to get a “list” of all the values ​​to search, I have code that runs FT Search to get the doc collection, then loop through each document to extract the value of the field that I want and add them to "SortedSet" (which does not automatically allow duplication), then loop the sorted set to insert bold tags around the search query and return them to the browser. It takes an average of 100 ms - it's great and barely noticeable. Are there any flaws in this approach - or reasons why I shouldn't do this?

Thanks for any feedback or advice on this. P.

Update Aug 14. 2013: I tried a different approach (inspired by the IBM / Tony McGuckin Insights application on OpenNtf ) as a Search type-ahead company that uses managed beans and quickly transfers a lot of data.

4. Although the Insights application deals with the distribution of data across multiple databases, the principle for forward is the same. I could not use the view with getAllEntriesByKey, although I also needed to look for a string in the text, and not just at the beginning of the record. I tried to create a ViewEntryCollection based on the FTSearch view, but since we have many duplicate names in the column, this did not give the unique list that I wanted. Then I tried using NotesViewNavigator in the categorized view and scrolling through it. This created a unique list that I needed, but it turned out to be slower than any of the other methods listed above. (I implemented these viewNavigator presentation tips.)

Answer 1

From my point of view, performance can affect any of the many levels that each Domino application (and not just XPages) consists of. Above is a browser (DOM, JS, CSS, HTML ...), a network (latency, DNS, SSO ...) to the application level (efficient algorithms, caches), database / API (amount of data, indexes, reader names. ..) and OS / hardware (disks, memory ...)

According to the tests you tested:

• This is interesting, but one would expect: SSJS is cached and can use the lower level API to retrieve data (NAPI).
• For your environment (32-bit / 3.5G RAM - I expect your statement about a 3.5M typo) I DO NOT recommend caching large lists, especially if you use them as a template for many fields / forms / applications. However, the cache in WeakHashMap may be more stable.
• Using FT search is great if you don't need data that is frequently updated. Updating the FT index takes some time and resources to update.

My suggestion: go to FT if it solves your problem. Definitely troubleshoot FT performance in some high performance tests on your server.

Answer 2

(I can not comment because of my low reputation)

I recently dealt with a similar problem. Here are some additional points to consider:

• Are there many duplicate keywords in the view? Consider creating a categorized view for @DbColumn.

• I find that FTS searches are often slower than a database. See article by Andre Guirara . Consider using db.FTSearch() and refine your FT query to enable @Formula, if possible.

• The FT index can be programmatically updated using db.updateFTIndex() . If keywords are rarely added, but they should be instantly available, you can perform an index update in the QuerySave event key document (or similar). We used this approach when keywords were stored in a different (much smaller) database and the update was very quick.

• Memory consumption can be checked as follows:

  • Install the XPages Toolbox from OpenNTF.
  • Open the app.
  • Create a JVM memory dump (session dumps - generate heap heaps).
  • Install Eclipse Memory Analysis Tool
  • Install the IBM Diagnostic Tool Framework in the memory analyzer.
  • Load the memory into MAT. You will see each Java object and their sizes.

In the end, I believe that there is no single general answer to your question. You need to test different approaches to find the fastest solution in your environment.

Answer 3

One of the problems with FT search is this error:

The full text index for this database is used.

Based on my experience, this will happen for a while (maybe a few seconds) when the task of the indexer starts indexing the database. If your users are not very demanding, they can just try again, and this will probably work.

But in many cases, you want to minimize the errors that users receive, and you will have to deal with this error. I created my own FTSearch method, which waits for a bit and FTSearch until an error is received. This will look like slowness for the user, not an error.