How does StackOverflow optimize performance for displaying questions?

Question

I am trying to learn C # .net for programming a web application.

And after learning that stackoverflow uses C # .net, I am happy to discover it.

I noticed that on the home page or in the questions section when I refresh the page. The page always returns me the latest information without failures and at acceptable speeds.

I'm not sure how you do it. Sorry for the long series of questions. I'm trying to find out what works best for data retrieval, swap, performance, etc.

I know that on the home page only a limited number of questions and their statistics are returned, but the question section actually returns everything.

How do you optimize it?

• For the homepage, do you always capture ALL statistics of recent questions? so is your query something like "select * from questions order by datetime_created limit 20"?

  Thus, * contains ALL information, including the name of the question, identifier, presentation, etc.

  Do you use HttpContext.Current.Server.cache for this?

• For questions, this is even more intriguing.

  How do you pager?

  Do you always retrieve only the results for a particular page from the database?

  Or do you get all the results and save them in a data set? Then do you use some kind of datagrid control to help with the paging call?

If this is the last, how do you maintain the update data?

Answer 1

Here at Stack Overflow, we try to use aggressive caching at many levels:

• pages fully cached by the IIS output cache, regardless of user authentication
• pages cached only for anonymous users; registered users see the latest content
• parts of the html page pages cached for all; HttpRuntime.Cache used for this.

Homepage consists of three cached html elements - last questions, last tags, last icons - each with a different duration.

The question list page will cache the identifiers ( Int32[] ) of all questions for a particular sorting / tag filter, making paging trivial. Additional caching is also performed according to statistics (for example, the number of questions, counting related tags).

Page <detailed description < will be fully cached for anonymous users, and registered users will see the latest products. In addition, related issues on the side are cached to disk for a longer time.

While we are trying to cache entire pages, where possible, we show the user information on the page up - some parts simply cannot be cached.

So, look at caching as a puzzle - what parts can be safely distributed between all my requests? What parts MUST be shared in all of my requests?

Answer 2

SO uses MVC and LINQ2SQL. I would listen to some of the podcasts to get more specific information. I know that they use a lot of caching, but are not sure if this includes a list of questions on the main page.

Answer 3

_{I have no idea how they did it - I did not write SO.}

Something like this, I would use some kind of caching mechanism for the entire Question class with all its answers. The cache will be short-lived, but since new / hot issues are looked up very often, they will survive. Older questions need to be requested from the database. It will also prevent the problem of streaming when one person answers the question and the other considers the question.

Another thing you may notice here is that they make heavy use of AJAX. But since AJAX.Net is extremely useless for bandwidth, they implemented AJAX calls so that they can return simple JSON objects, for example. when returning only a success object with a new number of votes or an error message it is returned, for example: _{(this is a compiled example and is not representative of what is happening because I cannot work to check the right at the moment) sub>}

 {"status": "ok", "votes": 3} 

AJAX.Net will return all the contents of the UpdatePanel , regardless of its size, which, no matter how small, will still be quite large.

Answer 4

You can also take a look at this Qaru Inspired Knowledge Exchange .

This is a group of articles trying to emulate stack overflows. You should be able to find codeaase code here in CodePlex.