We are trying to use Graphite for a (close) real-time web graphics system. However, we cannot accelerate graphite faster than 1 second. Ultimately, we would like to update 100 ms
From reading frequently asked questions, it sounds like graphite quickly - but it is either very misleading, or I don’t understand how to speed up graphite
whisper timing information seems to use UNIX timestamps
How scalable is graphite?
From a processor point of view, graphite scales horizontally on both the interface and the backend, which means that you can simply add more machines to the mix to get more bandwidth. It is also fault tolerant in the sense that loss of a backend machine will result in a minimum amount of data loss (no matter which machine was cached in memory) and will not disrupt the system if you have sufficient capacity to handle the load.
From the point of view of I / O under load, graphite very quickly performs many I / O operations on many different files. This is because each individual metric sent to Graphite is stored in its own database file, similar to how many tools (drraw, Cacti, Centreon, etc.) are built on top of RRD work. In fact, Graphite originally used RRD for storage until fundamental limitations requiring a new storage engine came up.
High volume (several thousand different metrics updated in the smallest detail) pretty much requires a good RAID array. A graphite backend caches incoming data if the disks cannot handle a large number of small write operations (each data point takes only a few bytes, but most disks cannot perform more than a few thousand I / O operations per second, even if they are tiny). When this happens, the Graphite database engine, in a whisper, allows carbon to record multiple data points at once, thereby increasing overall throughput only by storing excess data stored in memory until it is written.
How are real-time graphics?
Highly. Even under heavy load, where the number of metrics arriving at each time interval is much greater than the speed at which your storage system can perform I / O, and many data points are cached in the storage pipeline (see the previous question for an explanation), Graphite is still drawing real-time graphics. The trick is that when Graphite webapp receives a request to draw a graph, it simultaneously retrieves data from disk and also from the pre-memory cache (which can be allocated if you have several server servers), and combines two data sources to create real time graphics.
They also show only seconds and without fractional points: http://graphite.readthedocs.org/en/latest/config-carbon.html as well as from and until must be a time specification conforming to the AT-STYLE time specification described here: http://oss.oetiker.ch/rrdtool/doc/rrdfetch.en.html . http://graphite.wikidot.com/url-api-reference
So what is it? Is graphite fast? or just quickly process large amounts of data - we are looking for an easy-to-use web-based packet data receiver for visual display. Graphite seemed like a great solution, but now that we're all configured and running, I guess we just wasted a lot of time
Thank!