What is the maximum bandwidth of Loggly?

Question

How many requests per second from a client can Loggly handle? I can only handle 10-20 requests per second, and I wonder if this is normal.

Answer 1

I just checked a bunch of tests and found that it cannot handle using tcp connection using syslog-ng.

Here are my test results for those who want to try. I used the loggen program for this and sent 200-byte messages to the tcp port assigned to me by loggly. Please note that although syslog RFC (at least 3164) indicates that the log message should not exceed 1024 bytes, I used 200 byte packets just to be fair and because many messages are so small.

Signed up for a free account. Configured TCP connection for testing. Tried to send various amounts, results:

Test 1: FAIL

loggen -iS -r 6000 -s 200 -I 100 logs.loggly.com 16225 Send error Broken pipe, results may be skewed. average rate = 1392.13 msg/sec, count=18296, time=13.142, (average) msg size=200, bandwidth=271.74 kB/sec 

Test 2: FAIL

 loggen -iS -r 4000 -s 200 -I 100 logs.loggly.com 16225 Send error Broken pipe, results may be skewed. average rate = 2767.16 msg/sec, count=121146, time=43.779, (average) msg size=200, bandwidth=540.15 kB/sec 

Test 3: FAIL

 loggen -iS -r 2500 -s 200 -I 100 logs.loggly.com 16225 Send error Broken pipe, results may be skewed. average rate = 1931.27 msg/sec, count=85878, time=44.467, (average) msg size=200, bandwidth=376.98 kB/sec 

Test 4: FAIL

 loggen -iS -r 2000 -s 200 -I 100 logs.loggly.com 16225 Send error Broken pipe, results may be skewed. average rate = 1617.72 msg/sec, count=83134, time=51.389, (average) msg size=200, bandwidth=315.78 kB/sec 

Test 5: FAIL

 loggen -iS -r 1000 -s 200 -I 100 logs.loggly.com 16225 Send error Broken pipe, results may be skewed. average rate = 936.50 msg/sec, count=63331, time=67.624, (average) msg size=200, bandwidth=182.81 kB/sec 

Test 6: PASS for duration, FAIL> 100 seconds - SEE TEST 7

 loggen -iS -r 500 -s 200 -I 100 logs.loggly.com 16225 average rate = 325.00 msg/sec, count=32501, time=100.001, (average) msg size=200, bandwidth=63.44 kB/sec 

Test 7: FAIL - earned a new test @ 500 EPS for a longer period, and the pipe broke after 255 seconds:

 loggen -iS -r 500 -s 200 -I 10000 logs.loggly.com 16225 Send error Broken pipe, results may be skewed. average rate = 323.35 msg/sec, count=82642, time=255.577, (average) msg size=200, bandwidth=63.12 kB/sec 

Test 8: FAIL (lasting more than 200 EPS, but still failed)

 loggen -iS -r 200 -s 200 -I 10000 logs.loggly.com 16225 Send error Broken pipe, results may be skewed. average rate = 163.53 msg/sec, count=234090, time=1431.470, (average) msg size=200, bandwidth=31.92 kB/sec 

Test 9: FAIL (again, lasted longer, but still failed)

 loggen -iS -r 50 -s 200 -I 10000 logs.loggly.com 16225 Send error Broken pipe, results may be skewed. average rate = 47.36 msg/sec, count=89325, time=1886.014, (average) msg size=200, bandwidth=9.25 kB/sec 

Test 10: FAIL? (same results but lost connection again. Is it hard to believe that they cannot handle 10 eps?)

 loggen -iS -r 10 -s 200 -I 10000 logs.loggly.com 16225 Send error Broken pipe, results may be skewed. average rate = 9.94 msg/sec, count=1568, time=157.770, (average) msg size=200, bandwidth=1.94 kB/sec 

Was there some kind of web search to see what loggly can do, but theres only marketing material that says it is scalable and not how scalable it is. I found this: http://twitter.com/jordansissel/status/5948244626509824 This is a total of 22 events per second ...

Full disclosure: I am the founder of LogZilla, so I tested the competition because we are running the syslog cloud solution. My tests show that our software can handle 2,000 to 12,000 events per second, depending on which servers we use in the cloud.

Answer 2

I really don't know, but I was looking for a logging solution for node.js, as well as no luck.

Why?

Since all the ones I checked (did not check everything) use synchronous writing to disk! ...... which is PERFECT, degrades performance.

So, if you ask me - you should review your needs and log only what you really need.

Answer 3

I ran tests similar to those in Clayton's answer , as his results made me worried that Loggly would drop messages if I sent too many at the same time. I wanted to see if there were problems Clayton had in 2012.

So, this is what I found working loggen for 60 seconds, generating 100,000 messages per second.

 $ loggen -iS -r 100000 -s 200 -I 60 logs-01.loggly.com port average rate = 34885.98 msg/sec, count=2093163, time=60.000, (average) msg size=200, bandwidth=6809.74 kB/sec 

I was also curious that some competitors would return for similar tests, and I found the following:

Papertrail

 loggen -iS -D -r 100000 -s 200 -I 60 logs2.papertrailapp.com PORT average rate = 24344.71 msg/sec, count=1461327, time=60.026, (average) msg size=200, bandwidth=4752.09 kB/sec 

Logentries

 $ loggen -iS -D -r 100000 -s 200 -I 60 api.logentries.com PORT average rate = 14076.76 msg/sec, count=844609, time=60.000, (average) msg size=200, bandwidth=2747.78 kB/sec 

Obviously, these are not hard numbers that will always be the same as systems change over time. It just gives us a point in time how they answered when I ran the tests. Your mileage will vary!

Update . I did a longer (almost 3 hour) test with Loggly and got the following:

 loggen -iS -r 100000 -s 200 -I 10000 logs-01.loggly.com port average rate = 15869.22 msg/sec, count=158692177, time=10000.000, (average) msg size=200, bandwidth=3097.67 kB/sec