The correct way to remotely monitor / control the server via http in real time

Question

On my client (phone with a browser) I want to look at the statistics of the server, RAM and HDD and collect information from different logs.

I am using ajax poll.

On the client, every 5 seconds (setInterval) I call the PHP file:

• scan folder containing N logs

• read the last line of each magazine

• convert it to json

Problems:

• Open a new connection every 5 seconds.
• Multiple AJAX Calls.
• Request headers (they are also data and consume bandwidth)
• Answer Headers (^)
• Use PHP to read files every 5 seconds. even if nothing has changed.

The final JSON data is less than 5 KB, but I send it every 5 seconds, and headers and a new connection appear every time, so basically every 5 seconds I have to send 5-10 KB to get 5 KB, which are 10-20 KB .

This is 60 sec / 5 sec = 12 new connections per minute and about 15 MB per hour of traffic if I leave the application open.

Suppose I have 100 users whom I allow to control / control my server, which will be about 1.5 GB of outgoing traffic in one hour.

Not to mention the fact that the PHP server reads several files 100 times every 5 seconds.

I need something on the server that reads the last lines of these logs every 5 seconds and possibly writes them to a file, then I want to send this data to the client only if it has changed.

---

SSE (events sent by server) with PHP

 header('Content-Type: text/event-stream'); header('Cache-Control: no-cache'); while(true){ echo "id: ".time()."\ndata: ".ReadTheLogs()."\n\n"; ob_flush(); flush(); sleep(1); } 

In this case, after establishing a connection with the first user, the connection remains open (PHP has not been created for this), so I save some space (request headers, response headers). This work on my server in most servers does not allow me to stay open for a long time.

Also with several users, I read the log several times (slowing down my old server) And I can not control the server ... I will need to use ajax to send the command ...

I need WebSockets !!!

---

node.js and websockets

using node.js, from what I understand, I can do all this without consuming a lot of resources and bandish. The connection remains open, so there are no unnecessary headers, I can receive and send data.it handles several users very well.

And here I need your help.

• the node.js server should be in the background update and store log data every 5 seconds if the files are modified. OR should do the operating system with (iwatch, dnotify ...)

• data should only be pressed when changing.

• reading logs should occur only once after 5 seconds ... therefore, each user does not start.

---

this is the first example i found ..... and modified ..

 var ws=require("nodejs-websocket"); var server=ws.createServer(function(conn){ var data=read(whereToStoreTheLogs); conn.sendText(data)// send the logs data to the user //on first connection. setTimeout(checkLogs,5000); /* here i need to continuosly check if the logs are changed. but if i use setInterval(checkLogs,5000) or setTimeout every user invokes a new timer and so having lots of timers on the server can i do that in background? */ conn.on("text",function(str){ doStuff(str); // various commands to control the server. }) conn.on("close",function(code,reason){ console.log("Connection closed") }) }).listen(8001); var checkLogs=function(){ var data=read(whereToStoreTheLogs); if(data!=oldData){ conn.sendText(data) } setTimeout(checkLogs,5000); } 

the above script will be a notification server, but I also need to find a solution to store somwhere information of these several logs and do it every time something changes in the background.

How would you do to support low-user as well as server resources.

How would you do?

EDIT Btw. Is there a way to stream use this data simultaneously in all clinics?

EDIT

About logs: I also want to be able to scale the time extension between updates ... I mean, if I read the ffmpeg logs, which are updated every second, if possible ... but when the conversion is not active. I need to get basic information about the car every 5 minutes, maybe ... and so on ...

OBJECTIVES: 1. a way to read and store log data (only if the clinics are connected ... [mysql, file, it is possible to save this information inside RAM (using node.js ??)]). 2. An effective way to transfer data to different clients (at the same time). 3. be able to send commands to the server .. (bi-directional) 4. use of web languages ​​(js, php ...), lunix commands (something that is easy to implement on multiple machines) .. if necessary, free software.

best approach:

read the logs based on current activity , into the system memory and stream simultaneously and constantly , with an already open connection to various clients with websites .

I do not know anything that could be faster.

UPDATE

node.js server is up and running using http://einaros.imtqy.com/ws/ webSocketServer as it seems to be the fastest. I wrote the following code with @HeadCode to properly handle the client situation and keep the process as low as possible. checking out various things inside the broadcast cycle. Now pushing and handling the client are at a good point.

 var wss=new (require('ws').Server)({port:8080}), isBusy, logs, clients, i, checkLogs=function(){ if(wss.clients&&(clients=wss.clients.length)){ isBusy||(logs=readLogs()/*,isBusy=true*/); if(logs){ i=0; while(i<clients){ wss.clients[i++].send(logs) } } } }; setInterval(checkLogs,2000); 

But atm I use a really bad way to parse logs .. (nodejs-> httpRequest-> php) .. lol. After some googling, I found out that I can completely transfer linux software output directly to the nodejs application ... I did not check ... but maybe this would be the best way to do this. node.js also has an api file system where icould read logs. linux has its own api file system.

the readLogs () function (may be asynchronous) is still not satisfied.

• nodejs file system?
• linuxSoftware-> nodejs implementation
• linux api file system.

Keep in mind that I need to scan various folders for magazines, and then analyze somehow the output data, and this is every 2 seconds.

ps .: I adde isBusy for server variables if the logReading system is asynchronous.

EDIT

The answer is not complete.

Missing:

• An easy way to read, parse and store logs somewhere (linux api or nodejs api file system, so I store directly in system memory)
• An explanation of whether it is possible to transfer data directly to multiple users. apparently nodejs loops through clients and therefore (i think) sending data several times.

btw is possible / worth closing the node server if there are no clients and restarting new connections on the apache side. (for example: if I connect to the html apache file and the script starts the nodejs server again). will this further reduce memory leak? correctly?

EDIT

After some experimenting with websockets (some videos in the comments), I learned some new things. Raspberry PI has the ability to use some of the processor's DMA channels for high-frequency broadcasting, such as PWM ... I need to somehow understand how this works.

When using sensors and similar things, I have to store everything in RAM, is nodejs already doing this? (in the variable inside the script)

websocket remains the best choice as it is basically easily accessible from any device now, just using a browser.

Answer 1

I have not used nodejs-websocket, but it looks like it will accept an HTTP connection and perform an upgrade, as well as create a server. If all you need is text / json, then I suppose everything will be fine, but it seems to me that you can use the web page with it.

Here is a way to use express and socket.io to achieve what you are asking:

 var express = require('express'); var app = express(); var http = require('http').Server(app); var io = require('socket.io')(http); app.use(express.static(__dirname + '/')); app.get('/', function(req, res){ res.sendfile('index.html'); }); io.on('connection', function(socket){ // This is where we should emit the cached values of everything // that has been collected so far so this user doesn't have to // wait for a changed value on the monitored host to see // what is going on. // This code is based on something I wrote for myself so it not // going to do anything for you as is. You'll have to implement // your own caching mechanism. for (var stat in cache) { if (cache.hasOwnProperty(stat)) { socket.emit('data', JSON.stringify(cache[stat])); } } }); http.listen(3000, function(){ console.log('listening on *:3000'); }); (function checkLogs(){ var data=read(whereToStoreTheLogs); if(data!=oldData){ io.emit(data) } setTimeout(checkLogs,5000); })(); 

Of course, the checkLogs function should be highlighted by you. I just cut and pasted it here for context. Calling the emit function of the io object will send a message to all connected users, but the checkLogs function will be launched only once (and then continue to call itself), and not every time someone connects.

On the index.html page, you might have something like this. It should be included in the html page below, immediately before the closing tag.

 <script src="/path/to/socket.io.js"></script> <script> // Set up the websocket for receiving updates from the server var socket = io(); socket.on('data', function(msg){ // Do something with your message here, such as using javascript // to display it in an appropriate spot on the page. document.getElementById("content").innerHTML = msg; }); </script> 

By the way, check out the Nodejs documentation for various built-in methods for checking system resources ( https://nodejs.org/api/os.html ).

There is also a solution that is more consistent with what you think. Use this for your html page:

 <!DOCTYPE HTML> <html> <head> <meta charset="utf-8"> <title>WS example</title> </head> <body> <script> var connection; window.addEventListener("load", function () { connection = new WebSocket("ws://"+window.location.hostname+":8001") connection.onopen = function () { console.log("Connection opened") } connection.onclose = function () { console.log("Connection closed") } connection.onerror = function () { console.error("Connection error") } connection.onmessage = function (event) { var div = document.createElement("div") div.textContent = event.data document.body.appendChild(div) } }); </script> </body> </html> 

And use this as the web socket server code recently configured to use the tail module (as shown in this post: How to make `tail -f logfile.txt` -like processing in node.js? ) That you will need to install using npm (Note: tail uses fs.watch, which is not guaranteed to work the same everywhere):

 var ws = require("nodejs-websocket") var os = require('os'); Tail = require('tail').Tail; tail = new Tail('./testlog.txt'); var server = ws.createServer(function (conn) { conn.on("text", function (str) { console.log("Received " + str); }); conn.on("close", function (code, reason) { console.log("Connection closed"); }); }).listen(8001); setInterval(function(){ checkLoad(); }, 5000); function broadcast(mesg) { server.connections.forEach(function (conn) { conn.sendText(mesg) }) } var load = ''; function checkLoad(){ var new_load = os.loadavg().toString(); if (new_load === 'load'){ return; } load = new_load; broadcast(load); } tail.on("line", function(data) { broadcast(data); }); 

Obviously, this is very simple, and you will have to change it for your needs.

Answer 2

I recently made a similar implementation using Munin . Munin is a great open source server monitoring tool that also provides a REST API . There are several plugins available for your needs that track your computer’s use of your hard drive, hard drive, and RAM.

Answer 3

You need to create a push notification server. All clients who are listening will then receive a push notification when new data is updated. See this answer for more information: PHP - Push Notifications

As for how you will update the data, I would suggest using OS-based tools to run a PHP script (command line) that will generate a “push” json file for any client currently listening. Any new client connecting to the “listen” will receive the current json available until it is updated.

Thus, you will not be exposed to 100 users using 100 connections and how many bandwidths to poll your server every 5 seconds, and only update when they need to know that there is an update.

Answer 4

How about a service that reads all the log information (via IPMI , Nagios, or something else) and creates output files on a specific schedule. Then anyone who wants to connect can simply read this output, rather than clog server logs. Essentially one hit on the server logs, then everyone else just reads the web page.

This can be implemented quite easily.

BTW: Nagios has a nice free version

Answer 5

Answering these bits of your question:

2. efficient way to transfer data to different clients (at the same time).
3. be able to send commands to the server .. (bidirectional)
4. using web languages ​​(js, php ...), lunix commands (something that is easy to implement on multiple machines) .. if necessary, free software.

I will recommend the Bayeux protocol made by a simple CometD project. There are implementations in different languages, and it is very easy to use in its simplest form.

Meteor is generally similar. This is an application development environment, not a family of libraries, but it solves the same problems.

Answer 6

Some suggestions:

• Munin for charts
• NetSNMP (used by Munin, but you can also use Bash and Cron to create traps that send SMS text messages to warnings)
• Pingdom for remote alerts about how well the server responds to ping and HTTP checks. He can send SMS text messages or make phone calls, as well as call escalation rules.