node.js - Querying multiple sensors regularly using NodeJS


Question: 

I need to fetch the values of about 200 sensors every 15 seconds or so. To fetch the values I simply need to make an HTTP call with basic authentication and parse the response. The catch is that these sensors might be on slow connection so I need to wait at least 5 seconds for one sensor (but usually they respond a lot quicker, but there's always some that are slow and timeout). So right now I have the following setup for that:

There is a NodeJS process that is connected to my DB and knows all about the sensors. It checks regularly to see if there are new ones or there are some that got deleted. It spawns a child process for every sensor, and in case the child process dies it restarts it. Also it kills it if the sensor gets deleted. The child process makes the HTTP call to its sensor with a 5 second timeout value and if it receives the value, saves it to Redis. Also it is in an infinite loop with a 15 seconds setTimeout. And there is a third process that copies all the values from Redis to the main MySQL DB. So that has been a working solution for half a year, but after a major system upgrade (from Ubuntu 14.04 to 18.04 and thus every package upgraded as well) it seems to leak some memory and I can't seem to figure out where. After starting out, the processes summarised take about 1.5GB of memory. But after a day or so this goes up to 3GB and so on and before running out of memory I need to kill all node processes and restart the whole thing.

So now I am trying to figure out more efficient methods to achieve the same result (query around 2-300 URLs every 15 sec and store the result in MySQL). At the moment I'm thinking of ditching Redis and the child processes will communicate with their master process and the master process will write to MySQL directly. This way I don't need to load the Redis library into every child process and that might save me some time. So I need ideas on how to reduce memory usage for that application (I'm limited to PHP and NodeJS, mainly because of my knowledge, so writing a native daemon might be out of the question)

Thanks!




1 Answer: 

The solution was easier than I thought. I had to rewrite the child process into a native bash script and that brought down the memory usage to almost being zero.

 

More Articles


mysql - OTRS Fatal Error

hey i want to install otrs on an apache webserver with linux ubuntu.i created a new database with the otrs web installer but then I get the message: Can't open file /opt/otrs/Kernel/Config/Files/ZZZAAuto.pm.20881: Permission denied

node.js - Can't run NPM or ng build, or Apache in Ubuntu terminal

This question already has an answer here: env: bash\r: No such file or directory 4 answers Error with gradlew: /usr/bin/env: bash: No such file or directory 3 answe

php - MySql Install on Ubuntu root issue

Setting. I'm running Ubuntu 15.04 on a custom machine with an existing PHP / Apache / Sqlite setup. I recently installed MySql to fiddle around with Laravel a bit. It was working perfectly until I wasn't paying attention and did something incredibly stupid. I intended to export all of the user's


shell - Change the Particualy string value without string comparison in a file using Command line Ubuntu

I want to create a shell script to change the string values which is after '=' in my file Using command line.File is like: String name = "Max"; String age = "24"; String address = "Noida";OrString name=MaxString age=24String address=NoidaBut here, I don't wanna string comparison, Like this:$ sed -i

bash - How does the keyword “if” test if a value is true or false?

In bash scriptif [ 1 ]then echo "Yes"else echo "No"fiOutput: YesIt represents that '1' is treated as true value.But in code:word = Linuxletter = nuxiif echo "$word" | grep -q "$letter"then echo "Yes"else echo "No"fiOutput: NoBut echo "$word" | grep -q "$letter" will return 1, so why is the

Shell script - convert Excel (xlsx) to CSV - remove blank space / tab space

I receive excel file (xslx) with multiple sheets for my project. The number of records on these sheets ranges from 15k to 70k per sheet. I need to perform following tasks on this data and then convert it to CSV. Or covert to CSV and then process the data either way its fine.Input Example:call_no un


mysql - Apache2 / phpmyadmin - PHP isn't working

I was working on a project of mine . then i had to restart the computer.Once I did suddenly PHP stopped working on Apache. then i starting looking into the configuration files and error logs and fixed a spelling error in the config file, but STILL the problem persisted, then i decided to re-install

Why does the following code raise a SegFault. c(Linux)

This a code that would reverse the data of a document and save it in the same document itself.However I am getting a Segmentation Fault.Please Help,I don't know why it gives a SegFault.#include <stdio.h>#include <stdlib.h>#include <termios.h>#include <fcntl.h>#include <str

shell - monitor for file then copy to another directory

I'm relatively new to unix scripting, so apologies for the newbie question.I need to create a script which will permanently run in the background, and monitor for a file to arrive in an FTP landing directory, then copy it to a different directory, and lastly remove the file from the original directo

c - Whats the difference between the two command lines? (SegFault)

I have written a code that maps to a shared memory location,so that the first program opens a shared memory block and stores some data in it.And the second program reads the shared data.Whats the difference between the two command lines:1. if(argc<2) { printf("USAGE:%s text-to-share\n",argv[0