[ANSWERED] Pulling realtime numerical data from a webpage

[Woland]

@DusX is the king of this if I recall correctly

[dbini]

@woland

i assumed Ryan would be the person to ask, but i felt a bit cheeky asking him directly. i guess it can be done with Javascript or Websockets or something that i don't have a clue about.....

[Juriaan]

It depends a bit. If the webpage can be queried using JSON it is quite simple to do inside Isadora, if this isn't the case the question is or we can ask for the entire webpage using an TCP call and scrap the contents of the returned query to look for the content that we need. (Normally we search for the DIV since most of the time they are unique enough) and then get the elements behind that. 

Tl;dr; If it isn't JSON, possible but requires some thoughts / some knowledge regarding parsing strings using JavaScript and knowledge about the TCP actors of Izzy

[Juriaan]

https://api.safecast.org/

They have an JSON API

[DusX]

@juriaan

Just what I was going to post.

@dbini you can use the 'Get URL Text' actor to grab this data, and then process the data in a Javascript actor (this will allow you to grab the bits you want).

This URL get you the recent measurements: https://api.safecast.org/measu...

And this tutorial will help you deal with the data quickly: https://support.troikatronix.c...
You will want to ungroup it.

[dbini]

Ryan, Juriaan, thankyou, that's a top quality result. I'm on it. wish me luck.....

cheers,

john

 

[DusX]

Here is a quick showing of the data coming in.

Note I am using 2 versions of the Javascript Split to break out the data.. 
The second is the same as in the tutorial. The other 'upgraded for objects' is new, and includes a loop to deal with data groupings seen as objects in javascript.

[DusX]

Below is the upgraded code to handle the Objects. (note I only use this is required since the loop adds a little extra processing)

function debug(printString){
    if (DebugMode){
        print(printString + "\n");  
    }
}
function main()

{

DebugMode = false; //example usage: debug("string" + var + "\n");
// ensure input is not EMPTY
if (arguments[0].length < 1){
    debug("arg0 length: " + arguments[0].length + "\n");
    return array;
}
var parsed = JSON.parse(arguments[0]);
var array = [];

for (elem in parsed ) {
    if (typeof parsed[elem] == 'object') {
        parsed[elem] = JSON.stringify(parsed[elem])
    }
   array .push(parsed[elem]);
}
array.unshift(array.length); // make the first output value 
                            //a count of additional outputs needed
return array;

}

That should all be 1 script.. not sure why the format breaks it up.

[bonemap]

@dusx

I love this thread!

But how is that Trigger Text output being formatted?

Does it need a Text Formatter to get the final output?

Best Wishes

Russell

[Juriaan]

Hi there, 

I just spent some time on creating this little user actor that allows you to specify the area that you want to look and the distance. If you get any results it will return the smallest, biggest, sum, average and amount of sensors in that area.

This is meant as an example file that you can dig apart and see how to add new filters (like for example perhaps you want to get the readings of a few months ago, you only have to add the filter to the URL and you will get that data back from the system)

Let me know if you have any questions

IzzySafeCast.izz

[DusX]

Attached are TXT versions of the 2 SplitOutputs Scripts in a Zip (since dbini had some trouble getting it from the page).

JS_SplitOutputs_scripts.zip

You should be able to just copy past the code in to the JavaScript actor

@Juriaan s example is perfect for showing you how to create URLs for specific data sets, as well as processing that data. 
However, I suspect you will want to use the Split Output scripts first since they allow you to more easily see the data. 
Another option would be to copy the entire data set output from the get URL text actor, and paste it into an editor that give you JSON formatting.
https://jsonformatter.org does a nice job of this. Paste the copied JSON on the left, read the data on the right.

[dbini]

Thanks Ryan and Juriaan, I've studied your actors, but I'm afraid the internal workings of the Javascript actor still remain a complete mystery to me. 

As far as I can tell, the data from each of the radiation monitors consists of something like this:

  {
    "id": 78490936,
    "user_id": 759,
    "value": 18,
    "unit": "cpm",
    "location_name": null,
    "device_id": 100012,
    "original_id": 78490936,
    "measurement_import_id": null,
    "captured_at": "2017-01-22T07:23:26.000Z",
    "height": null,
    "devicetype_id": null,
    "sensor_id": null,
    "station_id": null,
    "channel_id": null,
    "latitude": 37.659,
    "longitude": 140.459
  },

some of them seem to be measuring different things or working with different units, so its not sensible to sample all of the data on the page at the same time.

I think that what would be useful for this project would be to have an actor where i could trigger a specific ¨id¨ to an input and receive the associated current  ¨value¨ - maybe with the option to sample a number of devices simultaneously (like 8 inputs and 8 outputs)

would one of you be able to make me an actor (or TXT) that does that? i would be eternally grateful.....

(i tried triggering specific longditude/latitude numbers into Juriaan's actor but it was outputting something that didn't correlate, and i don't know why)