New project: Platform Health Viewer
Platform Health Viewer is my current Ruby on Rails pet project.
Once stable, it will allow users to easily collect and visualize different types of statistical data which is typically produced by internet platforms, like CPU performance, user logins, HTTP requests etc.
The main application is build on Rails, the server used for data collection is written in node.js, the web interface makes heavy use of jQuery and uses Raphaël to create SVG graphs. Mass data is saved in a SQL db, other data is stored using CouchDB.
The project’s code is hosted on Github at https://github.com/manuelkiessling/PlatformHealthViewer.
This video is a short introduction to the current alpha version of the project. A funny voice and lots of grammatical shortcomings are included for free:
Transcription of the video:
Hi. Platform Health Viewer – or PHV – is my current pet project.I need an easy and lightweight way to collect and visualize the different key performance indicators of the web platform I’m responsible for – stuff like CPU performance of important systems, user logins, http requests.
So I started to play around with Ruby on Rails, jQuery, CouchDB and node.js, and here is an early alpha I would like to demonstrate.
My primary goal was to make the process from feeding data into the system to visualizing that data as simple as possible.
In order to get data into the system, all you need to make is an HTTP call, which makes it very easy to collect data from very different sources.
Let’s try an example. I would like to visualize the cpu usage of my local machine.
I’m going to collect this data using a standard unix command, sar.
That’s an important aspect of my approach: It doesn’t play any role for the Platform Health Viewer where the data comes from, you’re completely free to choose how to collect it. This way you can feed really anything into the system, from generic data like CPU load to highly individual stuff like the user logins of your specific web site.
Ok, here is how I can get my cpu “usr” value on my OS X command line:
sar 1 1| grep Average| cut -b 14-15
Great, that will do.
How do we push these values into the system? It’s a simple http post request using curl:
curl –data “event[value]=
sar 1 1| grep Average| cut -b 14-15
&event[source]=macbook&event[name]=cpu_usr_percentage” http://localhost:3000/queue_eventAs you can see, the payload of the post requests is just 3 parameters: the source of the event, the name of the event, and its value.
Again, you’re completely free here, you don’t need to configure event names and sources inside PHV – just define them when pushing data into the system, it will happily accept it. We will see in a moment how to make sense of different events that were pushed into the system.
Ok, let’s use a small helper script I wrote in order to feed the CPU sys, idle, usr and nice values into my system:
cat script/agents/macosx/cpu_overview_percent.sh
As you can see, this is all done using only standard unix commands.
Let’s start the script:
bash ./script/agents/macosx/cpu_overview_percent.sh http://localhost:3000/ macbook
I’m just providing two parameters here, the URL to my platform health viewer installation, which resides on the same host for this demo, and the source name, which I call “macbook”.
As you can see, my script pushes all four CPU usage values into the system. Now let’s have a look at this data within platform health viewer.
Well, the Dashboard is still empty, because we did not yet define any visualizations. But the “Tageditor” doesn’t show any events, too. This is because the events I pushed into the system have not yet been normalized to event-types.
This is an additional step, because it will allow the system to push incoming events into the database as quickly as possible without the need to normalize those events regarding their name and source. This normalization is done using a rake task:
rake queue:convert
This task reads the events from the incoming queue, creates new event-types as needed, or connects the event values with existing event types if they already exists. It then deletes the incoming queue.
Getting back to our tageditor, we can now see our 4 event types.
An event type is the combination of an event source and an event name, so “macbook – cpu_idle_percentage” is one event type.
Let’s see how we can use the tag editor to create something useful. Grouping one or more event types into a tag is what makes our data suitable for visualization. I’m not quite happy with the term “tag” by the way, maybe I will come up with something better.
Anyway, let’s create a very simple tag which we can use to visualize exactly one value.
I’m going to name my tag “macbook_cpu_usr”. It will hold all events whose source matches “macbook”, and whose name matches “cpu_usr_percentage”. I could type those parameters into the text box, but it’s easier to just drag’n’drop them there.
Ok, let’s add this tag.
Now we have this first tag, and to check if it works as expected, I can preview the values of the matching events.
Let’s push some new values into our system and check if they are visible here.
Ok, I’m starting my helper skript again in order to post new values to the server, and I start my rake task in order to normalize these values.
Clicking again on “Show latest events” now shows these values.
I will now start data push and normalization in a loop in order to get a lot of values.
Ok, we still have no data visualization, so let’s do this now. Let’s switch to the Dashboard and add a frame, which is a container that will hold our graph.
A frame is the visualization of all values connected to a tag, so I need to provide the name of the tag I want to visualize with this frame.
“Add frame”, and here we go. A simple line graph representing one of my CPU values. The graph is actually an SVG, created using Raphael, an awesome JavaScript library.
And thanks to jQuery, I can freely move and resize the graph.
Let’s create a graph with all my CPU values in it. Back to the Tageditor, I’m going to drag all my values together.
I can also create tags by combining event-sources and -names with already existing tags, as you can see here.
Let’s check the values of my new tag, and there are all the different CPU values my script collects.
Back to the Dashboard, I’m going to create another frame for my new tag. As you can see, this one contains 4 linegraphs and gives me a nice overview of my system’s CPU performance. Of course, a graph legend is needed, something that’s not yet implemented.
Well, that’s it, that’s the current state of this project, I would love to hear your feedback, you can fork the code on github and drop me an e-mail.
Thanks for your interest.