Monday, August 2, 2021

Grafana - You get the gist

 At my current company we're using a self-managed instance of Grafana to quickly and easily create flexible, good-looking dashboards for reporting on important production metrics. Over the last few years it's become our core monitoring and analysis platform for infrastructure. It's no good at alerting, but for ad-hoc monitoring and quick overviews, we've been very happy with it. 

Recently we've also been working to decommission an old, on-premise server and replace it with some AWS infrastructure. Because a lot of our Grafana dashboards were talking to this server, we had to go through and migrate or remove each one. Because every Grafana dashboard can have multiple panels, and every panel can use different (or even multiple) data sources, we quickly realized there were hundreds of places the server could be referenced that we were going to have to search through. This would be incredibly tedious and error-prone, plus we were going to need to do it several times because the project was going to take a while and people would almost certainly add or remove panels during that transition period. We needed something automated and reproducible.

Now there are probably scenarios where you can search Grafana's data store directly, but due to our security setup, that option was not available. The good news is that Grafana has a rich set of APIs for querying (and modifying) its dashboards (and other objects). So I ended up writing a little project that would search the JSON model of each dashboard in the instance for a specific keyword and write the results to a file. It's a pretty simple .NET Core 3.1 C# project, and you can find it here.

It uses Active Directory credentials to authenticate with the instance, but you could easily modify it to use an API key or some other method.

Hope someone finds that useful!