About Radar

What is Radar?

Welcome to Radar! This is a project that attempts to quantify and analyze the OStatus Fediverse - the collection of OStatus-compatible communities on the internet.

The main objective of Radar is to provide insight into how big the Fediverse is growing, where the users are, what software they use, and in time, what languages and topics are commonly discussed.

What does Radar monitor?

The intention is to eventually monitor all public instances of OStatus-compatible community software. At the moment, all the data being displayed is gathered from Mastodon instances, but more software will be added over time.

For each instance, Radar attempts to identify:

  • The number of registered users
  • The amount of status updates shared
  • The number of domains the instance is federated with

This information is gathered every hour, and all of it is made available through aggregate dashboards, and instance-specific pages.

What is Radar built with?

Radar is a Laravel 5.4 web application, and will eventually use NodeJS for stream subscriptions.

The charts are a combination of d3.js (for more complex visuals) and charts.js (for the chart types it supports out of the box).

Can I have access to the raw data?

Eventually, yes! This is a tricky one, however.

There is no centralized authority that makes the rules for how OStatus instances are meant to be run. This means there is no global policy on how data from those instances are handled. There's always the possibility that some admins may not want their instances tracked, or may not want historical data available to the public.

The plan is to eventually allow instance administrators to "claim" their instances in the Radar database, and give them the option of whether or not to display their instance information in charts, or in JSON API exports.

I don't like my instance being tracked

If you're an instance administrator and you'd like to remove your data from Radar right now, you can message me at https://wogan.im/@wogan from the instance you want removed.

How is the data gathered?

At present there are two means of obtaining data.

1. Scraping - Metrics

A scheduled task runs every hour to visit the "about" pages of all the Mastodon instances in the database, and reads the Users, Statuses and Connection numbers off the page. There's a PR in progress to expose that information via the Instances API instead, which will make this easier.

The next challenge will be scraping GNUSocial/postActiv and StatusNet instances, since they do not neatly provide this information in a standardized way.

2. Spidering - Domains

A substantial portion of Mastodon servers have their public timelines, well, public - it's possible to read them without authentication. Every 15 minutes, Radar pings as many as it can find, and scans the posts to identify new domains that host OStatus-compatible content.

In the future...

Radar will eventually run a realtime indexer. The idea is to have instances opt-in to be monitored, and have Radar deploy a bot that authenticates to the instance and subscribes to activity streams. This data would be used to feed a realtime display of trending topics, as well as identify new OStatus domains. This is still quite a long way off though.

What's on the Radar roadmap?

There's quite a lot I still want to do:

  • Identify fast-growing instances
  • Identify the dominant language by messages posted to OStatus instances
  • Identify trending/dominant topics of conversation
  • Detect federation and defederation between instances, to map out the connectivity of the Fediverse
  • Make as much data as possible available via a REST/JSON API
  • Launch an instance search engine that indexes site descriptions and public posts
  • Deploy a bot to automatically post these stats to a dedicated timeline
  • More stuff I haven't thought of yet

How can I support Radar?

Share it out, with as many people as you can!

Someday I hope to have easily-boostable stats posted to a dedicated Mastodon account to make that a bit easier.

If you have any questions, comments or complaints, I'm on the Fediverse at https://wogan.im/@wogan