SocialFeeds: An Overview

SocialFeeds was a data aggregation platform for routing posts from Reddit, RSS, Twitter, Twitch, StatusPage, YouTube and Roblox. The platform was built using a microservice architecture with a focus on scalability and performance as I wanted to ensure that is was built to scale from the start so I didn’t have to worry about a million and one re-writes later down the line. Though I do admit, I did have to perform at least one re-write anyway, so I’m not sure how much that helped - a lesson learned for the future.

What was the goal?

The goal was to allow users to subscribe to a feed from any of the supported providers and have the posts from that feed sent to a Discord channel of their choice. I argue that it’s a relatively simple concept but it was a lot of fun to build and I learnt a lot along the way. It also had a relatively large user base with over 7,500 servers using the platform with just under 37,000 unique feeds subscribed to. This meant around it’s peak we were sending around 250,000 updates per day to Discord servers. You can read more on the numbers in the statistics article but I would like to think that’s a pretty impressive number, especially for a side project.

Was it a success?

I think as a whole it was, you can’t really argue with the numbers. Sure, they aren’t on the same levels of Google, but if you’re aiming to be the next Google you’re probably going to be disappointed. For a side project, worked on purely in our own time (co-founder’s and I) it really was super motivating. I also had the opportunity to learn a great deal about real-world microservices which was ultimately my personal aim for the project.

Financially? Not so much. We had a few ideas on how to monetise the platform but we never really fully implemented any of them. We were spending maybe £30/month on a server so there wasn’t a huge push in all honesty. I think we were both happy with the platform being a side project and didn’t want to ruin it by trying to monetise it too heavily. I think we both knew that if we tried to monetise it, it would become a chore and we would lose interest in it. The reasoning being that, fundamentally, as soon as you start charging for something you have to provide a certain level of support and service. We didn’t want to have to deal with that, we just wanted to build something cool and see where it went.

We did eventually try to monetise by adding a premium subscription out of necessity due to the sudden change to monetised APIs. More on this is outlined in the end. As I’m sure you can guess from the article name, this didn’t go too well and we ended up shutting down the platform.

How did it work?

This was achieved by the user creating a subscription on the SocialFeeds website or via the Discord bot and then periodically fetching the data from the feed and sending it to the Discord webhook URL. The user could then see the posts in their Discord channel.

There were many different components to the platform and I mainly worked on the aggregator, router and dispatcher. Out of these cryptic component names, the aggregator was the main component that read a list of feeds from a database and spawned processes to fetch the feed data. This data could have been a new tweet, a new post on Reddit or a new video on YouTube etc. The aggregator would then send this data to the dispatcher which, using the data passed would build the Discord messages for each subscription and send them to their respective Discord webhooks. Feel to check out the technical bits for a better understanding of how the platform worked.

Still writing