development
How I built TweetTop.com in 6 hours with Drupal.
TweetTop was an idea I had when struggling with managing the hoardes of people who I liked to follow with twitter. I use TweetDeck for daily conversation and Hootsuite for posting, but even with groups bundling people together it was getting a bit ridiculous. I wanted a place to be able to just see what a group of people were saying, all in one page, without "committing" to them. (i.e. Without following them)
It also stemmed from the fact that when most people join twitter, they have no idea what to do (or whom to follow).
So, in order to kill two birds with one stone, I decided to create TweetTop one sleepless night. My goal was to more or less be "inspired by Alltop's interface" but have the near real-time feel of Twitter.
I eagerly went to work, wondering how far I could get with Drupal without having to write any code.
I got pretty far.
The first thing I did was do a standard 6.x Drupal installation, with the Zen theme, admin_menu, and Views (something I figure I was going to need regardless).
Once I had that whole setup installed, I played around with the Twitter Module. The Twitter Module essentially allows each Drupal User Account to have one-to-many associated Twitter accounts. Once associated, it can be set to automatically "scrape" the twitter API at every cron run for each of these twitter accounts.
As "messy" as it was, I made each user account a topic. E.g. "Social Media" is actually a Drupal user account that has many twitter accounts associated with it. This was great for a few reasons: i) I could hand off control of any given topic to have it edited and vetted by someone else (by just giving them the topic login and password), and ii) I can make it so anyone can register and create their own "MyTweetTop" page featuring accounts they want to follow. Yet to do, but solid.
So, at this point I had exactly what I needed happening in the backend. Twitter module was scraping the data and now I just needed to pull it from the database and present it in one cohesive page.
I played around for about an hour with Panels and Views trying to figure out how to create "topic pages" without code, but finally I had to drop down to a text editor and write about 30 lines of code.
Effectively, the code just did this:
1) Load up all the twitter accounts for a particular user account (aka "topic") e.g. Get all associated twitter accounts for "Social Media"
2) Order them according to a ratio including followers, following and status updates (to identify and rank the "best").
3) Loop through them, print a custom header for the avatar and user description, and then dynamically call a Drupal View that displays tweets for a particular twitter user account.
4) Bam. Literally, that was it.
For fun I added in a sidebar that auto-updates every 30 seconds or so for all members of the page, so we could see a sort of "fire hose" of everything everyone was saying.
Finally, I tweaked some CSS, turned on CSS and Javascript caching, created 10 or so more topics and LAUNCHED.
We got a great response from everyone in twitter-land. Nevermind the fact that Kevin Roses' WeFollow.com launched the next day, I was still stoked, because WeFollow's first release sucked IMO (anyone with lots of followers could dominate any category, since authors classify themselves, not the other way around). TweetTop.com on the other hand, is vetted and controlled by editors. Quality over quantity.
The Challenge
Continuously, we added more topics and more topics (meaning that more and more twitter user accounts were being followed). Right now we follow some 1000 accounts or so (give or take).
That means, with the current Twitter Module setup, we'd do 1000 API calls every 15 minutes. Twitter only allows 200 an hour, so we were hooped. This thing wasn't going to scale up for us to add more accounts. Twitter started rejecting our API calls.
So, we came up with a great idea. Whenever a user is added to TweetTop.com, that user is also FOLLOWED by our "tweettop" user account (this is done via the Twitter API). Then we simply poll the Twitter API every 30 seconds and say "which one of the peeps we follow on twitter has updates?". We then take those updates, write them with the existing Drupal Twitter Module and we are done. All in all about 20 lines of additional code.
Now we can follow up to 2000 people without any problems. We can likely follow more but we'll need to get the Twitter boys to up our limit.
Drupal seriously rocks. I am constantly amazed by the fact that I can accomplish very complicated things with limited programming knowledge and do 95% of things without touching code.
If you have any questions about how TweetTop was done, or if you need any Drupal help, don't be shy and give me an "@" on twitter. I'm a busy dude, but I'd been more than willing to walk you through it (even send you some code).
