Propeller Launch Week Reflections

Posted by blainegarrett

July 25, 2008

It is 11:30CST on Thursday. I should probably go to bed, but i feel like I need to do SOMETHING. This is my first moment really to breathe in over a week. At work, the fruits of nearly a year and a half of labor came to coalesced this previous Monday at around 1AM CST - the New Propeller.com Site HAS LAUNCHED!

Check out the site, it is pretty cool. For those of you who are don’t know what Propeller is, it’s a “Social News Network” where its members submit links to 3rd party news articles and these stories can be commented on and more. It is an advancement in the news industry (or a bastardization for the same reason, depending who you ask) since there is not just a single/authoritative entity that controls what you read. We get news from many many varied sources - everything from corporate newspapers to bloggers, from mainstream sources to specialty niche news. Like a lot of “social” sites, you can have friends and message users and, now with the new platform you can form groups and also manage blogs (called “reflections”).

Ok, enough for the sales pitch, for the remainder of the entry, I am going to ramble about the past week’s adventures. However, yes, check out propeller, you will like it.

So yes, on a personal front, this last week and a half has been hell. I should be more used to insane hours with launching sites for Adamantine Arts and JR Corps’ clients where I would work non-stop for weeks at a time. However, I might just be getting old because, god damn, I don’t remember getting this frustrated doing web development ever.

Luckily, my special lady friend Toni as well as my brother and my friend Eric were around at various stages to keep me company. I have been bouncing back and forth between home and the coffee shop just to get out of my house and see human beings. Otherwise, I have been sitting on my front porch in a camping chair with my laptop scalding my thighs. There is so much ash on the porch from me chain smoking out of frustration that it looks like small nuclear bomb went off and incinerated everything in its path. I have also collected a nice pile of empty to-go coffee cups, soda bottles, and junk food wrappers (this has not been a healthy week for me). I’m also pretty sure my neighbors think I am an unemployed World of Warcraft addict for as much time as I have been sitting out here lately (yes, I am sitting on the porch in the dark right now). Little do they know…

To get ready for the Tuesday morning launch, we started putting in long hours early last week. As the rest of the team started to scour the site doing QA testing, we were putting in 15 hour days by Thursday finishing up. The loooong days continued through the weekend, pretty much up until today around 8:30PM this evening when I took a break finally to do some yard work and stretch my legs. I looked at some bugs around 10:30, but just couldn’t force myself to work on anything. I guess you could say I am burnt out, but who wouldn’t be after pretty much either staring at the back of my eye lids or staring at a computer screen for the last week and a half.

Friday night I was hitting bugs until around 11pm when I decided to go to the gym (did I tell you I signed up at a 24 hour gym? Well I did). It was pretty awesome since no one was there. I started working again around 10AM Saturday until my brother Matt arrived. He came to town to visit his lady friend who was here in the Twin Cities for the summer. We chatted for a wee bit and he took off to go visit her. Later I went to Toni’s to work for a bit and she ordered pizza. That was short lived as I had to rush home to let my brother in since he lady friend’s parents kicked him out at 11PM (he thought he would be out until 1AM…). I continued to work until around 2AM. Sunday found me working again, but Toni was hanging out which was welcome human interaction. Later in the evening, we went to dinner and I went to go see my friend Josh’s band at the Triple Rock where I ran into a bunch of old friends from when I was in band ages ago. That plus beer, plus some awesome bands made for a great time. Monday, my brother took off in the morning and my friend Eric (who just got back from Japan teaching English) came to visit while he was looking for apartments. I have not seen Eric in ages (ya, know since he was in Japan). He went on his way, and I continued to work. Around, 9PM, my boss asked me if I would be willing to get up in the morning early to be available to handle issues in exchange for going to bed at a reasonable hour. I agreed. So after visiting Toni to kiss her goodnight, I was off to bed. Meanwhile, the rest of the team toiled away with the launch.

And then Tuesday (aka 1st day of launch) came. I woke up at 6ish to find most of the rest of the team still online prepping the launch. They had been up all night getting the site moved over, running migration scripts, and dealing with hosting/database issues. Shortly, after I woke up, things were finally stable enough for the rest of the team to get some shut eye. From about 7:30am to 10am, I was the only line of defense and it wasn’t pretty.Murphy’s law right? In spite of all the QA testing, in spite of all the long hours, in spite of all the planning - whatever could go wrong, did go wrong.

There was a lot of key functionality broken - mostly due to a lot of DB indexes not getting moved over and a lot of slowness with master/slave db replication. This was causing lots of “hey, where did my story/comment/login go?” as the load balancer bounced people around to machines where the slave DB hadn’t updated yet. Most of this was out of our hands as we are required to use corporate hosting that is managed by a central hosting service. The climax of “out of our hands” came when we had to push a critical fix and I had to wake up my co-worker to do so only for him to find that hosting nuked his account rendering us incapable of fixing our own site. Meanwhile, I don’t know as much about the hosting/db side of things, so I am just silently screaming OhGodOhGodOhGod as the error reports and angry feedback come in.

In spite of it being out of our hands, the users don’t understand this or cut us any slack (nor should they really - they are our “customers”). Even so, I still can’t wrap my brain around how emphatic people are about a service that is free for them. I understand it, but I don’t “get” it. I also think I take anonymous feedback like that too personally since I really love the site/community and it is easy to feel like they are personal attacks since I had a hand in its creation. I don’t remember any death threats, but there was lots of talk of how people hated the redesign, people hated that things were broken, people lingered posting threats to leave, etc etc. In the end, I would say a lot of the constructive criticism I read was in regard to missing features such as who voted for comments or friend’s activity - most of which I feel is merited. Even so, a lot of the negative feed back was in regard to things that were out of our hands.

That is not to say that we didn’t make some booboos. Our spam learning engine was auto closing stories that were appearing on the homepage, comment replies were randomly not posting, people signing up with openID were unable to do so, etc. Also, some unfortunate errors in our CMS prevented our team from nixing spam and the front page was cluttered for a bit with porn and viagra spam. I spent a lot of my morning helping nix spam.Yarg.

Finally, a lot of our user base are bloggers and social media consultants - so of course the less-than-awesome launch spread like wild fire in the blogosphere world. Probably the first story to break was Michael Arrington’s Tech Crunch post, which unlike previous scathing articles on Propeller, was actually quote neutral and featured a decent straight forward review of the site (he was given the opportunity to reivew the site in QA which didn’t suffer the load balancing and db issues). After the actual launch, many bloggers put in their two cents. Most of the gripes were about the redesign itself (again, merited), rather than the buggy launch, so I suppose that is good. Heck, they say bad press is better than no press right?

In the aftermath of the launch, my fellow teammates and I have been working at a frantic pace to solve the hiccoups with the site. Most of the critical ones were solved the day of the launch, but there are still minor ones that remain. Our error email list is getting one email per hour from perpetually. Most of the remaining errors are caused by spam bots throwing in junk data. Most of the things on the short term to-do list are enhancements or bugs caused by junk data. The team has been working diligently and beyond the call of duty on a site that we love (yes it also pays the bills, but we are salary… ). I couldn’t ask for a more awesomer team (except maybe the Dream Team). There has been laughs, there have been cries, there have been awesome non-sequiters, and there have been lots of LOLcat references (bugs - pew pew pew)… oh and of course FIXITFIXTITFIXT. So, any of the dev team who may be reading this, you guys rock… and the anchor team, and the scouts and our members. I’m really excited for the future of Propeller and hope to be a bigger part of the community there now that I am not working so hard on a code base that hadn’t, then, seen the light of day. Now that we are on the django platform, I am sure you will see a ton of awesome features popping up and we will close the gap on that #1 social news spot.

Funny bits include someone re-designing our page (read the stories in the mockup - hilarious), someone quoting this Penny Arcade comic, a former contractor sad his work didn’t make it into the final site, a feedback email saying simply “You Suck”, me being frazzled and nearly assaulting a Sierra Club door to door solicitor who was being overly aggressive to get me to donate (there is “no soliciting” at most other people’s places of employment, right?), and someone claming that Bush is in charge of our site and that was why our site sucked. hahahha.

For me, the next step is continuing to work hard until things calm down. I’m sure tomorrow will be a busy day, but I have tickets to see Dark Knight at IMAX that neither hell or high water will prevent me from seeing. This weekend is the Aquatential, which should be off the hook as it is the 150th anniversary of Minneapolis. Saturday around noonish, Jon is having his move out pool party where I hope to soak up some sun and some gin. Festivities continue with Toni’s annual “sausage fest” where they grill and then do a pub crawl in NE Minneapolis. Sunday, my dad is coming down with his crawler to help me grade the area that I will install my awesome Gecko Stone driveway. Somewhere in there, I need to finish up a Surplus Bio-Freaks comic for the DIM Media Coffecrumbs.

After that, I need to get back to finishing up my house since Jon and Julia will be moving in Sept 1. Then my birthday is the 2nd which will hopefully follow a huge joint party with Joe and Guanzon. I think vacation is in order as well around then. The last time I took a vacation day (last November), I went to the doc and they told me I needed surgery. I’m hoping this vacation is much nicer. I also need to get back to working out more now that I have a gym membership and quit that damn smoking. I think a couple weeks of detox is in order too after all the fast food, caffeinated beverages, and nicotine I have consumed these last few weeks. I need to do some summertime stuff like camping since I have done none this summer so far. I also need to hang out more with Toni and friends, all of whom I have been neglecting a lot lately. I should also put more engery into my music, art, and renewable energy experiments. Yarg.

Anywho, it is 2:30 now and this is probably too long winded. I’ll start posting again more, I promise

Lastly, some reviews:

AOL’s Propeller changes look, algorithm - CNET
Propeller 2.0 Launches: Ditching The Vote Count, Adding A Mascot - TechCrunch
Top Story On Propeller Says It All - Pop Fail
How Propeller Should Be - Auto Blog
The New Propeller: Less Digglike, more Yahoo Buzz - Venture Beat
After Re-Launch, not so Digglike - BetaNews
Propeller, FAIL - Social News Watch (note he claims “AOL Decison Makers” “accepted” the story, hmm..)
Revolt: Day 2 - Soshable (for the record the story in question got nixed by over eager spam learning engine, not a TOS Violation… but try to tell that to anyone)
Propeller.com New Layout Sucks!!! - Blogging Tips and Tricks (The title is uncuth , the content is good, the comments are lame - yay for social web)
Alexa Graph data for the last month - Notice the increase in “reach” and “traffic rank”. Hard to argue stats
Same Alexa Compared to Netscape.com - Our old Branding was netscape.com
Alexa Graph of the competition - Newsvine, Reddit, and Mixx (yes, Digg is still #1 by a lot)