It’s been a tumultuous month. For us – and unfortunately for you.
We’re sorry for the frustration and the less than great service. While not an excuse, here is the reason:
We changed co-location facilities. We, well the four members of our ops team, moved 1,000 servers and ALL of our real-time search infrastructure. Things are now returning to normal, although slower than we would have liked.
Looking forward:
We haven’t lost any data – so when things are back to normal your link counts and authority will be exactly where they should be.
We’ll follow up in more detail so you have information on exactly which services have been affected and how, and when they’ll be completely functional again.
Our colo move from one end of San Francisco to the other was completed this past Monday. We have successfully moved nearly 900 machines while reducing our overall rack footprint by more than 50%. This major reduction will help us manage our systems better, keep things running better, save a ton of money, and, as an added benefit, consume a lot less power, thus being a wee bit more friendly to the planet.
We had a hard deadline and only 5 weeks to complete the move. We moved quickly and as cautiously as possible. We did have outages and I apologize for any inconvenience that may have caused you, our Technorati users. We still have cleanup, networking to optimize, systems to stabilize, and the routine work of running a real-time blog search engine and ad platform, but things are calming down. This move makes great economic sense for us going forward and the pain of the last few weeks, though unpleasant, was worth enduring since we are now much better positioned for success going forward.
I want to thank the Technorati Operations team for their die hard dedication, positive attitude, and professionalism. I'd also like to thank all of you for sticking with us. We look forward to serving you, the bloggers, readers, and advertisers, for a long, long time.
Yesterday we completed 3 move phases, relocating about 60 machines. In particular we have moved the vast majority of our front end web farm and the back end search and crawling infrastructure. The team has been working non-stop to execute our adjusted plan.
If you have been trying to use the site in the last 24 hours, you have noticed we've had our difficulties. Despite rigorous configuration test scenarios and checks prior to the move, some of our internal network components failed to do what they were supposed to. This meant that, although everything we moved was up and running, requests coming from the internet were not getting everything they needed to complete a meaningful page of HTML.
You may have noticed that your claimed blogs didn't show up any more or that we seem to have forgotten that your favorite blogs exist. Believe, they are all safe and nothing has been lost.
The ops team has rerouted various pieces to start bringing the site online again and full results are now beginning to show up. Unfortunately, you may still see monster pages or slow responses.
We appreciate your patience and I'd like to say thank you to the team for their hard work and ingenuity. They are a small group dedicated to keeping Technorati running smoothly. A move of this magnitude is always a challenge and they are doing everything they can to get it done quickly and return all services to normalcy.
