When Brian Chesky and Joe Gebbia pushed the code for the first Airbnb deployment in their San Francisco apartment, they had no users to release to. Fast forward four crowded years which included successful capital funding and Airbnb deploys again except this time, millions of users would be swallowing a complete overhaul of the interface. Since 2008 it has changed it’s appetite for data tools to cope with the demand for the site.
Airbnb has turned into a company which allows it’s employees to get immersed in data using a cluster management project called Mesos which is now a top level Apache project made famous by Twitter. It lets users share resource isolation across multiple applications on a dynamically shared pool of nodes. Airbnb engineers are able to use more than just Hadoop to get the most out of it’s AWS infrastructure allowing engineers to be exploratory with other systems such as Spark in order to run queries up to 100 times faster than Hadoop, all part of streamlining Airbnb into a slicker and more efficient engineering outfit and enable the automation of resource allocation.*
The use of Mesos means that Airbnb have been able to migrate away from the Elastic MapReduce Hadoop service, despite still using AWS and it means that they can manage other frameworks too including Hadoop without having to rely on AWS for tweaks to Elastic Mapreduce. The following Slideshare from Brendan Matthews explains how Mesos provides application level distributed computing and how they now run Chronos, Hadoop and Storm on Mesos.
[slideshare id=24659104&style=border:1px solid #CCC; border-width:1px 1px 0; margin-bottom:5px; max-width: 100%;&sc=no]
Whilst Airbnb are getting on with scalability issues that they are able to control, the explosion in the short term rental market has highlighted the need for regulation. An initiative in SF is looking at restricting short term rentals to areas with commercial zoning and it could impose insurance requirements for people looking to use the service as well as rewarding informants that expose people that do not adhere to new guidelines.
Regardless, last month Airbnb secured an investment of $450 million which has valued the company at $10 billion, an endorsement of the engineers and tools which have been used to look after the data which the business model has generated so far.
