Spotify hosted this week’s Elasticsearch Meetup in Stockholm featuring a talk by Senior DevOps Engineer Shay Weiss who presented about Spotify’s experience so far with enterprise search engine Elasticsearch.
The use of Elasticsearch at Spotify started as a Hack Day project, extremely easy to set up but Weiss nevertheless advises that inherited infrastructure needs to be brought up to a maintainable level in order to allow any team to feel comfortable using Elasticsearch for event driven data. Once in use developers at Spotify were building dashboards for themselves because it was so easy to use.
Weiss runs through and demonstrates some of the features and benefits that they have been enjoying at Spotify including drill down (go to 4:58) which allows you to see the specific fields you need through log messages followed by Visibility (go to 10:00) which means knowing what your systems are doing, finding bottlenecks, capacity planning, correlating changes across different integration points and creating dashboards that aggregate data.
The talk also shows one of the customised dashboards that these guys use with build statuses, operating systems, projects, repositories and so on with pie chart visualisations which expand into full screen displays of the data required (go to 15:07) and also includes the points about ELK which could improved upon.
Points at the end of the talk look at general pointers from what Spotify have learned including the pain of migrating any schema and other smaller issues.
