LinkedIn has released Dr. Elephant, an internally developed tool for tuning Hadoop and Spark, under an open source license. “At LinkedIn, we made it compulsory for the developers to use Dr. Elephant as part of their development cycle,” blogged Akshay Rai, a software engineer at LinkedIn. “It is mandatory to get a green signal from Dr. Elephant for a flow to run in production. For any user issues, we first ask for Dr. Elephant’s report. This encourages them to write their jobs optimally and try to make all their jobs appear green in Dr. Elephant. Dr. Elephant has been a part of LinkedIn’s culture for more than a year and has been helping everyone.”
LinkedIn runs approximately 100,000 Hadoop and Spark jobs per day, and it says that Dr. Elephant has made those big data analytics jobs more efficient. Rai added, “Dr. Elephant is very popular at LinkedIn. People love it for its simplicity. Like a family doctor, it is always reachable and solves around 80 percent of the problems through simple diagnosis.”
