There is a lot of similarities between detective stories (from Sherlock Holmes, James Bond,???) and troubleshooting production problems. Detective stories need to have a very complex/burning problem. If your application is experiencing issues in production, it automatically becomes a burning problem in the enterprise and gets attention from Senior Management. A detective uses very basic clues, extrapolates them, rules out the odd possibilities, puts a lot of hard work and identifies the villain. He fights against all odds, takes risks and eradicates the evil. A lot of heroism is involved. This is no way different from debugging/troubleshooting complex production problems. Thus I am going to introduce a fictional troubleshooting character: ???Jack Che???. Through this fictional character ??? I am going to narrate how complex real world production problems faced by major enterprises are solved. Feel free to share your comments and let me know whether you like it. If not I can always revert back to regular writing style.
While twitter, Google and others are talking about 10 milliseconds, 20 milliseconds response time, still there are significant enterprises whose response time runs for several seconds. There is one such enterprise, whose response time was running for several seconds for their ???search??? transactions. Recently, this enterprise ported their application to AWS Elastic Beanstalk environment in Java 8/Tomcat 8.
When a customer performs ???search??? operation on this application, a progress bar is displayed on the browser. Once search completes, progress bar vanishes and search results are displayed. After porting to AWS Elastic Beanstalk for certain data conditions, the customer was seeing a progress bar on the screen forever. Management didn???t know what was causing this issue and how to go about solving it. Thus they engaged Tier1app LLC to solve the problem. Tier1app LLC sent out their top notch troubleshooting detective ???Jack Che??? to solve the problem.
HTTP 504 Gateway Time-out Error Code
Just like every time, Jack Che was super excited to solve this problem. He assessed the situation quickly. He wanted to understand what interaction was going on between the Server and the browser. Thus he launched the developer console in the chrome browser and triggered the search transaction. A few seconds later, he saw HTTP 504 error code thrown from the server. (HTTP 504 is a time-out error thrown from the backend). Ah, he got his first clue.
Seeing the smoke, where is the fire?
Now Jack Che was curious to figure out from where this HTTP 504 error code is thrown? Jack Che found a second clue now; exactly at 60th second of the search transaction, this HTTP 504 error code was thrown. Since exactly at 60th second, HTTP 504 error code was thrown, Jack Che believed there is some sort of timeout is kicking in. But he wasn???t sure where this timeout value is configured. He searched all throughout the application source code to see whether any 60 seconds timeout is configured. He checked with the application development team. But there was no 60 seconds timeout configured anywhere within the application source code.
Elastic Beanstalk Architecture
Now he came to the conclusion that timeout is triggered by some component that is outside of the source code. Thus he started to examine each layer in the technology stack. Below is a very quick overview of the Elastic Beanstalk architecture.