I spent this week debugging an issue with an internal web tool that our company’s support team relies upon to pull up information about our customers, making changes on their behalf to their subscriptions of the various packages and services that we offer. Trying to view one particular customer — one! — would always crash the application, leaving an unhelpful message that said, ​“Internal Server Error: Please contact the server administrator. More information about this error may be available in the server error log.” Unfortunately, there was only a single line in the log, ​“Error 500,” which only indicated that yes, an error had happened on the server. No details from the application.

Luckily, this application was written in Perl, an expressive programming language with a rich ecosystem of open-​source libraries. It also has a built-​in debugger mode that can run your program step by step, line by line. Any program that can be run from the text command line can be paused, have its variables and objects examined, new code interactively entered, and then continue its execution as if nothing had happened.

However, this was a web application that assumed it was running in a web server environment, and the customer’s information was in our production database, safe from prying eyes (including curious developers like me) due to financial compliance rules. I could not simply run this program on my desktop and reproduce the problem with this one customer — I had to somehow tease out more information from a running system and report it back using the only tool available: the server error log mentioned above.

But still, the Perl debugger approach was appealing. Could I somehow get the application to log each line of code as it was executed? Could I then see what was running the moment before it crashed, the offending line printed in the log like a smoking gun that had just murdered its victim? And assuming that the problem was in our code and not in the millions of lines of third-​party code it depended upon, could I filter out all the noise?

The answer, thankfully, was yes; since the debugger itself is written in Perl and designed to be extended or replaced, I could add code at the beginning of our application that intercepted each line as it was run, throw out anything that came from a file outside of our application’s directory folder, and then report the rest (along with helpful line numbers) to the error log. Then turn on the ​“debug” switch on the web server running the application, and voilà, the log would dutifully fill up with (slower, more memory-​consuming) code reported by the debugger.

We set up our staging server to use the branch of code with debugging enabled, and then instructed the application to display the problematic customer’s records. As expected, the error log immediately began filling up with line after line of our application’s code and then bang, crashed right after issuing a particular database query for services tied to the account. I had my smoking gun! After extracting the query and running it on a redacted copy of our database, I found that it was returning some 1.9 million rows of data as it retrieved provisioning, billing, and renewal history for every service owned by the customer. This was far too many than necessary — the application only cares about current status, and it was running out of memory as it created service objects for each row.

The database expert on my team was able to adjust the query to return only current information, and after a quick test on the redacted database, the change is now waiting for quality assurance testing before launching to our production servers. The debugging code branch will be saved until it’s needed again, and our team was once again grateful that we were working in such a powerful programming language as Perl.