Worst Problem Encountered till date

I am into a support team, and i am much interested in knowing the worst kind of problems that you have encountered till date, the problems/issues which were night mares to you.


Let me start with my exp:

We had encountered a problem like the site was performing slow, we had checked all the possible things but nothing did work. all the applications were working fine like the web servers, app servers, third party tools and also the data base, we didn't find any problems any where.

later on we found that the problem was because of the Apache/IHS. it was not releasing the semaphore which was the result of the problem. We had to restart the web server which was the route cause of the issue.

While debugging we had went thru the following things

Network status (num of connections)
App Svr status and the app status
App log files and App Svr log files
HTTP log files
Plugin Log files
JDBC Conn status
Database status
JVM status and amount of memory being utilised

The problem was solved after a long checking of about 6 Hrs. But those 6 hrs were awesome as we found that everything was working perfectly fine.

Regards,
VarmA