30 July 2011

OMG What Do I Do? - "The database is slow!" - Part III

So, the last time I wrote an epic adventure about my databasing, it was like, three months ago. I've learned a LOT in the last three months.  More vi practicing, how to refresh a database, how to do datapump imports and exports, moving lobs... in fact I am going to move a lob today! Four of them in fact!  I'll post about that later.  What I really wanted to discuss quickly was how the smallest of changes can really F*&^ up how a database coughs up information to users.

So here's the situation. Last week I got an email, then a CALL, from one of my key clients.  "My database is running slow!!"  Oh god.  Again.  But this time, I know a little bit more about databases and slowness, and of all things, I can open Toad AND I have Embarcadero. Two tools!  That makes me like... twice as powerful, right? In fact, I also have something called Precise InDepth!! I am loaded with tools to look at things. So I have all these wonderful tools and I spend literally two days comparing guiful windows stats and numbers.  I just want to say that I was also in the middle of  a 14 day nonstop project so MAYBE I wasn't as thoughtful as I could have been.

I get in late one morning and the senior DBA who sits next to me, who is TOTALLY sweet, mentions to me, "Oh, I opened up topas on your server this morning.  You may want to look at this one process, Hedgehog, that seems to be chewing up some cpu.." then he swivels back around to play some solitaire.

All my tools and he opens up a command line window and runs one unix command and finds my issue.  Yep, turns out that Security is monitoring my database and forgot to turn full debugging off and JUST when my client was running some larger loads, it just HAPPENED to be the straw on the camels back.  Which is why the slowness issue only popped up occasionally... thresholds and all that.

Which is why it takes years to be a senior DBA.

But now I know... before you start using all your fancy tools.  Look at topas.

Like they say, "In the beginning.... was the command line..."

No comments:

Post a Comment