I had a similar problem, and the solution lied on the IIS application pools.
My advice to you is to create a new app pool based on the default one and change settings according to your needs, my suggestion would be to read a bit about app pools and configuring it then set your own customized app pool for your web site, taking into account server memory and other factors.
Also check: I was surprised that when I installed my WinSrv2k3 it was giving foreground processes priority over memory, and a server runs mainly background services, so after I change that and got a proper app pool configured in IIS, my web sites have been running for months and months on end without the slightest problem.