Apache processes all stuck in lockf state (via top)

Hi,

I have an Apache server that's accessing source code over NFS (from a separate dedicated NFS file server). Periodically the Apache server stops accepting new requests (effectively bringing down the server), and when I look into it, it's filling up MaxClients. Upon further research, the site is not that busy... the reason the stack filled up was each Apache process is stuck in a "sending Reply" state (all W's via mod_status). Additionally in top, each apache process is in state "lockf", which makes me think it's an NFS issue. I can't find much information about it online, and I've been pulling my hair out this weekend trying to figure it out. Does anyone have any ideas on what might be going on, or how I can further diagnose the problem?

Thanks so much!

From serverfault Harry

• We've just had an issue with similar symptoms here, servers get to MaxClients then the number of apache processes doesn't drop and new clients can't connect.

  It appears the issue was caused by deadlock from the interaction of our web applications and the MaxClients setting on Apache. I was pointed in the right direction by another question here on ServerFault: http://serverfault.com/questions/133561/apache-reaching-maxclients-and-locking-the-server

  edit: Incidentally since we can't fix our application at the moment we've had to settle for adding more web servers and un-capping the MaxClients setting.

  Harry : Does this cause your apache processes to go into a lockf state?
  From James Yale

• If your logs are being written to your nfs server, make sure you specify the Lockfile directive to be on a local disk.

  Harry : Logs are not being written to the NFS server (all logs are written locally, normal configuration).
  From karmawhore