Hello all,
Often throughout the day I see instances where an httpd process will spike up to 100% cpu utilization and stay there for an extended period of time. eg:
19068 apache 1 0 0 71M 49M cpu/1 9:05 48.11% httpd (that's a 2-cpu machine)
This is a Roundcube process. If I run a truss on that pid I can see that it is doing absolutely nothing.
Also lsof tells me:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF
NODE NAME
httpd 19068 apache 39u IPv4 0x30012fd2380 0t0
TCP mail.example.org:63204->mail.example.org:imap (CLOSE_WAIT)
This connection will stay in the CLOSE_WAIT state (and the CPU pegged) indefinitely, or at least until...
...eventually it times out and I may see this in the Apache error log: [Thu Apr 03 11:46:44 2008] [error] [client 209.222.195.177] PHP Fatal error: Maximum execution time of 300 seconds exceeded in /group/www/mail.example.org/htdocs/program/lib/imap.inc on line 169, referer: https://mail.example.org/?_task=mail&_action=show&_uid=114766&_m...
Any ideas what is causing this? It's bringing the server to it's knees fairly often, and contributing to a perception among my users that Roundcube is "slow".
This is Roundcube 0.1 stable running under PHP 5.2.5 / Apache 2.2.8 on Solaris 10. Our IMAP server is UW and lives on the same box.
I'd appreciate any troubleshooting advice. Cheers, Roy
Roy McMorran wrote:
Often throughout the day I see instances where an httpd process will spike up to 100% cpu utilization and stay there for an extended period of time.
After correlating information from a bunch of log files, I think I've found the circumstances that precipitate this problem. It seems to be a case of one or more users running Roundcube *and* another email program at the same time.
UW-IMAP doesn't want to serve the same inbox to both imap clients. One of them gets cut off at the knees. Eg: Apr 3 14:51:10 mail imapd[12962]: [ID 289526 mail.info] Killed (lost mailbox lock) user=redacted host=mail.example.org [xxx.xxx.xxx.xxx]
When the loser is Roundcube we get into the 100% CPU / CLOSE_WAIT scenario I described earlier.
Is there any way we can make Roundcube more resilient in this situation? Or is there an IMAP server that isn't quite so brutal?
Thanks, Roy
On Thu, Apr 3, 2008 at 9:40 PM, Roy McMorran mcmorran@mdibl.org wrote:
Roy McMorran wrote:
Often throughout the day I see instances where an httpd process will spike up to 100% cpu utilization and stay there for an extended period of time.
After correlating information from a bunch of log files, I think I've found the circumstances that precipitate this problem. It seems to be a case of one or more users running Roundcube *and* another email program at the same time.
UW-IMAP doesn't want to serve the same inbox to both imap clients. One of them gets cut off at the knees. Eg: Apr 3 14:51:10 mail imapd[12962]: [ID 289526 mail.info] Killed (lost mailbox lock) user=redacted host=mail.example.org [xxx.xxx.xxx.xxx]
When the loser is Roundcube we get into the 100% CPU / CLOSE_WAIT scenario I described earlier.
Is there any way we can make Roundcube more resilient in this situation? Or is there an IMAP server that isn't quite so brutal?
Wow, thanks for tracking this down! I have never seen this with courier-imap. I often have it open on my Mac, the notebook and in RoundCube. But I could also be lucky.
Till _______________________________________________ List info: http://lists.roundcube.net/dev/
till wrote:
Wow, thanks for tracking this down! I have never seen this with courier-imap. I often have it open on my Mac, the notebook and in RoundCube. But I could also be lucky.
Hi till, thanks for the reply.
Does your courier-imap use mbox format, or something else?
This: http://mailman1.u.washington.edu/pipermail/imap-uw/2007-December/001809.html suggests the behavior is more due to the limitations of mbox rather than a particular imap server.
Nevertheless, mbox is still pretty popular and I don't think I can get away from it in my environment because I have many Unix command-line users (with pine, mutt, etc).
Looking through my apache logs, all the hangs have occurred inside imap.inc, in various functions. Usually iil_Readline or iil_C_Select though.
Best wishes,
-r
Roy McMorran Systems Administrator MDI Biological Laboratory mcmorran@mdibl.org
List info: http://lists.roundcube.net/dev/
On Apr 3, 2008, at 2:40 PM, Roy McMorran wrote:
UW-IMAP doesn't want to serve the same inbox to both imap clients.
One of them gets cut off at the knees. Eg: Apr 3 14:51:10 mail imapd[12962]: [ID 289526 mail.info] Killed (lost mailbox lock) user=redacted host=mail.example.org [xxx.xxx.xxx.xxx]When the loser is Roundcube we get into the 100% CPU / CLOSE_WAIT scenario I described earlier.
Is there any way we can make Roundcube more resilient in this situation? Or is there an IMAP server that isn't quite so brutal?
I switched to dovecot after Fedora dropped uw-imap due to licensing
issues.
I found dovecot to be faster and to use less memory. YMMV.
Policies here pretty much prevent two IMAP logins to the same
account, so I can't say I've run across this problem, or how dovecot
handles it.
Charles Dostale System Admin - Silver Oaks Communications http://www.silveroaks.com/ 824 17th Street, Moline IL 61265
List info: http://lists.roundcube.net/dev/
On Thu, Apr 3, 2008 at 10:47 PM, Roy McMorran mcmorran@mdibl.org wrote:
till wrote:
Wow, thanks for tracking this down! I have never seen this with courier-imap. I often have it open on my Mac, the notebook and in RoundCube. But I could also be lucky.
Hi till, thanks for the reply.
Does your courier-imap use mbox format, or something else?
Maildir.
(...) Nevertheless, mbox is still pretty popular and I don't think I can get away from it in my environment because I have many Unix command-line users (with pine, mutt, etc).
I haven't seen pine and mutt in ages. I can't comment on this though I think an MUA uses IMAP as the interface and doesn't care about Mailbox or Mbox in the backend.
Looking through my apache logs, all the hangs have occurred inside imap.inc, in various functions. Usually iil_Readline or iil_C_Select though.
There are some loops in those functions which expect sequences/characters and if those don't come around we have potential leaks and breaks. I see what you mean, but don't hold your breath for a fix soon. We are looking for a replacement of our IMAP backend lib and will work on this as soon as we released the 0.1.1 (this weekend).
Till _______________________________________________ List info: http://lists.roundcube.net/dev/
using dovecot with Maildir here. never encountered any issues as described in this thread, although when using roundcube there is always another imap client (thunderbird) connected with the same user.
chasd wrote:
On Apr 3, 2008, at 2:40 PM, Roy McMorran wrote:
I switched to dovecot after Fedora dropped uw-imap due to licensing
issues. I found dovecot to be faster and to use less memory. YMMV.Policies here pretty much prevent two IMAP logins to the same
account, so I can't say I've run across this problem, or how dovecot
handles it.
List info: http://lists.roundcube.net/dev/
On Apr 3, 2008, at 4:01 PM, till wrote:
I haven't seen pine and mutt in ages.
We use a desktop graphical MUA ( Thunderbird, Evolution, or Apple's
Mail.app ), RoundCube, and also alpine ( the successor to pine ) for
text logins.
Example of text login use :
A video editor is working in Final Cut Pro on OS X and wants to
maximize the computer resources for that, but still remain attentive
to incoming e-mail. A telnet session to the internal mail host and
running alpine does that without the overhead of a modern browser or
graphical desktop MUA. Instead of 40 to 100 MB, a Terminal session is
less than 10 MB of RAM, and much less CPU overhead. A terminal window
takes up less screen real estate too.
I can't comment on this though I think an MUA uses IMAP as the interface and doesn't care about Mailbox or Mbox in the backend.
A remote MUA doesn't care about the type of mail storage, but
something like alpine that accesses a local mail store does. Although
alpine, pine or mutt can be configured to use either maildir or mbox.
I believe you have to choose one, you can't mix.
<http://en.wikipedia.org/wiki/Comparison_of_e-mail_clients#Database. 2C_folders_and_customization> The first column lists the storage formats supported.
Actually one of the big additions to alpine over pine is support for
remote connections, it doesn't have to access a local mail store.
I'm off in the weeds here now, so I'll go away.
Charles Dostale System Admin - Silver Oaks Communications http://www.silveroaks.com/ 824 17th Street, Moline IL 61265
List info: http://lists.roundcube.net/dev/
47F513B3.4020403@mdibl.org 47F532B0.40805@mdibl.org 234b196e0804031301k4b950b4et4fca2c0a068cc140@mail.gmail.com 47F5425B.8040308@mdibl.org 234b196e0804031401q13c81f4ex9348109ed4343f92@mail.gmail.com Message-ID: 333ab0fb986d0c3549e0bcc30064a1e5@mdibl.org X-Sender: mcmorran@mdibl.org Received: from cpe-72-224-141-4.maine.res.rr.com [72.224.141.4] with HTTP/1.1 (POST); Thu, 03 Apr 2008 19:19:11 -0400 User-Agent: RoundCube Webmail/0.1 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit
On Thu, 3 Apr 2008 23:01:25 +0200, till klimpong@gmail.com wrote:
There are some loops in those functions which expect sequences/characters and if those don't come around we have potential leaks and breaks. I see what you mean, but don't hold your breath for a fix soon. We are looking for a replacement of our IMAP backend lib
For now I'm looking at a workaround using stream_set_timeout on the imap socket. I'll let you know how that goes.
and will work on this as soon as we released the 0.1.1 (this weekend).
Another new release? Awesome!
47F513B3.4020403@mdibl.org 47F532B0.40805@mdibl.org 234b196e0804031301k4b950b4et4fca2c0a068cc140@mail.gmail.com 47F5425B.8040308@mdibl.org 234b196e0804031401q13c81f4ex9348109ed4343f92@mail.gmail.com BB6EDD66-6C65-426C-8890-96E8AEF8D8DC@silveroaks.com Message-ID: 21e70725578ac33e7b142a4201abe90a@mdibl.org X-Sender: mcmorran@mdibl.org Received: from cpe-72-224-141-4.maine.res.rr.com [72.224.141.4] with HTTP/1.1 (POST); Thu, 03 Apr 2008 19:25:59 -0400 User-Agent: RoundCube Webmail/0.1 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit
Charles & Rainer,
Thanks for the info about your configurations. Here, pine and mutt use mbox, but I will look into the possibilities you suggest. I think maildir would definitely clear up this issue.
Cheers,
Roy McMorran wrote:
For now I'm looking at a workaround using stream_set_timeout on the imap socket. I'll let you know how that goes.
I'm using the following as a workaround and it seems to help prevent the deadlock. This isn't tested thoroughly at all, and there are probably unintended consequences that I have not observed yet. Use at your own risk.
--- imap.inc.dist 2008-04-03 17:59:33.108020000 -0400 +++ imap.inc 2008-04-03 18:42:08.009825000 -0400 @@ -467,6 +467,9 @@ $iil_errornum = -1; return false; }
//set socket timeout 10 secs
stream_set_blocking($conn->fp,FALSE);
stream_set_timeout($conn->fp,10);
$iil_error .= "Socket connection established\r\n"; $line = iil_ReadLine($conn->fp, 300);
Looking forward to that new imap lib ;-) Cheers, -r
-
chasd -
Rainer Klaffenboeck -
Roy McMorran -
till