On Fri, 13 Nov 2009 18:27:23 +0100, "A.L.E.C" alec@alec.pl wrote:
Chris January wrote:
I gave the threading a quick test and looks ok to so far.
However with this revision Roundcube runs out of memory viewing a
mailbox
with about 160,000 messages in it (with threading *off*). It works fine in v0.3.
No problem here. What IMAP server. What sorting column? Enable imap_debug and check the log.
IMAP server: dovecot-1.2.4-1.fc11.i586
Sorting column: SORT (DATE) US-ASCII ALL
errors log:
[14-Nov-2009 09:03:17] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 24 bytes) in /home/cjanuary/public_html/devel-threads/program/lib/imap.inc on line 901
imap log:
[14-Nov-2009 09:03:47 +0000]: C: a001 LOGIN "***" "***" [14-Nov-2009 09:03:47 +0000]: S: a001 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH] Logged in [14-Nov-2009 09:03:47 +0000]: C: sel1 SELECT "Gmail" [14-Nov-2009 09:03:48 +0000]: S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft) [14-Nov-2009 09:03:48 +0000]: S: * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft *)] Flags permitted. [14-Nov-2009 09:03:48 +0000]: S: * 163517 EXISTS [14-Nov-2009 09:03:48 +0000]: S: * 0 RECENT [14-Nov-2009 09:03:48 +0000]: S: * OK [UNSEEN 82179] First unseen. [14-Nov-2009 09:03:48 +0000]: S: * OK [UIDVALIDITY 1234311067] UIDs valid [14-Nov-2009 09:03:48 +0000]: S: * OK [UIDNEXT 163559] Predicted next UID [14-Nov-2009 09:03:48 +0000]: S: * OK [HIGHESTMODSEQ 1] [14-Nov-2009 09:03:48 +0000]: S: sel1 OK [READ-WRITE] Select completed. [14-Nov-2009 09:03:48 +0000]: C: s SORT (DATE) US-ASCII ALL [14-Nov-2009 09:03:50 +0000]: S: * SORT 54693 ... ... 1Mb worth of output - ~160,000 messages ... [14-Nov-2009 09:03:51 +0000]: S: s OK Sort completed (1.440 secs).
Chris
Chris January wrote:
IMAP server: dovecot-1.2.4-1.fc11.i586
I'm currently using dovecot-1.1.x
Sorting column: SORT (DATE) US-ASCII ALL
errors log:
[14-Nov-2009 09:03:17] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 24 bytes) in /home/cjanuary/public_html/devel-threads/program/lib/imap.inc on line 901
So, it is in iil_C_Sort()
imap log:
[14-Nov-2009 09:03:48 +0000]: C: s SORT (DATE) US-ASCII ALL [14-Nov-2009 09:03:50 +0000]: S: * SORT 54693 ... ... 1Mb worth of output - ~160,000 messages ... [14-Nov-2009 09:03:51 +0000]: S: s OK Sort completed (1.440 secs).
The log looks ok, but after SORT there should be FETCH, of course there's no FETCH because the error. It's clear. There was no changes in iil_C_Sort() and I have no idea what's wrong.
Please add before line 900: write_log('imap', 'Mem: '.memory_get_usage());
And test with devel-thread and svn-trunk, please.
The log looks ok, but after SORT there should be FETCH, of course there's no FETCH because the error. It's clear. There was no changes in iil_C_Sort() and I have no idea what's wrong.
Please add before line 900: write_log('imap', 'Mem: '.memory_get_usage());
And test with devel-thread and svn-trunk, please.
It appears to fail on x86_64 platforms, but not on i686. I am guessing PHP uses more memory on x86_64 and that pushes it over the limit.
devel-threads (x86_64 php-5.2.9-2.fc11.i586 httpd-2.2.13-1.fc11.i586 32M memory limit)
[16-Nov-2009 09:53:40 +0000]: Mem usage before SORT: 7536816 [16-Nov-2009 09:53:40 +0000]: C: s SORT (DATE) US-ASCII ALL [16-Nov-2009 09:53:41 +0000]: S: * SORT 54693 ... ... [16-Nov-2009 09:53:42 +0000]: Mem usage after SORT: 8572984 [16-Nov-2009 09:53:42 +0000]: Length of $data: 1033513 [16-Nov-2009 09:53:42 +0000]: Calling preg_split
[16-Nov-2009 09:53:42] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 71 bytes) in /home/cjanuary/public_html/devel-threads/program/lib/imap.inc on line 906
trunk (x86_64 php-5.2.9-2.fc11.i586 httpd-2.2.13-1.fc11.i586 32M memory limit)
[16-Nov-2009 10:03:20 +0000]: Mem usage before SORT: 755018 [16-Nov-2009 10:03:20 +0000]: C: s SORT (DATE) US-ASCII ALL [16-Nov-2009 10:03:22 +0000]: S: * SORT 54693 ... ... [16-Nov-2009 10:03:22 +0000]: S: s OK Sort completed (1.497 secs).4 [16-Nov-2009 10:03:22 +0000]: Mem usage after SORT: 8586864 [16-Nov-2009 10:03:22 +0000]: Length of $data: 1033513 [16-Nov-2009 10:03:22 +0000]: Calling preg_split
[16-Nov-2009 10:03:23] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 71 bytes) in /home/cjanuary/public_html/trunk/roundcubemail/program/lib/imap.inc on line 911
0.31 (x86_64 php-5.2.9-2.fc11.i586 httpd-2.2.13-1.fc11.i586 32M memory limit)
[16-Nov-2009 10:08:27 +0000]: Mem usage before SORT: 7476952 [16-Nov-2009 10:08:27 +0000]: C: s SORT (DATE) US-ASCII ALL [16-Nov-2009 10:08:29 +0000]: S: * SORT 54693 ... ... [16-Nov-2009 10:08:30 +0000]: S: s OK Sort completed (1.477 secs). [16-Nov-2009 10:08:30 +0000]: Mem usage after SORT: 8513400 [16-Nov-2009 10:08:30 +0000]: Length of $data: 1033513 [16-Nov-2009 10:08:30 +0000]: Calling preg_split
0.31 (i686 php-5.2.9-2.fc11.i586 httpd-2.2.13-1.fc11.i586 32M memory limit)
[16-Nov-2009 10:12:35 +0000]: Mem usage before SORT: 4714528 [16-Nov-2009 10:12:35 +0000]: C: s SORT (DATE) US-ASCII ALL [16-Nov-2009 10:12:36 +0000]: S: * SORT 54693 ... ... [16-Nov-2009 10:12:36 +0000]: S: s OK Sort completed (1.520 secs). [16-Nov-2009 10:12:36 +0000]: Mem usage after SORT: 5748980 [16-Nov-2009 10:12:36 +0000]: Length of $data: 1033513 [16-Nov-2009 10:12:36 +0000]: Calling preg_split [16-Nov-2009 10:12:37 +0000]: Mem after preg_split: 20530140 [16-Nov-2009 10:12:37 +0000]: Items in result: 163517
Chris
List info: http://lists.roundcube.net/dev/
Chris January wrote:
It appears to fail on x86_64 platforms, but not on i686. I am guessing PHP uses more memory on x86_64 and that pushes it over the limit.
[16-Nov-2009 10:12:37 +0000]: Mem after preg_split: 20530140 [16-Nov-2009 10:12:37 +0000]: Items in result: 163517
Was 20MB, with x86_64 you have more than 32 MB. Maybe this should be requested as PHP bug to PHP's bugtracker? In the mean time you can try replace line:
return preg_split('/\s+/', $data, -1, PREG_SPLIT_NO_EMPTY);
with:
return array_values(array_filter(explode(' ', $data)));
or with:
$res = array(); $tok = strtok($data, ' '); while ($tok !== false) { $res[] = $tok; $tok = strtok(' '); } return $res;
Check memory for each, please.
Was 20MB, with x86_64 you have more than 32 MB. Maybe this should be requested as PHP bug to PHP's bugtracker? In the mean time you can try replace line:
return preg_split('/\s+/', $data, -1, PREG_SPLIT_NO_EMPTY);
with:
return array_values(array_filter(explode(' ', $data)));
or with:
$res = array(); $tok = strtok($data, ' '); while ($tok !== false) { $res[] = $tok; $tok = strtok(' '); } return $res;
I tried both methods, but PHP runs out of memory in both cases.
Chris
List info: http://lists.roundcube.net/dev/
Chris January wrote:
I tried both methods, but PHP runs out of memory in both cases.
I've made some tests and I think we can do nothing about this. The problem is not in preg_split, the problem is in PHP's arrays handling. Array with 150k items requires 20-30 MB of memory. Also at x86_64 the footprint is 25% bigger.
Simple test, an array with empty items:
$b = array_fill(0, 100000, NULL);
in-memory size for this array = 10MB!!!
So, if you're working with such big mailboxes, you'll need to increase memory_limit.
On Fri, 20 Nov 2009 09:00:02 +0100, "A.L.E.C" alec@alec.pl wrote:
Chris January wrote:
I tried both methods, but PHP runs out of memory in both cases.
I've made some tests and I think we can do nothing about this. The problem is not in preg_split, the problem is in PHP's arrays handling. Array with 150k items requires 20-30 MB of memory. Also at x86_64 the footprint is 25% bigger.
Simple test, an array with empty items:
$b = array_fill(0, 100000, NULL);
in-memory size for this array = 10MB!!!
So, if you're working with such big mailboxes, you'll need to increase memory_limit.
Ok, that's what I expected as soon as I realised that the different versions were running on different architectures. Thanks for investigating the problem.
Chris
List info: http://lists.roundcube.net/dev/
-
A.L.E.C -
Chris January