My INBOX has > 10,000 messages. When doing a text search in the body across those messages, I'm ALWAYS getting a timeout the first time. Sometimes also when doing it a second time.
But when doing it at least the third time (but sometimes at the second attempt already), I get a result within a few seconds.
Is there anything we can do server-side which can improve this? IMAP server is Cyrus.
Furthermore, how does the text search work at all in detail? If I enter for instance "term1 term2" (without the quotation marks), does RC look for "term1" OR "term2", or does it look for "term1" AND "term2"? And if the latter, does it look for "term1 term2" as an exact phrase, or does it look separately for "term1" AND "term2" anywhere in the body? And do quotation marks have a meaning at all?
Our users are asking us this, and currently we don't have a good answer for them.
Last but not least: Will there be a header only search available in the near future?
Cheers,
Michael Heydekamp Co-Admin freexp.de Düsseldorf/Germany
Michael Heydekamp wrote:
My INBOX has > 10,000 messages. When doing a text search in the body across those messages, I'm ALWAYS getting a timeout the first time. Sometimes also when doing it a second time.
Is this a timeout displayed in the browser?
By default no socket timeout is set thus you might run into the default time limit for PHP processes which is defined by the max_execution_time php.ini setting.
The timeout for IMAP connections can explicitly set with
// IMAP connection timeout, in seconds. // Default: 0 (use default_socket_timeout) $config['imap_timeout'] = 30;
@Alec, we might consider setting the max_execution_time to max(max_execution_time, imap_timeout) in rcube_.
But when doing it at least the third time (but sometimes at the second attempt already), I get a result within a few seconds.
This sounds like the IMAP server has finally loaded your inbox into disk cache after the second search and is now fast enough to read all of it.
Is there anything we can do server-side which can improve this? IMAP server is Cyrus.
Searching is entirely delegated to the IMAP server and optimizations such as indexing need to be done there. Cyrus also has means to do indexing. Search for cyrus-squatter for more information on this.
Furthermore, how does the text search work at all in detail? If I enter for instance "term1 term2" (without the quotation marks), does RC look for "term1" OR "term2", or does it look for "term1" AND "term2"? And if the latter, does it look for "term1 term2" as an exact phrase, or does it look separately for "term1" AND "term2" anywhere in the body? And do quotation marks have a meaning at all?
Search strings are passed as-is to the IMAP server with either the SEARCH or SORT command like
A0004 UID SORT (ARRIVAL) US-ASCII TEXT "image test"
or
A0004 UID SEARCH RETURN (ALL) OR BODY "some test" HEADER SUBJECT "some test"
If you selected multiple headers sections (headers and/or body) to search in, these will be search with OR but the search term is taken as one string. Roundcube currently doesn't support operators like AND or OR or quotaion.
But again, IMAP servers might react differently depending on how they implemented the search command.
Our users are asking us this, and currently we don't have a good answer for them.
Last but not least: Will there be a header only search available in the near future?
What exactly does that mean? You can already select headers to search in.
Kind regards, Thomas
Am 13.10.2013 12:04, schrieb Thomas Bruederli:
Michael Heydekamp wrote:
My INBOX has > 10,000 messages. When doing a text search in the body across those messages, I'm ALWAYS getting a timeout the first time. Sometimes also when doing it a second time.
Is this a timeout displayed in the browser?
Yes, it's the Ajax message of Roundcube, see image attached.
It appears (if it appears at all) after 180 secs apparently, but I believe to have seen it earlier already.
By default no socket timeout is set thus you might run into the default time limit for PHP processes which is defined by the max_execution_time php.ini setting.
Currently I have no access to the server, but it may be set to 180 secs, will check.
But when doing it at least the third time (but sometimes at the second attempt already), I get a result within a few seconds.
This sounds like the IMAP server has finally loaded your inbox into disk cache after the second search and is now fast enough to read all of it.
Probably. I just did a test with this list (11,144 messages), the first search resulted in a timeout after 180 secs, any consecutive search was successful within 25-35 secs.
Is there anything we can do server-side which can improve this? IMAP server is Cyrus.
Searching is entirely delegated to the IMAP server and optimizations such as indexing need to be done there. Cyrus also has means to do indexing. Search for cyrus-squatter for more information on this.
Seen, thanks, will test. Unfortunately new messages will not be indexed automatically, which I do not find very convenient.
Plus: I have not yet found any information if and how Cyrus handles different 8bit charsets or UTF-8/HTML encoded messages, possibly on top being qp/b64 encoded. Do you have any information about or experience with this?
Furthermore, how does the text search work at all in detail? If I enter for instance "term1 term2" (without the quotation marks), does RC look for "term1" OR "term2", or does it look for "term1" AND "term2"? And if the latter, does it look for "term1 term2" as an exact phrase, or does it look separately for "term1" AND "term2" anywhere in the body? And do quotation marks have a meaning at all?
Search strings are passed as-is to the IMAP server with either the SEARCH or SORT command like
A0004 UID SORT (ARRIVAL) US-ASCII TEXT "image test"
or
A0004 UID SEARCH RETURN (ALL) OR BODY "some test" HEADER SUBJECT "some test"
If you selected multiple headers sections (headers and/or body) to search in, these will be search with OR but the search term is taken as one string. Roundcube currently doesn't support operators like AND or OR or quotaion.
But it would generally be possible, right?
Anyway, thanks for this helpful information. (Such things should go to a Roundcube documentation, by the way.)
Last but not least: Will there be a header only search available in the near future?
What exactly does that mean? You can already select headers to search in.
But only the standard headers From, To, Cc, Bcc and Subject. Apart from that, I can currently search a) in the entire message (all headers + body), and b) in the body only. "Header only" would search c) in the entire header section, but not in the body. For instance if I want to find all messages with a header "X-Envelope-To: listuser@freexp.de", where a search in the entire message (including the body) would simply make no sense.
Would that be feasible?
Cheers,
Michael Heydekamp Co-Admin freexp.de Düsseldorf/Germany
On Sunday 13 October 2013 15:54:19 Michael Heydekamp wrote:
Anyway, thanks for this helpful information. (Such things should go to a Roundcube documentation, by the way.)
Please add it to the Roundcube documentation! ;)
Kind Regards, Torsten
-
Michael Heydekamp -
Thomas Bruederli -
Torsten Grote