On Fri, Feb 27, 2009 at 5:27 PM, emi daruma tai emidaruma@gmail.com wrote:
Hi! Nice patches, Chris!
You can do some AJAX loading for the first time you expand a thread. With that, you'll be able to do a thread counter for pagesize and pagination and make a growing per message counter, if it's needed. And, sure, it will look very well, only transmitting lots of info when it's needed (server-side thread caching also enabled).
Testing with MySQL I found it's very slow to cache the threads and was faster simply calling IMAP THREADS again each time.
Chris _______________________________________________ List info: http://lists.roundcube.net/dev/
On Sun, 1 Mar 2009 15:38:35 +0000, Chris January chris@atomice.net wrote:
<snip> > Testing with MySQL I found it's very slow to cache the threads and was > faster simply calling IMAP THREADS again each time. > > Chris
Chris,
Where are you at on your TODO list for threading support? I ask because if you have already started, then I was thinking I might take a stab at a couple of them, particularly sorting the message list based on the most recently receive message in a thread. But if you have already started in on it, then I don't want to duplicate work. I just started using RoundCube and I haven't looked at the code yet, so I imagine there would be some ramp-up time needed for me to get my bearings.
Thanks,
Nathan _______________________________________________ List info: http://lists.roundcube.net/dev/
Where are you at on your TODO list for threading support? I ask because if you have already started, then I was thinking I might take a stab at a couple of them, particularly sorting the message list based on the most recently receive message in a thread. But if you have already started in
I haven't implemented this yet. The TODO list should be up to date.
Cheers, Chris _______________________________________________ List info: http://lists.roundcube.net/dev/
On Thu, 5 Mar 2009 12:17:24 +0000, Chris January chris@atomice.net wrote:
Where are you at on your TODO list for threading support? I ask
because
if you have already started, then I was thinking I might take a stab at a couple of them, particularly sorting the message list based on the most recently receive message in a thread. But if you have already started in
I haven't implemented this yet. The TODO list should be up to date.
Cheers, Chris
So as I began to investigate the sorting of threads by the date of the most recent message in a thread rather than the thread's root message, I discovered that there is beginning to be support for this at the IMAP server level:
http://tools.ietf.org/html/draft-ietf-morg-inthread-00
I happen to use Dovecot, and this functionality was introduced in the first alpha release of version 1.2 back in Sept. 2008 (http://dovecot.org/list/dovecot-news/2008-September/000084.html). I also use Debian, and v1.2 of Dovecot isn't even available in unstable or experimental, but I was able to build/install if from a source package referenced on the Dovecot site. If anyone wants the amd64 Debian packages for Dovecot 1.2beta1, just let me know. The functionality is introduced as a new and non-standard THREAD algorithm referenced by X-REFERENCES2, which I have verified to work on my own install.
Please note that all of these patches are dependant on Chris January's patch set to implement a threaded view: http://www.atomice.com/blog/?page_id=34, as indicated by the first message in this thread. Also, for this to work, your IMAP server must support a thread sorting algorithm that sorts based on the newest in-thread message rather than just by the date of a thread's root message. In Dovecot (since v1.2) this is supported as references above, but I have no idea how or even if any other IMAP servers implement this. I'd be interested to know if anyone else has more information on this. Here are the patches:
http://code.nkinka.de/gitweb?p=roundcube.git;a=commitdiff;h=61d007a3c6867766... http://code.nkinka.de/gitweb?p=roundcube.git;a=commitdiff;h=945604dcb6805965... http://code.nkinka.de/gitweb?p=roundcube.git;a=commitdiff;h=8018757b75262d00...
Nathan
List info: http://lists.roundcube.net/dev/
On Sun, 15 Mar 2009 22:43:20 +0000, Nathan Kinkade nath@nkinka.de wrote:
On Thu, 5 Mar 2009 12:17:24 +0000, Chris January chris@atomice.net
wrote:
Where are you at on your TODO list for threading support? I ask
because
if you have already started, then I was thinking I might take a stab at a couple of them, particularly sorting the message list based on the most recently receive message in a thread. But if you have already started in
I haven't implemented this yet. The TODO list should be up to date.
Cheers, Chris
So as I began to investigate the sorting of threads by the date of the
most
recent message in a thread rather than the thread's root message, I discovered that there is beginning to be support for this at the IMAP server level:
http://tools.ietf.org/html/draft-ietf-morg-inthread-00
I happen to use Dovecot, and this functionality was introduced in the
first
alpha release of version 1.2 back in Sept. 2008 (http://dovecot.org/list/dovecot-news/2008-September/000084.html). I
also
use Debian, and v1.2 of Dovecot isn't even available in unstable or experimental, but I was able to build/install if from a source package referenced on the Dovecot site. If anyone wants the amd64 Debian
packages
for Dovecot 1.2beta1, just let me know. The functionality is introduced
as
a new and non-standard THREAD algorithm referenced by X-REFERENCES2,
which
I have verified to work on my own install.
Please note that all of these patches are dependant on Chris January's patch set to implement a threaded view: http://www.atomice.com/blog/?page_id=34, as indicated by the first
message
in this thread. Also, for this to work, your IMAP server must support a thread sorting algorithm that sorts based on the newest in-thread message rather than just by the date of a thread's root message. In Dovecot
(since
v1.2) this is supported as references above, but I have no idea how or
even
if any other IMAP servers implement this. I'd be interested to know if anyone else has more information on this. Here are the patches:
http://code.nkinka.de/gitweb?p=roundcube.git;a=commitdiff;h=61d007a3c6867766...
http://code.nkinka.de/gitweb?p=roundcube.git;a=commitdiff;h=945604dcb6805965...
http://code.nkinka.de/gitweb?p=roundcube.git;a=commitdiff;h=8018757b75262d00...
Nathan
I just discovered a bug in the patches I linked to above. I believe it is fixed by the following:
http://code.nkinka.de/gitweb?p=roundcube.git;a=commitdiff;h=8f612c34a9356eb0...
Nathan
List info: http://lists.roundcube.net/dev/
So as I began to investigate the sorting of threads by the date of the most recent message in a thread rather than the thread's root message, I discovered that there is beginning to be support for this at the IMAP server level:
The Dovecot support for X-REFERENCES2 is very useful to know about!
anyone else has more information on this. Here are the patches:
http://code.nkinka.de/gitweb?p=roundcube.git;a=commitdiff;h=61d007a3c6867766... http://code.nkinka.de/gitweb?p=roundcube.git;a=commitdiff;h=945604dcb6805965... http://code.nkinka.de/gitweb?p=roundcube.git;a=commitdiff;h=8018757b75262d00...
May I ask what is the rationale behind the second patch?
I took a look at all three patches and the follow up. I think the idea of a 'default' sort order is a good one. I have been testing the threading implementation with a 70,000+ message mailbox and having the option to set the sort order to 'default' should improve load times even if threading is turned off. I would like to integrate this with the GUI so that, e.g. clicking the Date column toggles between ascending, descending and 'default' (since the default sort order sorts by date in some fashion). I also saw you had tweaked which messages are expanded by default. My personal preference is to use a slightly different algorithm, namely: unread = collapsed read + any descendants unread = expanded read + all descendants read = collapsed This assumes the threads you are interested in are the ones you have read. If any new replies are added to those threads you want to see them. However threads where you have not read the root of are probably not interesting so shouldn't be expanded.
Cheers, Chris _______________________________________________ List info: http://lists.roundcube.net/dev/
On Mon, 16 Mar 2009 12:50:55 +0000, Chris January chris@atomice.net wrote:
<snip> >> http://code.nkinka.de/gitweb?p=roundcube.git;a=commitdiff;h=61d007a3c6867766391afb7c5d02b85ff44df1e8 >> http://code.nkinka.de/gitweb?p=roundcube.git;a=commitdiff;h=945604dcb6805965ed498e690b003770fa619210 >> http://code.nkinka.de/gitweb?p=roundcube.git;a=commitdiff;h=8018757b75262d009bc6611be76bf4696d549caa > > May I ask what is the rationale behind the second patch? > > I took a look at all three patches and the follow up. I think the idea > of a 'default' sort order is a good one. I have been testing the > threading implementation with a 70,000+ message mailbox and having the > option to set the sort order to 'default' should improve load times > even if threading is turned off. > I would like to integrate this with the GUI so that, e.g. clicking the > Date column toggles between ascending, descending and 'default' (since > the default sort order sorts by date in some fashion). > I also saw you had tweaked which messages are expanded by default. My > personal preference is to use a slightly different algorithm, namely: > unread = collapsed > read + any descendants unread = expanded > read + all descendants read = collapsed > This assumes the threads you are interested in are the ones you have > read. If any new replies are added to those threads you want to see > them. However threads where you have not read the root of are probably > not interesting so shouldn't be expanded. > > Cheers, > Chris
Sorry, I should have explained some of the changes a little more. The 2nd patch is there because the default behavior of RoundCube is to store any sort order you select in both the session and in the database. With X-REFERENCES2, and presumably any other similar algorithms, there really is no sorting needed on the client side by default. However, I had no way to tell RoundCube to drop what it had stored and do no sorting at all. That is, once a sort order gets saved in the database you seem to be locked in to sort fields/orders that are available through the interface and there was no obvious way to make it forget that and just let the server do all the sorting. In my case, 99.99% of the time I'm going to want to sort DESC by date, but in the special way that X-REFERENCES2 implements it. The 2nd patch just means that if you do a special sorting through the interface that it will only implement it until you make some other action, at which point it will just automatically revert to no sort, letting the IMAP server do it's thing.
About the expanding of messages: my intention is that any thread with an unread message will be expanded as necessary to reveal the unread message. If it encounters sub-threads along the way whose children are all read then it will leave that sub-thread collapsed. Basically, expand as necessary so that no unread messages is ever hidden, yet only expand just enough so that this is true. It seems to be working fine.
A note about my follow up message. I was incorrect. The follow up patch I sent caused it appear to start working, but on closer inspection later, the sorting was not quite correct. The problem I was experiencing was like this: a folder seems to be working fine, the INBOX, for example, but as soon I would move a message from there to another folder suddenly RoundCube would tell me that there were no messages in the INBOX, yet the unread count next to the folder name would reveal the truth. I still haven't 100% figured it out, but it's related to the value of $max that gets set around line 590 of ./program/include/rcube_imap.php. That $max value when using the X-REFERENCES algorithm gets set to a value 1 greater than the actual correct number, because the function iil_C_Thread() seems to be returning this spurious value. I can get around the problem in one of two ways for now: 1) delete the files dovecot.* in the actual INBOX maildir directory 2) set $max = $this->_messagecount($mailbox, 'ALL');, which seems to work fine even with the threaded implementation. Any thoughts on this?
Nathan _______________________________________________ List info: http://lists.roundcube.net/dev/
On Mon, Mar 16, 2009 at 2:02 PM, Nathan Kinkade nath@nkinka.de wrote:
Sorry, I should have explained some of the changes a little more. The 2nd patch is there because the default behavior of RoundCube is to store any sort order you select in both the session and in the database. With X-REFERENCES2, and presumably any other similar algorithms, there really is no sorting needed on the client side by default. However, I had no way to tell RoundCube to drop what it had stored and do no sorting at all. That is, once a sort order gets saved in the database you seem to be locked in to sort fields/orders that are available through the interface and there was no obvious way to make it forget that and just let the server do all the sorting. In my case, 99.99% of the time I'm going to want to sort DESC by date, but in the special way that X-REFERENCES2 implements it. The 2nd patch just means that if you do a special sorting through the interface that it will only implement it until you make some other action, at which point it will just automatically revert to no sort, letting the IMAP server do it's thing.
I am just finishing a patch that adds a new sort column 'default' that will trigger default sorting. You can enable it by clicking the Date column DESC -> ASC -> 'default' -> DESC... This should give us what we want (default sorting) and work with the existing infrastructure.
About the expanding of messages: my intention is that any thread with an unread message will be expanded as necessary to reveal the unread message. If it encounters sub-threads along the way whose children are all read then it will leave that sub-thread collapsed. Basically, expand as necessary so that no unread messages is ever hidden, yet only expand just enough so that this is true. It seems to be working fine.
Ok. This is not what I would want, but is what you would want, so there's obviously a case here for making the policy configurable.
A note about my follow up message. I was incorrect. The follow up patch I sent caused it appear to start working, but on closer inspection later, the sorting was not quite correct. The problem I was experiencing was like this: a folder seems to be working fine, the INBOX, for example, but as soon I would move a message from there to another folder suddenly RoundCube would tell me that there were no messages in the INBOX, yet the unread count next to the folder name would reveal the truth. I still haven't 100% figured it out, but it's related to the value of $max that gets set around line 590 of ./program/include/rcube_imap.php. That $max value when using the X-REFERENCES algorithm gets set to a value 1 greater than the actual correct number, because the function iil_C_Thread() seems to be returning this spurious value. I can get around the problem in one of two ways for now: 1) delete the files dovecot.* in the actual INBOX maildir directory 2) set $max = $this->_messagecount($mailbox, 'ALL');, which seems to work fine even with the threaded implementation. Any thoughts on this?
I think $max being +1 is intentional (I can't remember why). As for moving messages, I only tested that briefly so there's likely to be bugs in my original implementation.
Regards, Chris _______________________________________________ List info: http://lists.roundcube.net/dev/
this spurious value. I can get around the problem in one of two ways for now: 1) delete the files dovecot.* in the actual INBOX maildir directory 2) set $max = $this->_messagecount($mailbox, 'ALL');, which seems to work fine even with the threaded implementation. Any thoughts on this?
I think $max = $this->_messagecount($mailbox, 'ALL'); is the correct fix but it needs to go just before the call to clear_message_cache. Something like the following: // delete cached messages with a higher index than $max+1 // Changed $max to $max+1 to fix this bug : #1484295 if ($message_threading) $max = $this->_messagecount($mailbox, 'ALL'); $this->clear_message_cache($cache_key, $max + 1); This may fix it. Previously $max was the number of threads so any messages with indices > no. threads got cleared form the cache.
Chris _______________________________________________ List info: http://lists.roundcube.net/dev/
On Sun, Mar 15, 2009 at 10:43 PM, Nathan Kinkade nath@nkinka.de wrote:
On Thu, 5 Mar 2009 12:17:24 +0000, Chris January chris@atomice.net wrote: Please note that all of these patches are dependant on Chris January's patch set to implement a threaded view: http://www.atomice.com/blog/?page_id=34, as indicated by the first message
I've refreshed the patches on the page. Changes:
- Ability to set sort order to 'default' by clicking on the Date
column (goes DESC -> ASC -> default) (this is a different implementation from Nathan's original patch).
- Ability to change IMAP threading algorithm (as suggested by Alec and
implemented in one of Nathan's patches).
- Optimised iil_C_ParseThreads.
If you use Dovecot you can set the threading algorithm to X-REFERENCES2 (as discovered by Nathan) and the sort order to 'default' and the threads will be sorted by most recent message in each thread.
Chris _______________________________________________ List info: http://lists.roundcube.net/dev/
On Tue, 17 Mar 2009 14:09:49 +0000, Chris January chris@atomice.net wrote:
this spurious value. I can get around the problem in one of two ways for now: 1) delete the files dovecot.* in the actual INBOX maildir directory 2) set $max = $this->_messagecount($mailbox, 'ALL');, which seems to work fine even with the threaded implementation. Any thoughts on this?
I think $max = $this->_messagecount($mailbox, 'ALL'); is the correct fix but it needs to go just before the call to clear_message_cache. Something like the following: // delete cached messages with a higher index than $max+1 // Changed $max to $max+1 to fix this bug : #1484295 if ($message_threading) $max = $this->_messagecount($mailbox, 'ALL'); $this->clear_message_cache($cache_key, $max + 1); This may fix it. Previously $max was the number of threads so any messages with indices > no. threads got cleared form the cache.
Chris
Turns out this was a bug in Dovecot. You can follow the report and resulting path on this thread:
http://markmail.org/message/x46suaawhcsefykg
Fore convenience the patch is here:
http://hg.dovecot.org/dovecot-1.2/rev/bf16646ec312
Nathan _______________________________________________ List info: http://lists.roundcube.net/dev/
-
Chris January -
Nathan Kinkade