Webmail competition

Brennan Stehling brennan at offwhite.net
Thu Dec 21 04:53:40 CET 2006


I would not be concerned with testing multiple versions of Apache, PHP or MySQL with RC.  I think most bugs we will encounter will be related to coding errors which will work universally the same in various hosting combinations.  The same goes for browsers to a lesser degree.  I would expect that the developers working on the project are already skillful enough to avoid the browser-specific problems that are well known at this point.  That said, I think testing FireFox and MSIE on Windows will cover a vast majority of user environments.  It would be nice to test Opera and Safari, but in my experience those browsers track pretty well with features in FireFox, such as a very similar AJAX system.  When I develop something advanced with Javascript I do it test it FireFox during development and when I am wrapping it up I test it on Safari and very infrequently have a problem.  And most of the time it is an easy fix.

As sending out test messages for the automatically updated RC installs, you could just use the mail command on Linux/FreeBSD server.  It is something like...

cat message1.txt | mail -t person at here.com

The text file just has the contents of an email message including the mail headers.  I would expect we can set it up to do attachments as well as add spam headers for that sort of testing.  I think we just need to configure the mail server to bypass the spam filter so that we can set our own headers specifically for testing purposes.

As for RC versions, I would track the current development and the active branches.  Right now I believe we do not have a major releases.  I do not see specific releases under branches yet.


Here is how I do releases...


Once we get past the 0.1 release we can track 0.1 for a while until we have 0.2 and 0.1 goes inactive.  So we just need 2 actively running test environments.  However, if you want to try out multiple IMAP backends and databases I do see a reason to do that.  There are significant differences between MySQL and Postgres and the various IMAP servers.  That is where multiple servers could be useful.  In the short term I would focus on the most actively used db and IMAP server and expand as the project grows.  I think the testing of the IMAP and db communications could be done outside the confines of a full web deployment.  Adding layers of functionality on top of these core integration points will make it harder to get at the root cause of a problem.  If we can discover that RC has trouble talking to a specific db version before it goes out to the full RC setup than we will save a good deal of time and effort.


On Wed, 20 Dec 2006 22:08:18 -0500, Jim Pingle <lists at pingle.org> wrote:
> Brennan Stehling wrote:
>> You can learn more about the ticket reports here...
>> http://trac.roundcube.net./trac.cgi/wiki/TracReports
>> I think the voting idea would work but we can simply put up a vote on
> specific
>> issues occasionally.  On Apache projects they do it simply by posting an
> email
>> to the list and letting everyone vote on each item.  The team lead keeps
> track
>> of it and adjust the priorities based on the results.  That way it stays
> within
>> the team.
> What I had in mind was to use the voting as a means to measure how much of
> the RoundCube user-base is affected by a given ticket. The periodic
> posting
> may work well for feature requests, but keeping the voting within the team
> doesn't necessarily help with checking how widespread a problem is with
> the
> users.
>> I do like the idea of a few test environments.  I think we can do it all
> on the
>> same server.  It just needs a separate website per install with a
> separate db
>> for each one.
> One server may be enough, but some situations may require some odd
> configurations to make that work. I'd prefer to minimize the customization
> needed, and to keep it closer to a standard user install. It might be
> possible to do using jails or just multiple IPs. However, when you start
> talking about all of the configuration possibilities, obviously there
> would
> need to be limits as to just how far we'd have to go. Apache (1.3.x, 2.x),
> PHP (4.x, 5.x), MySQL (4.x, 5.x), Postgres, not to mention IMAP servers:
> Courier, Dovecot, etc, etc. It could get out of hand fast no matter how it
> is done. :) The most popular are the most important though (Apache 2, PHP
> 5,
> MySQL 4, etc)
>> I have a FreeBSD server and could set up a test environment.
> I have quite a few active servers at work (~20), and a lot more sitting
> around collecting dust. Most of the ones collecting dust are at least dual
> CPU PIII-800s with a GB or two of ram. Not speed demons by today's
> standards, but for what needs done here it wouldn't have to be fast.
>> Perhaps you could help me with an automatic provisioning system to
> deploy
>> updates daily or weekly and automate some tests using Selenium. 
> Dropping the db
>> and rebuilding it with the latest update, generating a series of test
> emails and
>> loading up the Selenium tests would not be all that difficult.  And that
> does
>> not require programming experience, but with your SA background you
> would know
>> what to do.
> I haven't messed with Selenium at all, but getting the backends in place
> shouldn't be that hard. If you're just talking about getting a current SVN
> revision in a directory and wiping/reloading the DB, that's really easy.
> The test e-mail would be a little tougher to do, but still not hard.
> Copying
> the messages into the right place is easy, but coming up with all of the
> test cases might be rough. There are a lot of test cases that will need to
> be considered (Plain vs HTML, MIME/base64, Attachments of all shapes and
> sizes and each of those cases from multiple clients.)
>> What I do not know yet is how to run the Selenium scripts on a server
> like
>> FreeBSD or Linux.  I suppose I could have a job run on a Windows server
> which
>> has FireFox and MSIE installed so it can run the Selenium scripts.  Of
> course
>> someone could start the Selenium tests manually.
> Because Selenium is a UI testing package, I'm not sure if there is a way
> to
> automate it on a remote box unless you have VNC access or something
> similar,
> because the tests run inside a browser. It might be best to have tests for
> all platforms, though I do not have access to any Mac hardware. (And
> either
> their page is out of date or their browser support is really lagging...
> Only
> Opera 8 and Firefox 1.5?)
>> But I also do not have time till mid January.
> I'm not even sure when my schedule will clear up, but at this rate,
> probably
> early January. I still find time now and then to poke around the trouble
> tickets, I just wish I was more familiar with the code so I could jump in
> and help out more directly. Coding isn't a problem, it's knowing where to
> look and the how it all flows, which takes time to get into.
> And of course you also have to consider the delicate balance where all of
> this extra testing and overhead stops being helpful and starts being a
> burden to developers. That's where making it quick, easy, and painless
> would
> really help. The last thing we'd want is to discourage someone from coding
> because they don't want to go through all the "trouble" of testing... (I'm
> as guilty of putting off testing because of that as anyone!)
> Jim
Brennan Stehling
Offwhite.net LLC
brennan at offwhite.net

More information about the Dev mailing list