Jim,
I would not be concerned with testing multiple versions of Apache, PHP or MySQL with RC. I think most bugs we will encounter will be related to coding errors which will work universally the same in various hosting combinations. The same goes for browsers to a lesser degree. I would expect that the developers working on the project are already skillful enough to avoid the browser-specific problems that are well known at this point. That said, I think testing FireFox and MSIE on Windows will cover a vast majority of user environments. It would be nice to test Opera and Safari, but in my experience those browsers track pretty well with features in FireFox, such as a very similar AJAX system. When I develop something advanced with Javascript I do it test it FireFox during development and when I am wrapping it up I test it on Safari and very infrequently have a problem. And most of the time it is an easy fix.
As sending out test messages for the automatically updated RC installs, you could just use the mail command on Linux/FreeBSD server. It is something like...
cat message1.txt | mail -t person@here.com
The text file just has the contents of an email message including the mail headers. I would expect we can set it up to do attachments as well as add spam headers for that sort of testing. I think we just need to configure the mail server to bypass the spam filter so that we can set our own headers specifically for testing purposes.
As for RC versions, I would track the current development and the active branches. Right now I believe we do not have a major releases. I do not see specific releases under branches yet.
http://trac.roundcube.net./trac.cgi/browser/branches
Here is how I do releases...
http://svn.offwhite.net/trac/SmallSharpTools.ImageResizer/browser/branches
Once we get past the 0.1 release we can track 0.1 for a while until we have 0.2 and 0.1 goes inactive. So we just need 2 actively running test environments. However, if you want to try out multiple IMAP backends and databases I do see a reason to do that. There are significant differences between MySQL and Postgres and the various IMAP servers. That is where multiple servers could be useful. In the short term I would focus on the most actively used db and IMAP server and expand as the project grows. I think the testing of the IMAP and db communications could be done outside the confines of a full web deployment. Adding layers of functionality on top of these core integration points will make it harder to get at the root cause of a problem. If we can discover that RC has trouble talking to a specific db version before it goes out to the full RC setup than we will save a good deal of time and effort.
Brennan
On Wed, 20 Dec 2006 22:08:18 -0500, Jim Pingle lists@pingle.org wrote:
Brennan Stehling wrote:
You can learn more about the ticket reports here... http://trac.roundcube.net./trac.cgi/wiki/TracReports
I think the voting idea would work but we can simply put up a vote on
specific
issues occasionally. On Apache projects they do it simply by posting an
to the list and letting everyone vote on each item. The team lead keeps
track
of it and adjust the priorities based on the results. That way it stays
within
the team.
What I had in mind was to use the voting as a means to measure how much of the RoundCube user-base is affected by a given ticket. The periodic posting may work well for feature requests, but keeping the voting within the team doesn't necessarily help with checking how widespread a problem is with the users.
I do like the idea of a few test environments. I think we can do it all
on the
same server. It just needs a separate website per install with a
separate db
for each one.
One server may be enough, but some situations may require some odd configurations to make that work. I'd prefer to minimize the customization needed, and to keep it closer to a standard user install. It might be possible to do using jails or just multiple IPs. However, when you start talking about all of the configuration possibilities, obviously there would need to be limits as to just how far we'd have to go. Apache (1.3.x, 2.x), PHP (4.x, 5.x), MySQL (4.x, 5.x), Postgres, not to mention IMAP servers: Courier, Dovecot, etc, etc. It could get out of hand fast no matter how it is done. :) The most popular are the most important though (Apache 2, PHP 5, MySQL 4, etc)
I have a FreeBSD server and could set up a test environment.
I have quite a few active servers at work (~20), and a lot more sitting around collecting dust. Most of the ones collecting dust are at least dual CPU PIII-800s with a GB or two of ram. Not speed demons by today's standards, but for what needs done here it wouldn't have to be fast.
Perhaps you could help me with an automatic provisioning system to
deploy
updates daily or weekly and automate some tests using Selenium.
Dropping the db
and rebuilding it with the latest update, generating a series of test
emails and
loading up the Selenium tests would not be all that difficult. And that
does
not require programming experience, but with your SA background you
would know
what to do.
I haven't messed with Selenium at all, but getting the backends in place shouldn't be that hard. If you're just talking about getting a current SVN revision in a directory and wiping/reloading the DB, that's really easy.
The test e-mail would be a little tougher to do, but still not hard. Copying the messages into the right place is easy, but coming up with all of the test cases might be rough. There are a lot of test cases that will need to be considered (Plain vs HTML, MIME/base64, Attachments of all shapes and sizes and each of those cases from multiple clients.)
What I do not know yet is how to run the Selenium scripts on a server
like
FreeBSD or Linux. I suppose I could have a job run on a Windows server
which
has FireFox and MSIE installed so it can run the Selenium scripts. Of
course
someone could start the Selenium tests manually.
Because Selenium is a UI testing package, I'm not sure if there is a way to automate it on a remote box unless you have VNC access or something similar, because the tests run inside a browser. It might be best to have tests for all platforms, though I do not have access to any Mac hardware. (And either their page is out of date or their browser support is really lagging... Only Opera 8 and Firefox 1.5?)
But I also do not have time till mid January.
I'm not even sure when my schedule will clear up, but at this rate, probably early January. I still find time now and then to poke around the trouble tickets, I just wish I was more familiar with the code so I could jump in and help out more directly. Coding isn't a problem, it's knowing where to look and the how it all flows, which takes time to get into.
And of course you also have to consider the delicate balance where all of this extra testing and overhead stops being helpful and starts being a burden to developers. That's where making it quick, easy, and painless would really help. The last thing we'd want is to discourage someone from coding because they don't want to go through all the "trouble" of testing... (I'm as guilty of putting off testing because of that as anyone!)
Jim