On 3/6/15 11:10 AM, Thomas Bruederli wrote:

Dear developers and list lurkers

The Roundcube project will celebrate its 10th anniversary this year and a decade is just right to stop for a while and let your thoughts flow into new directions. Some of you may already have read my random thoughts [1] about the future or Roundcube development and since I wrote this down, the idea of a fresh new successor to Roundcube has settled in my mind and won't go away that easily. The initial responses on that were exclusively positive and this encourages us to take it one step further now. Meanwhile I looked into many other applications, libraries, frameworks, apis and programming languages to find the best set of tools that would form a new top-noch webmail application for the next decade. A loose and admittedly rather limited collection has manifested on a wiki page [2] already.

The primary goal of rethinking Roundcube and webmail in general is certainly to improve the user experience for everyday communication. The second important aspect (at least for me as a developer) is to have a slick, structured and flexible framework to build new features and applications on top of. The current Roundcube codebase comes with a variety of technological and conceptual constraints which rule the ongoing development in a rather negative manner. Therefore thinking of a successor that is completely independent and unrelated to the current software stack is a tempting exercise which I'd like to invite you to.

As already outlined in the blog post [1], the new Roundcube should move most of the application logic from the server to the client. Browsers are way more capable than they've been 10 years ago and times of server-side template rendering are definitely over. Those who are familiar with the Roundcube codebase will immediately understand that such a change in paradigms also means a complete break of compatibility with all existing components of Roundcube, including plugins.

So let's free our mind from what we know about Roundcube and start over. This can even go as far as choosing something else than PHP for the server part which is supposed to be reduced to a simple wrapper for data exchange between the backend storage (IMAP et al) and the client application. Possible candidates are PHP (still), Python, Node.js. At least these are the ones I'm currently familiar with. So let's take a deeper look into these as I'm particularly interested in your opinions, also from a deployment and hosting perspective.

• PHP

The major disadvantage of PHP that has become evident in Roundcube is its short process lifetime requiring a full startup, database and IMAP connection+authentication and closing all connections again for every single HTTP request. Even FPM and persistent socket connections (which IMO are still unreliable and should be avoided) don't really help to overcome this.

The advantage of sticking with PHP certainly is that we can re-use some of the existing parts from the Roundcube codebase, such as the IMAP library, the plugin infrastructure etc. And in terms of deployment: PHP is everywhere and lots of sysadmins are familiar with deploying and maintaing and tweaking PHP applications.

• Python

Although Python isn't perfect for running as long-term processes either, there are some well developed frameworks out there (e.g. Twisted or Flask) which could be used to manage persistent connections between the client (think websockets) and storage backends. In addition to that, its easier to implement reactive applications with an asynchronous architecture.

Regarding deployment: the Python universe has some well established package repositories which make installation and deployment pretty easy. From a sysadmin point of view, I have no idea how easy or painful the maintenance of Python applications is. But the number of hosting providers offering Python is certainly lower that for PHP.

• Node.js

The new star on the web app sky. Its architecture seems to make node.js perfect for long-living server processes and maximum throughput. Although we could use Javascript, which we're all familiar with, the number of pitfalls to use that for server applications is quite high. The learning curve here shouldn't be underestimated.

And opinions certainly go into two distinct directions when in comes to deploying and scaling node applications. Installation should come easy as there's basically only npm for all the package management and everybody doing node is familiar with that.

OK, so question #1: what's your take on the programming language choice?

The client part luckily doesn't need such a decision as there's only Javascript for this. But the variety of ways how to build a single-page web application and the number of frameworks helping you to do this is mind-boggling. I have to admit that in this regard, I'm one of the old fashioned web developers from the late 90ies and yet have to learn a lot about the brave new world of web development. I guess we can all agree that nobody wants to build a new web application framework from scratch as we did for Roundcube due to the lack of sophisticated solutions back in 2005. But choosing the right tools might be essential for the timeline and the success of Roundcube Next.

So here's question #2: what are your favorite javascript application libraries and frameworks? What experience did you make let's say with Angular or Backbone? What are the pitfalls?

Finally another big challenge we have when aiming to create the perfect webmail application is the way we make the client and the server talk together. We should carefully draft a proper data model and an according protocol for this. For the data model, I already tried to create a first draft in regards of what we already know the application should be capable of. This includes multi-account support, tagging, sharing and the extensibility for calendaring and other groupware-like tasks.

For the client-server communication, there are some interesting concepts out there but none of them actually seems to suit the purpose we're after. They either lack the concept of sharing (i.e. ACL) or do not account for collections such as multiple different address books one account my have access to. The candidates I'm referring to are: Nilas.com (former InboxApp), JMAP (by Fastmail) or the Gmail API (also to be supported by Dovecot). All are good approaches and generally point into the same direction but I couldn't say that one of them would already solve the use-cases we want to cover in Roundcube. And yes, we're seeking for a protocol and not an API [3] :-)

And this is question #3: who's willing to help us draft the perfect protocol for this? I guess this should be a highly collaborative and iterative process and I'm keen on any input from people who are experienced with this type of task.

Thank you all for reading and participating in this survey. I'm really excited to move this forward. All free and open source of course!

Best, Thomas

[1] https://roundcubeinbox.wordpress.com/2014/09/12/roundcube-next-if-we-would-s... [2] http://trac.roundcube.net/wiki/Roundcube_Next [3] https://vimeo.com/71278954 _______________________________________________ Roundcube Development discussion mailing list dev@lists.roundcube.net http://lists.roundcube.net/mailman/listinfo/dev

#1 : They're all fine, it really boils down to which language best fits the mold of hosting a thin backend protocol server that you described.
I prefer node.js for all of the most recent web or socket based applications we've developed due to it's simple concurrency and the package manager (which is awesome). Using Node you could have plugins published as packages, which are very simple for developers to build and deploy. Also IMO a rich plugin architecture would spring more naturally from Javascript as it's a more ubiquitous language for the people who are already working with RC. Since Node is also hosting its own (web/protocol) socket you can drop many dependencies like an http server which fits in nicely with PaaS architectures for hosting. Though we have never built anything nearly as complex as a webmail server framework, Node is very capable and could be molded to do just that.
You might look at the Haraka SMTP server as an example of a plugin based architecture in Node.

#2 : Not being much of a client side aficionado I can't really comment, but most of the work I have seen is done in backbone which does not impose a structure on the developer which has actually spawned other frameworks on top of backbone (e.g. option to roll your own). Also from what I have read Angular is still evloving so fast that newer versions are breaking backwards compatibility (e.g. version 1.X -> 2.0). Don't discount Ember.js either, it's probably the closest to MVC that server side folks understand.

#3 : Sorry, very busy father.

#X : PLEASE consider incorporating support for more than just SQL backends, and look at a data model on top of a NoSQL backend such as Cassandra or MongoDB. We leverage Cassandra more and more every day for multi-site application deployments (mostly in node.js) and having state persisted between data centers. It would be fantastic to see applications like RC start looking at the future of geo-distributed services and support those of us that host managed services that require 100% uptime across geographies.

Congrats on the 10 year anniversary!

12345678901234567890123456789012345678901234567890123456789012345678901234567890

Hi Thomas, great to see such an effort. Although roundcube works well, I would agree with you from what you've said in an earlier blog post that the rc codebase could use modernisation. Like you, Ive often though.. 'if only…'.

Some comments.

• PHP

The major disadvantage of PHP that has become evident in Roundcube is its short process lifetime requiring a full startup, database and IMAP connection+authentication and closing all connections again for every single HTTP request. Even FPM and persistent socket connections (which IMO are still unreliable and should be avoided) don't really help to overcome this.

The advantage of sticking with PHP certainly is that we can re-use some of the existing parts from the Roundcube codebase, such as the IMAP library, the plugin infrastructure etc. And in terms of deployment: PHP is everywhere and lots of sysadmins are familiar with deploying and maintaing and tweaking PHP applications.

• Python

Although Python isn't perfect for running as long-term processes either, there are some well developed frameworks out there (e.g. Twisted or Flask) which could be used to manage persistent connections between the client (think websockets) and storage backends. In addition to that, its easier to implement reactive applications with an asynchronous architecture.

Regarding deployment: the Python universe has some well established package repositories which make installation and deployment pretty easy. From a sysadmin point of view, I have no idea how easy or painful the maintenance of Python applications is. But the number of hosting providers offering Python is certainly lower that for PHP.

• Node.js

The new star on the web app sky. Its architecture seems to make node.js perfect for long-living server processes and maximum throughput. Although we could use Javascript, which we're all familiar with, the number of pitfalls to use that for server applications is quite high. The learning curve here shouldn't be underestimated.

And opinions certainly go into two distinct directions when in comes to deploying and scaling node applications. Installation should come easy as there's basically only npm for all the package management and everybody doing node is familiar with that.

OK, so question #1: what's your take on the programming language choice?

I would not switch from PHP to any other similar (scripting) language like Python, Ruby or Perl. PHP can hold its own, you just have to embrace modern PHP.

You can debate the pros and cons of each for ever, and never even agree, but in the end, it really doesnt matter that much in the context of roundcube. Id say the benefit of already being a PHP project outweighs most arguments

Now, node.js is a different story. I can see the benefits in theory. Or even Go, although thats probably a really big step to take. Im not sure if the benefits outweigh switching, I havent used node or go myself.

So here's question #2: what are your favorite javascript application libraries and frameworks? What experience did you make let's say with Angular or Backbone? What are the pitfalls?

I havent used any JS frameworks, but what i read doesnt really inspire confidence that they'll be a decade long stable platform. Which i think should be the primary goal, above anything else. Look at the Angular 1->2 debacle. Can you imagine having to rewrite all of the RC UI, which will be the bulk of the code in RCN every time some framework decides to break backwards compatibility. Or worse, you end up in an unsupported EOL version of said platform. Maybe backbone is different, and id love to hear what people think about the longevity of these frameworks.

And this is question #3: who's willing to help us draft the perfect protocol for this? I guess this should be a highly collaborative and iterative process and I'm keen on any input from people who are experienced with this type of task.

For most people I think this depends on what route RCN decides to take. Although it certainly is possible to contribute if you arent proficient in a specific language or framework, it sure helps.

Cor

I know I'm a bit late to the conversation but I still wanted to add my two cents. While I think a fresh start for Roundcube is a good idea I think its still a bit premature to be talking about what should be used to develop the new system when there isn't a clear set of goals for what Roundcube Next should include. So far the only goals I've seen laid out are:

1. Move to a flexible framework to reduce Roundcube specific framework code
2. Move more code like templates to the client side
3. Move to a persistent based data connection

Those are all good development goals but in my mind that isn't necessarily enough to validate re-coding the whole system. While its been touched on we don't have any written goals like adding multi-account support, GPG/PGP encryption support, or any of the other long outstanding feature requests[1] that aren't possible with the current system. Now seems like the perfect time to define what functionality we want out of a new system so we can make better decisions about what frameworks and languages to use.

Now on to the questions at hand.

#1.

I think the question that needs to be asked is: Who is the target audience of Roundcube Next going to be? That in my mind has a profound effect on what language is used since it will limit who can simply setup and install Roundcube without advanced knowledge. Because Roundcube currently uses PHP it has allowed almost anyone to grab a copy of Roundcube and set it up on any shared host, bundle it with another system that has PHP like cPanel or to go all out and do a custom install that supports hundreds of thousands of users. That flexibility has allowed Roundcube to become ubiquitous as a webmail option. With that in mind I'll layout my opinions on each language.

PHP - While PHP is very flexible and could allow the majority of the current Roundcube users to continue using Roundcube with the next version it handles persistent connections badly. While it is improving I don't know that I'd want to bet a rebuild on it. To make matters worse all of the approaches to handling persistent connections that I've seen require a server to be setup in a special way which could make it even harder for people to be able to setup a Roundcube system that would run along side other applications.

Python - I know the least about Python out of the three language choices so I can't really comment on its capabilities. I think its fair to say that using Python would limit Roundcube to users with advanced server knowledge to be able to setup the correct Python environment.

Node.js - Unlike PHP Node is very good at handling persistent connections and would allow Roundcube a lot of flexibility since the web server could be written for the exact needs of the new system. One of my concerns with using Node is performance, while Node performs well it requires a lot of resources to do it. Some of the Node applications I've seen remind me of the early days of Roundcube where Apache would max out or hang on IMAP connections causing performance bottlenecks. While Node.js isn't as common as PHP I'm seeing it showing up in more places and I could see it becoming more ubiquitous in the coming years. In my mind Node.js is the best candidate if Roundcube Next is to have any form of persistent connection support.

#2

If Roundcube Next is going to have the overhead of persistent connections with a server side data model I would like the ability to install just the server side of Roundcube and use the data model as an API to build mail support into other applications without having to use any of the Roundcube frontend code. The data model IMO needs to sit on top of standard IMAP without any special mail server requirements, but it would be neat if parts of the data model could be switched out for special mail server components when available for better performance. As for frontend libraries I haven't had a chance to be able to play with enough of them to have any useful insight to what is best.

#3

I would love to help however possible! Although I'm not sure what the best way to collaborate on something like a data model would be, I think the first step is to defined the features that we want to include in the new system.

[1] http://trac.roundcube.net/report/10?sort=created&asc=1&page=1

Thanks, -Scott Kidder

On Mar 6, 2015, at 10:10 AM, Thomas Bruederli wrote:

Dear developers and list lurkers

The Roundcube project will celebrate its 10th anniversary this year and a decade is just right to stop for a while and let your thoughts flow into new directions. Some of you may already have read my random thoughts [1] about the future or Roundcube development and since I wrote this down, the idea of a fresh new successor to Roundcube has settled in my mind and won't go away that easily. The initial responses on that were exclusively positive and this encourages us to take it one step further now. Meanwhile I looked into many other applications, libraries, frameworks, apis and programming languages to find the best set of tools that would form a new top-noch webmail application for the next decade. A loose and admittedly rather limited collection has manifested on a wiki page [2] already.

The primary goal of rethinking Roundcube and webmail in general is certainly to improve the user experience for everyday communication. The second important aspect (at least for me as a developer) is to have a slick, structured and flexible framework to build new features and applications on top of. The current Roundcube codebase comes with a variety of technological and conceptual constraints which rule the ongoing development in a rather negative manner. Therefore thinking of a successor that is completely independent and unrelated to the current software stack is a tempting exercise which I'd like to invite you to.

As already outlined in the blog post [1], the new Roundcube should move most of the application logic from the server to the client. Browsers are way more capable than they've been 10 years ago and times of server-side template rendering are definitely over. Those who are familiar with the Roundcube codebase will immediately understand that such a change in paradigms also means a complete break of compatibility with all existing components of Roundcube, including plugins.

So let's free our mind from what we know about Roundcube and start over. This can even go as far as choosing something else than PHP for the server part which is supposed to be reduced to a simple wrapper for data exchange between the backend storage (IMAP et al) and the client application. Possible candidates are PHP (still), Python, Node.js. At least these are the ones I'm currently familiar with. So let's take a deeper look into these as I'm particularly interested in your opinions, also from a deployment and hosting perspective.

• PHP

The major disadvantage of PHP that has become evident in Roundcube is its short process lifetime requiring a full startup, database and IMAP connection+authentication and closing all connections again for every single HTTP request. Even FPM and persistent socket connections (which IMO are still unreliable and should be avoided) don't really help to overcome this.

The advantage of sticking with PHP certainly is that we can re-use some of the existing parts from the Roundcube codebase, such as the IMAP library, the plugin infrastructure etc. And in terms of deployment: PHP is everywhere and lots of sysadmins are familiar with deploying and maintaing and tweaking PHP applications.

• Python

Although Python isn't perfect for running as long-term processes either, there are some well developed frameworks out there (e.g. Twisted or Flask) which could be used to manage persistent connections between the client (think websockets) and storage backends. In addition to that, its easier to implement reactive applications with an asynchronous architecture.

Regarding deployment: the Python universe has some well established package repositories which make installation and deployment pretty easy. From a sysadmin point of view, I have no idea how easy or painful the maintenance of Python applications is. But the number of hosting providers offering Python is certainly lower that for PHP.

• Node.js

The new star on the web app sky. Its architecture seems to make node.js perfect for long-living server processes and maximum throughput. Although we could use Javascript, which we're all familiar with, the number of pitfalls to use that for server applications is quite high. The learning curve here shouldn't be underestimated.

And opinions certainly go into two distinct directions when in comes to deploying and scaling node applications. Installation should come easy as there's basically only npm for all the package management and everybody doing node is familiar with that.

OK, so question #1: what's your take on the programming language choice?

The client part luckily doesn't need such a decision as there's only Javascript for this. But the variety of ways how to build a single-page web application and the number of frameworks helping you to do this is mind-boggling. I have to admit that in this regard, I'm one of the old fashioned web developers from the late 90ies and yet have to learn a lot about the brave new world of web development. I guess we can all agree that nobody wants to build a new web application framework from scratch as we did for Roundcube due to the lack of sophisticated solutions back in 2005. But choosing the right tools might be essential for the timeline and the success of Roundcube Next.

So here's question #2: what are your favorite javascript application libraries and frameworks? What experience did you make let's say with Angular or Backbone? What are the pitfalls?

Finally another big challenge we have when aiming to create the perfect webmail application is the way we make the client and the server talk together. We should carefully draft a proper data model and an according protocol for this. For the data model, I already tried to create a first draft in regards of what we already know the application should be capable of. This includes multi-account support, tagging, sharing and the extensibility for calendaring and other groupware-like tasks.

For the client-server communication, there are some interesting concepts out there but none of them actually seems to suit the purpose we're after. They either lack the concept of sharing (i.e. ACL) or do not account for collections such as multiple different address books one account my have access to. The candidates I'm referring to are: Nilas.com (former InboxApp), JMAP (by Fastmail) or the Gmail API (also to be supported by Dovecot). All are good approaches and generally point into the same direction but I couldn't say that one of them would already solve the use-cases we want to cover in Roundcube. And yes, we're seeking for a protocol and not an API [3] :-)

And this is question #3: who's willing to help us draft the perfect protocol for this? I guess this should be a highly collaborative and iterative process and I'm keen on any input from people who are experienced with this type of task.

Thank you all for reading and participating in this survey. I'm really excited to move this forward. All free and open source of course!

Best, Thomas

[1] https://roundcubeinbox.wordpress.com/2014/09/12/roundcube-next-if-we-would-s... [2] http://trac.roundcube.net/wiki/Roundcube_Next [3] https://vimeo.com/71278954 _______________________________________________ Roundcube Development discussion mailing list dev@lists.roundcube.net http://lists.roundcube.net/mailman/listinfo/dev

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512

Hello,

Thanks for making such awesome project available to the rest of the world ;)

I love the idea and agree on the server-client architecture part, but building such complex software one should also think about scalability. I currently have no idea, who are the main users of RC, but we should assume that the new solution should work flawlessly with thousands of mailboxes accessed simultaneously.

NB: Since I'm a sysadmin for a hosting company I'm writing everything from the hosting provider view.

NBB: I might sometimes use a bit of imperious/authoritarian phraseology (should, must etc), so please don't be offended, it's just like I write

• • probably from reading too much RFCs :)

NBBB: Also, I have ended a some sentences with question mark, I don't want answers for those "questions", these are more like parts "what if" style things ergo my brain going into cosmos of ideas.

### Back-end

To be honest, connection caching isn't really an issue for me, there are utils specifically designed for that (local imapproxy), so are there good ways to fasten the authentication process (local Nginx IMAP proxy with HTTP authentication to Nginx http server with custom auth script, makes it also possible to load balance). Yes of course it would be better, if back-end could do those things, but the question is, that can we do it better and with less performance and development time cost than existing software? Or instead it we could use this time to make a documentation how create a good performance RC setup using the existing technologies?

Creating such complex back-end one also should take in mind that it should be possible to load balance it - run multiple back-end servers which communicate with each other. Keep in mind, that in the future the backend could be directly used by mobile apps, desktop clients, user made scripts, or even by your refrigerator - that's one of the point of making separate back-end server.

Back end should be extendable by plug-ins and this raises a lot of questions. Someone just don't want the back-end with by default calendar support. Some even don't use address-books. Maybe someone wants to make a plug-in to create friends list, how will actions and data described for the protocol for that plug-in? How strict will be? How to avoid protocol conflicts?

Also, as for data caching (eg Redis) one should think about the stored data. Is it mostly key-value based or does it require complex relations 1? What data will be cached?

### Back-end language

OK, so question #1: what's your take on the programming language choice?

The eternal question of which language is the best.

PHP has good enough performance, sorry to disappoint You, but "good enough" just don't do on large systems. You have to have best performance You can get. Also, my personal opinion is (as a long time developer), that if I'd had a time machine the programming language which I would make disappear is PHP (next one will be of course Javascript :).

I'd go with Python. Easy to write (and IMHO more supported than Node.js) and in my experience it runs long-term processes spotless unless You write it bad. Most *nix systems have it on by default as dependencies for providing any web services at all (same with Perl). Plug-ins could easily be installed as Python Packages using pip. Lots of well documented and stable libraries for network communication with protocol definitions. I have been PHP dev longer than Python dev and I'd prefer Python any time. Some say Python is hard to learn (yes, we have many PHP devs out there, more than Python's). IMHO a good dev can write any language and also, we can always make a good documentation.

Node.js - sorry, it might be good but first, it's javascript - who even uses javascript for back-end and second, it seems too hipster for old me. Usually needs a separate installation unless you are providing as part of hosting service.

Large number of hosting providers do not allow long lasting scripts for their clients (in virtual/shared hosting). Max execution times are usually about 5 minutes (10 mins max). Therefore running any server applications in virtual/shared hosting is not possible (most of common clients use shared hosting). So taking this in consideration the "PHP is everywhere" is quite useless in my opinion, since You probably need VPS for client-server application anyway and there You can install anything You want.

On the other hand, most of clients use the webmail provided by hosting provider (who has capabilities for complex set-ups).

### Front-end

For frameworks I would recommend Backbone.js for structure and data modeling and Twitter Bootstrap as UI toolkit.

Does keeping back-end and front-end separate would also mean that every plug-in developer must release the back-end and front-end version of their plug-in or will the protocol support UI data transportation (eg client will request UI data from back-end and then caches it)?

### Data model

Creating the data model should we take in consideration that maybe there are people, who only would like to use the company's global calendar and address-book without any e-mail? Just log in to see company's contacts. Maybe they want to add personal mailbox later? The model should be abstract as possible, but extendable by various plug-ins. Even authentication methods (IMAP, HTTP, Kerberos, government issued ID-card etc) should be provided as plug-ins.

The only thing that is certain, is that we have some abstract "client" (not a physical entity) what will have all the provided services (mailboxes, calendars, contacts, identities?, chat messaging system etc).

Every client can have one or more users (company "clients" can have more than one for example). Users with authentication and authorization methods defined by specific plug-ins designated to those actions and handled via strictly defined User API. We also should keep in mind that the required data should be as little as possible, since many auth protocols don't support mother maiden name or even address. Strictly data required for authentication and authorization.

### Protocol

For the sake of simplicity, I think that using HTTP protocol would be the easies way to go. Yes, it's overhead, but on the other side it's easy to handle and develop on. And for the architecture REST is a good way to go. Well documented, easy to use.

Best regards, Günter Kits

It's kind of a late reply, but still...

Some of you asked the question: Who is the intended user of Roundcube? Who is going to install it on his server, or who is going to provide it as an installable on his webhosting platform?

To find out which programming language suits best, there are probably several ways to find out what your intended users want/need. One of them might simply be to look at what is supported most. http://w3techs.com/technologies/overview/programming_language/all

The link above will tell you the same thing as any other similar websites will tell you: "PHP is used everywhere, the rest of the languages don't seem to matter". What might also be of interest, is that most of the websites use PHP 5.x

Moving away from PHP might result in that the new RC will no longer be the number 1 choice. Demand for the "old" (the current) version might even stay higher. Is it already decided how (and if) the "old" version will remain supported? and for how long?

Furthermore, there is already a lot of good and reusable and thoroughly tested code in the current version. It would be a waste to recode it all.

I really like Node.JS, but I think it would be a poor choice.

Martijn Thie