Hello,
Like most web applications, roundcube is intended to be used on a
single machine.
In a high load environment, simple LVS source ip balancing may not be
enough to simply dispatch evenly end users to several roundcube
instances.
I happen to have a situation where many customers (multiple 10s if not
100s) are logging in all at once and using the same proxy server,
hence all hitting the same backend roundcube machine.
It may be possible to get around this problem using a shared DB for
the session and use an NFS partition for the upload folder, but it
then creates single points of failure (and I happen to already have
some nginx reverse proxy installed).
In order to perform per user balancing of request, ticket 1485721 [1]
contains a small patch [2] to roundcube to add a new cookie to the
roundcube session.
This cookie can be used by nginx (for example) to perform a layer 7
routing of the request, an example of such configuration is attached
to the ticket [3]
This patch is used in production and is as harmless as harmless can
be. It does not change anything used by roundcube itself.
I would be interrested in hearing any comment the list may have about
it.
Another benefit for me is that it allowed me to go from an active/ passive lvs setup to an active/active one (using wackamole [4])
Regards,
Thomas Mangin
[1] http://trac.roundcube.net/ticket/1485721 [2] http://trac.roundcube.net/attachment/ticket/1485721/roundcube-nginx-cluster.... [3] http://trac.roundcube.net/attachment/ticket/1485721/nginx-config-example.txt [4] http://packages.ubuntu.com/hardy/net/wackamole _______________________________________________ List info: http://lists.roundcube.net/dev/
Thomas Mangin wrote:
Hello,
Like most web applications, roundcube is intended to be used on a
single machine.
This is not to discount the validity of your points, but I would like to point out that roundcube (svn trunk) is ready to be used on more than a single web server.
Using the database attachments plugin recently merged into head in combination with the already existing database backed sessions, we are able to run one instance of roundcube across many machines with DNS load balancing.
In a high load environment, simple LVS source ip balancing may not be
enough to simply dispatch evenly end users to several roundcube
instances. I happen to have a situation where many customers (multiple 10s if not
100s) are logging in all at once and using the same proxy server,
hence all hitting the same backend roundcube machine.It may be possible to get around this problem using a shared DB for
the session and use an NFS partition for the upload folder, but it
then creates single points of failure (and I happen to already have
some nginx reverse proxy installed).In order to perform per user balancing of request, ticket 1485721 [1]
contains a small patch [2] to roundcube to add a new cookie to the
roundcube session. This cookie can be used by nginx (for example) to perform a layer 7
routing of the request, an example of such configuration is attached
to the ticket [3]This patch is used in production and is as harmless as harmless can
be. It does not change anything used by roundcube itself.I would be interrested in hearing any comment the list may have about
it.Another benefit for me is that it allowed me to go from an active/ passive lvs setup to an active/active one (using wackamole [4])
Regards,
Thomas Mangin
[1] http://trac.roundcube.net/ticket/1485721 [2] http://trac.roundcube.net/attachment/ticket/1485721/roundcube-nginx-cluster.... [3] http://trac.roundcube.net/attachment/ticket/1485721/nginx-config-example.txt [4] http://packages.ubuntu.com/hardy/net/wackamole _______________________________________________ List info: http://lists.roundcube.net/dev/
List info: http://lists.roundcube.net/dev/
Hello Ziba,
This is not to discount the validity of your points, but I would
like to point out that roundcube (svn trunk) is ready to be used on more
than a single web server.
Thank you for the information. I had not noticed this feature and
assumed that attachments would cause a problem with load balancing. I
would still prefer not to put the sessions and attachments in a DB but
I can see how this is a good way to cluster roundcube.
Using the database attachments plugin recently merged into head in combination with the already existing database backed sessions, we are able to run one instance of roundcube across many machines with DNS
load balancing.
DNS balancing is a good solution but I still believe that the patch -
and cookie based routing - has its advantages.
- You can force new sessions on the machines with the lowest load/
number of connection, DNS load balancing does not provide as much
control over the repartition of the load.
- Not all DNS will round-robin multiple A records ( dnscache does
not for example ) so you may still find one of your machine getting
most of the load if you do not control your users.
Thomas _______________________________________________ List info: http://lists.roundcube.net/dev/
On Thursday 07 May 2009 00:08:34 Thomas Mangin wrote:
(and I happen to already have some nginx reverse proxy installed).
to make the most out of your reverse proxies the directives/info in ticket 1485800 will probably help.
Daniel Black
Email/List Administrator CAcert _______________________________________________ List info: http://lists.roundcube.net/dev/
On Wed, May 6, 2009 at 6:06 PM, Thomas Mangin thomas.mangin@exa-networks.co.uk wrote:
Hello Ziba,
This is not to discount the validity of your points, but I would like to point out that roundcube (svn trunk) is ready to be used on more than a single web server.
Thank you for the information. I had not noticed this feature and assumed that attachments would cause a problem with load balancing. I would still prefer not to put the sessions and attachments in a DB but I can see how this is a good way to cluster roundcube.
Using the database attachments plugin recently merged into head in combination with the already existing database backed sessions, we are able to run one instance of roundcube across many machines with DNS load balancing.
DNS balancing is a good solution but I still believe that the patch - and cookie based routing - has its advantages. (...)
I think DNS is just the "cheapest" in most cases. Besides you should probably use a "real" load balancer (like HAProxy or Pound, or maybe even hardware) to spread out. And because of the availability of those tools/solutions, I wouldn't necessarily wrap that part into the application as well.
I don't want to talk you into another solution, but e.g. if one of your backends fails, the user will be send to it regardless. Round-robin doesn't safe guard against that, but e.g. a load balancer would. Pushing attachments and sessions into the database gives you lots of flexibility (with a trade-off in terms of resources).
But anyway, did you progress with the plugin api?
Till _______________________________________________ List info: http://lists.roundcube.net/dev/
Hi Till,
Thank you for taking the time to read my mail.
I think DNS is just the "cheapest" in most cases.
DNS is simpler to put in place.
Besides you should probably use a "real" load balancer (like HAProxy or Pound, or maybe even hardware) to spread out.
HAProxy or Pound (or even Varnish [1]) are no more "real load
balancer" than NGINX.
And I see no need to invest in an "hardware solution" (ie: which is
just a nice way to say an expensive hardware + software bundle) when I
have everything I need in the OSS landscape.
And because of the availability of those tools/solutions, I wouldn't necessarily wrap that part into the application as well.
I explained previously why source IP based load balancing is not
enough in my case and why I _had_ to use a round robin algorithm and
cookie solution to spread the load.
I can see how the new DB code is a simpler solution but AFAIK it did
not exist when I first presented my patch three months ago.
I was able to load balancing roundcube a few months ago after only a
few hours work, well before the SQL sharing code was in place.
I don't want to talk you into another solution, but e.g. if one of your backends fails, the user will be send to it regardless.
Yes existing connection could routed to a dead server if you have
nothing to pervent it.
One could use wackamole[2] on the roundcube servers to prevent it (the
patch plan for it).
Round-robin doesn't safe guard against that, but e.g. a load balancer would.
I assume you meant "e.g. load balancer with roundcube using a shared
DB would" :)
I was not focusing on high availability - like the need of DB
replication, etc.
Pushing attachments and sessions into the database gives you lots of flexibility (with a trade-off in terms of resources).
It is another design, do not get me wrong it is a good solution but
not what _I_ need
But anyway, did you progress with the plugin api?
I explained why I thought it was not possible. Do you think I am
mistaken ?
Thomas
[1] http://varnish.projects.linpro.no/ [2] http://www.backhand.org/wackamole/
List info: http://lists.roundcube.net/dev/
On Fri, May 8, 2009 at 9:48 AM, Thomas Mangin thomas.mangin@exa-networks.co.uk wrote:
Hi Till,
Thank you for taking the time to read my mail. (...)
Besides you should probably use a "real" load balancer (like HAProxy or Pound, or maybe even hardware) to spread out.
HAProxy or Pound (or even Varnish [1]) are no more "real load balancer" than NGINX. And I see no need to invest in an "hardware solution" (ie: which is just a nice way to say an expensive hardware + software bundle) when I have everything I need in the OSS landscape.
Unless you use the fair-loadbalancer-module in nginx -- and even that one has a couple minor issues -- nginx based load balancing is no better than DNS.
The difference is that a "real" loadbalancer like pound or haxproxy would route a request to the backend that has the least load and also provide what is called "stickyness" (which is basically what your patch does) and at the same time providing the failover in case the backend is overloaded or becomes unavailable.
(...)
Round-robin doesn't safe guard against that, but e.g. a load balancer would.
I assume you meant "e.g. load balancer with roundcube using a shared DB would" :)
Yeah, a loadbalancer and a shared database.
I was not focusing on high availability - like the need of DB replication, etc.
Did I mention it?
But anyway, did you progress with the plugin api?
I explained why I thought it was not possible. Do you think I am mistaken ?
http://trac.roundcube.net/wiki/Plugin_Hooks
The bottom line is to use the plugin hooks. E.g. use login_after to set your cookie. If you happen to need another hook, just let us know, and we'll roll it in. :-)
Till _______________________________________________ List info: http://lists.roundcube.net/dev/
Hi Till,
Unless you use the fair-loadbalancer-module in nginx -- and even that one has a couple minor issues -- nginx based load balancing is no better than DNS.
The difference is that a "real" loadbalancer like pound or haxproxy would route a request to the backend that has the least load and also provide what is called "stickyness" (which is basically what your patch does) and at the same time providing the failover in case the backend is overloaded or becomes unavailable.
I knew apache could proxy with "sticky session", I did not notice
Pound had that feature, so I will have recall my previous mail :D
It would seems that my path of least resistance is to simply replace
nginx by pound for http/https proxying and just keep it for imap/pop
load dispatching.
I hate to be shown wrong :D
http://trac.roundcube.net/wiki/Plugin_Hooks
The bottom line is to use the plugin hooks. E.g. use login_after to set your cookie. If you happen to need another hook, just let us know, and we'll roll it in. :-)
Thank you for the time you took to answer my email and clarify your
statements.
Should a configuration of pound for such deployment be added to the
wiki ?
(coming back to the point made about Daniel about helping roundcube
users with their deployment).
Regards,
Thomas _______________________________________________ List info: http://lists.roundcube.net/dev/
On Fri, May 8, 2009 at 1:12 PM, Thomas Mangin thomas.mangin@exa-networks.co.uk wrote:
Hi Till,
Unless you use the fair-loadbalancer-module in nginx -- and even that one has a couple minor issues -- nginx based load balancing is no better than DNS.
The difference is that a "real" loadbalancer like pound or haxproxy would route a request to the backend that has the least load and also provide what is called "stickyness" (which is basically what your patch does) and at the same time providing the failover in case the backend is overloaded or becomes unavailable.
I knew apache could proxy with "sticky session", I did not notice Pound had that feature, so I will have recall my previous mail :D It would seems that my path of least resistance is to simply replace nginx by pound for http/https proxying and just keep it for imap/pop load dispatching. I hate to be shown wrong :D
No worries! :D I also enjoy being 'right' for once... At your expense! ;)
http://trac.roundcube.net/wiki/Plugin_Hooks
The bottom line is to use the plugin hooks. E.g. use login_after to set your cookie. If you happen to need another hook, just let us know, and we'll roll it in. :-)
Thank you for the time you took to answer my email and clarify your statements.
Should a configuration of pound for such deployment be added to the wiki ? (coming back to the point made about Daniel about helping roundcube users with their deployment).
Yeah, maybe we need a new 'performance' section. I'd welcome that very much! If you want to start it -- please go for it. :)
Have a great weekend!
Till _______________________________________________ List info: http://lists.roundcube.net/dev/
-
Daniel Black -
Thomas Mangin -
till -
Ziba Scott