Mercurial > kallithea
changeset 6762:716e53c085ff stable
config: clarify that we only recommend and support single threaded operation
Sad, but true. Especially because we reuse Repository instances between
threads.
| author | Mads Kiilerich <madski@unity3d.com> |
|---|---|
| date | Tue, 06 Sep 2016 00:51:18 +0200 |
| parents | 25dae19e0719 |
| children | c25392980f91 |
| files | development.ini docs/setup.rst docs/usage/performance.rst kallithea/bin/template.ini.mako kallithea/config/deployment.ini_tmpl kallithea/tests/test.ini |
| diffstat | 6 files changed, 35 insertions(+), 18 deletions(-) [+] |
line wrap: on
line diff
--- a/development.ini Thu Jul 28 16:28:34 2016 +0200 +++ b/development.ini Tue Sep 06 00:51:18 2016 +0200 @@ -71,16 +71,16 @@ ## PASTE ## #use = egg:Paste#http ## nr of worker threads to spawn -#threadpool_workers = 5 +#threadpool_workers = 1 ## max request before thread respawn -#threadpool_max_requests = 10 +#threadpool_max_requests = 100 ## option to use threads of process #use_threadpool = true ## WAITRESS ## use = egg:waitress#main ## number of worker threads -threads = 5 +threads = 1 ## MAX BODY SIZE 100GB max_request_body_size = 107374182400 ## use poll instead of select, fixes fd limits, may not work on old
--- a/docs/setup.rst Thu Jul 28 16:28:34 2016 +0200 +++ b/docs/setup.rst Tue Sep 06 00:51:18 2016 +0200 @@ -735,8 +735,7 @@ .. code-block:: apache - WSGIDaemonProcess kallithea \ - threads=4 \ + WSGIDaemonProcess kallithea processes=5 threads=1 maximum-requests=100 \ python-home=/srv/kallithea/venv WSGIProcessGroup kallithea WSGIScriptAlias / /srv/kallithea/dispatch.wsgi @@ -746,7 +745,7 @@ .. code-block:: apache - WSGIDaemonProcess kallithea threads=4 + WSGIDaemonProcess kallithea processes=5 threads=1 maximum-requests=100 WSGIProcessGroup kallithea WSGIScriptAlias / /srv/kallithea/dispatch.wsgi WSGIPassAuthorization On
--- a/docs/usage/performance.rst Thu Jul 28 16:28:34 2016 +0200 +++ b/docs/usage/performance.rst Tue Sep 06 00:51:18 2016 +0200 @@ -35,8 +35,26 @@ Scaling horizontally can give huge performance benefits when dealing with large amounts of traffic (many users, CI servers, etc.). Kallithea can be - scaled horizontally on one (recommended) or multiple machines. In order - to scale horizontally you need to do the following: + scaled horizontally on one (recommended) or multiple machines. + + It is generally possible to run WSGI applications multithreaded, so that + several HTTP requests are served from the same Python process at once. That + can in principle give better utilization of internal caches and less + process overhead. + + One danger of running multithreaded is that program execution becomes much + more complex; programs must be written to consider all combinations of + events and problems might depend on timing and be impossible to reproduce. + + Kallithea can't promise to be thread-safe, just like the embedded Mercurial + backend doesn't make any strong promises when used as Kallithea uses it. + Instead, we recommend scaling by using multiple server processes. + + Web servers with multiple worker processes (such as ``mod_wsgi`` with the + ``WSGIDaemonProcess`` ``processes`` parameter) will work out of the box. + + In order to scale horizontally on multiple machines, you need to do the + following: - Each instance needs its own .ini file and unique ``instance_id`` set. - Each instance's ``data`` storage needs to be configured to be stored on a @@ -45,7 +63,7 @@ task locking (so it is safe across multiple instances). Set the ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir`` variables in each .ini file to a shared location across Kallithea instances - - If celery is used each instance should run a separate Celery instance, but + - If using several Celery instances, the message broker should be common to all of them (e.g., one shared RabbitMQ server) - Load balance using round robin or IP hash, recommended is writing LB rules
--- a/kallithea/bin/template.ini.mako Thu Jul 28 16:28:34 2016 +0200 +++ b/kallithea/bin/template.ini.mako Tue Sep 06 00:51:18 2016 +0200 @@ -66,9 +66,9 @@ <%text>## PASTE ##</%text> use = egg:Paste#http <%text>## nr of worker threads to spawn</%text> -threadpool_workers = 5 +threadpool_workers = 1 <%text>## max request before thread respawn</%text> -threadpool_max_requests = 10 +threadpool_max_requests = 100 <%text>## option to use threads of process</%text> use_threadpool = true @@ -76,7 +76,7 @@ <%text>## WAITRESS ##</%text> use = egg:waitress#main <%text>## number of worker threads</%text> -threads = 5 +threads = 1 <%text>## MAX BODY SIZE 100GB</%text> max_request_body_size = 107374182400 <%text>## use poll instead of select, fixes fd limits, may not work on old</%text>
--- a/kallithea/config/deployment.ini_tmpl Thu Jul 28 16:28:34 2016 +0200 +++ b/kallithea/config/deployment.ini_tmpl Tue Sep 06 00:51:18 2016 +0200 @@ -66,16 +66,16 @@ ## PASTE ## #use = egg:Paste#http ## nr of worker threads to spawn -#threadpool_workers = 5 +#threadpool_workers = 1 ## max request before thread respawn -#threadpool_max_requests = 10 +#threadpool_max_requests = 100 ## option to use threads of process #use_threadpool = true ## WAITRESS ## use = egg:waitress#main ## number of worker threads -threads = 5 +threads = 1 ## MAX BODY SIZE 100GB max_request_body_size = 107374182400 ## use poll instead of select, fixes fd limits, may not work on old
--- a/kallithea/tests/test.ini Thu Jul 28 16:28:34 2016 +0200 +++ b/kallithea/tests/test.ini Tue Sep 06 00:51:18 2016 +0200 @@ -70,16 +70,16 @@ ## PASTE ## #use = egg:Paste#http ## nr of worker threads to spawn -#threadpool_workers = 5 +#threadpool_workers = 1 ## max request before thread respawn -#threadpool_max_requests = 10 +#threadpool_max_requests = 100 ## option to use threads of process #use_threadpool = true ## WAITRESS ## use = egg:waitress#main ## number of worker threads -threads = 5 +threads = 1 ## MAX BODY SIZE 100GB max_request_body_size = 107374182400 ## use poll instead of select, fixes fd limits, may not work on old