changeset 6153:d6942b2b421c

config: clarify that we only recommend and support single threaded operation Sad, but true. Especially because we reuse Repository instances between threads.
author Mads Kiilerich <madski@unity3d.com>
date Tue, 06 Sep 2016 00:51:18 +0200
parents 246ac058da2c
children e8bd3f074b4e
files development.ini docs/setup.rst docs/usage/performance.rst kallithea/bin/template.ini.mako kallithea/config/deployment.ini_tmpl kallithea/tests/test.ini
diffstat 6 files changed, 35 insertions(+), 18 deletions(-) [+]
line wrap: on
line diff
--- a/development.ini	Fri Aug 12 10:01:12 2016 +0200
+++ b/development.ini	Tue Sep 06 00:51:18 2016 +0200
@@ -70,16 +70,16 @@
 ## PASTE ##
 #use = egg:Paste#http
 ## nr of worker threads to spawn
-#threadpool_workers = 5
+#threadpool_workers = 1
 ## max request before thread respawn
-#threadpool_max_requests = 10
+#threadpool_max_requests = 100
 ## option to use threads of process
 #use_threadpool = true
 
 ## WAITRESS ##
 use = egg:waitress#main
 ## number of worker threads
-threads = 5
+threads = 1
 ## MAX BODY SIZE 100GB
 max_request_body_size = 107374182400
 ## use poll instead of select, fixes fd limits, may not work on old
--- a/docs/setup.rst	Fri Aug 12 10:01:12 2016 +0200
+++ b/docs/setup.rst	Tue Sep 06 00:51:18 2016 +0200
@@ -785,8 +785,7 @@
 
 .. code-block:: apache
 
-    WSGIDaemonProcess kallithea \
-        threads=4 \
+    WSGIDaemonProcess kallithea processes=5 threads=1 maximum-requests=100 \
         python-home=/srv/kallithea/venv
     WSGIProcessGroup kallithea
     WSGIScriptAlias / /srv/kallithea/dispatch.wsgi
@@ -796,7 +795,7 @@
 
 .. code-block:: apache
 
-    WSGIDaemonProcess kallithea threads=4
+    WSGIDaemonProcess kallithea processes=5 threads=1 maximum-requests=100
     WSGIProcessGroup kallithea
     WSGIScriptAlias / /srv/kallithea/dispatch.wsgi
     WSGIPassAuthorization On
--- a/docs/usage/performance.rst	Fri Aug 12 10:01:12 2016 +0200
+++ b/docs/usage/performance.rst	Tue Sep 06 00:51:18 2016 +0200
@@ -31,8 +31,26 @@
 
     Scaling horizontally can give huge performance benefits when dealing with
     large amounts of traffic (many users, CI servers, etc.). Kallithea can be
-    scaled horizontally on one (recommended) or multiple machines. In order
-    to scale horizontally you need to do the following:
+    scaled horizontally on one (recommended) or multiple machines.
+
+    It is generally possible to run WSGI applications multithreaded, so that
+    several HTTP requests are served from the same Python process at once. That
+    can in principle give better utilization of internal caches and less
+    process overhead.
+    
+    One danger of running multithreaded is that program execution becomes much
+    more complex; programs must be written to consider all combinations of
+    events and problems might depend on timing and be impossible to reproduce.
+
+    Kallithea can't promise to be thread-safe, just like the embedded Mercurial
+    backend doesn't make any strong promises when used as Kallithea uses it.
+    Instead, we recommend scaling by using multiple server processes.
+
+    Web servers with multiple worker processes (such as ``mod_wsgi`` with the
+    ``WSGIDaemonProcess`` ``processes`` parameter) will work out of the box.
+
+    In order to scale horizontally on multiple machines, you need to do the
+    following:
 
     - Each instance's ``data`` storage needs to be configured to be stored on a
       shared disk storage, preferably together with repositories. This ``data``
@@ -40,7 +58,7 @@
       task locking (so it is safe across multiple instances). Set the
       ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir``
       variables in each .ini file to a shared location across Kallithea instances
-    - If celery is used each instance should run a separate Celery instance, but
+    - If using several Celery instances,
       the message broker should be common to all of them (e.g.,  one
       shared RabbitMQ server)
     - Load balance using round robin or IP hash, recommended is writing LB rules
--- a/kallithea/bin/template.ini.mako	Fri Aug 12 10:01:12 2016 +0200
+++ b/kallithea/bin/template.ini.mako	Tue Sep 06 00:51:18 2016 +0200
@@ -65,9 +65,9 @@
 <%text>## PASTE ##</%text>
 use = egg:Paste#http
 <%text>## nr of worker threads to spawn</%text>
-threadpool_workers = 5
+threadpool_workers = 1
 <%text>## max request before thread respawn</%text>
-threadpool_max_requests = 10
+threadpool_max_requests = 100
 <%text>## option to use threads of process</%text>
 use_threadpool = true
 
@@ -75,7 +75,7 @@
 <%text>## WAITRESS ##</%text>
 use = egg:waitress#main
 <%text>## number of worker threads</%text>
-threads = 5
+threads = 1
 <%text>## MAX BODY SIZE 100GB</%text>
 max_request_body_size = 107374182400
 <%text>## use poll instead of select, fixes fd limits, may not work on old</%text>
--- a/kallithea/config/deployment.ini_tmpl	Fri Aug 12 10:01:12 2016 +0200
+++ b/kallithea/config/deployment.ini_tmpl	Tue Sep 06 00:51:18 2016 +0200
@@ -65,16 +65,16 @@
 ## PASTE ##
 #use = egg:Paste#http
 ## nr of worker threads to spawn
-#threadpool_workers = 5
+#threadpool_workers = 1
 ## max request before thread respawn
-#threadpool_max_requests = 10
+#threadpool_max_requests = 100
 ## option to use threads of process
 #use_threadpool = true
 
 ## WAITRESS ##
 use = egg:waitress#main
 ## number of worker threads
-threads = 5
+threads = 1
 ## MAX BODY SIZE 100GB
 max_request_body_size = 107374182400
 ## use poll instead of select, fixes fd limits, may not work on old
--- a/kallithea/tests/test.ini	Fri Aug 12 10:01:12 2016 +0200
+++ b/kallithea/tests/test.ini	Tue Sep 06 00:51:18 2016 +0200
@@ -68,16 +68,16 @@
 ## PASTE ##
 #use = egg:Paste#http
 ## nr of worker threads to spawn
-#threadpool_workers = 5
+#threadpool_workers = 1
 ## max request before thread respawn
-#threadpool_max_requests = 10
+#threadpool_max_requests = 100
 ## option to use threads of process
 #use_threadpool = true
 
 ## WAITRESS ##
 use = egg:waitress#main
 ## number of worker threads
-threads = 5
+threads = 1
 ## MAX BODY SIZE 100GB
 max_request_body_size = 107374182400
 ## use poll instead of select, fixes fd limits, may not work on old