kallithea: docs/usage/performance.rst annotate

annotate docs/usage/performance.rst @ 7611:9efcf6b78f71

dependencies: remove duplicate dependencies in dev_requirements.txt Babel and WebOb were present both in setup.py and dev_requirements.txt. When explicitly setting minimum dependencies in setup.py and running: pip install --upgrade -e . -r dev_requirements.txt python-ldap python-pam this duplication replaced useful diagnostic message from pip with less useful ones. For example, the following message was displayed when the Babel dependency duplication is present: kallithea 0.4.0rc1 has requirement Babel<2.7,==0.9.6, but you'll have babel 2.6.0 which is incompatible. When removing the duplication in dev_requirements.txt, this becomes: sphinx 1.7.9 has requirement babel!=2.0,>=1.3, but you'll have babel 0.9.6 which is incompatible. which makes it clear that to solve this problem, we need to bump the minimum dependency for Babel in setup.py from 0.9.6 to 1.3.

author	Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
date	Mon, 25 Mar 2019 20:57:39 +0100
parents	692dddf298e2
children	19af3fef3b34

rev	line source
2517 fa88997aa421 Added simple docs for optimizing RhodeCode performance Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	1 .. _performance:
fa88997aa421 Added simple docs for optimizing RhodeCode performance Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	2
fa88997aa421 Added simple docs for optimizing RhodeCode performance Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	3 ================================
5413 22a3fa3c4254 docs: cleanup of casing, markup and spacing of headings Mads Kiilerich <madski@unity3d.com> parents: 5400 diff changeset	4 Optimizing Kallithea performance
2517 fa88997aa421 Added simple docs for optimizing RhodeCode performance Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	5 ================================
fa88997aa421 Added simple docs for optimizing RhodeCode performance Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	6
6374 d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	7 When serving a large amount of big repositories, Kallithea can start performing
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	8 slower than expected. Because of the demanding nature of handling large amounts
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	9 of data from version control systems, here are some tips on how to get the best
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	10 performance.
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	11
2775 5d12768a0aa1 docs improvements Marcin Kuzminski <marcin@python-works.com> parents: 2680 diff changeset	12
6374 d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	13 Fast storage
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	14 ------------
2517 fa88997aa421 Added simple docs for optimizing RhodeCode performance Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	15
6374 d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	16 Kallithea is often I/O bound, and hence a fast disk (SSD/SAN) and plenty of RAM
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	17 is usually more important than a fast CPU.
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	18
5769 6afa528ee30e db: get rid of vcs_full_cache - it should always be used Mads Kiilerich <madski@unity3d.com> parents: 5534 diff changeset	19
6374 d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	20 Caching
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	21 -------
2517 fa88997aa421 Added simple docs for optimizing RhodeCode performance Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	22
6374 d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	23 Tweak beaker cache settings in the ini file. The actual effect of that is
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	24 questionable.
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	25
2517 fa88997aa421 Added simple docs for optimizing RhodeCode performance Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	26
6374 d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	27 Database
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	28 --------
3224 8b8edfc25856 whitespace cleanup Marcin Kuzminski <marcin@python-works.com> parents: 2775 diff changeset	29
6374 d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	30 SQLite is a good option when having a small load on the system. But due to
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	31 locking issues with SQLite, it is not recommended to use it for larger
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	32 deployments.
3224 8b8edfc25856 whitespace cleanup Marcin Kuzminski <marcin@python-works.com> parents: 2775 diff changeset	33
6374 d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	34 Switching to MySQL or PostgreSQL will result in an immediate performance
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	35 increase. A tool like SQLAlchemyGrate_ can be used for migrating to another
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	36 database platform.
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	37
2517 fa88997aa421 Added simple docs for optimizing RhodeCode performance Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	38
6374 d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	39 Horizontal scaling
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	40 ------------------
6153 d6942b2b421c config: clarify that we only recommend and support single threaded operation Mads Kiilerich <madski@unity3d.com> parents: 5880 diff changeset	41
6374 d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	42 Scaling horizontally means running several Kallithea instances and let them
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	43 share the load. That can give huge performance benefits when dealing with large
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	44 amounts of traffic (many users, CI servers, etc.). Kallithea can be scaled
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	45 horizontally on one (recommended) or multiple machines.
6216 49c82acd30b2 scripts: fix X mode on code formatting scripts Mads Kiilerich <madski@unity3d.com> parents: 6153 diff changeset	46
6374 d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	47 It is generally possible to run WSGI applications multithreaded, so that
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	48 several HTTP requests are served from the same Python process at once. That can
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	49 in principle give better utilization of internal caches and less process
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	50 overhead.
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	51
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	52 One danger of running multithreaded is that program execution becomes much more
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	53 complex; programs must be written to consider all combinations of events and
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	54 problems might depend on timing and be impossible to reproduce.
6153 d6942b2b421c config: clarify that we only recommend and support single threaded operation Mads Kiilerich <madski@unity3d.com> parents: 5880 diff changeset	55
6374 d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	56 Kallithea can't promise to be thread-safe, just like the embedded Mercurial
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	57 backend doesn't make any strong promises when used as Kallithea uses it.
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	58 Instead, we recommend scaling by using multiple server processes.
6153 d6942b2b421c config: clarify that we only recommend and support single threaded operation Mads Kiilerich <madski@unity3d.com> parents: 5880 diff changeset	59
6374 d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	60 Web servers with multiple worker processes (such as ``mod_wsgi`` with the
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	61 ``WSGIDaemonProcess`` ``processes`` parameter) will work out of the box.
6153 d6942b2b421c config: clarify that we only recommend and support single threaded operation Mads Kiilerich <madski@unity3d.com> parents: 5880 diff changeset	62
6374 d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	63 In order to scale horizontally on multiple machines, you need to do the
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	64 following:
3413 d79f3505549e whitespace cleanup Marcin Kuzminski <marcin@python-works.com> parents: 3390 diff changeset	65
4955 4e6dfdb3fa01 docs: English and consistency corrections Michael V. DePalatis <mike@depalatis.net> parents: 4192 diff changeset	66 - Each instance's ``data`` storage needs to be configured to be stored on a
4e6dfdb3fa01 docs: English and consistency corrections Michael V. DePalatis <mike@depalatis.net> parents: 4192 diff changeset	67 shared disk storage, preferably together with repositories. This ``data``
4e6dfdb3fa01 docs: English and consistency corrections Michael V. DePalatis <mike@depalatis.net> parents: 4192 diff changeset	68 dir contains template caches, sessions, whoosh index and is used for
4e6dfdb3fa01 docs: English and consistency corrections Michael V. DePalatis <mike@depalatis.net> parents: 4192 diff changeset	69 task locking (so it is safe across multiple instances). Set the
4e6dfdb3fa01 docs: English and consistency corrections Michael V. DePalatis <mike@depalatis.net> parents: 4192 diff changeset	70 ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir``
4e6dfdb3fa01 docs: English and consistency corrections Michael V. DePalatis <mike@depalatis.net> parents: 4192 diff changeset	71 variables in each .ini file to a shared location across Kallithea instances
6153 d6942b2b421c config: clarify that we only recommend and support single threaded operation Mads Kiilerich <madski@unity3d.com> parents: 5880 diff changeset	72 - If using several Celery instances,
4955 4e6dfdb3fa01 docs: English and consistency corrections Michael V. DePalatis <mike@depalatis.net> parents: 4192 diff changeset	73 the message broker should be common to all of them (e.g., one
4e6dfdb3fa01 docs: English and consistency corrections Michael V. DePalatis <mike@depalatis.net> parents: 4192 diff changeset	74 shared RabbitMQ server)
4e6dfdb3fa01 docs: English and consistency corrections Michael V. DePalatis <mike@depalatis.net> parents: 4192 diff changeset	75 - Load balance using round robin or IP hash, recommended is writing LB rules
3390 4cd84f4f28fb performance section docs update Marcin Kuzminski <marcin@python-works.com> parents: 3224 diff changeset	76 that will separate regular user traffic from automated processes like CI
4cd84f4f28fb performance section docs update Marcin Kuzminski <marcin@python-works.com> parents: 3224 diff changeset	77 servers or build bots.
5060 778f7ae3b6eb docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs Anatoly Bubenkov <bubenkoff@gmail.com> parents: 4955 diff changeset	78
6374 d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	79
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	80 Serve static files directly from the web server
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections Mads Kiilerich <mads@kiilerich.com> parents: 6216 diff changeset	81 -----------------------------------------------
5880 61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	82
61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	83 With the default ``static_files`` ini setting, the Kallithea WSGI application
6375 692dddf298e2 middleware: drop gzipper middleware Mads Kiilerich <mads@kiilerich.com> parents: 6374 diff changeset	84 will take care of serving the static files from ``kallithea/public/`` at the
692dddf298e2 middleware: drop gzipper middleware Mads Kiilerich <mads@kiilerich.com> parents: 6374 diff changeset	85 root of the application URL.
5880 61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	86
6375 692dddf298e2 middleware: drop gzipper middleware Mads Kiilerich <mads@kiilerich.com> parents: 6374 diff changeset	87 The actual serving of the static files is very fast and unlikely to be a
692dddf298e2 middleware: drop gzipper middleware Mads Kiilerich <mads@kiilerich.com> parents: 6374 diff changeset	88 problem in a Kallithea setup - the responses generated by Kallithea from
692dddf298e2 middleware: drop gzipper middleware Mads Kiilerich <mads@kiilerich.com> parents: 6374 diff changeset	89 database and repository content will take significantly more time and
692dddf298e2 middleware: drop gzipper middleware Mads Kiilerich <mads@kiilerich.com> parents: 6374 diff changeset	90 resources.
5880 61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	91
61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	92 To serve static files from the web server, use something like this Apache config
61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	93 snippet::
61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	94
61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	95 Alias /images/ /srv/kallithea/kallithea/kallithea/public/images/
61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	96 Alias /css/ /srv/kallithea/kallithea/kallithea/public/css/
61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	97 Alias /js/ /srv/kallithea/kallithea/kallithea/public/js/
61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	98 Alias /codemirror/ /srv/kallithea/kallithea/kallithea/public/codemirror/
61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	99 Alias /fontello/ /srv/kallithea/kallithea/kallithea/public/fontello/
61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	100
61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	101 Then disable serving of static files in the ``.ini`` ``app:main`` section::
61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	102
61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	103 static_files = false
61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	104
61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	105 If using Kallithea installed as a package, you should be able to find the files
6375 692dddf298e2 middleware: drop gzipper middleware Mads Kiilerich <mads@kiilerich.com> parents: 6374 diff changeset	106 under ``site-packages/kallithea``, either in your Python installation or in your
5880 61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	107 virtualenv. When upgrading, make sure to update the web server configuration
61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	108 too if necessary.
61954577a0df docs: mention use of static_files, it's setup and implications Mads Kiilerich <madski@unity3d.com> parents: 5769 diff changeset	109
6375 692dddf298e2 middleware: drop gzipper middleware Mads Kiilerich <mads@kiilerich.com> parents: 6374 diff changeset	110 It might also be possible to improve performance by configuring the web server
692dddf298e2 middleware: drop gzipper middleware Mads Kiilerich <mads@kiilerich.com> parents: 6374 diff changeset	111 to compress responses (served from static files or generated by Kallithea) when
692dddf298e2 middleware: drop gzipper middleware Mads Kiilerich <mads@kiilerich.com> parents: 6374 diff changeset	112 serving them. That might also imply buffering of responses - that is more
692dddf298e2 middleware: drop gzipper middleware Mads Kiilerich <mads@kiilerich.com> parents: 6374 diff changeset	113 likely to be a problem; large responses (clones or pulls) will have to be fully
692dddf298e2 middleware: drop gzipper middleware Mads Kiilerich <mads@kiilerich.com> parents: 6374 diff changeset	114 processed and spooled to disk or memory before the client will see any
692dddf298e2 middleware: drop gzipper middleware Mads Kiilerich <mads@kiilerich.com> parents: 6374 diff changeset	115 response. See the documentation for your web server.
692dddf298e2 middleware: drop gzipper middleware Mads Kiilerich <mads@kiilerich.com> parents: 6374 diff changeset	116
5433 fbbe80e3322b docs: consistent spacing around headings Mads Kiilerich <madski@unity3d.com> parents: 5425 diff changeset	117
5060 778f7ae3b6eb docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs Anatoly Bubenkov <bubenkoff@gmail.com> parents: 4955 diff changeset	118 .. _SQLAlchemyGrate: https://github.com/shazow/sqlalchemygrate

Mercurial > kallithea

annotate docs/usage/performance.rst @ 7611:9efcf6b78f71