Mercurial > kallithea

annotate docs/usage/performance.rst @ 7596:18d9479bc93a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
i18n: regenerate translations
author Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
date Wed, 20 Mar 2019 20:59:59 +0100
parents 692dddf298e2
children 19af3fef3b34
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
1 .. _performance:
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
2
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
3 ================================
5413
22a3fa3c4254 docs: cleanup of casing, markup and spacing of headings
Mads Kiilerich <madski@unity3d.com>
parents: 5400
diff changeset
4 Optimizing Kallithea performance
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
5 ================================
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
6
6374
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
7 When serving a large amount of big repositories, Kallithea can start performing
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
8 slower than expected. Because of the demanding nature of handling large amounts
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
9 of data from version control systems, here are some tips on how to get the best
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
10 performance.
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
11
2775
5d12768a0aa1 docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents: 2680
diff changeset
12
6374
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
13 Fast storage
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
14 ------------
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
15
6374
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
16 Kallithea is often I/O bound, and hence a fast disk (SSD/SAN) and plenty of RAM
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
17 is usually more important than a fast CPU.
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
18
5769
6afa528ee30e db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents: 5534
diff changeset
19
6374
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
20 Caching
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
21 -------
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
22
6374
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
23 Tweak beaker cache settings in the ini file. The actual effect of that is
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
24 questionable.
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
25
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
26
6374
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
27 Database
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
28 --------
3224
8b8edfc25856 whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2775
diff changeset
29
6374
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
30 SQLite is a good option when having a small load on the system. But due to
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
31 locking issues with SQLite, it is not recommended to use it for larger
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
32 deployments.
3224
8b8edfc25856 whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2775
diff changeset
33
6374
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
34 Switching to MySQL or PostgreSQL will result in an immediate performance
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
35 increase. A tool like SQLAlchemyGrate_ can be used for migrating to another
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
36 database platform.
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
37
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
38
6374
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
39 Horizontal scaling
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
40 ------------------
6153
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
41
6374
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
42 Scaling horizontally means running several Kallithea instances and let them
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
43 share the load. That can give huge performance benefits when dealing with large
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
44 amounts of traffic (many users, CI servers, etc.). Kallithea can be scaled
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
45 horizontally on one (recommended) or multiple machines.
6216
49c82acd30b2 scripts: fix X mode on code formatting scripts
Mads Kiilerich <madski@unity3d.com>
parents: 6153
diff changeset
46
6374
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
47 It is generally possible to run WSGI applications multithreaded, so that
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
48 several HTTP requests are served from the same Python process at once. That can
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
49 in principle give better utilization of internal caches and less process
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
50 overhead.
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
51
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
52 One danger of running multithreaded is that program execution becomes much more
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
53 complex; programs must be written to consider all combinations of events and
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
54 problems might depend on timing and be impossible to reproduce.
6153
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
55
6374
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
56 Kallithea can't promise to be thread-safe, just like the embedded Mercurial
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
57 backend doesn't make any strong promises when used as Kallithea uses it.
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
58 Instead, we recommend scaling by using multiple server processes.
6153
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
59
6374
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
60 Web servers with multiple worker processes (such as ``mod_wsgi`` with the
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
61 ``WSGIDaemonProcess`` ``processes`` parameter) will work out of the box.
6153
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
62
6374
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
63 In order to scale horizontally on multiple machines, you need to do the
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
64 following:
3413
d79f3505549e whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 3390
diff changeset
65
4955
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
66 - Each instance's ``data`` storage needs to be configured to be stored on a
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
67 shared disk storage, preferably together with repositories. This ``data``
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
68 dir contains template caches, sessions, whoosh index and is used for
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
69 task locking (so it is safe across multiple instances). Set the
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
70 ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir``
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
71 variables in each .ini file to a shared location across Kallithea instances
6153
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
72 - If using several Celery instances,
4955
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
73 the message broker should be common to all of them (e.g., one
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
74 shared RabbitMQ server)
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
75 - Load balance using round robin or IP hash, recommended is writing LB rules
3390
4cd84f4f28fb performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents: 3224
diff changeset
76 that will separate regular user traffic from automated processes like CI
4cd84f4f28fb performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents: 3224
diff changeset
77 servers or build bots.
5060
778f7ae3b6eb docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents: 4955
diff changeset
78
6374
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
79
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
80 Serve static files directly from the web server
d02c715e2805 docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents: 6216
diff changeset
81 -----------------------------------------------
5880
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
82
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
83 With the default ``static_files`` ini setting, the Kallithea WSGI application
6375
692dddf298e2 middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents: 6374
diff changeset
84 will take care of serving the static files from ``kallithea/public/`` at the
692dddf298e2 middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents: 6374
diff changeset
85 root of the application URL.
5880
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
86
6375
692dddf298e2 middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents: 6374
diff changeset
87 The actual serving of the static files is very fast and unlikely to be a
692dddf298e2 middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents: 6374
diff changeset
88 problem in a Kallithea setup - the responses generated by Kallithea from
692dddf298e2 middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents: 6374
diff changeset
89 database and repository content will take significantly more time and
692dddf298e2 middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents: 6374
diff changeset
90 resources.
5880
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
91
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
92 To serve static files from the web server, use something like this Apache config
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
93 snippet::
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
94
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
95 Alias /images/ /srv/kallithea/kallithea/kallithea/public/images/
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
96 Alias /css/ /srv/kallithea/kallithea/kallithea/public/css/
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
97 Alias /js/ /srv/kallithea/kallithea/kallithea/public/js/
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
98 Alias /codemirror/ /srv/kallithea/kallithea/kallithea/public/codemirror/
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
99 Alias /fontello/ /srv/kallithea/kallithea/kallithea/public/fontello/
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
100
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
101 Then disable serving of static files in the ``.ini`` ``app:main`` section::
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
102
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
103 static_files = false
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
104
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
105 If using Kallithea installed as a package, you should be able to find the files
6375
692dddf298e2 middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents: 6374
diff changeset
106 under ``site-packages/kallithea``, either in your Python installation or in your
5880
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
107 virtualenv. When upgrading, make sure to update the web server configuration
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
108 too if necessary.
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
109
6375
692dddf298e2 middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents: 6374
diff changeset
110 It might also be possible to improve performance by configuring the web server
692dddf298e2 middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents: 6374
diff changeset
111 to compress responses (served from static files or generated by Kallithea) when
692dddf298e2 middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents: 6374
diff changeset
112 serving them. That might also imply buffering of responses - that is more
692dddf298e2 middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents: 6374
diff changeset
113 likely to be a problem; large responses (clones or pulls) will have to be fully
692dddf298e2 middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents: 6374
diff changeset
114 processed and spooled to disk or memory before the client will see any
692dddf298e2 middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents: 6374
diff changeset
115 response. See the documentation for your web server.
692dddf298e2 middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents: 6374
diff changeset
116
5433
fbbe80e3322b docs: consistent spacing around headings
Mads Kiilerich <madski@unity3d.com>
parents: 5425
diff changeset
117
5060
778f7ae3b6eb docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents: 4955
diff changeset
118 .. _SQLAlchemyGrate: https://github.com/shazow/sqlalchemygrate