Mercurial > kallithea
annotate docs/usage/performance.rst @ 7387:c677d5839bcc
make-config: tweak description - don't call it "bare" and don't reference setup-app
author | Mads Kiilerich <mads@kiilerich.com> |
---|---|
date | Tue, 09 Oct 2018 11:45:55 +0200 |
parents | 692dddf298e2 |
children | 19af3fef3b34 |
rev | line source |
---|---|
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
1 .. _performance: |
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
2 |
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
3 ================================ |
5413
22a3fa3c4254
docs: cleanup of casing, markup and spacing of headings
Mads Kiilerich <madski@unity3d.com>
parents:
5400
diff
changeset
|
4 Optimizing Kallithea performance |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
5 ================================ |
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
6 |
6374
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
7 When serving a large amount of big repositories, Kallithea can start performing |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
8 slower than expected. Because of the demanding nature of handling large amounts |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
9 of data from version control systems, here are some tips on how to get the best |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
10 performance. |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
11 |
2775
5d12768a0aa1
docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents:
2680
diff
changeset
|
12 |
6374
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
13 Fast storage |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
14 ------------ |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
15 |
6374
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
16 Kallithea is often I/O bound, and hence a fast disk (SSD/SAN) and plenty of RAM |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
17 is usually more important than a fast CPU. |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
18 |
5769
6afa528ee30e
db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents:
5534
diff
changeset
|
19 |
6374
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
20 Caching |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
21 ------- |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
22 |
6374
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
23 Tweak beaker cache settings in the ini file. The actual effect of that is |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
24 questionable. |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
25 |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
26 |
6374
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
27 Database |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
28 -------- |
3224
8b8edfc25856
whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2775
diff
changeset
|
29 |
6374
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
30 SQLite is a good option when having a small load on the system. But due to |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
31 locking issues with SQLite, it is not recommended to use it for larger |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
32 deployments. |
3224
8b8edfc25856
whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2775
diff
changeset
|
33 |
6374
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
34 Switching to MySQL or PostgreSQL will result in an immediate performance |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
35 increase. A tool like SQLAlchemyGrate_ can be used for migrating to another |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
36 database platform. |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
37 |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
38 |
6374
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
39 Horizontal scaling |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
40 ------------------ |
6153
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
41 |
6374
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
42 Scaling horizontally means running several Kallithea instances and let them |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
43 share the load. That can give huge performance benefits when dealing with large |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
44 amounts of traffic (many users, CI servers, etc.). Kallithea can be scaled |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
45 horizontally on one (recommended) or multiple machines. |
6216
49c82acd30b2
scripts: fix X mode on code formatting scripts
Mads Kiilerich <madski@unity3d.com>
parents:
6153
diff
changeset
|
46 |
6374
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
47 It is generally possible to run WSGI applications multithreaded, so that |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
48 several HTTP requests are served from the same Python process at once. That can |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
49 in principle give better utilization of internal caches and less process |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
50 overhead. |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
51 |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
52 One danger of running multithreaded is that program execution becomes much more |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
53 complex; programs must be written to consider all combinations of events and |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
54 problems might depend on timing and be impossible to reproduce. |
6153
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
55 |
6374
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
56 Kallithea can't promise to be thread-safe, just like the embedded Mercurial |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
57 backend doesn't make any strong promises when used as Kallithea uses it. |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
58 Instead, we recommend scaling by using multiple server processes. |
6153
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
59 |
6374
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
60 Web servers with multiple worker processes (such as ``mod_wsgi`` with the |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
61 ``WSGIDaemonProcess`` ``processes`` parameter) will work out of the box. |
6153
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
62 |
6374
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
63 In order to scale horizontally on multiple machines, you need to do the |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
64 following: |
3413
d79f3505549e
whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
3390
diff
changeset
|
65 |
4955
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
66 - Each instance's ``data`` storage needs to be configured to be stored on a |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
67 shared disk storage, preferably together with repositories. This ``data`` |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
68 dir contains template caches, sessions, whoosh index and is used for |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
69 task locking (so it is safe across multiple instances). Set the |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
70 ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir`` |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
71 variables in each .ini file to a shared location across Kallithea instances |
6153
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
72 - If using several Celery instances, |
4955
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
73 the message broker should be common to all of them (e.g., one |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
74 shared RabbitMQ server) |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
75 - Load balance using round robin or IP hash, recommended is writing LB rules |
3390
4cd84f4f28fb
performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
3224
diff
changeset
|
76 that will separate regular user traffic from automated processes like CI |
4cd84f4f28fb
performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
3224
diff
changeset
|
77 servers or build bots. |
5060
778f7ae3b6eb
docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents:
4955
diff
changeset
|
78 |
6374
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
79 |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
80 Serve static files directly from the web server |
d02c715e2805
docs: tweak formatting of the performance page - replace the odd numbered list with sections
Mads Kiilerich <mads@kiilerich.com>
parents:
6216
diff
changeset
|
81 ----------------------------------------------- |
5880
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
82 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
83 With the default ``static_files`` ini setting, the Kallithea WSGI application |
6375
692dddf298e2
middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents:
6374
diff
changeset
|
84 will take care of serving the static files from ``kallithea/public/`` at the |
692dddf298e2
middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents:
6374
diff
changeset
|
85 root of the application URL. |
5880
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
86 |
6375
692dddf298e2
middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents:
6374
diff
changeset
|
87 The actual serving of the static files is very fast and unlikely to be a |
692dddf298e2
middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents:
6374
diff
changeset
|
88 problem in a Kallithea setup - the responses generated by Kallithea from |
692dddf298e2
middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents:
6374
diff
changeset
|
89 database and repository content will take significantly more time and |
692dddf298e2
middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents:
6374
diff
changeset
|
90 resources. |
5880
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
91 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
92 To serve static files from the web server, use something like this Apache config |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
93 snippet:: |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
94 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
95 Alias /images/ /srv/kallithea/kallithea/kallithea/public/images/ |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
96 Alias /css/ /srv/kallithea/kallithea/kallithea/public/css/ |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
97 Alias /js/ /srv/kallithea/kallithea/kallithea/public/js/ |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
98 Alias /codemirror/ /srv/kallithea/kallithea/kallithea/public/codemirror/ |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
99 Alias /fontello/ /srv/kallithea/kallithea/kallithea/public/fontello/ |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
100 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
101 Then disable serving of static files in the ``.ini`` ``app:main`` section:: |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
102 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
103 static_files = false |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
104 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
105 If using Kallithea installed as a package, you should be able to find the files |
6375
692dddf298e2
middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents:
6374
diff
changeset
|
106 under ``site-packages/kallithea``, either in your Python installation or in your |
5880
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
107 virtualenv. When upgrading, make sure to update the web server configuration |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
108 too if necessary. |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
109 |
6375
692dddf298e2
middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents:
6374
diff
changeset
|
110 It might also be possible to improve performance by configuring the web server |
692dddf298e2
middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents:
6374
diff
changeset
|
111 to compress responses (served from static files or generated by Kallithea) when |
692dddf298e2
middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents:
6374
diff
changeset
|
112 serving them. That might also imply buffering of responses - that is more |
692dddf298e2
middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents:
6374
diff
changeset
|
113 likely to be a problem; large responses (clones or pulls) will have to be fully |
692dddf298e2
middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents:
6374
diff
changeset
|
114 processed and spooled to disk or memory before the client will see any |
692dddf298e2
middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents:
6374
diff
changeset
|
115 response. See the documentation for your web server. |
692dddf298e2
middleware: drop gzipper middleware
Mads Kiilerich <mads@kiilerich.com>
parents:
6374
diff
changeset
|
116 |
5433
fbbe80e3322b
docs: consistent spacing around headings
Mads Kiilerich <madski@unity3d.com>
parents:
5425
diff
changeset
|
117 |
5060
778f7ae3b6eb
docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents:
4955
diff
changeset
|
118 .. _SQLAlchemyGrate: https://github.com/shazow/sqlalchemygrate |