Mercurial > kallithea
annotate docs/usage/performance.rst @ 6360:b2e9677577ca
style: remove extra unused wrapper table in user group settings
author | Mads Kiilerich <mads@kiilerich.com> |
---|---|
date | Sun, 04 Dec 2016 22:53:37 +0100 |
parents | 49c82acd30b2 |
children | d02c715e2805 |
rev | line source |
---|---|
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
1 .. _performance: |
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
2 |
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
3 ================================ |
5413
22a3fa3c4254
docs: cleanup of casing, markup and spacing of headings
Mads Kiilerich <madski@unity3d.com>
parents:
5400
diff
changeset
|
4 Optimizing Kallithea performance |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
5 ================================ |
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
6 |
4955
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
7 When serving a large amount of big repositories, Kallithea can start |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
8 performing slower than expected. Because of the demanding nature of handling large |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
9 amounts of data from version control systems, here are some tips on how to get |
2775
5d12768a0aa1
docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents:
2680
diff
changeset
|
10 the best performance. |
5d12768a0aa1
docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents:
2680
diff
changeset
|
11 |
4192
e73a69cb98dc
Rename some strings examples and commands in documentation
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents:
3413
diff
changeset
|
12 Follow these few steps to improve performance of Kallithea system. |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
13 |
5769
6afa528ee30e
db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents:
5534
diff
changeset
|
14 1. Kallithea is often I/O bound, and hence a fast disk (SSD/SAN) is |
6afa528ee30e
db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents:
5534
diff
changeset
|
15 usually more important than a fast CPU. |
6afa528ee30e
db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents:
5534
diff
changeset
|
16 |
6afa528ee30e
db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents:
5534
diff
changeset
|
17 2. Increase cache |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
18 |
5400
6257de126ec7
docs: improve documentation of beaker session configuration
Mads Kiilerich <madski@unity3d.com>
parents:
5399
diff
changeset
|
19 Tweak beaker cache settings in the ini file. The actual effect of that |
5399
4610a39d3be9
cleanup: kill unused beaker cache regions
Mads Kiilerich <madski@unity3d.com>
parents:
5060
diff
changeset
|
20 is questionable. |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
21 |
5769
6afa528ee30e
db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents:
5534
diff
changeset
|
22 3. Switch from SQLite to PostgreSQL or MySQL |
3224
8b8edfc25856
whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2775
diff
changeset
|
23 |
5425
5ae8e644aa88
docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents:
5413
diff
changeset
|
24 SQLite is a good option when having a small load on the system. But due to |
5ae8e644aa88
docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents:
5413
diff
changeset
|
25 locking issues with SQLite, it is not recommended to use it for larger |
5ae8e644aa88
docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents:
5413
diff
changeset
|
26 deployments. Switching to MySQL or PostgreSQL will result in an immediate |
5060
778f7ae3b6eb
docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents:
4955
diff
changeset
|
27 performance increase. A tool like SQLAlchemyGrate_ can be used for |
778f7ae3b6eb
docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents:
4955
diff
changeset
|
28 migrating to another database platform. |
3224
8b8edfc25856
whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2775
diff
changeset
|
29 |
5769
6afa528ee30e
db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents:
5534
diff
changeset
|
30 4. Scale Kallithea horizontally |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
31 |
5425
5ae8e644aa88
docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents:
5413
diff
changeset
|
32 Scaling horizontally can give huge performance benefits when dealing with |
5ae8e644aa88
docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents:
5413
diff
changeset
|
33 large amounts of traffic (many users, CI servers, etc.). Kallithea can be |
6153
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
34 scaled horizontally on one (recommended) or multiple machines. |
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
35 |
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
36 It is generally possible to run WSGI applications multithreaded, so that |
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
37 several HTTP requests are served from the same Python process at once. That |
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
38 can in principle give better utilization of internal caches and less |
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
39 process overhead. |
6216
49c82acd30b2
scripts: fix X mode on code formatting scripts
Mads Kiilerich <madski@unity3d.com>
parents:
6153
diff
changeset
|
40 |
6153
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
41 One danger of running multithreaded is that program execution becomes much |
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
42 more complex; programs must be written to consider all combinations of |
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
43 events and problems might depend on timing and be impossible to reproduce. |
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
44 |
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
45 Kallithea can't promise to be thread-safe, just like the embedded Mercurial |
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
46 backend doesn't make any strong promises when used as Kallithea uses it. |
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
47 Instead, we recommend scaling by using multiple server processes. |
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
48 |
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
49 Web servers with multiple worker processes (such as ``mod_wsgi`` with the |
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
50 ``WSGIDaemonProcess`` ``processes`` parameter) will work out of the box. |
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
51 |
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
52 In order to scale horizontally on multiple machines, you need to do the |
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
53 following: |
3413
d79f3505549e
whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
3390
diff
changeset
|
54 |
4955
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
55 - Each instance's ``data`` storage needs to be configured to be stored on a |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
56 shared disk storage, preferably together with repositories. This ``data`` |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
57 dir contains template caches, sessions, whoosh index and is used for |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
58 task locking (so it is safe across multiple instances). Set the |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
59 ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir`` |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
60 variables in each .ini file to a shared location across Kallithea instances |
6153
d6942b2b421c
config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents:
5880
diff
changeset
|
61 - If using several Celery instances, |
4955
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
62 the message broker should be common to all of them (e.g., one |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
63 shared RabbitMQ server) |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
64 - Load balance using round robin or IP hash, recommended is writing LB rules |
3390
4cd84f4f28fb
performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
3224
diff
changeset
|
65 that will separate regular user traffic from automated processes like CI |
4cd84f4f28fb
performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
3224
diff
changeset
|
66 servers or build bots. |
5060
778f7ae3b6eb
docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents:
4955
diff
changeset
|
67 |
5880
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
68 5. Serve static files directly from the web server |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
69 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
70 With the default ``static_files`` ini setting, the Kallithea WSGI application |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
71 will take care of serving the static files found in ``kallithea/public`` from |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
72 the root of the application URL. While doing that, it will currently also |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
73 apply buffering and compression of all the responses it is serving. |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
74 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
75 The actual serving of the static files is unlikely to be a problem in a |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
76 Kallithea setup. The buffering of responses is more likely to be a problem; |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
77 large responses (clones or pulls) will have to be fully processed and spooled |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
78 to disk or memory before the client will see any response. |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
79 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
80 To serve static files from the web server, use something like this Apache config |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
81 snippet:: |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
82 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
83 Alias /images/ /srv/kallithea/kallithea/kallithea/public/images/ |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
84 Alias /css/ /srv/kallithea/kallithea/kallithea/public/css/ |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
85 Alias /js/ /srv/kallithea/kallithea/kallithea/public/js/ |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
86 Alias /codemirror/ /srv/kallithea/kallithea/kallithea/public/codemirror/ |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
87 Alias /fontello/ /srv/kallithea/kallithea/kallithea/public/fontello/ |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
88 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
89 Then disable serving of static files in the ``.ini`` ``app:main`` section:: |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
90 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
91 static_files = false |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
92 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
93 If using Kallithea installed as a package, you should be able to find the files |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
94 under site-packages/kallithea, either in your Python installation or in your |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
95 virtualenv. When upgrading, make sure to update the web server configuration |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
96 too if necessary. |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
97 |
5433
fbbe80e3322b
docs: consistent spacing around headings
Mads Kiilerich <madski@unity3d.com>
parents:
5425
diff
changeset
|
98 |
5060
778f7ae3b6eb
docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents:
4955
diff
changeset
|
99 .. _SQLAlchemyGrate: https://github.com/shazow/sqlalchemygrate |