Mercurial > kallithea
annotate docs/usage/performance.rst @ 6109:23ff4e66391a
routing: use POST to 'edit_user_api_keys_update' instead of PUT
author | Mads Kiilerich <madski@unity3d.com> |
---|---|
date | Thu, 04 Aug 2016 14:23:36 +0200 |
parents | 61954577a0df |
children | d6942b2b421c |
rev | line source |
---|---|
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
1 .. _performance: |
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
2 |
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
3 ================================ |
5413
22a3fa3c4254
docs: cleanup of casing, markup and spacing of headings
Mads Kiilerich <madski@unity3d.com>
parents:
5400
diff
changeset
|
4 Optimizing Kallithea performance |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
5 ================================ |
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
6 |
4955
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
7 When serving a large amount of big repositories, Kallithea can start |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
8 performing slower than expected. Because of the demanding nature of handling large |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
9 amounts of data from version control systems, here are some tips on how to get |
2775
5d12768a0aa1
docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents:
2680
diff
changeset
|
10 the best performance. |
5d12768a0aa1
docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents:
2680
diff
changeset
|
11 |
4192
e73a69cb98dc
Rename some strings examples and commands in documentation
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents:
3413
diff
changeset
|
12 Follow these few steps to improve performance of Kallithea system. |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
13 |
5769
6afa528ee30e
db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents:
5534
diff
changeset
|
14 1. Kallithea is often I/O bound, and hence a fast disk (SSD/SAN) is |
6afa528ee30e
db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents:
5534
diff
changeset
|
15 usually more important than a fast CPU. |
6afa528ee30e
db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents:
5534
diff
changeset
|
16 |
6afa528ee30e
db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents:
5534
diff
changeset
|
17 2. Increase cache |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
18 |
5400
6257de126ec7
docs: improve documentation of beaker session configuration
Mads Kiilerich <madski@unity3d.com>
parents:
5399
diff
changeset
|
19 Tweak beaker cache settings in the ini file. The actual effect of that |
5399
4610a39d3be9
cleanup: kill unused beaker cache regions
Mads Kiilerich <madski@unity3d.com>
parents:
5060
diff
changeset
|
20 is questionable. |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
21 |
5769
6afa528ee30e
db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents:
5534
diff
changeset
|
22 3. Switch from SQLite to PostgreSQL or MySQL |
3224
8b8edfc25856
whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2775
diff
changeset
|
23 |
5425
5ae8e644aa88
docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents:
5413
diff
changeset
|
24 SQLite is a good option when having a small load on the system. But due to |
5ae8e644aa88
docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents:
5413
diff
changeset
|
25 locking issues with SQLite, it is not recommended to use it for larger |
5ae8e644aa88
docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents:
5413
diff
changeset
|
26 deployments. Switching to MySQL or PostgreSQL will result in an immediate |
5060
778f7ae3b6eb
docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents:
4955
diff
changeset
|
27 performance increase. A tool like SQLAlchemyGrate_ can be used for |
778f7ae3b6eb
docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents:
4955
diff
changeset
|
28 migrating to another database platform. |
3224
8b8edfc25856
whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2775
diff
changeset
|
29 |
5769
6afa528ee30e
db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents:
5534
diff
changeset
|
30 4. Scale Kallithea horizontally |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
31 |
5425
5ae8e644aa88
docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents:
5413
diff
changeset
|
32 Scaling horizontally can give huge performance benefits when dealing with |
5ae8e644aa88
docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents:
5413
diff
changeset
|
33 large amounts of traffic (many users, CI servers, etc.). Kallithea can be |
3390
4cd84f4f28fb
performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
3224
diff
changeset
|
34 scaled horizontally on one (recommended) or multiple machines. In order |
4cd84f4f28fb
performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
3224
diff
changeset
|
35 to scale horizontally you need to do the following: |
3413
d79f3505549e
whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
3390
diff
changeset
|
36 |
4955
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
37 - Each instance's ``data`` storage needs to be configured to be stored on a |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
38 shared disk storage, preferably together with repositories. This ``data`` |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
39 dir contains template caches, sessions, whoosh index and is used for |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
40 task locking (so it is safe across multiple instances). Set the |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
41 ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir`` |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
42 variables in each .ini file to a shared location across Kallithea instances |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
43 - If celery is used each instance should run a separate Celery instance, but |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
44 the message broker should be common to all of them (e.g., one |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
45 shared RabbitMQ server) |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
46 - Load balance using round robin or IP hash, recommended is writing LB rules |
3390
4cd84f4f28fb
performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
3224
diff
changeset
|
47 that will separate regular user traffic from automated processes like CI |
4cd84f4f28fb
performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
3224
diff
changeset
|
48 servers or build bots. |
5060
778f7ae3b6eb
docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents:
4955
diff
changeset
|
49 |
5880
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
50 5. Serve static files directly from the web server |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
51 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
52 With the default ``static_files`` ini setting, the Kallithea WSGI application |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
53 will take care of serving the static files found in ``kallithea/public`` from |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
54 the root of the application URL. While doing that, it will currently also |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
55 apply buffering and compression of all the responses it is serving. |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
56 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
57 The actual serving of the static files is unlikely to be a problem in a |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
58 Kallithea setup. The buffering of responses is more likely to be a problem; |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
59 large responses (clones or pulls) will have to be fully processed and spooled |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
60 to disk or memory before the client will see any response. |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
61 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
62 To serve static files from the web server, use something like this Apache config |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
63 snippet:: |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
64 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
65 Alias /images/ /srv/kallithea/kallithea/kallithea/public/images/ |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
66 Alias /css/ /srv/kallithea/kallithea/kallithea/public/css/ |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
67 Alias /js/ /srv/kallithea/kallithea/kallithea/public/js/ |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
68 Alias /codemirror/ /srv/kallithea/kallithea/kallithea/public/codemirror/ |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
69 Alias /fontello/ /srv/kallithea/kallithea/kallithea/public/fontello/ |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
70 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
71 Then disable serving of static files in the ``.ini`` ``app:main`` section:: |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
72 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
73 static_files = false |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
74 |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
75 If using Kallithea installed as a package, you should be able to find the files |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
76 under site-packages/kallithea, either in your Python installation or in your |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
77 virtualenv. When upgrading, make sure to update the web server configuration |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
78 too if necessary. |
61954577a0df
docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents:
5769
diff
changeset
|
79 |
5433
fbbe80e3322b
docs: consistent spacing around headings
Mads Kiilerich <madski@unity3d.com>
parents:
5425
diff
changeset
|
80 |
5060
778f7ae3b6eb
docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents:
4955
diff
changeset
|
81 .. _SQLAlchemyGrate: https://github.com/shazow/sqlalchemygrate |