Mercurial > kallithea

annotate docs/usage/performance.rst @ 6360:b2e9677577ca

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
style: remove extra unused wrapper table in user group settings
author Mads Kiilerich <mads@kiilerich.com>
date Sun, 04 Dec 2016 22:53:37 +0100
parents 49c82acd30b2
children d02c715e2805
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
1 .. _performance:
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
2
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
3 ================================
5413
22a3fa3c4254 docs: cleanup of casing, markup and spacing of headings
Mads Kiilerich <madski@unity3d.com>
parents: 5400
diff changeset
4 Optimizing Kallithea performance
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
5 ================================
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
6
4955
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
7 When serving a large amount of big repositories, Kallithea can start
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
8 performing slower than expected. Because of the demanding nature of handling large
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
9 amounts of data from version control systems, here are some tips on how to get
2775
5d12768a0aa1 docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents: 2680
diff changeset
10 the best performance.
5d12768a0aa1 docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents: 2680
diff changeset
11
4192
e73a69cb98dc Rename some strings examples and commands in documentation
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3413
diff changeset
12 Follow these few steps to improve performance of Kallithea system.
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
13
5769
6afa528ee30e db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents: 5534
diff changeset
14 1. Kallithea is often I/O bound, and hence a fast disk (SSD/SAN) is
6afa528ee30e db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents: 5534
diff changeset
15 usually more important than a fast CPU.
6afa528ee30e db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents: 5534
diff changeset
16
6afa528ee30e db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents: 5534
diff changeset
17 2. Increase cache
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
18
5400
6257de126ec7 docs: improve documentation of beaker session configuration
Mads Kiilerich <madski@unity3d.com>
parents: 5399
diff changeset
19 Tweak beaker cache settings in the ini file. The actual effect of that
5399
4610a39d3be9 cleanup: kill unused beaker cache regions
Mads Kiilerich <madski@unity3d.com>
parents: 5060
diff changeset
20 is questionable.
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
21
5769
6afa528ee30e db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents: 5534
diff changeset
22 3. Switch from SQLite to PostgreSQL or MySQL
3224
8b8edfc25856 whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2775
diff changeset
23
5425
5ae8e644aa88 docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents: 5413
diff changeset
24 SQLite is a good option when having a small load on the system. But due to
5ae8e644aa88 docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents: 5413
diff changeset
25 locking issues with SQLite, it is not recommended to use it for larger
5ae8e644aa88 docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents: 5413
diff changeset
26 deployments. Switching to MySQL or PostgreSQL will result in an immediate
5060
778f7ae3b6eb docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents: 4955
diff changeset
27 performance increase. A tool like SQLAlchemyGrate_ can be used for
778f7ae3b6eb docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents: 4955
diff changeset
28 migrating to another database platform.
3224
8b8edfc25856 whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2775
diff changeset
29
5769
6afa528ee30e db: get rid of vcs_full_cache - it should always be used
Mads Kiilerich <madski@unity3d.com>
parents: 5534
diff changeset
30 4. Scale Kallithea horizontally
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
31
5425
5ae8e644aa88 docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents: 5413
diff changeset
32 Scaling horizontally can give huge performance benefits when dealing with
5ae8e644aa88 docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents: 5413
diff changeset
33 large amounts of traffic (many users, CI servers, etc.). Kallithea can be
6153
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
34 scaled horizontally on one (recommended) or multiple machines.
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
35
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
36 It is generally possible to run WSGI applications multithreaded, so that
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
37 several HTTP requests are served from the same Python process at once. That
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
38 can in principle give better utilization of internal caches and less
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
39 process overhead.
6216
49c82acd30b2 scripts: fix X mode on code formatting scripts
Mads Kiilerich <madski@unity3d.com>
parents: 6153
diff changeset
40
6153
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
41 One danger of running multithreaded is that program execution becomes much
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
42 more complex; programs must be written to consider all combinations of
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
43 events and problems might depend on timing and be impossible to reproduce.
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
44
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
45 Kallithea can't promise to be thread-safe, just like the embedded Mercurial
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
46 backend doesn't make any strong promises when used as Kallithea uses it.
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
47 Instead, we recommend scaling by using multiple server processes.
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
48
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
49 Web servers with multiple worker processes (such as ``mod_wsgi`` with the
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
50 ``WSGIDaemonProcess`` ``processes`` parameter) will work out of the box.
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
51
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
52 In order to scale horizontally on multiple machines, you need to do the
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
53 following:
3413
d79f3505549e whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 3390
diff changeset
54
4955
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
55 - Each instance's ``data`` storage needs to be configured to be stored on a
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
56 shared disk storage, preferably together with repositories. This ``data``
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
57 dir contains template caches, sessions, whoosh index and is used for
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
58 task locking (so it is safe across multiple instances). Set the
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
59 ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir``
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
60 variables in each .ini file to a shared location across Kallithea instances
6153
d6942b2b421c config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5880
diff changeset
61 - If using several Celery instances,
4955
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
62 the message broker should be common to all of them (e.g., one
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
63 shared RabbitMQ server)
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
64 - Load balance using round robin or IP hash, recommended is writing LB rules
3390
4cd84f4f28fb performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents: 3224
diff changeset
65 that will separate regular user traffic from automated processes like CI
4cd84f4f28fb performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents: 3224
diff changeset
66 servers or build bots.
5060
778f7ae3b6eb docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents: 4955
diff changeset
67
5880
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
68 5. Serve static files directly from the web server
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
69
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
70 With the default ``static_files`` ini setting, the Kallithea WSGI application
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
71 will take care of serving the static files found in ``kallithea/public`` from
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
72 the root of the application URL. While doing that, it will currently also
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
73 apply buffering and compression of all the responses it is serving.
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
74
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
75 The actual serving of the static files is unlikely to be a problem in a
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
76 Kallithea setup. The buffering of responses is more likely to be a problem;
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
77 large responses (clones or pulls) will have to be fully processed and spooled
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
78 to disk or memory before the client will see any response.
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
79
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
80 To serve static files from the web server, use something like this Apache config
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
81 snippet::
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
82
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
83 Alias /images/ /srv/kallithea/kallithea/kallithea/public/images/
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
84 Alias /css/ /srv/kallithea/kallithea/kallithea/public/css/
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
85 Alias /js/ /srv/kallithea/kallithea/kallithea/public/js/
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
86 Alias /codemirror/ /srv/kallithea/kallithea/kallithea/public/codemirror/
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
87 Alias /fontello/ /srv/kallithea/kallithea/kallithea/public/fontello/
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
88
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
89 Then disable serving of static files in the ``.ini`` ``app:main`` section::
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
90
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
91 static_files = false
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
92
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
93 If using Kallithea installed as a package, you should be able to find the files
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
94 under site-packages/kallithea, either in your Python installation or in your
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
95 virtualenv. When upgrading, make sure to update the web server configuration
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
96 too if necessary.
61954577a0df docs: mention use of static_files, it's setup and implications
Mads Kiilerich <madski@unity3d.com>
parents: 5769
diff changeset
97
5433
fbbe80e3322b docs: consistent spacing around headings
Mads Kiilerich <madski@unity3d.com>
parents: 5425
diff changeset
98
5060
778f7ae3b6eb docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents: 4955
diff changeset
99 .. _SQLAlchemyGrate: https://github.com/shazow/sqlalchemygrate