Mercurial > kallithea

annotate docs/usage/performance.rst @ 7552:e74aa69f6827 stable

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
lib: sanitize HTML for all types of README rendering, not only markdown The repository summary page will display a rendered version of the repository 'readme' based on its file extension. In commit 5746cc3b3fa5, the rendered output was already sanitized when the input was markdown. However, also readmes written in other formats, like ReStructuredText (RST) or plain text could have content that we want sanitized. Therefore, move the sanitizing one level up so it covers all renderers, for now and the future. This fixes an XSS issue when a repository readme contains javascript code, which would be executed when the repository summary page is visited by a user. Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
author Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
date Sat, 26 Jan 2019 20:27:50 +0100
parents 716e53c085ff
children 19af3fef3b34
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
1 .. _performance:
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
2
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
3 ================================
5413
22a3fa3c4254 docs: cleanup of casing, markup and spacing of headings
Mads Kiilerich <madski@unity3d.com>
parents: 5400
diff changeset
4 Optimizing Kallithea performance
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
5 ================================
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
6
4955
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
7 When serving a large amount of big repositories, Kallithea can start
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
8 performing slower than expected. Because of the demanding nature of handling large
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
9 amounts of data from version control systems, here are some tips on how to get
2775
5d12768a0aa1 docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents: 2680
diff changeset
10 the best performance.
5d12768a0aa1 docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents: 2680
diff changeset
11
5425
5ae8e644aa88 docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents: 5413
diff changeset
12 * Kallithea is often I/O bound, and hence a fast disk (SSD/SAN) is
5ae8e644aa88 docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents: 5413
diff changeset
13 usually more important than a fast CPU.
2775
5d12768a0aa1 docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents: 2680
diff changeset
14
5425
5ae8e644aa88 docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents: 5413
diff changeset
15 * Sluggish loading of the front page can easily be fixed by grouping repositories or by
4955
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
16 increasing cache size (see below). This includes using the lightweight dashboard
5425
5ae8e644aa88 docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents: 5413
diff changeset
17 option and ``vcs_full_cache`` setting in .ini file.
2775
5d12768a0aa1 docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents: 2680
diff changeset
18
4192
e73a69cb98dc Rename some strings examples and commands in documentation
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3413
diff changeset
19 Follow these few steps to improve performance of Kallithea system.
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
20
2680
3b179f1ac3a2 doc fixes
Marcin Kuzminski <marcin@python-works.com>
parents: 2517
diff changeset
21 1. Increase cache
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
22
5400
6257de126ec7 docs: improve documentation of beaker session configuration
Mads Kiilerich <madski@unity3d.com>
parents: 5399
diff changeset
23 Tweak beaker cache settings in the ini file. The actual effect of that
5399
4610a39d3be9 cleanup: kill unused beaker cache regions
Mads Kiilerich <madski@unity3d.com>
parents: 5060
diff changeset
24 is questionable.
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
25
5425
5ae8e644aa88 docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents: 5413
diff changeset
26 2. Switch from SQLite to PostgreSQL or MySQL
3224
8b8edfc25856 whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2775
diff changeset
27
5425
5ae8e644aa88 docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents: 5413
diff changeset
28 SQLite is a good option when having a small load on the system. But due to
5ae8e644aa88 docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents: 5413
diff changeset
29 locking issues with SQLite, it is not recommended to use it for larger
5ae8e644aa88 docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents: 5413
diff changeset
30 deployments. Switching to MySQL or PostgreSQL will result in an immediate
5060
778f7ae3b6eb docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents: 4955
diff changeset
31 performance increase. A tool like SQLAlchemyGrate_ can be used for
778f7ae3b6eb docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents: 4955
diff changeset
32 migrating to another database platform.
3224
8b8edfc25856 whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2775
diff changeset
33
4192
e73a69cb98dc Rename some strings examples and commands in documentation
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3413
diff changeset
34 3. Scale Kallithea horizontally
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
35
5425
5ae8e644aa88 docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents: 5413
diff changeset
36 Scaling horizontally can give huge performance benefits when dealing with
5ae8e644aa88 docs: spelling, grammar, content and typography
Søren Løvborg <sorenl@unity3d.com>
parents: 5413
diff changeset
37 large amounts of traffic (many users, CI servers, etc.). Kallithea can be
6762
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
38 scaled horizontally on one (recommended) or multiple machines.
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
39
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
40 It is generally possible to run WSGI applications multithreaded, so that
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
41 several HTTP requests are served from the same Python process at once. That
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
42 can in principle give better utilization of internal caches and less
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
43 process overhead.
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
44
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
45 One danger of running multithreaded is that program execution becomes much
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
46 more complex; programs must be written to consider all combinations of
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
47 events and problems might depend on timing and be impossible to reproduce.
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
48
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
49 Kallithea can't promise to be thread-safe, just like the embedded Mercurial
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
50 backend doesn't make any strong promises when used as Kallithea uses it.
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
51 Instead, we recommend scaling by using multiple server processes.
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
52
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
53 Web servers with multiple worker processes (such as ``mod_wsgi`` with the
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
54 ``WSGIDaemonProcess`` ``processes`` parameter) will work out of the box.
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
55
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
56 In order to scale horizontally on multiple machines, you need to do the
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
57 following:
3413
d79f3505549e whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 3390
diff changeset
58
4955
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
59 - Each instance needs its own .ini file and unique ``instance_id`` set.
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
60 - Each instance's ``data`` storage needs to be configured to be stored on a
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
61 shared disk storage, preferably together with repositories. This ``data``
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
62 dir contains template caches, sessions, whoosh index and is used for
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
63 task locking (so it is safe across multiple instances). Set the
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
64 ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir``
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
65 variables in each .ini file to a shared location across Kallithea instances
6762
716e53c085ff config: clarify that we only recommend and support single threaded operation
Mads Kiilerich <madski@unity3d.com>
parents: 5434
diff changeset
66 - If using several Celery instances,
4955
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
67 the message broker should be common to all of them (e.g., one
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
68 shared RabbitMQ server)
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
69 - Load balance using round robin or IP hash, recommended is writing LB rules
3390
4cd84f4f28fb performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents: 3224
diff changeset
70 that will separate regular user traffic from automated processes like CI
4cd84f4f28fb performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents: 3224
diff changeset
71 servers or build bots.
5060
778f7ae3b6eb docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents: 4955
diff changeset
72
5433
fbbe80e3322b docs: consistent spacing around headings
Mads Kiilerich <madski@unity3d.com>
parents: 5425
diff changeset
73
5060
778f7ae3b6eb docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents: 4955
diff changeset
74 .. _SQLAlchemyGrate: https://github.com/shazow/sqlalchemygrate