Mercurial > kallithea
annotate docs/usage/performance.rst @ 4997:9885bbacf99c
.hgignore: Add .eggs/ and tarballcache/
Setuptools >= 7.0 and .eggs is created and populated with eggs if setup.py has
setup_requires of packages that are not installed and setuptools has to
download them to satisfy the requirements. See
https://bitbucket.org/pypa/setuptools/pull-request/97/cache-eggs-required-for-building-in-eggs/diff#
tarballcache comes from test.ini : archive_cache_dir = %(here)s/tarballcache
author | Marc Abramowitz <marc@marc-abramowitz.com> |
---|---|
date | Wed, 08 Apr 2015 10:15:05 -0700 |
parents | 4e6dfdb3fa01 |
children | 778f7ae3b6eb |
rev | line source |
---|---|
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
1 .. _performance: |
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
2 |
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
3 ================================ |
4192
e73a69cb98dc
Rename some strings examples and commands in documentation
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents:
3413
diff
changeset
|
4 Optimizing Kallithea Performance |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
5 ================================ |
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
6 |
4955
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
7 When serving a large amount of big repositories, Kallithea can start |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
8 performing slower than expected. Because of the demanding nature of handling large |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
9 amounts of data from version control systems, here are some tips on how to get |
2775
5d12768a0aa1
docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents:
2680
diff
changeset
|
10 the best performance. |
5d12768a0aa1
docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents:
2680
diff
changeset
|
11 |
4192
e73a69cb98dc
Rename some strings examples and commands in documentation
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents:
3413
diff
changeset
|
12 * Kallithea will perform better on machines with faster disks (SSD/SAN). It's |
4955
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
13 more important to have a faster disk than a faster CPU. |
2775
5d12768a0aa1
docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents:
2680
diff
changeset
|
14 |
5d12768a0aa1
docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents:
2680
diff
changeset
|
15 * Slowness on initial page can be easily fixed by grouping repositories, and/or |
4955
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
16 increasing cache size (see below). This includes using the lightweight dashboard |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
17 option and ``vcs_full_cache`` setting in .ini file |
2775
5d12768a0aa1
docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents:
2680
diff
changeset
|
18 |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
19 |
4192
e73a69cb98dc
Rename some strings examples and commands in documentation
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents:
3413
diff
changeset
|
20 Follow these few steps to improve performance of Kallithea system. |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
21 |
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
22 |
2680 | 23 1. Increase cache |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
24 |
4955
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
25 In the .ini file:: |
3224
8b8edfc25856
whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2775
diff
changeset
|
26 |
2680 | 27 beaker.cache.sql_cache_long.expire=3600 <-- set this to higher number |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
28 |
4955
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
29 This option affects the cache expiration time for the main |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
30 page. Having several hundreds of repositories on main page can |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
31 sometimes make the system behave slowly when the cache expires for |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
32 all of them. Increasing the ``expire`` option to a day (86400) or a |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
33 week (604800) will improve general response times for the main |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
34 page. Kallithea has an intelligent cache expiration system and it |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
35 will expire the cache for repositories that have been changed. |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
36 |
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
37 2. Switch from sqlite to postgres or mysql |
3224
8b8edfc25856
whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2775
diff
changeset
|
38 |
4955
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
39 sqlite is a good option when having a small load on the system. But due to |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
40 locking issues with sqlite, it is not recommended to use it for larger |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
41 deployments. Switching to mysql or postgres will result in an immediate |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
42 performance increase. |
3224
8b8edfc25856
whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2775
diff
changeset
|
43 |
4192
e73a69cb98dc
Rename some strings examples and commands in documentation
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents:
3413
diff
changeset
|
44 3. Scale Kallithea horizontally |
2517
fa88997aa421
Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
45 |
4955
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
46 Scaling horizontally can give huge performance increases when dealing with |
4192
e73a69cb98dc
Rename some strings examples and commands in documentation
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents:
3413
diff
changeset
|
47 large traffic (large amount of users, CI servers etc). Kallithea can be |
3390
4cd84f4f28fb
performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
3224
diff
changeset
|
48 scaled horizontally on one (recommended) or multiple machines. In order |
4cd84f4f28fb
performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
3224
diff
changeset
|
49 to scale horizontally you need to do the following: |
3413
d79f3505549e
whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
3390
diff
changeset
|
50 |
4955
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
51 - Each instance needs its own .ini file and unique ``instance_id`` set. |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
52 - Each instance's ``data`` storage needs to be configured to be stored on a |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
53 shared disk storage, preferably together with repositories. This ``data`` |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
54 dir contains template caches, sessions, whoosh index and is used for |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
55 task locking (so it is safe across multiple instances). Set the |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
56 ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir`` |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
57 variables in each .ini file to a shared location across Kallithea instances |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
58 - If celery is used each instance should run a separate Celery instance, but |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
59 the message broker should be common to all of them (e.g., one |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
60 shared RabbitMQ server) |
4e6dfdb3fa01
docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents:
4192
diff
changeset
|
61 - Load balance using round robin or IP hash, recommended is writing LB rules |
3390
4cd84f4f28fb
performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
3224
diff
changeset
|
62 that will separate regular user traffic from automated processes like CI |
4cd84f4f28fb
performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
3224
diff
changeset
|
63 servers or build bots. |