Mercurial > kallithea

annotate docs/usage/performance.rst @ 5060:778f7ae3b6eb

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
author Anatoly Bubenkov <bubenkoff@gmail.com>
date Wed, 22 Apr 2015 14:57:12 +0000
parents 4e6dfdb3fa01
children 4610a39d3be9
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
1 .. _performance:
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
2
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
3 ================================
4192
e73a69cb98dc Rename some strings examples and commands in documentation
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3413
diff changeset
4 Optimizing Kallithea Performance
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
5 ================================
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
6
4955
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
7 When serving a large amount of big repositories, Kallithea can start
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
8 performing slower than expected. Because of the demanding nature of handling large
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
9 amounts of data from version control systems, here are some tips on how to get
2775
5d12768a0aa1 docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents: 2680
diff changeset
10 the best performance.
5d12768a0aa1 docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents: 2680
diff changeset
11
4192
e73a69cb98dc Rename some strings examples and commands in documentation
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3413
diff changeset
12 * Kallithea will perform better on machines with faster disks (SSD/SAN). It's
4955
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
13 more important to have a faster disk than a faster CPU.
2775
5d12768a0aa1 docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents: 2680
diff changeset
14
5d12768a0aa1 docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents: 2680
diff changeset
15 * Slowness on initial page can be easily fixed by grouping repositories, and/or
4955
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
16 increasing cache size (see below). This includes using the lightweight dashboard
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
17 option and ``vcs_full_cache`` setting in .ini file
2775
5d12768a0aa1 docs improvements
Marcin Kuzminski <marcin@python-works.com>
parents: 2680
diff changeset
18
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
19
4192
e73a69cb98dc Rename some strings examples and commands in documentation
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3413
diff changeset
20 Follow these few steps to improve performance of Kallithea system.
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
21
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
22
2680
3b179f1ac3a2 doc fixes
Marcin Kuzminski <marcin@python-works.com>
parents: 2517
diff changeset
23 1. Increase cache
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
24
4955
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
25 In the .ini file::
3224
8b8edfc25856 whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2775
diff changeset
26
2680
3b179f1ac3a2 doc fixes
Marcin Kuzminski <marcin@python-works.com>
parents: 2517
diff changeset
27 beaker.cache.sql_cache_long.expire=3600 <-- set this to higher number
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
28
4955
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
29 This option affects the cache expiration time for the main
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
30 page. Having several hundreds of repositories on main page can
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
31 sometimes make the system behave slowly when the cache expires for
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
32 all of them. Increasing the ``expire`` option to a day (86400) or a
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
33 week (604800) will improve general response times for the main
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
34 page. Kallithea has an intelligent cache expiration system and it
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
35 will expire the cache for repositories that have been changed.
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
36
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
37 2. Switch from sqlite to postgres or mysql
3224
8b8edfc25856 whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2775
diff changeset
38
4955
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
39 sqlite is a good option when having a small load on the system. But due to
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
40 locking issues with sqlite, it is not recommended to use it for larger
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
41 deployments. Switching to mysql or postgres will result in an immediate
5060
778f7ae3b6eb docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents: 4955
diff changeset
42 performance increase. A tool like SQLAlchemyGrate_ can be used for
778f7ae3b6eb docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents: 4955
diff changeset
43 migrating to another database platform.
3224
8b8edfc25856 whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2775
diff changeset
44
4192
e73a69cb98dc Rename some strings examples and commands in documentation
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3413
diff changeset
45 3. Scale Kallithea horizontally
2517
fa88997aa421 Added simple docs for optimizing RhodeCode performance
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
46
4955
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
47 Scaling horizontally can give huge performance increases when dealing with
4192
e73a69cb98dc Rename some strings examples and commands in documentation
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3413
diff changeset
48 large traffic (large amount of users, CI servers etc). Kallithea can be
3390
4cd84f4f28fb performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents: 3224
diff changeset
49 scaled horizontally on one (recommended) or multiple machines. In order
4cd84f4f28fb performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents: 3224
diff changeset
50 to scale horizontally you need to do the following:
3413
d79f3505549e whitespace cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 3390
diff changeset
51
4955
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
52 - Each instance needs its own .ini file and unique ``instance_id`` set.
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
53 - Each instance's ``data`` storage needs to be configured to be stored on a
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
54 shared disk storage, preferably together with repositories. This ``data``
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
55 dir contains template caches, sessions, whoosh index and is used for
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
56 task locking (so it is safe across multiple instances). Set the
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
57 ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir``
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
58 variables in each .ini file to a shared location across Kallithea instances
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
59 - If celery is used each instance should run a separate Celery instance, but
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
60 the message broker should be common to all of them (e.g., one
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
61 shared RabbitMQ server)
4e6dfdb3fa01 docs: English and consistency corrections
Michael V. DePalatis <mike@depalatis.net>
parents: 4192
diff changeset
62 - Load balance using round robin or IP hash, recommended is writing LB rules
3390
4cd84f4f28fb performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents: 3224
diff changeset
63 that will separate regular user traffic from automated processes like CI
4cd84f4f28fb performance section docs update
Marcin Kuzminski <marcin@python-works.com>
parents: 3224
diff changeset
64 servers or build bots.
5060
778f7ae3b6eb docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents: 4955
diff changeset
65
778f7ae3b6eb docs: add a reference to https://github.com/shazow/sqlalchemygrate for migration from sqlite to other dbs
Anatoly Bubenkov <bubenkoff@gmail.com>
parents: 4955
diff changeset
66 .. _SQLAlchemyGrate: https://github.com/shazow/sqlalchemygrate