annotate docs/dev/dbmigrations.rst @ 6554:2c3d30095d5e

gearbox: replace paster with something TurboGears2-ish that still works with the Pylons stack This is a step towards moving away from the Pylons stack to TurboGears2, but still independent of it. Some notes from the porting - it could perhaps be the missing(?) documentation for migrating from paster to gearbox: Note: 'gearbox' without parameters will crash - specify '-h' to get started testing. Replace paster summary = 'yada yada' with the first line of the docstring of the Command class ... or override get_description. Note: All newlines in the docstring will be collapsed and mangle the long help text. Grouping of commands is not possible. Standard commands (for development) can't be customized under the same name or hidden. (Like for paster, the conceptual model also assumes that the sub-command naming is namespaced so commands from other packages won't conflict.) The usage help is fully automated from the declared options. For all deprecated Commands, replace paster hidden = True with gearbox deprecated = True Note: config_file, takes_config_file, min_args and max_args are not available / relevant. The gearbox parser is customized by overriding get_parser - there is nothing like paster update_parser. Gearbox is using argparse instead of optparse ... but argparse add_argument is mostly backwards compatible with optparse add_option. Instead of overriding command or run as in paster, override take_action in gearbox. The parsed arguments are passed to take_action, not available on the command instance. Paster BadCommand is not available and must be handled manually, terminating with sys.exit(1). There is no standard make-config command in gearbox. Paster appinstall has been replaced by the somewhat different setup_app module in gearbox. There is still no clean way to pass parameters to SetupAppCommand and it relies on websetup and other apparently unnecessary complexity. Instead, implement setup-db from scratch. Minor change by Thomas De Schampheleire: add gearbox logging configuration. Because we use logging.config.fileConfig(.inifile) during gearbox command execution, the logging settings need to be correct and contain a block for gearbox logging itself. Otherwise, errors in command processing are not even visible and the command exits silently.
author Mads Kiilerich <madski@unity3d.com>
date Tue, 25 Oct 2016 21:32:24 +0200
parents 9fd64dd2617d
children 213085032127
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6023
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
1 =======================
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
2 Database schema changes
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
3 =======================
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
4
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
5 Kallithea uses Alembic for :ref:`database migrations <upgrade_db>`
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
6 (upgrades and downgrades).
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
7
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
8 If you are developing a Kallithea feature that requires database schema
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
9 changes, you should make a matching Alembic database migration script:
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
10
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
11 1. :ref:`Create a Kallithea configuration and database <setup>` for testing
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
12 the migration script, or use existing ``development.ini`` setup.
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
13
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
14 Ensure that this database is up to date with the latest database
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
15 schema *before* the changes you're currently developing. (Do not
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
16 create the database while your new schema changes are applied.)
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
17
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
18 2. Create a separate throwaway configuration for iterating on the actual
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
19 database changes::
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
20
6554
2c3d30095d5e gearbox: replace paster with something TurboGears2-ish that still works with the Pylons stack
Mads Kiilerich <madski@unity3d.com>
parents: 6023
diff changeset
21 TODO make-config Kallithea temp.ini
6023
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
22
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
23 Edit the file to change database settings. SQLite is typically fine,
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
24 but make sure to change the path to e.g. ``temp.db``, to avoid
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
25 clobbering any existing database file.
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
26
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
27 3. Make your code changes (including database schema changes in ``db.py``).
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
28
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
29 4. After every database schema change, recreate the throwaway database
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
30 to test the changes::
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
31
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
32 rm temp.db
6554
2c3d30095d5e gearbox: replace paster with something TurboGears2-ish that still works with the Pylons stack
Mads Kiilerich <madski@unity3d.com>
parents: 6023
diff changeset
33 gearbox setup-db -c temp.ini --repos=/var/repos --user=doe --email doe@example.com --password=123456 --no-public-access --force-yes
2c3d30095d5e gearbox: replace paster with something TurboGears2-ish that still works with the Pylons stack
Mads Kiilerich <madski@unity3d.com>
parents: 6023
diff changeset
34 gearbox repo-scan -c temp.ini
6023
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
35
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
36 5. Once satisfied with the schema changes, auto-generate a draft Alembic
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
37 script using the development database that has *not* been upgraded.
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
38 (The generated script will upgrade the database to match the code.)
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
39
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
40 ::
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
41
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
42 alembic -c development.ini revision -m "area: add cool feature" --autogenerate
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
43
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
44 6. Edit the script to clean it up and fix any problems.
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
45
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
46 Note that for changes that simply add columns, it may be appropriate
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
47 to not remove them in the downgrade script (and instead do nothing),
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
48 to avoid the loss of data. Unknown columns will simply be ignored by
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
49 Kallithea versions predating your changes.
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
50
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
51 7. Run ``alembic -c development.ini upgrade head`` to apply changes to
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
52 the (non-throwaway) database, and test the upgrade script. Also test
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
53 downgrades.
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
54
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
55 The included ``development.ini`` has full SQL logging enabled. If
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
56 you're using another configuration file, you may want to enable it
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
57 by setting ``level = DEBUG`` in section ``[handler_console_sql]``.
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
58
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
59 The Alembic migration script should be committed in the same revision as
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
60 the database schema (``db.py``) changes.
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
61
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
62 See the `Alembic documentation`__ for more information, in particular
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
63 the tutorial and the section about auto-generating migration scripts.
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
64
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
65 .. __: http://alembic.zzzcomputing.com/en/latest/
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
66
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
67
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
68 Troubleshooting
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
69 ---------------
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
70
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
71 * If ``alembic --autogenerate`` responds "Target database is not up to
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
72 date", you need to either first use Alembic to upgrade the database
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
73 to the most recent version (before your changes), or recreate the
9fd64dd2617d docs: document how to use Alembic for database migrations
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
74 database from scratch (without your schema changes applied).