annotate scripts/dbmigrate-test @ 6554:2c3d30095d5e

gearbox: replace paster with something TurboGears2-ish that still works with the Pylons stack This is a step towards moving away from the Pylons stack to TurboGears2, but still independent of it. Some notes from the porting - it could perhaps be the missing(?) documentation for migrating from paster to gearbox: Note: 'gearbox' without parameters will crash - specify '-h' to get started testing. Replace paster summary = 'yada yada' with the first line of the docstring of the Command class ... or override get_description. Note: All newlines in the docstring will be collapsed and mangle the long help text. Grouping of commands is not possible. Standard commands (for development) can't be customized under the same name or hidden. (Like for paster, the conceptual model also assumes that the sub-command naming is namespaced so commands from other packages won't conflict.) The usage help is fully automated from the declared options. For all deprecated Commands, replace paster hidden = True with gearbox deprecated = True Note: config_file, takes_config_file, min_args and max_args are not available / relevant. The gearbox parser is customized by overriding get_parser - there is nothing like paster update_parser. Gearbox is using argparse instead of optparse ... but argparse add_argument is mostly backwards compatible with optparse add_option. Instead of overriding command or run as in paster, override take_action in gearbox. The parsed arguments are passed to take_action, not available on the command instance. Paster BadCommand is not available and must be handled manually, terminating with sys.exit(1). There is no standard make-config command in gearbox. Paster appinstall has been replaced by the somewhat different setup_app module in gearbox. There is still no clean way to pass parameters to SetupAppCommand and it relies on websetup and other apparently unnecessary complexity. Instead, implement setup-db from scratch. Minor change by Thomas De Schampheleire: add gearbox logging configuration. Because we use logging.config.fileConfig(.inifile) during gearbox command execution, the logging settings need to be correct and contain a block for gearbox logging itself. Otherwise, errors in command processing are not even visible and the command exits silently.
author Mads Kiilerich <madski@unity3d.com>
date Tue, 25 Oct 2016 21:32:24 +0200
parents e285bb7abb28
children 3158cf0dafb7
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6025
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
1 #!/bin/sh -e
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
2
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
3 if [ $# -lt 2 ] || [ $# -gt 3 ]; then
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
4 cat >&2 <<EOD
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
5 usage: $0 CONFIG_FILE FROM_REV [TO_REV]
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
6
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
7 Runs a database migration from FROM_REV to TO_REV (default: current
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
8 working directory parent), using the specified CONFIG_FILE (.ini file).
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
9
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
10 Test is run using a clean Kallithea install, in a temporary virtual
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
11 environment. FROM_REV and (optional) TO_REV should be Mercurial revision
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
12 identifiers (e.g. changeset hash or a version number tag). The working
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
13 directory is not touched, but the database referenced in the config file
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
14 will be (re)created.
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
15
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
16 Only SQLite is available out of the box; for MySQL or PostgreSQL, set
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
17 the EXTRA environment variable to the required package(s), and it'll
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
18 be installed in the virtual environment. (E.g. EXTRA=MySQL-python or
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
19 EXTRA=psycopg2.)
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
20
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
21 The temporary directory is not removed, allowing follow-up examination
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
22 of the upgrade results. It is, however, created in /tmp by default,
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
23 which many Linux distributions automatically clean at regular intervals.
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
24 EOD
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
25 exit 1
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
26 fi
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
27
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
28 config_file=$(readlink -f "$1")
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
29 from_rev=$2
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
30 to_rev=$3
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
31 source_repo=$(dirname "$(dirname "$(readlink -f "$0")")")
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
32
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
33 announce() {
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
34 echo
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
35 echo "$1"
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
36 echo
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
37 }
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
38
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
39 quiet_if_ok() (
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
40 local output
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
41 local st
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
42 set +e
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
43 output=$("$@" < /dev/null 2>&1)
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
44 st=$?
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
45 if [ $st -ne 0 ]; then
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
46 echo "$output" >&2
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
47 echo "Command $@ returned exit status $st." >&2
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
48 exit 1
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
49 fi
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
50 )
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
51
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
52 HG() {
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
53 "${HG:-hg}" --repository "$source_repo" "$@"
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
54 }
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
55
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
56 # If upgrading to "current revision", warn if working directory is dirty.
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
57 if [ ! "$to_rev" ] && [ "$(HG status -mard)" ]; then
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
58 announce "Warning: Uncommitted changes in working directory will be ignored!"
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
59 fi
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
60
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
61 from_rev_hash=$(HG id --id --rev "${from_rev:-.}")
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
62 to_rev_hash=$(HG id --id --rev "${to_rev:-.}")
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
63 temp=$(readlink -f "$(mktemp --tmpdir -d 'dbmigrate-test.XXXXXX')")
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
64
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
65 cat <<EOD
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
66 Config file: $config_file
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
67 EOD
6214
f973b866fffc Turbogears2 migration: use sqlalchemy.url iso sqlalchemy.db1.url
Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
parents: 6025
diff changeset
68 sed -n -e 's/^sqlalchemy\.url *= */Database URL: /p' "$config_file"
6025
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
69 cat <<EOD
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
70 Working dir: $temp
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
71 Repository: $source_repo
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
72 Upgrade from: $from_rev_hash (${from_rev:-current})
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
73 Upgrade to: $to_rev_hash (${to_rev:-current})
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
74 Extra packages: ${EXTRA:-(none)}
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
75 EOD
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
76
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
77 mkdir "$temp/repos" # empty
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
78
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
79 # Enable caching for old pip versions (this will cache the pip upgrade)
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
80 # Newer pip versions cache automatically, and don't use this variable.
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
81 if [ ! "$PIP_DOWNLOAD_CACHE" ]; then
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
82 export PIP_DOWNLOAD_CACHE=$HOME/.cache/pip/legacy
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
83 fi
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
84
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
85 install_kallithea() {
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
86 local prefix=$1
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
87 local rev=$2
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
88
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
89 announce "Installing Kallithea $rev in $prefix..."
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
90
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
91 "${VIRTUALENV:-virtualenv}" --quiet "$prefix-env"
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
92 HG archive --rev "$rev" "$prefix"
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
93
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
94 (
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
95 cd "$prefix"
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
96 . "$prefix-env/bin/activate"
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
97 pip install --quiet --upgrade pip setuptools mercurial $EXTRA
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
98 pip install --quiet -e .
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
99 )
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
100 }
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
101
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
102 install_kallithea "$temp/from" "$from_rev_hash"
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
103 (
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
104 cd "$temp/from"
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
105 . "$temp/from-env/bin/activate"
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
106 announce "Initializing database..."
6554
2c3d30095d5e gearbox: replace paster with something TurboGears2-ish that still works with the Pylons stack
Mads Kiilerich <madski@unity3d.com>
parents: 6370
diff changeset
107 quiet_if_ok gearbox setup-db -c "$config_file" --repos="$temp/repos" --user=doe --email=doe@example.com --password=123456 --no-public-access --force-yes
6025
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
108 alembic -c "$config_file" current -v
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
109 )
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
110
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
111 install_kallithea "$temp/to" "$to_rev_hash"
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
112 (
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
113 cd "$temp/to"
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
114 . "$temp/to-env/bin/activate"
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
115
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
116 announce "Commencing database upgrade from shown Alembic revision to head..."
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
117 alembic -c "$config_file" current -v
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
118 alembic -c "$config_file" upgrade head
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
119 announce "Upgrade complete, now at the shown Alembic revision:"
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
120 alembic -c "$config_file" current -v
51b1af58589b db: script to test database upgrades
Søren Løvborg <sorenl@unity3d.com>
parents:
diff changeset
121 )