Mercurial > gemma

annotate pkg/scheduler/boot.go @ 1995:59055c8301df

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Move import queue to its own database namespace Authorisation of the import queue has to be handled differently from the waterway-related data in the waterway schema. This is easier to handle, if both are in their own schema/namespace.
author Tom Gottfried <tom@intevation.de>
date Thu, 24 Jan 2019 12:56:31 +0100
parents 8bbdad3f9af7
children c06a2e5397fc
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1557
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018 by via donau
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
13
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14 package scheduler
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 import (
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
17 "context"
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
18 "database/sql"
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
19 "log"
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
20
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
21 "gemma.intevation.de/gemma/pkg/auth"
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
22 "gemma.intevation.de/gemma/pkg/config"
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
23 )
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
24
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
25 const (
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
26 bootRole = "sys_admin"
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
27
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
28 selectImportConfSQL = `
1742
8bbdad3f9af7 Fixed SQL for booting scheduled imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1660
diff changeset
29 SELECT id, username, cron
1995
59055c8301df Move import queue to its own database namespace
Tom Gottfried <tom@intevation.de>
parents: 1742
diff changeset
30 FROM import.import_configuration
1557
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
31 WHERE cron IS NOT NULL`
1660
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
32
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
33 scheduledIDsSQL = `
1995
59055c8301df Move import queue to its own database namespace
Tom Gottfried <tom@intevation.de>
parents: 1742
diff changeset
34 SELECT id from import.import_configuration
1660
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
35 WHERE username = $1 AND cron IS NOT NULL`
1557
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
36 )
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
37
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
38 func init() { go boot() }
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
39
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
40 // boot starts the scheduler with the configurations from
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
41 // the database which have a schedule.
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
42 func boot() {
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
43 config.WaitReady()
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
44 log.Println("info: booting scheduler from database.")
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
45 ctx := context.Background()
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
46 err := auth.RunAs(
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
47 ctx, bootRole,
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
48 func(conn *sql.Conn) error {
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
49 rows, err := conn.QueryContext(ctx, selectImportConfSQL)
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
50 if err != nil {
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
51 return err
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
52 }
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
53 defer rows.Close()
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
54 err = BootActions(func(ba *BoundAction) (bool, error) {
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
55 if err != nil {
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
56 return false, err
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
57 }
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
58 if !rows.Next() {
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
59 return false, nil
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
60 }
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
61 var id int64
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
62 if err = rows.Scan(
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
63 &id,
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
64 &ba.Name,
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
65 &ba.Spec,
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
66 ); err != nil {
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
67 return false, err
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
68 }
1652
f39957ea08aa Scheduled imports: The configuration id is not optional as it used to load the concrete configuration from the database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1557
diff changeset
69 ba.CfgID = id
1557
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
70 return true, nil
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
71 })
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
72 if err != nil {
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
73 return err
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
74 }
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
75 return rows.Err()
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
76 })
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
77 if err != nil {
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
78 log.Printf("error: %v\n", err)
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
79 }
62171cd9a42b Import scheduler: Start scheduler a gemma boot time with configurations from database which have a schedule.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
80 }
1660
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
81
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
82 // ScheduledUserIDs returns the IDs with a schedule for a given user.
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
83 func ScheduledUserIDs(
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
84 ctx context.Context,
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
85 conn *sql.Conn,
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
86 user string,
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
87 ) (map[int64]struct{}, error) {
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
88 ids := map[int64]struct{}{}
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
89 rows, err := conn.QueryContext(ctx, scheduledIDsSQL, user)
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
90 if err != nil {
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
91 return nil, nil
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
92 }
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
93 defer rows.Close()
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
94 for rows.Next() {
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
95 var id int64
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
96 if err := rows.Scan(&id); err != nil {
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
97 return nil, err
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
98 }
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
99 ids[id] = struct{}{}
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
100 }
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
101 if err := rows.Err(); err != nil {
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
102 return nil, err
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
103 }
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
104 return ids, nil
f32e8a973b48 Scheduler: Moved code to figure out the scheduled jobs per user from the database to a more suited place.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
105 }