Mercurial > gemma

annotate pkg/imports/scheduled.go @ 1705:dcbe2a7dc532

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Scheduled imports: Take extra configuration parameters from attributes table.
author Sascha L. Teichmann <sascha.teichmann@intevation.de>
date Mon, 07 Jan 2019 16:11:07 +0100
parents 8fdb57173e3f
children 49e047c2106e
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1551
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018 by via donau
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
13
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14 package imports
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 import (
1705
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
17 "errors"
1551
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
18 "log"
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
19
1661
51a0ba4ede41 Scheduled imports: Made gauge measurement import able to be scheduled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
20 "gemma.intevation.de/gemma/pkg/common"
1551
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
21 "gemma.intevation.de/gemma/pkg/scheduler"
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
22 )
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
23
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
24 func init() {
1705
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
25 registerAction(GMJobKind, func(cfg *IDConfig) (interface{}, error) {
1665
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
26 log.Println("info: schedule 'gm' import")
1705
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
27 insecure := cfg.Attributes.Bool("insecure")
1665
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
28 return &GaugeMeasurement{
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
29 URL: *cfg.URL,
1705
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
30 Insecure: insecure,
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
31 }, nil
1665
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
32 })
1705
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
33 registerAction(FAJobKind, func(cfg *IDConfig) (interface{}, error) {
1665
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
34 log.Println("info: schedule 'fa' import")
1705
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
35 insecure := cfg.Attributes.Bool("insecure")
1665
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
36 return &FairwayAvailability{
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
37 URL: *cfg.URL,
1705
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
38 Insecure: insecure,
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
39 }, nil
1665
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
40 })
1705
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
41 registerAction(BNJobKind, func(cfg *IDConfig) (interface{}, error) {
1666
56b29406a163 Bottleneck import: Made schedulable, too. Even if it's not required.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1665
diff changeset
42 log.Println("info: schedule 'bn' import")
1705
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
43 insecure := cfg.Attributes.Bool("insecure")
1666
56b29406a163 Bottleneck import: Made schedulable, too. Even if it's not required.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1665
diff changeset
44 return &Bottleneck{
56b29406a163 Bottleneck import: Made schedulable, too. Even if it's not required.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1665
diff changeset
45 URL: *cfg.URL,
1705
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
46 Insecure: insecure,
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
47 }, nil
1666
56b29406a163 Bottleneck import: Made schedulable, too. Even if it's not required.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1665
diff changeset
48 })
1705
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
49 registerAction(WXJobKind, func(cfg *IDConfig) (interface{}, error) {
1678
8fdb57173e3f Waterway axis import: Made schedulable, too. TODO: Remove hard-coded parameters.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1666
diff changeset
50 log.Println("info: schedule 'wx' import")
1705
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
51
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
52 ft, found := cfg.Attributes.Get("feature-type")
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
53 if !found {
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
54 return nil, errors.New("cannot find 'feature-type' attribute")
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
55 }
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
56 sb, found := cfg.Attributes.Get("sort-by")
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
57 if !found {
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
58 return nil, errors.New("cannot find 'sort-by' attribute")
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
59 }
1678
8fdb57173e3f Waterway axis import: Made schedulable, too. TODO: Remove hard-coded parameters.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1666
diff changeset
60 return &WaterwayAxis{
8fdb57173e3f Waterway axis import: Made schedulable, too. TODO: Remove hard-coded parameters.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1666
diff changeset
61 URL: *cfg.URL,
1705
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
62 FeatureType: ft,
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
63 SortBy: sb,
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
64 }, nil
1678
8fdb57173e3f Waterway axis import: Made schedulable, too. TODO: Remove hard-coded parameters.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1666
diff changeset
65 })
1551
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
66 }
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
67
1705
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
68 func registerAction(kind JobKind, setup func(cfg *IDConfig) (interface{}, error)) {
1665
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
69
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
70 action := func(id int64) {
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
71 cfg, err := loadIDConfig(id)
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
72 if err != nil {
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
73 log.Printf("error: %v\n", err)
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
74 return
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
75 }
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
76 if cfg == nil {
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
77 log.Printf("error: No config found for id %d.\n", id)
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
78 return
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
79 }
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
80 if cfg.URL == nil {
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
81 log.Println("error: No URL specified")
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
82 return
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
83 }
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
84
1705
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
85 what, err := setup(cfg)
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
86 if err != nil {
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
87 log.Printf("error: setup failed %v.\n", err)
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
88 return
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1678
diff changeset
89 }
1661
51a0ba4ede41 Scheduled imports: Made gauge measurement import able to be scheduled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
90
1665
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
91 var serialized string
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
92 if serialized, err = common.ToJSONString(what); err != nil {
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
93 log.Printf("error: %v\n", err)
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
94 return
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
95 }
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
96
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
97 var jobID int64
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
98 if jobID, err = AddJob(
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
99 kind,
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
100 cfg.User,
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
101 cfg.SendEMail, cfg.AutoAccept,
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
102 serialized,
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
103 ); err != nil {
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
104 log.Printf("error: %v\n", err)
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
105 return
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
106 }
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
107
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
108 log.Printf("info: added import #%d to queue\n", jobID)
1661
51a0ba4ede41 Scheduled imports: Made gauge measurement import able to be scheduled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
109 }
51a0ba4ede41 Scheduled imports: Made gauge measurement import able to be scheduled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1654
diff changeset
110
1665
da0d1a19ebe6 Fairway availability import: Made schedulable, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1661
diff changeset
111 scheduler.RegisterAction(string(kind), action)
1551
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
112 }