Mercurial > gemma
annotate pkg/imports/ugm.go @ 4113:f39d20427e89 request_hist_bns
WIP: Reworked bottleneck import:
- all available bottlenecks from the service are loaded (including
historic data) - kudos to Tom Gottfried
- Bottlenecks which are identical to an exiting (non staged) one are
recognized as "unchanged" and ignored.
- All remaining Bottlenecks are inserted into Staging area
- On "staging done" all existing bottlenecks with conflicting
(bottleneck_id,validity) are deleted (replaced by the new
bottlenecks).
Known Issues:
- Sometimes the identification of unchanged bottlenecks does not work,
needs debugging.
- The front end does not handle historic data correctly (we need to
define what "correctly" should be).
- Save points should be used to distinguish ignored errors from fatal
ones.
author | Sascha Wilde <wilde@intevation.de> |
---|---|
date | Wed, 31 Jul 2019 17:04:31 +0200 |
parents | 4acbee65275d |
children | 8b75ac5e243e |
rev | line source |
---|---|
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2018 by via donau |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package imports |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 "context" |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 "database/sql" |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "errors" |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "os" |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
21 "path/filepath" |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
22 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
23 "gemma.intevation.de/gemma/pkg/soap" |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
24 "gemma.intevation.de/gemma/pkg/soap/nts" |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 ) |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
26 |
2249
35db2e11c966
Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2246
diff
changeset
|
27 type UploadedGaugeMeasurement struct { |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
28 Dir string `json:"dir"` |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
29 } |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
30 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
31 const UGMJobKind JobKind = "ugm" |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
32 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
33 type ugmJobCreator struct{} |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
34 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
35 func init() { RegisterJobCreator(UGMJobKind, ugmJobCreator{}) } |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
36 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
37 func (ugmJobCreator) Description() string { return "uploaded gauge measurements" } |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
38 |
2249
35db2e11c966
Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2246
diff
changeset
|
39 func (ugmJobCreator) Create() Job { return new(UploadedGaugeMeasurement) } |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
40 |
3219
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2278
diff
changeset
|
41 func (ugmJobCreator) Depends() [2][]string { return gmJobCreator{}.Depends() } |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
42 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
43 func (ugmJobCreator) AutoAccept() bool { return true } |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
44 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 func (ugmJobCreator) StageDone(context.Context, *sql.Tx, int64) error { return nil } |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
46 |
2249
35db2e11c966
Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2246
diff
changeset
|
47 func (ugm *UploadedGaugeMeasurement) CleanUp() error { return os.RemoveAll(ugm.Dir) } |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
48 |
2249
35db2e11c966
Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2246
diff
changeset
|
49 // Do executes the actual uploaded gauge measurement import. |
35db2e11c966
Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2246
diff
changeset
|
50 func (ugm *UploadedGaugeMeasurement) Do( |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
51 ctx context.Context, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
52 importID int64, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
53 conn *sql.Conn, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
54 feedback Feedback, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
55 ) (interface{}, error) { |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
56 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 fetch := func() ([]*nts.RIS_Message_Type, error) { |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
58 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
59 var dst nts.Get_messages_result |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
60 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
61 if err := soap.ValidateFile( |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
62 filepath.Join(ugm.Dir, "data.xml"), |
2278
a3e7e727209a
UGM import: Use special schema.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2249
diff
changeset
|
63 "NtS.xsd", |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
64 &dst, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
65 ); err != nil { |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
66 return nil, err |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
67 } |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
68 |
2249
35db2e11c966
Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2246
diff
changeset
|
69 if len(dst.Result_message) == 0 { |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
70 return nil, errors.New("No gauge measurements found") |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
71 } |
2249
35db2e11c966
Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2246
diff
changeset
|
72 return dst.Result_message, nil |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
73 } |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
74 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
75 return storeGaugeMeasurements( |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
76 ctx, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
77 importID, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
78 fetch, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
79 conn, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
80 feedback, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
81 ) |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
82 } |