Mercurial > gemma

annotate pkg/imports/ugm.go @ 5591:0011f50cf216 surveysperbottleneckid

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Removed no longer used alternative api for surveys/ endpoint. As bottlenecks in the summary for SR imports are now identified by their id and no longer by the (not guarantied to be unique!) name, there is no longer the need to request survey data by the name+date tuple (which isn't reliable anyway). So the workaround was now reversed.
author Sascha Wilde <wilde@sha-bang.de>
date Wed, 06 Apr 2022 13:30:29 +0200
parents 56c589f7435d
children 6270951dda28
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2245
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018 by via donau
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
13
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14 package imports
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 import (
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
17 "context"
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
18 "database/sql"
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
19 "errors"
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
20 "os"
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
21 "path/filepath"
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
22
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
23 "gemma.intevation.de/gemma/pkg/soap"
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
24 "gemma.intevation.de/gemma/pkg/soap/nts"
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
25 )
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
26
4182
49012340336c Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4177
diff changeset
27 // UploadedGaugeMeasurement is an Jon to extract gauge measurement data
49012340336c Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4177
diff changeset
28 // from an uploaded XML file and stores it into the database.
2249
35db2e11c966 Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2246
diff changeset
29 type UploadedGaugeMeasurement struct {
2245
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
30 Dir string `json:"dir"`
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
31 }
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
32
4182
49012340336c Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4177
diff changeset
33 // UGMJobKind is the unique name of this import job type.
2245
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
34 const UGMJobKind JobKind = "ugm"
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
35
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
36 type ugmJobCreator struct{}
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
37
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
38 func init() { RegisterJobCreator(UGMJobKind, ugmJobCreator{}) }
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
39
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
40 func (ugmJobCreator) Description() string { return "uploaded gauge measurements" }
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
41
2249
35db2e11c966 Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2246
diff changeset
42 func (ugmJobCreator) Create() Job { return new(UploadedGaugeMeasurement) }
2245
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
43
3219
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2278
diff changeset
44 func (ugmJobCreator) Depends() [2][]string { return gmJobCreator{}.Depends() }
2245
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
45
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
46 func (ugmJobCreator) AutoAccept() bool { return true }
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
47
5035
56c589f7435d Enhance comments on implementations of StageDone()
Tom Gottfried <tom@intevation.de>
parents: 5034
diff changeset
48 // StageDone is a NOP for gauge measurements imports.
5034
59a99655f34d Added feedback support for StageDone.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4182
diff changeset
49 func (ugmJobCreator) StageDone(context.Context, *sql.Tx, int64, Feedback) error { return nil }
2245
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
50
4182
49012340336c Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4177
diff changeset
51 // CleanUp removes the temporary files from the filesystem.
2249
35db2e11c966 Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2246
diff changeset
52 func (ugm *UploadedGaugeMeasurement) CleanUp() error { return os.RemoveAll(ugm.Dir) }
2245
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
53
2249
35db2e11c966 Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2246
diff changeset
54 // Do executes the actual uploaded gauge measurement import.
35db2e11c966 Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2246
diff changeset
55 func (ugm *UploadedGaugeMeasurement) Do(
2245
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
56 ctx context.Context,
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
57 importID int64,
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
58 conn *sql.Conn,
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
59 feedback Feedback,
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
60 ) (interface{}, error) {
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
61
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
62 fetch := func() ([]*nts.RIS_Message_Type, error) {
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
63
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
64 var dst nts.Get_messages_result
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
65
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
66 if err := soap.ValidateFile(
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
67 filepath.Join(ugm.Dir, "data.xml"),
2278
a3e7e727209a UGM import: Use special schema.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2249
diff changeset
68 "NtS.xsd",
2245
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
69 &dst,
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
70 ); err != nil {
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
71 return nil, err
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
72 }
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
73
2249
35db2e11c966 Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2246
diff changeset
74 if len(dst.Result_message) == 0 {
4177
8b75ac5e243e Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3219
diff changeset
75 return nil, errors.New("no gauge measurements found")
2245
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
76 }
2249
35db2e11c966 Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2246
diff changeset
77 return dst.Result_message, nil
2245
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
78 }
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
79
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
80 return storeGaugeMeasurements(
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
81 ctx,
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
82 importID,
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
83 fetch,
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
84 conn,
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
85 feedback,
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
86 )
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
87 }