Mercurial > gemma

annotate pkg/imports/gm.go @ 5490:5f47eeea988d logging

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Use own logging package.
author Sascha L. Teichmann <sascha.teichmann@intevation.de>
date Mon, 20 Sep 2021 17:45:39 +0200
parents 56c589f7435d
children 133dc5b3076a
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
3 //
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
6 //
3176
1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import
Tom Gottfried <tom@intevation.de>
parents: 2849
diff changeset
7 // Copyright (C) 2018, 2019 by via donau
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
10 //
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
11 // Author(s):
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
12 // * Raimund Renkert <raimund.renkert@intevation.de>
2245
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2244
diff changeset
13 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
3176
1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import
Tom Gottfried <tom@intevation.de>
parents: 2849
diff changeset
14 // * Tom Gottfried <tom.gottfried@intevation.de>
1694
4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 1688
diff changeset
15
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
16 package imports
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
17
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
18 import (
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
19 "context"
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
20 "database/sql"
1772
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
21 "fmt"
3301
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
22 "sort"
1772
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
23 "strings"
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
24 "time"
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
25
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5035
diff changeset
26 "github.com/jackc/pgx/pgtype"
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5035
diff changeset
27
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5035
diff changeset
28 "gemma.intevation.de/gemma/pkg/log"
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
29 "gemma.intevation.de/gemma/pkg/models"
4062
6c760abcff0e Move handling of PostgreSQL errors to own package
Tom Gottfried <tom@intevation.de>
parents: 4058
diff changeset
30 "gemma.intevation.de/gemma/pkg/pgxutils"
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
31 "gemma.intevation.de/gemma/pkg/soap/nts"
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
32 )
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
33
1696
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
34 // GaugeMeasurement is an import job to import
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
35 // gauges measurement data from a NtS SOAP service.
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
36 type GaugeMeasurement struct {
1696
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
37 // URL is the URL of the SOAP service.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
38 URL string `json:"url"`
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
39 // Insecure indicates if HTTPS traffic
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
40 // should validate certificates or not.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
41 Insecure bool `json:"insecure"`
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
42 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
43
4852
046a07a33b19 Fixed the golint issues of the imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4848
diff changeset
44 // Description gives a short info about relevant facts of this import.
4798
ca6a5f722471 Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents: 4505
diff changeset
45 func (gm *GaugeMeasurement) Description() (string, error) {
4799
f32d086b5dbf Removed the mechanical touch of the last commit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4798
diff changeset
46 return gm.URL, nil
4798
ca6a5f722471 Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents: 4505
diff changeset
47 }
ca6a5f722471 Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents: 4505
diff changeset
48
1696
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
49 // GMJobKind is the import queue type identifier.
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
50 const GMJobKind JobKind = "gm"
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
51
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
52 const (
3301
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
53 listGaugesSQL = `
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
54 SELECT
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
55 (location).country_code,
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
56 (location).locode,
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
57 (location).fairway_section,
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
58 (location).orc,
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
59 (location).hectometre
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
60 FROM waterway.gauges
4158
5466562cca60 Remove utility function with possibly bad performance impact
Tom Gottfried <tom@intevation.de>
parents: 4147
diff changeset
61 WHERE (location).country_code = (
5466562cca60 Remove utility function with possibly bad performance impact
Tom Gottfried <tom@intevation.de>
parents: 4147
diff changeset
62 SELECT country FROM users.list_users WHERE username = current_user)
3301
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
63 OR pg_has_role('sys_admin', 'MEMBER')
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
64 `
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
65
2849
b8972e4671fa Preserve old predictions on raw gauge measurement import.
Sascha Wilde <wilde@intevation.de>
parents: 2245
diff changeset
66 // Note: we do not expect corrections of data through this service. So
3302
ec6163c6687d 'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents: 3301
diff changeset
67 // any constraint conflicts are triggered by redundant data which
2849
b8972e4671fa Preserve old predictions on raw gauge measurement import.
Sascha Wilde <wilde@intevation.de>
parents: 2245
diff changeset
68 // can be dropped.
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
69 insertGMSQL = `
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
70 INSERT INTO waterway.gauge_measurements (
3302
ec6163c6687d 'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents: 3301
diff changeset
71 location,
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
72 measure_date,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
73 sender,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
74 language_code,
1766
e51db2a365a2 Insert country code in gm import.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1761
diff changeset
75 country_code,
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
76 date_issue,
1775
fcb0106ec510 Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1772
diff changeset
77 reference_code,
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
78 water_level,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
79 date_info,
1780
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1776
diff changeset
80 source_organization,
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1776
diff changeset
81 staging_done
3302
ec6163c6687d 'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents: 3301
diff changeset
82 ) VALUES (
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
83 ($1, $2, $3, $4, $5),
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
84 $6,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
85 $7,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
86 $8,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
87 $9,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
88 $10,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
89 $11,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
90 $12,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
91 $13,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
92 $14,
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
93 true
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
94 )
2849
b8972e4671fa Preserve old predictions on raw gauge measurement import.
Sascha Wilde <wilde@intevation.de>
parents: 2245
diff changeset
95 ON CONFLICT DO NOTHING
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
96 RETURNING 1
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
97 `
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
98
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
99 insertGPSQL = `
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
100 INSERT INTO waterway.gauge_predictions (
3302
ec6163c6687d 'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents: 3301
diff changeset
101 location,
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
102 measure_date,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
103 sender,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
104 language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
105 country_code,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
106 date_issue,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
107 reference_code,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
108 water_level,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
109 conf_interval,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
110 date_info,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
111 source_organization
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
112 ) VALUES(
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
113 ($1, $2, $3, $4, $5),
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
114 $6,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
115 $7,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
116 $8,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
117 $9,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
118 $10,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
119 $11,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
120 $12,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
121 $13,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
122 $14,
3278
831193935739 Avoid storing of NtS messages other than water level
Tom Gottfried <tom@intevation.de>
parents: 3277
diff changeset
123 $15
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
124 )
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
125 ON CONFLICT DO NOTHING
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
126 RETURNING 1
1978
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1954
diff changeset
127 `
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
128 )
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
129
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
130 type gmJobCreator struct{}
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
131
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
132 func init() {
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
133 RegisterJobCreator(GMJobKind, gmJobCreator{})
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
134 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
135
2187
7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
136 func (gmJobCreator) Description() string { return "gauge measurements" }
1646
a0982c38eac0 Import queue: Implemented email notifications.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1641
diff changeset
137
2187
7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
138 func (gmJobCreator) Create() Job { return new(GaugeMeasurement) }
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
139
3219
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3176
diff changeset
140 func (gmJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3176
diff changeset
141 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3176
diff changeset
142 {"gauge_measurements"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3176
diff changeset
143 {"gauges"},
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
144 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
145 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
146
1754
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1696
diff changeset
147 func (gmJobCreator) AutoAccept() bool { return true }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1696
diff changeset
148
5035
56c589f7435d Enhance comments on implementations of StageDone()
Tom Gottfried <tom@intevation.de>
parents: 5034
diff changeset
149 // StageDone is a NOP for gauge measurements imports.
5034
59a99655f34d Added feedback support for StageDone.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4852
diff changeset
150 func (gmJobCreator) StageDone(context.Context, *sql.Tx, int64, Feedback) error {
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
151 return nil
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
152 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
153
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
154 // CleanUp of a gauge measurement import is a NOP.
1676
4407ecaa2192 Imports: Cosmetics.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1664
diff changeset
155 func (*GaugeMeasurement) CleanUp() error { return nil }
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
156
4845
8c46b845b406 Fix comment
Tom Gottfried <tom@intevation.de>
parents: 4813
diff changeset
157 // Do executes the actual import.
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
158 func (gm *GaugeMeasurement) Do(
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
159 ctx context.Context,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
160 importID int64,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
161 conn *sql.Conn,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
162 feedback Feedback,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
163 ) (interface{}, error) {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
164
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
165 fetch := func() ([]*nts.RIS_Message_Type, error) {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
166 client := nts.NewINtSMessageService(gm.URL, gm.Insecure, nil)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
167
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
168 mt := nts.Message_type_typeWRM
4504
b7e3d42fd9e7 When fetching gauge measurements look 1day into past and future.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4197
diff changeset
169 now := time.Now()
4505
30c499f996d7 Simplified code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4504
diff changeset
170 dis := []*nts.Date_pair{{
4504
b7e3d42fd9e7 When fetching gauge measurements look 1day into past and future.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4197
diff changeset
171 Date_start: nts.Date{Time: now.AddDate(0, 0, -1)},
b7e3d42fd9e7 When fetching gauge measurements look 1day into past and future.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4197
diff changeset
172 Date_end: nts.Date{Time: now.AddDate(0, 0, +1)},
4505
30c499f996d7 Simplified code.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4504
diff changeset
173 }}
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
174
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
175 req := &nts.Get_messages_query{
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
176 Message_type: &mt,
3709
d4a8d084eab4 import: Use dates_issue parameter in gm imports.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3583
diff changeset
177 Dates_issue: dis,
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
178 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
179
3582
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3580
diff changeset
180 const maxTries = 3
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3580
diff changeset
181
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3580
diff changeset
182 tries := 0
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3580
diff changeset
183
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3580
diff changeset
184 again:
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
185 resp, err := client.Get_messages(req)
3582
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3580
diff changeset
186
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
187 if err != nil {
3582
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3580
diff changeset
188 if t, ok := err.(interface{ Timeout() bool }); ok && t.Timeout() && tries < maxTries {
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5035
diff changeset
189 log.Warnln("NtS SOAP request timed out. Trying again.")
3582
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3580
diff changeset
190 tries++
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3580
diff changeset
191 goto again
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3580
diff changeset
192 }
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3580
diff changeset
193 return nil, fmt.Errorf(
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3580
diff changeset
194 "Error requesting NtS service: %v", err)
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
195 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
196
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
197 result := resp.Result_message
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
198 if result == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
199 for _, e := range resp.Result_error {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
200 if e != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
201 feedback.Error("Error code: %s", *e)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
202 } else {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
203 feedback.Error("Unknown error")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
204 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
205 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
206 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
207 return result, nil
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
208 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
209
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
210 return storeGaugeMeasurements(
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
211 ctx,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
212 importID,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
213 fetch,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
214 conn,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
215 feedback,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
216 )
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
217 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
218
3301
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
219 func loadGauges(ctx context.Context, conn *sql.Conn) ([]string, error) {
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
220
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
221 rows, err := conn.QueryContext(ctx, listGaugesSQL)
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
222 if err != nil {
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
223 return nil, err
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
224 }
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
225 defer rows.Close()
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
226
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
227 var gauges []string
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
228
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
229 for rows.Next() {
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
230 var g models.Isrs
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
231 if err = rows.Scan(
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
232 &g.CountryCode,
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
233 &g.LoCode,
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
234 &g.FairwaySection,
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
235 &g.Orc,
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
236 &g.Hectometre,
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
237 ); err != nil {
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
238 return nil, err
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
239 }
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
240 gauges = append(gauges, g.String())
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
241 }
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
242
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
243 if err = rows.Err(); err != nil {
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
244 return nil, err
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
245 }
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
246
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
247 if len(gauges) == 0 {
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
248 return nil, UnchangedError(
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
249 "No gauges for which measurements can be imported in database")
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
250 }
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
251
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
252 sort.Strings(gauges)
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
253
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
254 return gauges, nil
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
255 }
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
256
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
257 func storeGaugeMeasurements(
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
258 ctx context.Context,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
259 importID int64,
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
260 fetch func() ([]*nts.RIS_Message_Type, error),
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
261 conn *sql.Conn,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
262 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
263 ) (interface{}, error) {
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
264
2233
137addc77b1b Gauge measurement imports: Do all database ops in one transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
265 start := time.Now()
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
266
3301
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
267 // Get gauges from database, for which user is allowed to import data
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
268 gauges, err := loadGauges(ctx, conn)
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
269 if err != nil {
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
270 return nil, err
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
271 }
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
272
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
273 gids, err := doForGM(ctx, gauges, fetch, conn, feedback)
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
274 if err != nil {
1761
0145537d5474 Gauge measument import: Die with an error if something went wrong.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1754
diff changeset
275 return nil, err
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
276 }
2233
137addc77b1b Gauge measurement imports: Do all database ops in one transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
277
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
278 if len(gids) == 0 {
2233
137addc77b1b Gauge measurement imports: Do all database ops in one transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
279 return nil, UnchangedError("No new gauge measurements found")
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
280 }
2233
137addc77b1b Gauge measurement imports: Do all database ops in one transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
281
137addc77b1b Gauge measurement imports: Do all database ops in one transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
282 feedback.Info(
3176
1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import
Tom Gottfried <tom@intevation.de>
parents: 2849
diff changeset
283 "Importing gauge measurements took %s", time.Since(start))
2233
137addc77b1b Gauge measurement imports: Do all database ops in one transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
284
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
285 // TODO: needs to be filled more useful.
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
286 summary := struct {
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
287 GaugeMeasuremets []string `json:"gaugeMeasurements"`
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
288 }{
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
289 GaugeMeasuremets: gids,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
290 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
291 return &summary, err
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
292 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
293
1772
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
294 // rescale returns a scaling function to bring the unit all to cm.
3264
9ae43313b463 Handle some possibly missing elements in NtS response
Tom Gottfried <tom@intevation.de>
parents: 3219
diff changeset
295 func rescale(unit string) (func(*float32), error) {
1772
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
296
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
297 var scale float32
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
298
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
299 switch strings.ToLower(unit) {
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
300 case "mm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
301 scale = 0.1
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
302 case "cm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
303 scale = 1.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
304 case "dm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
305 scale = 10.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
306 case "m":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
307 scale = 100.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
308 case "hm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
309 scale = 10000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
310 case "km":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
311 scale = 100000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
312 default:
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
313 return nil, fmt.Errorf("unknown unit '%s'", unit)
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
314 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
315
3264
9ae43313b463 Handle some possibly missing elements in NtS response
Tom Gottfried <tom@intevation.de>
parents: 3219
diff changeset
316 fn := func(x *float32) {
9ae43313b463 Handle some possibly missing elements in NtS response
Tom Gottfried <tom@intevation.de>
parents: 3219
diff changeset
317 if x != nil {
3266
3dee5cf16a58 Cosmetics.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3264
diff changeset
318 *x *= scale
3264
9ae43313b463 Handle some possibly missing elements in NtS response
Tom Gottfried <tom@intevation.de>
parents: 3219
diff changeset
319 }
9ae43313b463 Handle some possibly missing elements in NtS response
Tom Gottfried <tom@intevation.de>
parents: 3219
diff changeset
320 }
1772
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
321 return fn, nil
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
322 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
323
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
324 func doForGM(
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
325 ctx context.Context,
3301
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
326 gauges []string,
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
327 fetch func() ([]*nts.RIS_Message_Type, error),
3176
1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import
Tom Gottfried <tom@intevation.de>
parents: 2849
diff changeset
328 conn *sql.Conn,
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
329 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
330 ) ([]string, error) {
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
331
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
332 insertGPStmt, err := conn.PrepareContext(ctx, insertGPSQL)
2234
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2233
diff changeset
333 if err != nil {
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2233
diff changeset
334 return nil, err
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2233
diff changeset
335 }
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
336 defer insertGPStmt.Close()
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
337
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
338 insertGMStmt, err := conn.PrepareContext(ctx, insertGMSQL)
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
339 if err != nil {
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
340 return nil, err
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
341 }
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
342 defer insertGMStmt.Close()
2234
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2233
diff changeset
343
3301
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
344 // lookup to see if data can be imported for gauge
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
345 isKnown := func(s string) bool {
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
346 idx := sort.SearchStrings(gauges, s)
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
347 return idx < len(gauges) && gauges[idx] == s
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
348 }
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
349
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
350 result, err := fetch()
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
351 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
352 return nil, err
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
353 }
1981
e89368aec538 Import: Make one request per gauge for gauge measurement data.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1978
diff changeset
354
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
355 var gids []string
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
356 for _, msg := range result {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
357 for _, wrm := range msg.Wrm {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
358 curr := string(*wrm.Geo_object.Id)
3718
7431c6680b90 import: Trim whitespaces in ISRS code and improved log messsage.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3709
diff changeset
359 curr = strings.TrimSpace(curr)
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
360 currIsrs, err := models.IsrsFromString(curr)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
361 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
362 feedback.Warn("Invalid ISRS code %v", err)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
363 continue
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
364 }
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
365 feedback.Info("Found measurements/predictions for %s", curr)
3301
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
366 if !isKnown(curr) {
3718
7431c6680b90 import: Trim whitespaces in ISRS code and improved log messsage.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3709
diff changeset
367 feedback.Warn("Cannot find gauge %q for import", curr)
3301
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
368 continue
6514b943654e Re-enable checking of gauge availability
Tom Gottfried <tom@intevation.de>
parents: 3284
diff changeset
369 }
1981
e89368aec538 Import: Make one request per gauge for gauge measurement data.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1978
diff changeset
370
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
371 var referenceCode string
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
372 if wrm.Reference_code == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
373 feedback.Info("'Reference_code' not specified. Assuming 'ZPG'")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
374 referenceCode = "ZPG"
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
375 } else {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
376 referenceCode = string(*wrm.Reference_code)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
377 }
2849
b8972e4671fa Preserve old predictions on raw gauge measurement import.
Sascha Wilde <wilde@intevation.de>
parents: 2245
diff changeset
378
3847
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999.
Sascha Wilde <wilde@intevation.de>
parents: 3723
diff changeset
379 badValue := 0
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
380 newM, newP := 0, 0
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
381 for _, measure := range wrm.Measure {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
382 var unit string
3723
9759355d7b90 import: Fixed data initialization for database rows.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3718
diff changeset
383 if *measure.Measure_code != nts.Measure_code_enumWAL {
9759355d7b90 import: Fixed data initialization for database rows.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3718
diff changeset
384 feedback.Warn("Ignored message with measure_code %s",
9759355d7b90 import: Fixed data initialization for database rows.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3718
diff changeset
385 *measure.Measure_code)
9759355d7b90 import: Fixed data initialization for database rows.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3718
diff changeset
386 continue
9759355d7b90 import: Fixed data initialization for database rows.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3718
diff changeset
387 }
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
388 if measure.Unit == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
389 feedback.Info("'Unit' not specified. Assuming 'cm'")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
390 unit = "cm"
1772
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
391 } else {
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
392 unit = string(*measure.Unit)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
393 }
4813
0644c2b3af54 Check for mandatory measurement value in GM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4812
diff changeset
394
0644c2b3af54 Check for mandatory measurement value in GM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4812
diff changeset
395 if measure.Value == nil {
0644c2b3af54 Check for mandatory measurement value in GM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4812
diff changeset
396 feedback.Warn("Missing mandatory value at %s. Ignored (bad service)",
0644c2b3af54 Check for mandatory measurement value in GM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4812
diff changeset
397 measure.Measuredate.Format(time.RFC3339))
0644c2b3af54 Check for mandatory measurement value in GM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4812
diff changeset
398 continue
0644c2b3af54 Check for mandatory measurement value in GM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4812
diff changeset
399 }
0644c2b3af54 Check for mandatory measurement value in GM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4812
diff changeset
400
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
401 convert, err := rescale(unit)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
402 if err != nil {
4848
8584197232ec Handle unknown units gracefully
Tom Gottfried <tom@intevation.de>
parents: 4845
diff changeset
403 feedback.Error(err.Error())
8584197232ec Handle unknown units gracefully
Tom Gottfried <tom@intevation.de>
parents: 4845
diff changeset
404 continue
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
405 }
3264
9ae43313b463 Handle some possibly missing elements in NtS response
Tom Gottfried <tom@intevation.de>
parents: 3219
diff changeset
406 convert(measure.Value)
9ae43313b463 Handle some possibly missing elements in NtS response
Tom Gottfried <tom@intevation.de>
parents: 3219
diff changeset
407 convert(measure.Value_min)
9ae43313b463 Handle some possibly missing elements in NtS response
Tom Gottfried <tom@intevation.de>
parents: 3219
diff changeset
408 convert(measure.Value_max)
9ae43313b463 Handle some possibly missing elements in NtS response
Tom Gottfried <tom@intevation.de>
parents: 3219
diff changeset
409
3847
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999.
Sascha Wilde <wilde@intevation.de>
parents: 3723
diff changeset
410 // -99999 is used by some gauges to signal an error
4813
0644c2b3af54 Check for mandatory measurement value in GM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4812
diff changeset
411 if *measure.Value == -99999 {
3847
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999.
Sascha Wilde <wilde@intevation.de>
parents: 3723
diff changeset
412 badValue++
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999.
Sascha Wilde <wilde@intevation.de>
parents: 3723
diff changeset
413 continue
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999.
Sascha Wilde <wilde@intevation.de>
parents: 3723
diff changeset
414 }
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999.
Sascha Wilde <wilde@intevation.de>
parents: 3723
diff changeset
415
3580
d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3533
diff changeset
416 var dummy int
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
417 if measure.Predicted {
3723
9759355d7b90 import: Fixed data initialization for database rows.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3718
diff changeset
418 confInterval := pgtype.Numrange{
9759355d7b90 import: Fixed data initialization for database rows.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3718
diff changeset
419 Lower: pgtype.Numeric{Status: pgtype.Null},
9759355d7b90 import: Fixed data initialization for database rows.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3718
diff changeset
420 Upper: pgtype.Numeric{Status: pgtype.Null},
9759355d7b90 import: Fixed data initialization for database rows.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3718
diff changeset
421 LowerType: pgtype.Inclusive,
9759355d7b90 import: Fixed data initialization for database rows.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3718
diff changeset
422 UpperType: pgtype.Inclusive,
9759355d7b90 import: Fixed data initialization for database rows.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3718
diff changeset
423 Status: pgtype.Null,
9759355d7b90 import: Fixed data initialization for database rows.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3718
diff changeset
424 }
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
425 if measure.Value_min != nil && measure.Value_max != nil {
3723
9759355d7b90 import: Fixed data initialization for database rows.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3718
diff changeset
426 valueMin := pgtype.Numeric{Status: pgtype.Null}
9759355d7b90 import: Fixed data initialization for database rows.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3718
diff changeset
427 valueMax := pgtype.Numeric{Status: pgtype.Null}
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
428 valueMin.Set(measure.Value_min)
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
429 valueMax.Set(measure.Value_max)
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
430 confInterval = pgtype.Numrange{
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
431 Lower: valueMin,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
432 Upper: valueMax,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
433 LowerType: pgtype.Inclusive,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
434 UpperType: pgtype.Inclusive,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
435 Status: pgtype.Present,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
436 }
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
437 }
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
438 err = insertGPStmt.QueryRowContext(
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
439 ctx,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
440 currIsrs.CountryCode,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
441 currIsrs.LoCode,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
442 currIsrs.FairwaySection,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
443 currIsrs.Orc,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
444 currIsrs.Hectometre,
3580
d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3533
diff changeset
445 measure.Measuredate.Time,
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
446 msg.Identification.From,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
447 msg.Identification.Language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
448 msg.Identification.Country_code,
3580
d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3533
diff changeset
449 msg.Identification.Date_issue.Time,
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
450 referenceCode,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
451 measure.Value,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
452 &confInterval,
3580
d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3533
diff changeset
453 msg.Identification.Date_issue.Time,
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
454 msg.Identification.Originator,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
455 ).Scan(&dummy)
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
456 switch {
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
457 case err == sql.ErrNoRows:
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
458 // thats expected, nothing to do
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
459 case err != nil:
4197
5d7ce7f926eb Log fatal errors as errors, not warnings, in imports
Tom Gottfried <tom@intevation.de>
parents: 4158
diff changeset
460 feedback.Error(pgxutils.ReadableError{Err: err}.Error())
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
461 default:
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
462 newP++
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
463 }
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
464 } else {
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
465 err = insertGMStmt.QueryRowContext(
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
466 ctx,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
467 currIsrs.CountryCode,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
468 currIsrs.LoCode,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
469 currIsrs.FairwaySection,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
470 currIsrs.Orc,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
471 currIsrs.Hectometre,
3580
d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3533
diff changeset
472 measure.Measuredate.Time,
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
473 msg.Identification.From,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
474 msg.Identification.Language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
475 msg.Identification.Country_code,
3580
d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3533
diff changeset
476 msg.Identification.Date_issue.Time,
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
477 referenceCode,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
478 measure.Value,
3580
d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3533
diff changeset
479 msg.Identification.Date_issue.Time,
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
480 msg.Identification.Originator,
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
481 ).Scan(&dummy)
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
482 switch {
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
483 case err == sql.ErrNoRows:
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
484 // thats expected, nothing to do
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
485 case err != nil:
4197
5d7ce7f926eb Log fatal errors as errors, not warnings, in imports
Tom Gottfried <tom@intevation.de>
parents: 4158
diff changeset
486 feedback.Error(pgxutils.ReadableError{Err: err}.Error())
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
487 default:
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
488 newM++
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
489 }
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
490 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
491 }
3847
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999.
Sascha Wilde <wilde@intevation.de>
parents: 3723
diff changeset
492 if badValue > 0 {
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999.
Sascha Wilde <wilde@intevation.de>
parents: 3723
diff changeset
493 feedback.Warn("Ignored %d measurements with value -99999",
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999.
Sascha Wilde <wilde@intevation.de>
parents: 3723
diff changeset
494 badValue)
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999.
Sascha Wilde <wilde@intevation.de>
parents: 3723
diff changeset
495 }
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
496 feedback.Info("Inserted %d measurements for %s",
3277
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
497 newM, curr)
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
498 feedback.Info("Inserted %d predictions for %s",
232fc90e6ee2 Disentangle gauge measurements and predictions
Tom Gottfried <tom@intevation.de>
parents: 3266
diff changeset
499 newP, curr)
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
500 gids = append(gids, curr)
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
501 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
502 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
503 return gids, nil
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
504 }