Mercurial > gemma

annotate pkg/imports/gm.go @ 3224:bfde4f8dd323

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Add missing table dependencies of imports with area generation
author Tom Gottfried <tom@intevation.de>
date Thu, 09 May 2019 15:50:07 +0200
parents 4acbee65275d
children 9ae43313b463
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
3 //
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
6 //
3176
1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import
Tom Gottfried <tom@intevation.de>
parents: 2849
diff changeset
7 // Copyright (C) 2018, 2019 by via donau
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
10 //
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
11 // Author(s):
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
12 // * Raimund Renkert <raimund.renkert@intevation.de>
2245
7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2244
diff changeset
13 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
3176
1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import
Tom Gottfried <tom@intevation.de>
parents: 2849
diff changeset
14 // * Tom Gottfried <tom.gottfried@intevation.de>
1694
4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 1688
diff changeset
15
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
16 package imports
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
17
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
18 import (
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
19 "context"
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
20 "database/sql"
1772
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
21 "fmt"
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
22 "strings"
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
23 "time"
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
24
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
25 "gemma.intevation.de/gemma/pkg/models"
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
26 "gemma.intevation.de/gemma/pkg/soap/nts"
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
27 )
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
28
1696
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
29 // GaugeMeasurement is an import job to import
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
30 // gauges measurement data from a NtS SOAP service.
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
31 type GaugeMeasurement struct {
1696
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
32 // URL is the URL of the SOAP service.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
33 URL string `json:"url"`
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
34 // Insecure indicates if HTTPS traffic
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
35 // should validate certificates or not.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
36 Insecure bool `json:"insecure"`
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
37 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
38
1696
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1694
diff changeset
39 // GMJobKind is the import queue type identifier.
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
40 const GMJobKind JobKind = "gm"
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
41
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
42 const (
2849
b8972e4671fa Preserve old predictions on raw gauge measurement import.
Sascha Wilde <wilde@intevation.de>
parents: 2245
diff changeset
43 // Note: we do not expect corrections of data through this service. So
b8972e4671fa Preserve old predictions on raw gauge measurement import.
Sascha Wilde <wilde@intevation.de>
parents: 2245
diff changeset
44 // any constraint conflicts are triggered by actual redundat data which
b8972e4671fa Preserve old predictions on raw gauge measurement import.
Sascha Wilde <wilde@intevation.de>
parents: 2245
diff changeset
45 // can be dropped.
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
46 insertGMSQL = `
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
47 INSERT INTO waterway.gauge_measurements (
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
48 fk_gauge_id,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
49 measure_date,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
50 sender,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
51 language_code,
1766
e51db2a365a2 Insert country code in gm import.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1761
diff changeset
52 country_code,
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
53 date_issue,
1775
fcb0106ec510 Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1772
diff changeset
54 reference_code,
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
55 water_level,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
56 predicted,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
57 is_waterlevel,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
58 value_min,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
59 value_max,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
60 date_info,
1780
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1776
diff changeset
61 source_organization,
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1776
diff changeset
62 staging_done
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
63 ) VALUES(
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
64 ($1, $2, $3, $4, $5),
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
65 $6,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
66 $7,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
67 $8,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
68 $9,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
69 $10,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
70 $11,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
71 $12,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
72 $13,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
73 $14,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
74 $15,
1766
e51db2a365a2 Insert country code in gm import.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1761
diff changeset
75 $16,
1775
fcb0106ec510 Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1772
diff changeset
76 $17,
1780
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1776
diff changeset
77 $18,
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1776
diff changeset
78 $19
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
79 )
2849
b8972e4671fa Preserve old predictions on raw gauge measurement import.
Sascha Wilde <wilde@intevation.de>
parents: 2245
diff changeset
80 ON CONFLICT DO NOTHING
1978
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1954
diff changeset
81 RETURNING id
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1954
diff changeset
82 `
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
83 )
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
84
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
85 type gmJobCreator struct{}
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
86
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
87 func init() {
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
88 RegisterJobCreator(GMJobKind, gmJobCreator{})
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
89 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
90
2187
7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
91 func (gmJobCreator) Description() string { return "gauge measurements" }
1646
a0982c38eac0 Import queue: Implemented email notifications.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1641
diff changeset
92
2187
7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
93 func (gmJobCreator) Create() Job { return new(GaugeMeasurement) }
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
94
3219
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3176
diff changeset
95 func (gmJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3176
diff changeset
96 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3176
diff changeset
97 {"gauge_measurements"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3176
diff changeset
98 {"gauges"},
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
99 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
100 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
101
1754
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1696
diff changeset
102 func (gmJobCreator) AutoAccept() bool { return true }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1696
diff changeset
103
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
104 // StageDone moves the imported gauge measurement out of the staging area.
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
105 // Currently doing nothing.
1676
4407ecaa2192 Imports: Cosmetics.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1664
diff changeset
106 func (gmJobCreator) StageDone(context.Context, *sql.Tx, int64) error {
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
107 return nil
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
108 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
109
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
110 // CleanUp of a gauge measurement import is a NOP.
1676
4407ecaa2192 Imports: Cosmetics.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1664
diff changeset
111 func (*GaugeMeasurement) CleanUp() error { return nil }
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
112
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
113 // Do executes the actual bottleneck import.
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
114 func (gm *GaugeMeasurement) Do(
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
115 ctx context.Context,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
116 importID int64,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
117 conn *sql.Conn,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
118 feedback Feedback,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
119 ) (interface{}, error) {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
120
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
121 fetch := func() ([]*nts.RIS_Message_Type, error) {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
122 client := nts.NewINtSMessageService(gm.URL, gm.Insecure, nil)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
123
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
124 mt := nts.Message_type_typeWRM
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
125
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
126 req := &nts.Get_messages_query{
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
127 Message_type: &mt,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
128 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
129
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
130 resp, err := client.Get_messages(req)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
131 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
132 return nil, err
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
133 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
134
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
135 result := resp.Result_message
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
136 if result == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
137 for _, e := range resp.Result_error {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
138 if e != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
139 feedback.Error("Error code: %s", *e)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
140 } else {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
141 feedback.Error("Unknown error")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
142 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
143 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
144 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
145 return result, nil
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
146 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
147
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
148 return storeGaugeMeasurements(
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
149 ctx,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
150 importID,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
151 fetch,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
152 conn,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
153 feedback,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
154 )
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
155 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
156
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
157 func storeGaugeMeasurements(
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
158 ctx context.Context,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
159 importID int64,
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
160 fetch func() ([]*nts.RIS_Message_Type, error),
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
161 conn *sql.Conn,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
162 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
163 ) (interface{}, error) {
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
164
2233
137addc77b1b Gauge measurement imports: Do all database ops in one transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
165 start := time.Now()
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
166
3176
1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import
Tom Gottfried <tom@intevation.de>
parents: 2849
diff changeset
167 // TODO get date_issue for selected gauges
1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import
Tom Gottfried <tom@intevation.de>
parents: 2849
diff changeset
168 gids, err := doForGM(ctx, fetch, conn, feedback)
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
169 if err != nil {
3176
1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import
Tom Gottfried <tom@intevation.de>
parents: 2849
diff changeset
170 feedback.Error("Error processing gauges: %v", err)
1761
0145537d5474 Gauge measument import: Die with an error if something went wrong.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1754
diff changeset
171 return nil, err
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
172 }
2233
137addc77b1b Gauge measurement imports: Do all database ops in one transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
173
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
174 if len(gids) == 0 {
2233
137addc77b1b Gauge measurement imports: Do all database ops in one transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
175 return nil, UnchangedError("No new gauge measurements found")
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
176 }
2233
137addc77b1b Gauge measurement imports: Do all database ops in one transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
177
137addc77b1b Gauge measurement imports: Do all database ops in one transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
178 feedback.Info(
3176
1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import
Tom Gottfried <tom@intevation.de>
parents: 2849
diff changeset
179 "Importing gauge measurements took %s", time.Since(start))
2233
137addc77b1b Gauge measurement imports: Do all database ops in one transaction.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
180
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
181 // TODO: needs to be filled more useful.
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
182 summary := struct {
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
183 GaugeMeasuremets []string `json:"gaugeMeasurements"`
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
184 }{
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
185 GaugeMeasuremets: gids,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
186 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
187 return &summary, err
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
188 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
189
1772
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
190 // rescale returns a scaling function to bring the unit all to cm.
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
191 func rescale(unit string) (func(float32) float32, error) {
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
192
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
193 var scale float32
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
194
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
195 switch strings.ToLower(unit) {
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
196 case "mm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
197 scale = 0.1
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
198 case "cm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
199 scale = 1.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
200 case "dm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
201 scale = 10.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
202 case "m":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
203 scale = 100.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
204 case "hm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
205 scale = 10000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
206 case "km":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
207 scale = 100000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
208 default:
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
209 return nil, fmt.Errorf("unknown unit '%s'", unit)
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
210 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
211
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
212 fn := func(x float32) float32 { return scale * x }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
213 return fn, nil
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
214 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
215
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
216 func doForGM(
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
217 ctx context.Context,
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
218 fetch func() ([]*nts.RIS_Message_Type, error),
3176
1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import
Tom Gottfried <tom@intevation.de>
parents: 2849
diff changeset
219 conn *sql.Conn,
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
220 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
221 ) ([]string, error) {
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
222
3176
1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import
Tom Gottfried <tom@intevation.de>
parents: 2849
diff changeset
223 insertStmt, err := conn.PrepareContext(ctx, insertGMSQL)
2234
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2233
diff changeset
224 if err != nil {
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2233
diff changeset
225 return nil, err
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2233
diff changeset
226 }
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2233
diff changeset
227 defer insertStmt.Close()
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2233
diff changeset
228
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
229 result, err := fetch()
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
230 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
231 return nil, err
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
232 }
1981
e89368aec538 Import: Make one request per gauge for gauge measurement data.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1978
diff changeset
233
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
234 var gids []string
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
235 for _, msg := range result {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
236 var gid int64
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
237 for _, wrm := range msg.Wrm {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
238 curr := string(*wrm.Geo_object.Id)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
239 currIsrs, err := models.IsrsFromString(curr)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
240 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
241 feedback.Warn("Invalid ISRS code %v", err)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
242 continue
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
243 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
244 feedback.Info("Found measurements for %s", curr)
1981
e89368aec538 Import: Make one request per gauge for gauge measurement data.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1978
diff changeset
245
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
246 var referenceCode string
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
247 if wrm.Reference_code == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
248 feedback.Info("'Reference_code' not specified. Assuming 'ZPG'")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
249 referenceCode = "ZPG"
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
250 } else {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
251 referenceCode = string(*wrm.Reference_code)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
252 }
2849
b8972e4671fa Preserve old predictions on raw gauge measurement import.
Sascha Wilde <wilde@intevation.de>
parents: 2245
diff changeset
253
b8972e4671fa Preserve old predictions on raw gauge measurement import.
Sascha Wilde <wilde@intevation.de>
parents: 2245
diff changeset
254 var newCnt int = 0
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
255 for _, measure := range wrm.Measure {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
256 var unit string
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
257 if measure.Unit == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
258 feedback.Info("'Unit' not specified. Assuming 'cm'")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
259 unit = "cm"
1772
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1766
diff changeset
260 } else {
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
261 unit = string(*measure.Unit)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
262 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
263 convert, err := rescale(unit)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
264 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
265 return nil, err
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
266 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
267 isWaterlevel := *measure.Measure_code == nts.Measure_code_enumWAL
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
268 err = insertStmt.QueryRowContext(
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
269 ctx,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
270 currIsrs.CountryCode,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
271 currIsrs.LoCode,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
272 currIsrs.FairwaySection,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
273 currIsrs.Orc,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
274 currIsrs.Hectometre,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
275 measure.Measuredate,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
276 msg.Identification.From,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
277 msg.Identification.Language_code,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
278 msg.Identification.Country_code,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
279 msg.Identification.Date_issue,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
280 referenceCode,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
281 convert(measure.Value),
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
282 measure.Predicted,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
283 isWaterlevel,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
284 convert(measure.Value_min),
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
285 convert(measure.Value_max),
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
286 msg.Identification.Date_issue,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
287 msg.Identification.Originator,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
288 true, // staging_done
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
289 ).Scan(&gid)
2849
b8972e4671fa Preserve old predictions on raw gauge measurement import.
Sascha Wilde <wilde@intevation.de>
parents: 2245
diff changeset
290 switch {
b8972e4671fa Preserve old predictions on raw gauge measurement import.
Sascha Wilde <wilde@intevation.de>
parents: 2245
diff changeset
291 case err == sql.ErrNoRows:
b8972e4671fa Preserve old predictions on raw gauge measurement import.
Sascha Wilde <wilde@intevation.de>
parents: 2245
diff changeset
292 // thats expected, nothing to do
b8972e4671fa Preserve old predictions on raw gauge measurement import.
Sascha Wilde <wilde@intevation.de>
parents: 2245
diff changeset
293 case err != nil:
3176
1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import
Tom Gottfried <tom@intevation.de>
parents: 2849
diff changeset
294 feedback.Warn(handleError(err).Error())
2849
b8972e4671fa Preserve old predictions on raw gauge measurement import.
Sascha Wilde <wilde@intevation.de>
parents: 2245
diff changeset
295 default:
b8972e4671fa Preserve old predictions on raw gauge measurement import.
Sascha Wilde <wilde@intevation.de>
parents: 2245
diff changeset
296 newCnt++
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
297 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
298 }
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
299 feedback.Info("Inserted %d measurements for %s",
2849
b8972e4671fa Preserve old predictions on raw gauge measurement import.
Sascha Wilde <wilde@intevation.de>
parents: 2245
diff changeset
300 newCnt, curr)
2242
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2238
diff changeset
301 gids = append(gids, curr)
1637
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
302 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
303 }
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
304 return gids, nil
dd31be75ce6d Implemented gauge measurement import.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
diff changeset
305 }