Mercurial > gemma
annotate pkg/imports/wg.go @ 5736:55892008ec96 default tip
Fixed a bunch of corner cases in WG import.
author | Sascha Wilde <wilde@sha-bang.de> |
---|---|
date | Wed, 29 May 2024 19:02:42 +0200 |
parents | 0500d76e074b |
children |
rev | line source |
---|---|
1808
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
5718
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
7 // Copyright (C) 2018, 2019, 2024 by via donau |
1808
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
5718
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
13 // * Sascha Wilde <wilde@intevation.de> |
2318
06c4e57435f1
Warn on import of unknown reference level codes
Tom Gottfried <tom@intevation.de>
parents:
2317
diff
changeset
|
14 // * Tom Gottfried <tom.gottfried@intevation.de> |
1808
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 package imports |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 import ( |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "context" |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "database/sql" |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
21 "errors" |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
22 "strings" |
1835
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
23 "time" |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
24 |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
25 "github.com/jackc/pgx/pgtype" |
1808
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
26 |
1829
b4b9089c2d79
Waterway gauges: Started with deleting old gauges to be overwritten.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1812
diff
changeset
|
27 "gemma.intevation.de/gemma/pkg/models" |
4062
6c760abcff0e
Move handling of PostgreSQL errors to own package
Tom Gottfried <tom@intevation.de>
parents:
4058
diff
changeset
|
28 "gemma.intevation.de/gemma/pkg/pgxutils" |
5609
e1936db6db8e
Adjust ta erdms data structure.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5565
diff
changeset
|
29 erdms "gemma.intevation.de/gemma/pkg/soap/erdms2" |
1808
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
30 ) |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
31 |
4182
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4147
diff
changeset
|
32 // WaterwayGauge is a Job to load gauge data from |
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4147
diff
changeset
|
33 // a specified NTS service and stores them into the database. |
1808
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
34 type WaterwayGauge struct { |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
35 // URL is the URL of the SOAP service. |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
36 URL string `json:"url"` |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
37 // Username is the username used to authenticate. |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
38 Username string `json:"username"` |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
39 // Passwort is the password to authenticate. |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
40 Password string `json:"password"` |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
41 // Insecure indicates if HTTPS traffic |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
42 // should validate certificates or not. |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
43 Insecure bool `json:"insecure"` |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
44 } |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 |
4852
046a07a33b19
Fixed the golint issues of the imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4799
diff
changeset
|
46 // Description gives a short info about relevant facts of this import. |
5565
ade07a3f2cfd
Forget to change signature of some imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5034
diff
changeset
|
47 func (wg *WaterwayGauge) Description([]string) (string, error) { |
4799
f32d086b5dbf
Removed the mechanical touch of the last commit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4798
diff
changeset
|
48 return wg.URL, nil |
4798
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4497
diff
changeset
|
49 } |
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4497
diff
changeset
|
50 |
4182
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4147
diff
changeset
|
51 // WGJobKind is the unique name of this import job type. |
1808
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
52 const WGJobKind JobKind = "wg" |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
53 |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
54 type wgJobCreator struct{} |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
55 |
4182
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4147
diff
changeset
|
56 func init() { RegisterJobCreator(WGJobKind, wgJobCreator{}) } |
1808
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
58 func (wgJobCreator) Description() string { return "waterway gauges" } |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
59 |
1829
b4b9089c2d79
Waterway gauges: Started with deleting old gauges to be overwritten.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1812
diff
changeset
|
60 func (wgJobCreator) AutoAccept() bool { return true } |
1808
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
61 |
2187
7c83b5277c1c
Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
62 func (wgJobCreator) Create() Job { return new(WaterwayGauge) } |
1808
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
63 |
3219
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3163
diff
changeset
|
64 func (wgJobCreator) Depends() [2][]string { |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3163
diff
changeset
|
65 return [2][]string{ |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3163
diff
changeset
|
66 {"gauges_reference_water_levels", "gauges"}, |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3163
diff
changeset
|
67 {"depth_references"}, |
1808
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
68 } |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
69 } |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
70 |
1829
b4b9089c2d79
Waterway gauges: Started with deleting old gauges to be overwritten.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1812
diff
changeset
|
71 // StageDone does nothing as there is no staging for gauges. |
5034
59a99655f34d
Added feedback support for StageDone.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4852
diff
changeset
|
72 func (wgJobCreator) StageDone(context.Context, *sql.Tx, int64, Feedback) error { return nil } |
1808
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
73 |
1829
b4b9089c2d79
Waterway gauges: Started with deleting old gauges to be overwritten.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1812
diff
changeset
|
74 // CleanUp does nothing as there is nothing to cleanup with gauges. |
1808
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
75 func (*WaterwayGauge) CleanUp() error { return nil } |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
76 |
1812
4910bcfab319
Waterway gauge import: Work around the expected (but nonetheless silly) namespace issues with the SOAP interface.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1810
diff
changeset
|
77 const ( |
3311
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
78 eraseObsoleteGaugesSQL = ` |
4028
5fed2f5bc104
End validity of gauge if it gets deleted during import.
Sascha Wilde <wilde@intevation.de>
parents:
3665
diff
changeset
|
79 UPDATE waterway.gauges SET erased = true, validity = validity - '[now,)' |
3551
a5448426e4e2
Only erase gauges that have been requested from ERDMS
Tom Gottfried <tom@intevation.de>
parents:
3529
diff
changeset
|
80 WHERE NOT erased |
5721
0500d76e074b
Fixed eraseObsoleteGaugesSQL by reintroducing matching on users CC.
Sascha Wilde <wilde@sha-bang.de>
parents:
5720
diff
changeset
|
81 AND (location).country_code = ANY($1) |
0500d76e074b
Fixed eraseObsoleteGaugesSQL by reintroducing matching on users CC.
Sascha Wilde <wilde@sha-bang.de>
parents:
5720
diff
changeset
|
82 AND isrs_astext(location) <> ALL($2) |
3311
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
83 RETURNING isrs_astext(location) |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
84 ` |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
85 |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
86 eraseGaugeSQL = ` |
3402
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
87 WITH upd AS ( |
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
88 UPDATE waterway.gauges SET |
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
89 erased = true |
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
90 WHERE isrs_astext(location) = $1 |
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
91 AND NOT erased |
3648
0ec5c8ec1e44
Avoid empty validity time ranges
Tom Gottfried <tom@intevation.de>
parents:
3645
diff
changeset
|
92 -- Don't touch old entry if new validity contains old: will be updated |
0ec5c8ec1e44
Avoid empty validity time ranges
Tom Gottfried <tom@intevation.de>
parents:
3645
diff
changeset
|
93 AND NOT validity <@ $2 |
3402
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
94 RETURNING 1 |
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
95 ) |
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
96 -- Decide whether a new version will be INSERTed |
5736
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
97 SELECT (EXISTS(SELECT 1 FROM upd) |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
98 AND NOT EXISTS(SELECT 1 FROM waterway.gauges |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
99 WHERE isrs_astext(location) = $1 |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
100 AND validity <@ $2)) |
3402
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
101 OR NOT EXISTS(SELECT 1 FROM waterway.gauges WHERE isrs_astext(location) = $1) |
3163
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
102 ` |
1829
b4b9089c2d79
Waterway gauges: Started with deleting old gauges to be overwritten.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1812
diff
changeset
|
103 |
1835
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
104 insertGaugeSQL = ` |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
105 INSERT INTO waterway.gauges ( |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
106 location, |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
107 objname, |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
108 geom, |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
109 applicability_from_km, |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
110 applicability_to_km, |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
111 validity, |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
112 zero_point, |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
113 geodref, |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
114 date_info, |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
115 source_organization, |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
116 lastupdate |
1835
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
117 ) VALUES ( |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
118 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int), |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
119 $6, |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
120 ST_SetSRID(ST_MakePoint($7, $8), 4326)::geography, |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
121 $9, |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
122 $10, |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
123 $11, |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
124 $12, |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
125 $13, |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
126 $14, |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
127 $15, |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
128 $16 |
3402
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
129 ) |
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
130 ` |
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
131 |
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
132 fixValiditySQL = ` |
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
133 UPDATE waterway.gauges SET |
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
134 -- Set enddate of old entry to new startdate in case of overlap: |
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
135 validity = validity - $2 |
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
136 WHERE isrs_astext(location) = $1 |
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
137 AND validity && $2 |
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
138 AND erased |
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
139 ` |
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
140 |
5736
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
141 deleteObsoleteSQL = ` |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
142 WITH dummy AS ( |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
143 DELETE FROM waterway.gauges_reference_water_levels |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
144 WHERE isrs_astext(location) = $1 |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
145 AND validity <@ $2 AND validity != $2 |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
146 ) |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
147 DELETE FROM waterway.gauges |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
148 WHERE isrs_astext(location) = $1 |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
149 AND validity <@ $2 AND validity != $2 |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
150 AND erased |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
151 ` |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
152 |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
153 updateGaugeSQL = ` |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
154 UPDATE waterway.gauges SET |
5670
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
155 objname = $2, |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
156 geom = ST_SetSRID(ST_MakePoint($3, $4), 4326), |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
157 applicability_from_km = $5, |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
158 applicability_to_km = $6, |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
159 zero_point = $7, |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
160 geodref = $8, |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
161 date_info = $9, |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
162 source_organization = $10, |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
163 lastupdate = $11, |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
164 validity = $12, |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
165 erased = false |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
166 WHERE isrs_astext(location) = $1 |
5736
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
167 AND validity = (SELECT validity FROM waterway.gauges |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
168 WHERE isrs_astext(location) = $1 AND validity <@ $12 |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
169 AND $11 > lastupdate ORDER BY validity DESC LIMIT 1) |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
170 RETURNING 1 |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
171 ` |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
172 |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
173 deleteReferenceWaterLevelsSQL = ` |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
174 DELETE FROM waterway.gauges_reference_water_levels |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
175 WHERE isrs_astext(location) = $1 |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
176 AND validity = $2 |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
177 AND depth_reference <> ALL($3) |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
178 RETURNING depth_reference |
1836
4dcdd8891770
Waterway gauges import: Fixed insert/update of gauges. TODO: Re-insert reference water levels.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1835
diff
changeset
|
179 ` |
3163
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
180 |
2318
06c4e57435f1
Warn on import of unknown reference level codes
Tom Gottfried <tom@intevation.de>
parents:
2317
diff
changeset
|
181 isNtSDepthRefSQL = ` |
06c4e57435f1
Warn on import of unknown reference level codes
Tom Gottfried <tom@intevation.de>
parents:
2317
diff
changeset
|
182 SELECT EXISTS(SELECT 1 FROM depth_references WHERE depth_reference = $1)` |
06c4e57435f1
Warn on import of unknown reference level codes
Tom Gottfried <tom@intevation.de>
parents:
2317
diff
changeset
|
183 |
1837
00d63eb9306a
Waterway gauges import: Finalized with re-inserting the reference water levels.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1836
diff
changeset
|
184 insertReferenceWaterLevelsSQL = ` |
00d63eb9306a
Waterway gauges import: Finalized with re-inserting the reference water levels.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1836
diff
changeset
|
185 INSERT INTO waterway.gauges_reference_water_levels ( |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
186 location, |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
187 validity, |
2317
8a8680e70d2e
Cleanup schema for reference water levels
Tom Gottfried <tom@intevation.de>
parents:
2187
diff
changeset
|
188 depth_reference, |
1837
00d63eb9306a
Waterway gauges import: Finalized with re-inserting the reference water levels.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1836
diff
changeset
|
189 value |
00d63eb9306a
Waterway gauges import: Finalized with re-inserting the reference water levels.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1836
diff
changeset
|
190 ) VALUES ( |
00d63eb9306a
Waterway gauges import: Finalized with re-inserting the reference water levels.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1836
diff
changeset
|
191 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int), |
00d63eb9306a
Waterway gauges import: Finalized with re-inserting the reference water levels.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1836
diff
changeset
|
192 $6, |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
193 $7, |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
194 $8 |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
195 ) ON CONFLICT (location, validity, depth_reference) DO UPDATE SET |
3163
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
196 value = EXCLUDED.value |
1837
00d63eb9306a
Waterway gauges import: Finalized with re-inserting the reference water levels.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1836
diff
changeset
|
197 ` |
1812
4910bcfab319
Waterway gauge import: Work around the expected (but nonetheless silly) namespace issues with the SOAP interface.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1810
diff
changeset
|
198 ) |
4910bcfab319
Waterway gauge import: Work around the expected (but nonetheless silly) namespace issues with the SOAP interface.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1810
diff
changeset
|
199 |
5683
31973f6f5cca
Get rid of deprecation warnings from staticcheck.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5670
diff
changeset
|
200 var errContinue = errors.New("continue") |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
201 |
5718
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
202 func storeWaterwayGauges( |
1808
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
203 ctx context.Context, |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
204 conn *sql.Conn, |
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
205 feedback Feedback, |
5718
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
206 fetch func(context.Context, *sql.Tx) ([]*erdms.GetRisDataXMLResponse, error), |
5712
6270951dda28
/interface{}/any/
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5711
diff
changeset
|
207 ) (any, error) { |
1837
00d63eb9306a
Waterway gauges import: Finalized with re-inserting the reference water levels.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1836
diff
changeset
|
208 start := time.Now() |
00d63eb9306a
Waterway gauges import: Finalized with re-inserting the reference water levels.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1836
diff
changeset
|
209 |
5718
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
210 tx, err := conn.BeginTx(ctx, nil) |
2633
eb1d119f253f
Fetch data from ERDMS for all allowed countries
Tom Gottfried <tom@intevation.de>
parents:
2518
diff
changeset
|
211 if err != nil { |
1812
4910bcfab319
Waterway gauge import: Work around the expected (but nonetheless silly) namespace issues with the SOAP interface.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1810
diff
changeset
|
212 return nil, err |
4910bcfab319
Waterway gauge import: Work around the expected (but nonetheless silly) namespace issues with the SOAP interface.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1810
diff
changeset
|
213 } |
5718
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
214 defer tx.Rollback() |
1812
4910bcfab319
Waterway gauge import: Work around the expected (but nonetheless silly) namespace issues with the SOAP interface.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1810
diff
changeset
|
215 |
3665
29ef6d41e4af
Use database triggers to move referencing objects to new versions
Tom Gottfried <tom@intevation.de>
parents:
3648
diff
changeset
|
216 var eraseGaugeStmt, insertStmt, |
5736
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
217 fixValidityStmt, deleteObsoleteStmt, updateStmt, |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
218 deleteReferenceWaterLevelsStmt, |
3163
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
219 isNtSDepthRefStmt, insertWaterLevelStmt *sql.Stmt |
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
220 for _, x := range []struct { |
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
221 sql string |
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
222 stmt **sql.Stmt |
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
223 }{ |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
224 {eraseGaugeSQL, &eraseGaugeStmt}, |
3163
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
225 {insertGaugeSQL, &insertStmt}, |
3402
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
226 {fixValiditySQL, &fixValidityStmt}, |
5736
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
227 {deleteObsoleteSQL, &deleteObsoleteStmt}, |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
228 {updateGaugeSQL, &updateStmt}, |
3163
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
229 {deleteReferenceWaterLevelsSQL, &deleteReferenceWaterLevelsStmt}, |
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
230 {isNtSDepthRefSQL, &isNtSDepthRefStmt}, |
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
231 {insertReferenceWaterLevelsSQL, &insertWaterLevelStmt}, |
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
232 } { |
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
233 var err error |
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
234 if *x.stmt, err = conn.PrepareContext(ctx, x.sql); err != nil { |
1829
b4b9089c2d79
Waterway gauges: Started with deleting old gauges to be overwritten.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1812
diff
changeset
|
235 return nil, err |
b4b9089c2d79
Waterway gauges: Started with deleting old gauges to be overwritten.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1812
diff
changeset
|
236 } |
3163
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
237 defer (*x.stmt).Close() |
1812
4910bcfab319
Waterway gauge import: Work around the expected (but nonetheless silly) namespace issues with the SOAP interface.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1810
diff
changeset
|
238 } |
1810
7ee9bdaac336
Waterway gauge import: Added support for username/password.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1808
diff
changeset
|
239 |
3310 | 240 var gauges []string |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
241 var unchanged int |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
242 |
5670
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
243 var invalidISRS, startEndOrder, missingObjname, missingZeropoint, updatedGauges, newGauges []string |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
244 |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
245 databaseErrors := map[string][]string{} |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
246 |
5718
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
247 wgs, err := fetch(ctx, tx) |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
248 if err != nil { |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
249 return nil, err |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
250 } |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
251 |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
252 for _, data := range wgs { |
3310 | 253 for _, dr := range data.RisdataReturn { |
1836
4dcdd8891770
Waterway gauges import: Fixed insert/update of gauges. TODO: Re-insert reference water levels.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1835
diff
changeset
|
254 |
3310 | 255 isrs := string(*dr.RisidxCode) |
256 code, err := models.IsrsFromString(isrs) | |
257 if err != nil { | |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
258 invalidISRS = append(invalidISRS, isrs) |
3310 | 259 continue |
1835
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
260 } |
3310 | 261 gauges = append(gauges, isrs) |
262 | |
3529
ba0339118d9c
Improve validation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3460
diff
changeset
|
263 // We need a valid, non-empty time range to identify gauge versions |
5609
e1936db6db8e
Adjust ta erdms data structure.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5565
diff
changeset
|
264 if dr.Enddate != nil && dr.Startdate != nil { |
5615
f6179b31e0a9
Use a type alias on date type to not loose XML serialisation.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5609
diff
changeset
|
265 ed := dr.Enddate.ToGoTime() |
f6179b31e0a9
Use a type alias on date type to not loose XML serialisation.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5609
diff
changeset
|
266 sd := dr.Startdate.ToGoTime() |
f6179b31e0a9
Use a type alias on date type to not loose XML serialisation.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5609
diff
changeset
|
267 // log.Debugf("start date: %v end date: %v\n", sd, ed) |
f6179b31e0a9
Use a type alias on date type to not loose XML serialisation.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5609
diff
changeset
|
268 if !ed.After(sd) { |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
269 startEndOrder = append(startEndOrder, isrs) |
5609
e1936db6db8e
Adjust ta erdms data structure.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5565
diff
changeset
|
270 unchanged++ |
e1936db6db8e
Adjust ta erdms data structure.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5565
diff
changeset
|
271 continue |
e1936db6db8e
Adjust ta erdms data structure.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5565
diff
changeset
|
272 } |
3458
aa3c83fb7018
Validate temporal validity of gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3402
diff
changeset
|
273 } |
aa3c83fb7018
Validate temporal validity of gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3402
diff
changeset
|
274 |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
275 if dr.Zeropoint == nil { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
276 missingZeropoint = append(missingZeropoint, isrs) |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
277 unchanged++ |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
278 continue |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
279 } |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
280 |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
281 if dr.Objname.Loc == nil { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
282 missingObjname = append(missingObjname, isrs) |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
283 unchanged++ |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
284 continue |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
285 } |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
286 |
3310 | 287 var from, to sql.NullInt64 |
1835
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
288 |
3310 | 289 if dr.Applicabilityfromkm != nil { |
290 from = sql.NullInt64{ | |
291 Int64: int64(*dr.Applicabilityfromkm), | |
292 Valid: true, | |
293 } | |
1836
4dcdd8891770
Waterway gauges import: Fixed insert/update of gauges. TODO: Re-insert reference water levels.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1835
diff
changeset
|
294 } |
3310 | 295 if dr.Applicabilitytokm != nil { |
296 to = sql.NullInt64{ | |
297 Int64: int64(*dr.Applicabilitytokm), | |
298 Valid: true, | |
299 } | |
300 } | |
1835
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
301 |
3310 | 302 var tfrom, tto, dateInfo pgtype.Timestamptz |
1835
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
303 |
3310 | 304 if dr.Startdate != nil { |
305 tfrom = pgtype.Timestamptz{ | |
5615
f6179b31e0a9
Use a type alias on date type to not loose XML serialisation.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5609
diff
changeset
|
306 Time: dr.Startdate.ToGoTime(), |
3310 | 307 Status: pgtype.Present, |
308 } | |
309 } else { | |
310 tfrom = pgtype.Timestamptz{ | |
311 Status: pgtype.Null, | |
312 } | |
313 } | |
1835
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
314 |
3310 | 315 if dr.Enddate != nil { |
316 tto = pgtype.Timestamptz{ | |
5615
f6179b31e0a9
Use a type alias on date type to not loose XML serialisation.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5609
diff
changeset
|
317 Time: dr.Enddate.ToGoTime(), |
3310 | 318 Status: pgtype.Present, |
319 } | |
320 } else { | |
321 tto = pgtype.Timestamptz{ | |
322 Status: pgtype.Null, | |
323 } | |
1836
4dcdd8891770
Waterway gauges import: Fixed insert/update of gauges. TODO: Re-insert reference water levels.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1835
diff
changeset
|
324 } |
1835
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
325 |
3310 | 326 validity := pgtype.Tstzrange{ |
327 Lower: tfrom, | |
328 Upper: tto, | |
329 LowerType: pgtype.Inclusive, | |
330 UpperType: pgtype.Exclusive, | |
331 Status: pgtype.Present, | |
1836
4dcdd8891770
Waterway gauges import: Fixed insert/update of gauges. TODO: Re-insert reference water levels.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1835
diff
changeset
|
332 } |
3163
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
333 |
3310 | 334 if dr.Infodate != nil { |
335 dateInfo = pgtype.Timestamptz{ | |
5615
f6179b31e0a9
Use a type alias on date type to not loose XML serialisation.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5609
diff
changeset
|
336 Time: dr.Infodate.ToGoTime(), |
3310 | 337 Status: pgtype.Present, |
338 } | |
339 } else { | |
340 dateInfo = pgtype.Timestamptz{ | |
341 Status: pgtype.Null, | |
342 } | |
343 } | |
344 | |
345 var geodref sql.NullString | |
346 if dr.Geodref != nil { | |
347 geodref = sql.NullString{ | |
348 String: string(*dr.Geodref), | |
349 Valid: true, | |
350 } | |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
351 } |
3310 | 352 |
353 var source sql.NullString | |
5670
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
354 if dr.RisidxSource != nil { |
3310 | 355 source = sql.NullString{ |
5670
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
356 String: string(*dr.RisidxSource), |
3310 | 357 Valid: true, |
358 } | |
359 } | |
360 | |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
361 err = func() error { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
362 tx, err := conn.BeginTx(ctx, nil) |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
363 if err != nil { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
364 return err |
3310 | 365 } |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
366 defer tx.Rollback() |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
367 |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
368 // Mark old entry of gauge as erased, if applicable |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
369 var isNew bool |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
370 err = tx.StmtContext(ctx, eraseGaugeStmt).QueryRowContext(ctx, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
371 code.String(), |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
372 validity, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
373 ).Scan(&isNew) |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
374 switch { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
375 case err != nil: |
4197
5d7ce7f926eb
Log fatal errors as errors, not warnings, in imports
Tom Gottfried <tom@intevation.de>
parents:
4182
diff
changeset
|
376 feedback.Error(pgxutils.ReadableError{Err: err}.Error()) |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
377 unchanged++ |
5683
31973f6f5cca
Get rid of deprecation warnings from staticcheck.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5670
diff
changeset
|
378 return errContinue |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
379 case isNew: |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
380 var lu *time.Time |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
381 if dr.Lastupdate != nil { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
382 t := dr.Lastupdate.ToGoTime() |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
383 lu = &t |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
384 } |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
385 // insert gauge version entry |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
386 if _, err = tx.StmtContext(ctx, insertStmt).ExecContext(ctx, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
387 code.CountryCode, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
388 code.LoCode, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
389 code.FairwaySection, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
390 code.Orc, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
391 code.Hectometre, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
392 dr.Objname.Loc, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
393 dr.Lon, dr.Lat, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
394 from, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
395 to, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
396 &validity, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
397 dr.Zeropoint, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
398 geodref, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
399 &dateInfo, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
400 source, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
401 lu, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
402 ); err != nil { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
403 key := pgxutils.ReadableError{Err: err}.Error() |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
404 databaseErrors[key] = append(databaseErrors[key], isrs) |
5683
31973f6f5cca
Get rid of deprecation warnings from staticcheck.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5670
diff
changeset
|
405 return errContinue |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
406 } |
5670
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
407 newGauges = append(newGauges, isrs) |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
408 //feedback.Info("insert new version") |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
409 case !isNew: |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
410 var lu *time.Time |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
411 if dr.Lastupdate != nil { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
412 t := dr.Lastupdate.ToGoTime() |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
413 lu = &t |
3402
c04b1409a596
Fix adaptation of gauge temporal validity
Tom Gottfried <tom@intevation.de>
parents:
3311
diff
changeset
|
414 } |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
415 // try to update |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
416 var dummy int |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
417 err2 := tx.StmtContext(ctx, updateStmt).QueryRowContext(ctx, |
5670
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
418 code.String(), |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
419 dr.Objname.Loc, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
420 dr.Lon, dr.Lat, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
421 from, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
422 to, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
423 dr.Zeropoint, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
424 geodref, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
425 &dateInfo, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
426 source, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
427 lu, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
428 &validity, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
429 ).Scan(&dummy) |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
430 switch { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
431 case err2 == sql.ErrNoRows: |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
432 //feedback.Info("unchanged") |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
433 unchanged++ |
5683
31973f6f5cca
Get rid of deprecation warnings from staticcheck.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5670
diff
changeset
|
434 return errContinue |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
435 case err2 != nil: |
5670
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
436 key := pgxutils.ReadableError{Err: err2}.Error() |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
437 databaseErrors[key] = append(databaseErrors[key], isrs) |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
438 unchanged++ |
5683
31973f6f5cca
Get rid of deprecation warnings from staticcheck.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5670
diff
changeset
|
439 return errContinue |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
440 default: |
5670
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
441 updatedGauges = append(updatedGauges, isrs) |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
442 //feedback.Info("update") |
3310 | 443 } |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
444 |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
445 // Remove obsolete reference water levels |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
446 var currLevels pgtype.VarcharArray |
5670
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
447 var reflevels []string |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
448 if dr.Reflevel1code != nil { |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
449 reflevels = append(reflevels, |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
450 string(*dr.Reflevel1code)) |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
451 } |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
452 if dr.Reflevel2code != nil { |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
453 reflevels = append(reflevels, |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
454 string(*dr.Reflevel2code)) |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
455 } |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
456 if dr.Reflevel3code != nil { |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
457 reflevels = append(reflevels, |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
458 string(*dr.Reflevel3code)) |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
459 } |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
460 currLevels.Set(reflevels) |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
461 rwls, err := tx.StmtContext(ctx, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
462 deleteReferenceWaterLevelsStmt).QueryContext(ctx, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
463 code.String(), |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
464 &validity, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
465 &currLevels, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
466 ) |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
467 if err != nil { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
468 return err |
3310 | 469 } |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
470 defer rwls.Close() |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
471 for rwls.Next() { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
472 var delRef string |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
473 if err := rwls.Scan(&delRef); err != nil { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
474 return err |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
475 } |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
476 feedback.Warn("Removed reference water level %s from %s", |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
477 delRef, code) |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
478 } |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
479 if err := rwls.Err(); err != nil { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
480 return err |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
481 } |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
482 } |
3310 | 483 |
5736
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
484 // Delete old versions if they are completely replaced by the new entrie |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
485 // (old validity is completely in new validity) an were not updated. |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
486 if _, err = tx.StmtContext(ctx, deleteObsoleteStmt).ExecContext( |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
487 ctx, |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
488 code.String(), |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
489 &validity, |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
490 ); err != nil { |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
491 key := pgxutils.ReadableError{Err: err}.Error() |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
492 databaseErrors[key] = append(databaseErrors[key], isrs) |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
493 unchanged++ |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
494 return errContinue |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
495 } |
55892008ec96
Fixed a bunch of corner cases in WG import.
Sascha Wilde <wilde@sha-bang.de>
parents:
5721
diff
changeset
|
496 |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
497 // Set end of validity of old version to start of new version |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
498 // in case of overlap |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
499 if _, err = tx.StmtContext(ctx, fixValidityStmt).ExecContext( |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
500 ctx, |
3310 | 501 code.String(), |
502 &validity, | |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
503 ); err != nil { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
504 key := pgxutils.ReadableError{Err: err}.Error() |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
505 databaseErrors[key] = append(databaseErrors[key], isrs) |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
506 unchanged++ |
5683
31973f6f5cca
Get rid of deprecation warnings from staticcheck.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5670
diff
changeset
|
507 return errContinue |
3310 | 508 } |
509 | |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
510 // "Upsert" reference water levels |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
511 for _, wl := range []struct { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
512 level **erdms.RisreflevelcodeType |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
513 value **erdms.RisreflevelvalueType |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
514 }{ |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
515 {&dr.Reflevel1code, &dr.Reflevel1value}, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
516 {&dr.Reflevel2code, &dr.Reflevel2value}, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
517 {&dr.Reflevel3code, &dr.Reflevel3value}, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
518 } { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
519 if *wl.level == nil || *wl.value == nil { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
520 continue |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
521 } |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
522 |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
523 var isNtSDepthRef bool |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
524 if err := tx.StmtContext( |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
525 ctx, isNtSDepthRefStmt).QueryRowContext(ctx, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
526 string(**wl.level), |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
527 ).Scan( |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
528 &isNtSDepthRef, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
529 ); err != nil { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
530 return err |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
531 } |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
532 if !isNtSDepthRef { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
533 feedback.Warn( |
5670
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
534 "Reference level code '%s' of %s is not in line "+ |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
535 "with the NtS reference_code table", |
5670
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
536 string(**wl.level), isrs) |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
537 } |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
538 |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
539 if _, err := tx.StmtContext( |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
540 ctx, insertWaterLevelStmt).ExecContext(ctx, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
541 code.CountryCode, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
542 code.LoCode, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
543 code.FairwaySection, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
544 code.Orc, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
545 code.Hectometre, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
546 &validity, |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
547 string(**wl.level), |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
548 int64(**wl.value), |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
549 ); err != nil { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
550 key := pgxutils.ReadableError{Err: err}.Error() |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
551 databaseErrors[key] = append(databaseErrors[key], isrs) |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
552 continue |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
553 } |
3310 | 554 } |
555 | |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
556 return tx.Commit() |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
557 }() |
3310 | 558 |
5683
31973f6f5cca
Get rid of deprecation warnings from staticcheck.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5670
diff
changeset
|
559 if err != nil && err != errContinue { |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
560 return err, nil |
3305
5932f9574493
Follow-up for 3350:e640f51b5a4e (Fix reporting of removed reference water levels).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3302
diff
changeset
|
561 } |
3163
d9903cb34842
Handle failing INSERTs gracefully during gauges import
Tom Gottfried <tom@intevation.de>
parents:
2633
diff
changeset
|
562 } |
1835
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
563 } |
f7b926440449
Waterway gauge import: More stuggling with inserting gauges. Not working, yet.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1829
diff
changeset
|
564 |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
565 if len(invalidISRS) > 0 { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
566 feedback.Error("Invalid ISRS code: '%s'", strings.Join(invalidISRS, "', '")) |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
567 } |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
568 |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
569 if len(startEndOrder) > 0 { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
570 feedback.Error("start date not before end date: %s", |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
571 strings.Join(startEndOrder, ", ")) |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
572 } |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
573 |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
574 if len(databaseErrors) > 0 { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
575 for err, iris := range databaseErrors { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
576 feedback.Error("%s: %s", err, strings.Join(iris, ", ")) |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
577 } |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
578 } |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
579 |
5621
cf1e8ffe1ed5
Fix missing zero point logging in wg import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5620
diff
changeset
|
580 if len(missingZeropoint) > 0 { |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
581 feedback.Error("Missing zeropoint: %s", strings.Join(missingZeropoint, ", ")) |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
582 } |
5621
cf1e8ffe1ed5
Fix missing zero point logging in wg import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5620
diff
changeset
|
583 |
5620
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
584 if len(missingObjname) > 0 { |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
585 feedback.Error("Missing objname: %s", strings.Join(missingObjname, ", ")) |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
586 } |
165e77c5736a
Simplified WG import logging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5615
diff
changeset
|
587 |
5670
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
588 if len(updatedGauges) > 0 { |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
589 feedback.Info("Updated gauges: %s", strings.Join(updatedGauges, ", ")) |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
590 } |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
591 |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
592 if len(newGauges) > 0 { |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
593 feedback.Info("New gauges: %s", strings.Join(newGauges, ", ")) |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
594 } |
b75d0b303328
Various fixes and improvements of gauges import:
Sascha Wilde <wilde@sha-bang.de>
parents:
5621
diff
changeset
|
595 |
3310 | 596 if len(gauges) == 0 { |
597 return nil, UnchangedError("No gauges returned from ERDMS") | |
598 } | |
599 | |
5721
0500d76e074b
Fixed eraseObsoleteGaugesSQL by reintroducing matching on users CC.
Sascha Wilde <wilde@sha-bang.de>
parents:
5720
diff
changeset
|
600 countries, err := userCountries(ctx, conn) |
0500d76e074b
Fixed eraseObsoleteGaugesSQL by reintroducing matching on users CC.
Sascha Wilde <wilde@sha-bang.de>
parents:
5720
diff
changeset
|
601 if err != nil { |
0500d76e074b
Fixed eraseObsoleteGaugesSQL by reintroducing matching on users CC.
Sascha Wilde <wilde@sha-bang.de>
parents:
5720
diff
changeset
|
602 return nil, err |
0500d76e074b
Fixed eraseObsoleteGaugesSQL by reintroducing matching on users CC.
Sascha Wilde <wilde@sha-bang.de>
parents:
5720
diff
changeset
|
603 } |
0500d76e074b
Fixed eraseObsoleteGaugesSQL by reintroducing matching on users CC.
Sascha Wilde <wilde@sha-bang.de>
parents:
5720
diff
changeset
|
604 |
0500d76e074b
Fixed eraseObsoleteGaugesSQL by reintroducing matching on users CC.
Sascha Wilde <wilde@sha-bang.de>
parents:
5720
diff
changeset
|
605 var pgCountries, pgGauges pgtype.VarcharArray |
0500d76e074b
Fixed eraseObsoleteGaugesSQL by reintroducing matching on users CC.
Sascha Wilde <wilde@sha-bang.de>
parents:
5720
diff
changeset
|
606 pgCountries.Set(countries) |
3311
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
607 pgGauges.Set(gauges) |
3551
a5448426e4e2
Only erase gauges that have been requested from ERDMS
Tom Gottfried <tom@intevation.de>
parents:
3529
diff
changeset
|
608 obsGauges, err := conn.QueryContext(ctx, |
a5448426e4e2
Only erase gauges that have been requested from ERDMS
Tom Gottfried <tom@intevation.de>
parents:
3529
diff
changeset
|
609 eraseObsoleteGaugesSQL, |
5721
0500d76e074b
Fixed eraseObsoleteGaugesSQL by reintroducing matching on users CC.
Sascha Wilde <wilde@sha-bang.de>
parents:
5720
diff
changeset
|
610 &pgCountries, |
3551
a5448426e4e2
Only erase gauges that have been requested from ERDMS
Tom Gottfried <tom@intevation.de>
parents:
3529
diff
changeset
|
611 &pgGauges) |
3311
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
612 if err != nil { |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
613 return nil, err |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
614 } |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
615 defer obsGauges.Close() |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
616 for obsGauges.Next() { |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
617 var isrs string |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
618 if err := obsGauges.Scan(&isrs); err != nil { |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
619 return nil, err |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
620 } |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
621 feedback.Info("Erased %s", isrs) |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
622 unchanged-- |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
623 } |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
624 if err := obsGauges.Err(); err != nil { |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
625 return nil, err |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
626 } |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
627 |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
628 if unchanged == len(gauges) { |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
629 return nil, UnchangedError("All gauges unchanged") |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
630 } |
5711
2dd155cc95ec
Fix all revive issue (w/o machine generated stuff).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5683
diff
changeset
|
631 feedback.Info("Unchanged gauges: %d", |
2dd155cc95ec
Fix all revive issue (w/o machine generated stuff).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5683
diff
changeset
|
632 unchanged) |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3286
diff
changeset
|
633 |
3311
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
634 feedback.Info("Importing gauges took %s", |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
635 time.Since(start)) |
0f6b156cff55
Mark gauges as erased if they are missing in ERDMS response
Tom Gottfried <tom@intevation.de>
parents:
3310
diff
changeset
|
636 |
1837
00d63eb9306a
Waterway gauges import: Finalized with re-inserting the reference water levels.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1836
diff
changeset
|
637 return nil, err |
1808
77582da3adb0
Waterway gauges import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
638 } |
5718
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
639 |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
640 // Do implements the actual import. |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
641 func (wg *WaterwayGauge) Do( |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
642 ctx context.Context, |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
643 _ int64, |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
644 conn *sql.Conn, |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
645 feedback Feedback, |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
646 ) (any, error) { |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
647 |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
648 fetch := func( |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
649 ctx context.Context, |
5720 | 650 _ *sql.Tx, |
5718
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
651 ) ([]*erdms.GetRisDataXMLResponse, error) { |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
652 responseData, _, err := getRisData( |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
653 ctx, |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
654 conn, |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
655 feedback, |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
656 wg.Username, |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
657 wg.Password, |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
658 wg.URL, |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
659 wg.Insecure, |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
660 "wtwgag") |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
661 if err != nil { |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
662 return nil, err |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
663 } |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
664 return responseData, nil |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
665 } |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
666 |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
667 return storeWaterwayGauges( |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
668 ctx, |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
669 conn, |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
670 feedback, |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
671 fetch, |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
672 ) |
3d497077f888
Implemented direct file upload as alternative import method for WG.
Sascha Wilde <wilde@sha-bang.de>
parents:
5712
diff
changeset
|
673 } |