Mercurial > gemma
annotate pkg/imports/bn.go @ 4850:18d5461bec5d
Fixed some golint issues.
author | Sascha L. Teichmann <sascha.teichmann@intevation.de> |
---|---|
date | Tue, 19 Nov 2019 12:45:52 +0100 |
parents | ca6a5f722471 |
children | faabfed7f0e9 |
rev | line source |
---|---|
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
7 // Copyright (C) 2018, 2019 by via donau |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
13 // * Tom Gottfried <tom.gottfried@intevation.de> |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
14 // * Sascha Wilde <sascha.wilde@intevation.de> |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 package imports |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 import ( |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "context" |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "database/sql" |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
21 "errors" |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
22 "regexp" |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
23 "strconv" |
3528
b9e331c1e616
Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents:
3423
diff
changeset
|
24 "strings" |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
25 "time" |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
26 |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
27 "github.com/jackc/pgx/pgtype" |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
28 |
4102
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
29 "gemma.intevation.de/gemma/pkg/common" |
4062
6c760abcff0e
Move handling of PostgreSQL errors to own package
Tom Gottfried <tom@intevation.de>
parents:
4058
diff
changeset
|
30 "gemma.intevation.de/gemma/pkg/pgxutils" |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
31 "gemma.intevation.de/gemma/pkg/soap/ifbn" |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
32 ) |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
33 |
1695
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
34 // Bottleneck is an import job to import |
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
35 // bottlenecks from an IFBN SOAP service. |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
36 type Bottleneck struct { |
1695
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
37 // URL is the URL of the SOAP service. |
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
38 URL string `json:"url"` |
2563
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
39 // Tolerance used for axis snapping |
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
40 Tolerance float64 `json:"tolerance"` |
1695
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
41 // Insecure indicates if HTTPS traffic |
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
42 // should validate certificates or not. |
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
43 Insecure bool `json:"insecure"` |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
44 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 |
1696
ad5e1cddaa09
Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1695
diff
changeset
|
46 // BNJobKind is the import queue type identifier. |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
47 const BNJobKind JobKind = "bn" |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
48 |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
49 const ( |
1658
07f996b0e14a
Bottleneck import: Cleaned up source a bit (removed outdated TODOs).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1646
diff
changeset
|
50 insertBottleneckSQL = ` |
3566
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
51 WITH |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
52 bounds (b) AS (VALUES (isrs_fromText($6)), (isrs_fromText($7))), |
3566
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
53 r AS (SELECT isrsrange( |
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
54 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY), |
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
55 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r) |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
56 INSERT INTO waterway.bottlenecks ( |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
57 bottleneck_id, |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
58 validity, |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3224
diff
changeset
|
59 gauge_location, |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
60 objnam, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
61 nobjnm, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
62 stretch, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
63 area, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
64 rb, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
65 lb, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
66 responsible_country, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
67 revisiting_time, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
68 limiting, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
69 date_info, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
70 source_organization |
4069
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
71 ) VALUES ( |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
72 $1, |
4069
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
73 $2::tstzrange, |
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
74 isrs_fromText($3), |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
75 $4, |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
76 $5, |
3566
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
77 (SELECT r FROM r), |
1631
6ee06eb37fa9
Use new function to calculate bn area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1611
diff
changeset
|
78 ISRSrange_area( |
3566
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
79 ISRSrange_axis((SELECT r FROM r), |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
80 $15), |
2248
cce158db02b0
Input area as multipolygons to generate area from stretch
Tom Gottfried <tom@intevation.de>
parents:
2195
diff
changeset
|
81 (SELECT ST_Collect(CAST(area AS geometry)) |
1902
c4af342be999
Use waterway area as basis for bottleneck area generation.
Sascha Wilde <wilde@intevation.de>
parents:
1754
diff
changeset
|
82 FROM waterway.waterway_area)), |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
83 $8, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
84 $9, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
85 $10, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
86 $11, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
87 $12, |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
88 $13, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
89 $14 |
4069
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
90 ) |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
91 RETURNING id |
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
92 ` |
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
93 |
4130
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
94 // We only check for NOT NULL values, for correct compairison with |
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
95 // values, which might be null (and then muyst not be compairt with `=' |
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
96 // but with `IS NULL' is comlicated and that we are checking more than |
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
97 // only (bottleneck_id, validity, date_info) is luxury already. |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
98 findExactMatchBottleneckSQL = ` |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
99 WITH |
4130
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
100 bounds (b) AS (VALUES (isrs_fromText($4)), (isrs_fromText($5))), |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
101 r AS (SELECT isrsrange( |
4130
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
102 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY), |
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
103 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r) |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
104 SELECT id FROM waterway.bottlenecks |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
105 WHERE ( |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
106 bottleneck_id, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
107 validity, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
108 gauge_location, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
109 stretch, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
110 responsible_country, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
111 limiting, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
112 date_info, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
113 source_organization, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
114 staging_done |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
115 ) = ( SELECT |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
116 $1, |
4069
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
117 $2::tstzrange, |
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
118 isrs_fromText($3), |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
119 (SELECT r FROM r), |
4130
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
120 $6, |
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
121 $7, |
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
122 $8::timestamptz, |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
123 $9, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
124 true |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
125 ) |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
126 ` |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
127 |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
128 findIntersectingBottleneckSQL = ` |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
129 SELECT id FROM waterway.bottlenecks |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
130 WHERE (bottleneck_id, staging_done) = ($1, true) |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
131 AND $2::tstzrange && validity |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
132 ` |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
133 |
2999
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
134 insertBottleneckMaterialSQL = ` |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
135 INSERT INTO waterway.bottlenecks_riverbed_materials ( |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
136 bottleneck_id, |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
137 riverbed |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
138 ) VALUES ($1, $2) |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
139 ON CONFLICT (bottleneck_id, riverbed) DO NOTHING |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
140 ` |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
141 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
142 bnStageDoneDeleteSQL = ` |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
143 DELETE FROM waterway.bottlenecks WHERE id IN ( |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
144 SELECT key |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
145 FROM import.track_imports |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
146 WHERE import_id = $1 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
147 AND relation = 'waterway.bottlenecks'::regclass |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
148 AND deletion |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
149 )` |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
150 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
151 bnStageDoneSQL = ` |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
152 UPDATE waterway.bottlenecks SET staging_done = true |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
153 WHERE id IN ( |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
154 SELECT key |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
155 FROM import.track_imports |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
156 WHERE import_id = $1 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
157 AND relation = 'waterway.bottlenecks'::regclass |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
158 AND NOT deletion |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
159 )` |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
160 ) |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
161 |
4798
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4197
diff
changeset
|
162 func (bn *Bottleneck) Description() (string, error) { |
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4197
diff
changeset
|
163 |
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4197
diff
changeset
|
164 var descs []string |
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4197
diff
changeset
|
165 |
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4197
diff
changeset
|
166 descs = append(descs, bn.URL) |
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4197
diff
changeset
|
167 return strings.Join(descs, "|"), nil |
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4197
diff
changeset
|
168 } |
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4197
diff
changeset
|
169 |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
170 type bnJobCreator struct{} |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
171 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
172 func init() { |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
173 RegisterJobCreator(BNJobKind, bnJobCreator{}) |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
174 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
175 |
1754
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
176 func (bnJobCreator) Description() string { return "bottlenecks" } |
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
177 |
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
178 func (bnJobCreator) AutoAccept() bool { return false } |
1646
a0982c38eac0
Import queue: Implemented email notifications.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1631
diff
changeset
|
179 |
2187
7c83b5277c1c
Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
180 func (bnJobCreator) Create() Job { return new(Bottleneck) } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
181 |
3219
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3164
diff
changeset
|
182 func (bnJobCreator) Depends() [2][]string { |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3164
diff
changeset
|
183 return [2][]string{ |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3164
diff
changeset
|
184 {"bottlenecks", "bottlenecks_riverbed_materials"}, |
3224
bfde4f8dd323
Add missing table dependencies of imports with area generation
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
185 {"gauges", "distance_marks_virtual", "waterway_axis", "waterway_area"}, |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
186 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
187 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
188 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
189 // StageDone moves the imported bottleneck out of the staging area. |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
190 func (bnJobCreator) StageDone( |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
191 ctx context.Context, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
192 tx *sql.Tx, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
193 id int64, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
194 ) error { |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
195 _, err := tx.ExecContext(ctx, bnStageDoneDeleteSQL, id) |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
196 if err == nil { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
197 _, err = tx.ExecContext(ctx, bnStageDoneSQL, id) |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
198 } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
199 return err |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
200 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
201 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
202 // CleanUp of a bottleneck import is a NOP. |
1676
4407ecaa2192
Imports: Cosmetics.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1664
diff
changeset
|
203 func (*Bottleneck) CleanUp() error { return nil } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
204 |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
205 var rblbRe = regexp.MustCompile(`(..)_(..)`) |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
206 |
2765
41f5ac76d642
BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents:
2752
diff
changeset
|
207 func splitRBLB(s string) (*string, *string) { |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
208 m := rblbRe.FindStringSubmatch(s) |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
209 if len(m) == 0 { |
2765
41f5ac76d642
BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents:
2752
diff
changeset
|
210 return nil, nil |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
211 } |
2765
41f5ac76d642
BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents:
2752
diff
changeset
|
212 return &m[1], &m[2] |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
213 } |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
214 |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
215 // Do executes the actual bottleneck import. |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
216 func (bn *Bottleneck) Do( |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
217 ctx context.Context, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
218 importID int64, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
219 conn *sql.Conn, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
220 feedback Feedback, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
221 ) (interface{}, error) { |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
222 |
2253
cfc523c70e90
Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2248
diff
changeset
|
223 fetch := func() ([]*ifbn.BottleNeckType, error) { |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
224 client := ifbn.NewIBottleneckService(bn.URL, bn.Insecure, nil) |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
225 |
4102
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
226 req := &ifbn.Export_bn_by_isrs{ |
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
227 Period: &ifbn.RequestedPeriod{Date_start: &time.Time{}}, |
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
228 } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
229 |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
230 resp, err := client.Export_bn_by_isrs(req) |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
231 if err != nil { |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
232 return nil, err |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
233 } |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
234 |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
235 if resp.Export_bn_by_isrsResult == nil { |
3562 | 236 return nil, errors.New( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4147
diff
changeset
|
237 "the requested service returned no bottlenecks") |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
238 } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
239 |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
240 return resp.Export_bn_by_isrsResult.BottleNeckType, nil |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
241 } |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
242 |
2563
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
243 return storeBottlenecks(ctx, fetch, importID, conn, feedback, bn.Tolerance) |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
244 } |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
245 |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
246 type bnStmts struct { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
247 insert *sql.Stmt |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
248 findExactMatch *sql.Stmt |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
249 findIntersecting *sql.Stmt |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
250 insertMaterial *sql.Stmt |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
251 track *sql.Stmt |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
252 } |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
253 |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
254 func (bs *bnStmts) close() { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
255 for _, s := range []**sql.Stmt{ |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
256 &bs.insert, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
257 &bs.findExactMatch, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
258 &bs.findIntersecting, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
259 &bs.insertMaterial, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
260 &bs.track, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
261 } { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
262 if *s != nil { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
263 (*s).Close() |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
264 *s = nil |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
265 } |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
266 } |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
267 } |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
268 |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
269 func (bs *bnStmts) prepare(ctx context.Context, conn *sql.Conn) error { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
270 for _, x := range []struct { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
271 sql string |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
272 stmt **sql.Stmt |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
273 }{ |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
274 {insertBottleneckSQL, &bs.insert}, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
275 {findExactMatchBottleneckSQL, &bs.findExactMatch}, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
276 {findIntersectingBottleneckSQL, &bs.findIntersecting}, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
277 {insertBottleneckMaterialSQL, &bs.insertMaterial}, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
278 {trackImportDeletionSQL, &bs.track}, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
279 } { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
280 var err error |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
281 if *x.stmt, err = conn.PrepareContext(ctx, x.sql); err != nil { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
282 return err |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
283 } |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
284 } |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
285 return nil |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
286 } |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
287 |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
288 func storeBottlenecks( |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
289 ctx context.Context, |
2253
cfc523c70e90
Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2248
diff
changeset
|
290 fetch func() ([]*ifbn.BottleNeckType, error), |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
291 importID int64, |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
292 conn *sql.Conn, |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
293 feedback Feedback, |
2563
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
294 tolerance float64, |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
295 ) (interface{}, error) { |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
296 start := time.Now() |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
297 |
2253
cfc523c70e90
Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2248
diff
changeset
|
298 bns, err := fetch() |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
299 if err != nil { |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
300 return nil, err |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
301 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
302 |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
303 feedback.Info("Found %d bottlenecks for import", len(bns)) |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
304 |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
305 var bs bnStmts |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
306 defer bs.close() |
1580
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
307 |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
308 if err := bs.prepare(ctx, conn); err != nil { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
309 return nil, err |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
310 } |
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
311 |
1611
1835d1c9eb9b
Store bottleneck_id's instead of id's in bottleneck import summary.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1595
diff
changeset
|
312 var nids []string |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
313 seenOldBnIDs := make(map[int64]bool) |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
314 |
2693 | 315 feedback.Info("Tolerance used to snap waterway axis: %g", tolerance) |
316 | |
4119
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
317 tx, err := conn.BeginTx(ctx, nil) |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
318 if err != nil { |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
319 return nil, err |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
320 } |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
321 defer tx.Rollback() |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
322 |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
323 savepoint := Savepoint(ctx, tx, "insert_bottlenck") |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
324 |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
325 for _, bn := range bns { |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
326 if err := storeBottleneck( |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
327 ctx, tx, |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
328 importID, feedback, bn, |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
329 &nids, seenOldBnIDs, tolerance, |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
330 &bs, |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
331 savepoint, |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
332 ); err != nil { |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
333 return nil, err |
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
334 } |
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
335 } |
4119
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
336 if err = tx.Commit(); err != nil { |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
337 return nil, err |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
338 } |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
339 |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
340 if len(nids) == 0 { |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
341 return nil, UnchangedError("No new bottlenecks inserted") |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
342 } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
343 |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
344 feedback.Info("Storing %d bottlenecks took %s", len(nids), time.Since(start)) |
1906
32c56e6c089a
Stretch import: Added forgotten source file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1902
diff
changeset
|
345 feedback.Info("Import of bottlenecks was successful") |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
346 summary := struct { |
1611
1835d1c9eb9b
Store bottleneck_id's instead of id's in bottleneck import summary.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1595
diff
changeset
|
347 Bottlenecks []string `json:"bottlenecks"` |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
348 }{ |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
349 Bottlenecks: nids, |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
350 } |
1906
32c56e6c089a
Stretch import: Added forgotten source file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1902
diff
changeset
|
351 return &summary, nil |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
352 } |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
353 |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
354 func storeBottleneck( |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
355 ctx context.Context, |
4119
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
356 tx *sql.Tx, |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
357 importID int64, |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
358 feedback Feedback, |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
359 bn *ifbn.BottleNeckType, |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
360 nids *[]string, |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
361 seenOldBnIDs map[int64]bool, |
2563
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
362 tolerance float64, |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
363 bs *bnStmts, |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
364 savepoint func(func() error) error, |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
365 ) error { |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
366 feedback.Info("Processing %s (%s)", bn.OBJNAM, bn.Bottleneck_id) |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
367 |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
368 var ( |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
369 tfrom, tto pgtype.Timestamptz |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
370 uBound pgtype.BoundType |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
371 ) |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
372 |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
373 if bn.AdditionalData == nil || bn.AdditionalData.KeyValuePair == nil { |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
374 // This is a workaround for the fact that most BN data does not |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
375 // contain the optional validity information. |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
376 // |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
377 // The current solution makes every change in the data an |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
378 // update, which might be not the best solution. Alternative |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
379 // approaches could include usingthe Date Info of the data as |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
380 // start date, and maybe even inspecting the details of changed |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
381 // data for hints wether an update or historisation with a new |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
382 // version is advisable. |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
383 // |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
384 // Never the less, the current solution "just works" for the |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
385 // time being and reflects the upstream systems... -- sw |
4134
9864d682ab47
bn-import: Downgraded info on missing validity from Warning to Info.
Sascha Wilde <wilde@intevation.de>
parents:
4133
diff
changeset
|
386 feedback.Info("No validity information, assuming infinite validity.") |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
387 tfrom.Set(time.Date(1970, time.January, 1, 0, 0, 0, 0, time.UTC)) |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
388 uBound = pgtype.Unbounded |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
389 } else { |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
390 const ( |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
391 fromKey = "Valid_from_date" |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
392 toKey = "Valid_to_date" |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
393 ) |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
394 fromTo := make(map[string]time.Time) |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
395 for _, kv := range bn.AdditionalData.KeyValuePair { |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
396 if kv.Key == fromKey || kv.Key == toKey { |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
397 t, err := time.Parse(time.RFC3339, kv.Value) |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
398 if err != nil { |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
399 return err |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
400 } |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
401 fromTo[kv.Key] = t |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
402 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
403 } |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
404 |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
405 if t, ok := fromTo[fromKey]; ok { |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
406 tfrom.Set(t) |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
407 } else { |
4197
5d7ce7f926eb
Log fatal errors as errors, not warnings, in imports
Tom Gottfried <tom@intevation.de>
parents:
4177
diff
changeset
|
408 feedback.Error("Missing start date") |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
409 return nil |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
410 } |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
411 |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
412 if t, ok := fromTo[toKey]; ok { |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
413 tto.Set(t) |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
414 uBound = pgtype.Exclusive |
4102
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
415 feedback.Info("Valid from %s to %s", |
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
416 fromTo[fromKey].Format(common.TimeFormat), |
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
417 fromTo[toKey].Format(common.TimeFormat)) |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
418 } else { |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
419 uBound = pgtype.Unbounded |
4102
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
420 feedback.Info("Valid from %s", |
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
421 fromTo[fromKey].Format(common.TimeFormat)) |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
422 } |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
423 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
424 |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
425 validity := pgtype.Tstzrange{ |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
426 Lower: tfrom, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
427 Upper: tto, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
428 LowerType: pgtype.Inclusive, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
429 UpperType: uBound, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
430 Status: pgtype.Present, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
431 } |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
432 |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
433 rb, lb := splitRBLB(bn.Rb_lb) |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
434 |
3423
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
435 var revisitingTime *int |
3528
b9e331c1e616
Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents:
3423
diff
changeset
|
436 if bn.Revisiting_time != nil && |
b9e331c1e616
Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents:
3423
diff
changeset
|
437 len(strings.TrimSpace(*bn.Revisiting_time)) > 0 { |
3423
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
438 i, err := strconv.Atoi(*bn.Revisiting_time) |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
439 if err != nil { |
3530 | 440 feedback.Warn( |
441 "Cannot convert given revisiting time '%s' to number of months", | |
3423
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
442 *bn.Revisiting_time) |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
443 } else { |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
444 revisitingTime = &i |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
445 } |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
446 } |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
447 |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
448 var limiting, country string |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
449 |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
450 if bn.Limiting_factor != nil { |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
451 limiting = string(*bn.Limiting_factor) |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
452 } |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
453 |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
454 if bn.Responsible_country != nil { |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
455 country = string(*bn.Responsible_country) |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
456 } |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
457 |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
458 var materials []string |
2999
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
459 if bn.Riverbed != nil { |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
460 for _, material := range bn.Riverbed.Material { |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
461 if material != nil { |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
462 materials = append(materials, string(*material)) |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
463 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
464 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
465 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
466 |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
467 // Check if a bottleneck identical to the one we would insert already |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
468 // exists: |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
469 var old int64 |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
470 err := tx.StmtContext(ctx, bs.findExactMatch).QueryRowContext( |
4119
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
471 ctx, |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
472 bn.Bottleneck_id, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
473 &validity, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
474 bn.Fk_g_fid, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
475 bn.From_ISRS, bn.To_ISRS, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
476 country, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
477 limiting, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
478 bn.Date_Info, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
479 bn.Source, |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
480 ).Scan(&old) |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
481 switch { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
482 case err == sql.ErrNoRows: |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
483 // We dont have a matching old. |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
484 case err != nil: |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
485 return err |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
486 default: |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
487 // We could check if the materials are also matching -- but per |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
488 // specification the Date_Info would hvae to change on that kind of |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
489 // change anyway. So actually we are already checking more in depth than |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
490 // required. |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
491 feedback.Info("unchanged") |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
492 return nil |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
493 } |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
494 |
3815
aa7bede70b96
Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents:
3705
diff
changeset
|
495 // Additional Information, only of interest when we change something, so |
aa7bede70b96
Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents:
3705
diff
changeset
|
496 // it can be used for debugging if something goes wrong... |
aa7bede70b96
Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents:
3705
diff
changeset
|
497 feedback.Info("Range: from %s to %s", bn.From_ISRS, bn.To_ISRS) |
aa7bede70b96
Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents:
3705
diff
changeset
|
498 |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
499 // Check if the new bottleneck intersects with the validity of existing |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
500 // for the same bottleneck_id we consider this an update and mark the |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
501 // old data for deletion. |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
502 bns, err := tx.StmtContext(ctx, bs.findIntersecting).QueryContext( |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
503 ctx, bn.Bottleneck_id, &validity, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
504 ) |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
505 if err != nil { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
506 return err |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
507 } |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
508 defer bns.Close() |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
509 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
510 // Mark old intersecting bottleneck data for deletion. Don't worry about |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
511 // materials, they will be deleted via cascading. |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
512 var oldBnIDs []int64 |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
513 for bns.Next() { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
514 var oldID int64 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
515 err := bns.Scan(&oldID) |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
516 if err != nil { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
517 return err |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
518 } |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
519 oldBnIDs = append(oldBnIDs, oldID) |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
520 } |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
521 |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
522 if err := bns.Err(); err != nil { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
523 return err |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
524 } |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
525 |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
526 switch { |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
527 case len(oldBnIDs) == 1: |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
528 feedback.Info("Bottleneck '%s' "+ |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
529 "with intersecting validity already exists: "+ |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
530 "UPDATING", bn.Bottleneck_id) |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
531 case len(oldBnIDs) > 1: |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
532 // This case is unexpected and should only happen when historic |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
533 // data in the bottleneck service was changed subsequently... |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
534 // We handle it gracefully anyway, but warn. |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
535 feedback.Warn("More than one Bottelneck '%s' "+ |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
536 "with intersecting validity already exists: "+ |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
537 "REPLACING all of them!", bn.Bottleneck_id) |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
538 } |
4133
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
539 // We write the actual tracking information for deletion of superseded |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
540 // bottlenecks later to the database -- AFTER the new bottleneck was |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
541 // created successfully. That way, we don't change the database, when |
4133
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
542 // an error arises during inserting the new data. |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
543 |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
544 var bnID int64 |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
545 // Add new BN data: |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
546 if err := savepoint(func() error { |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
547 if err := tx.StmtContext(ctx, bs.insert).QueryRowContext( |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
548 ctx, |
4119
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
549 bn.Bottleneck_id, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
550 &validity, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
551 bn.Fk_g_fid, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
552 bn.OBJNAM, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
553 bn.NOBJNM, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
554 bn.From_ISRS, bn.To_ISRS, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
555 rb, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
556 lb, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
557 country, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
558 revisitingTime, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
559 limiting, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
560 bn.Date_Info, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
561 bn.Source, |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
562 tolerance, |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
563 ).Scan(&bnID); err != nil { |
4119
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
564 return err |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
565 } |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
566 |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
567 // Add new materials |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
568 for _, material := range materials { |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
569 if _, err := tx.StmtContext(ctx, bs.insertMaterial).ExecContext( |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
570 ctx, |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
571 bnID, |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
572 material, |
4119
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
573 ); err != nil { |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
574 return err |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
575 } |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
576 } |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
577 return nil |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
578 }); err != nil { |
4197
5d7ce7f926eb
Log fatal errors as errors, not warnings, in imports
Tom Gottfried <tom@intevation.de>
parents:
4177
diff
changeset
|
579 feedback.Error(pgxutils.ReadableError{Err: err}.Error()) |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
580 return nil |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
581 } |
2999
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
582 |
4133
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
583 // Now that adding BNs to staging was successful, write import tracking |
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
584 // information to database: |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
585 for _, oldID := range oldBnIDs { |
4133
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
586 // It is possible, that two new bottlenecks intersect with the |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
587 // same old one, therefore we have to handle duplicates in |
4133
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
588 // oldBnIds. |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
589 if !seenOldBnIDs[oldID] { |
4133
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
590 if _, err := tx.StmtContext(ctx, bs.track).ExecContext( |
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
591 ctx, importID, "waterway.bottlenecks", oldID, true, |
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
592 ); err != nil { |
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
593 return err |
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
594 } |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
595 seenOldBnIDs[oldID] = true |
4133
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
596 } |
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
597 } |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
598 |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
599 if _, err := tx.StmtContext(ctx, bs.track).ExecContext( |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
600 ctx, importID, "waterway.bottlenecks", bnID, false, |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
601 ); err != nil { |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
602 return err |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
603 } |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
604 |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
605 *nids = append(*nids, bn.Bottleneck_id) |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
606 return nil |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
607 } |