Mercurial > gemma
annotate pkg/imports/bn.go @ 4550:aa2d0006e742 iso-areas
Write iso lines between classes to SVG for debugging.
author | Sascha L. Teichmann <sascha.teichmann@intevation.de> |
---|---|
date | Mon, 30 Sep 2019 16:34:55 +0200 |
parents | 5d7ce7f926eb |
children | ca6a5f722471 |
rev | line source |
---|---|
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
7 // Copyright (C) 2018, 2019 by via donau |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
13 // * Tom Gottfried <tom.gottfried@intevation.de> |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
14 // * Sascha Wilde <sascha.wilde@intevation.de> |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 package imports |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 import ( |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "context" |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "database/sql" |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
21 "errors" |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
22 "regexp" |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
23 "strconv" |
3528
b9e331c1e616
Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents:
3423
diff
changeset
|
24 "strings" |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
25 "time" |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
26 |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
27 "github.com/jackc/pgx/pgtype" |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
28 |
4102
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
29 "gemma.intevation.de/gemma/pkg/common" |
4062
6c760abcff0e
Move handling of PostgreSQL errors to own package
Tom Gottfried <tom@intevation.de>
parents:
4058
diff
changeset
|
30 "gemma.intevation.de/gemma/pkg/pgxutils" |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
31 "gemma.intevation.de/gemma/pkg/soap/ifbn" |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
32 ) |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
33 |
1695
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
34 // Bottleneck is an import job to import |
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
35 // bottlenecks from an IFBN SOAP service. |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
36 type Bottleneck struct { |
1695
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
37 // URL is the URL of the SOAP service. |
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
38 URL string `json:"url"` |
2563
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
39 // Tolerance used for axis snapping |
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
40 Tolerance float64 `json:"tolerance"` |
1695
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
41 // Insecure indicates if HTTPS traffic |
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
42 // should validate certificates or not. |
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
43 Insecure bool `json:"insecure"` |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
44 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 |
1696
ad5e1cddaa09
Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1695
diff
changeset
|
46 // BNJobKind is the import queue type identifier. |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
47 const BNJobKind JobKind = "bn" |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
48 |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
49 const ( |
1658
07f996b0e14a
Bottleneck import: Cleaned up source a bit (removed outdated TODOs).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1646
diff
changeset
|
50 insertBottleneckSQL = ` |
3566
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
51 WITH |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
52 bounds (b) AS (VALUES (isrs_fromText($6)), (isrs_fromText($7))), |
3566
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
53 r AS (SELECT isrsrange( |
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
54 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY), |
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
55 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r) |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
56 INSERT INTO waterway.bottlenecks ( |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
57 bottleneck_id, |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
58 validity, |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3224
diff
changeset
|
59 gauge_location, |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
60 objnam, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
61 nobjnm, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
62 stretch, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
63 area, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
64 rb, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
65 lb, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
66 responsible_country, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
67 revisiting_time, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
68 limiting, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
69 date_info, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
70 source_organization |
4069
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
71 ) VALUES ( |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
72 $1, |
4069
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
73 $2::tstzrange, |
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
74 isrs_fromText($3), |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
75 $4, |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
76 $5, |
3566
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
77 (SELECT r FROM r), |
1631
6ee06eb37fa9
Use new function to calculate bn area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1611
diff
changeset
|
78 ISRSrange_area( |
3566
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
79 ISRSrange_axis((SELECT r FROM r), |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
80 $15), |
2248
cce158db02b0
Input area as multipolygons to generate area from stretch
Tom Gottfried <tom@intevation.de>
parents:
2195
diff
changeset
|
81 (SELECT ST_Collect(CAST(area AS geometry)) |
1902
c4af342be999
Use waterway area as basis for bottleneck area generation.
Sascha Wilde <wilde@intevation.de>
parents:
1754
diff
changeset
|
82 FROM waterway.waterway_area)), |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
83 $8, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
84 $9, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
85 $10, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
86 $11, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
87 $12, |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
88 $13, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
89 $14 |
4069
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
90 ) |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
91 RETURNING id |
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
92 ` |
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
93 |
4130
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
94 // We only check for NOT NULL values, for correct compairison with |
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
95 // values, which might be null (and then muyst not be compairt with `=' |
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
96 // but with `IS NULL' is comlicated and that we are checking more than |
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
97 // only (bottleneck_id, validity, date_info) is luxury already. |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
98 findExactMatchBottleneckSQL = ` |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
99 WITH |
4130
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
100 bounds (b) AS (VALUES (isrs_fromText($4)), (isrs_fromText($5))), |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
101 r AS (SELECT isrsrange( |
4130
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
102 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY), |
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
103 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r) |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
104 SELECT id FROM waterway.bottlenecks |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
105 WHERE ( |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
106 bottleneck_id, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
107 validity, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
108 gauge_location, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
109 stretch, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
110 responsible_country, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
111 limiting, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
112 date_info, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
113 source_organization, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
114 staging_done |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
115 ) = ( SELECT |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
116 $1, |
4069
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
117 $2::tstzrange, |
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
118 isrs_fromText($3), |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
119 (SELECT r FROM r), |
4130
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
120 $6, |
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
121 $7, |
980f12d3c766
bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents:
4128
diff
changeset
|
122 $8::timestamptz, |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
123 $9, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
124 true |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
125 ) |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
126 ` |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
127 |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
128 findIntersectingBottleneckSQL = ` |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
129 SELECT id FROM waterway.bottlenecks |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
130 WHERE (bottleneck_id, staging_done) = ($1, true) |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
131 AND $2::tstzrange && validity |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
132 ` |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
133 |
2999
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
134 insertBottleneckMaterialSQL = ` |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
135 INSERT INTO waterway.bottlenecks_riverbed_materials ( |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
136 bottleneck_id, |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
137 riverbed |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
138 ) VALUES ($1, $2) |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
139 ON CONFLICT (bottleneck_id, riverbed) DO NOTHING |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
140 ` |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
141 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
142 bnStageDoneDeleteSQL = ` |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
143 DELETE FROM waterway.bottlenecks WHERE id IN ( |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
144 SELECT key |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
145 FROM import.track_imports |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
146 WHERE import_id = $1 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
147 AND relation = 'waterway.bottlenecks'::regclass |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
148 AND deletion |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
149 )` |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
150 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
151 bnStageDoneSQL = ` |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
152 UPDATE waterway.bottlenecks SET staging_done = true |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
153 WHERE id IN ( |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
154 SELECT key |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
155 FROM import.track_imports |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
156 WHERE import_id = $1 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
157 AND relation = 'waterway.bottlenecks'::regclass |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
158 AND NOT deletion |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
159 )` |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
160 ) |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
161 |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
162 type bnJobCreator struct{} |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
163 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
164 func init() { |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
165 RegisterJobCreator(BNJobKind, bnJobCreator{}) |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
166 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
167 |
1754
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
168 func (bnJobCreator) Description() string { return "bottlenecks" } |
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
169 |
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
170 func (bnJobCreator) AutoAccept() bool { return false } |
1646
a0982c38eac0
Import queue: Implemented email notifications.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1631
diff
changeset
|
171 |
2187
7c83b5277c1c
Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
172 func (bnJobCreator) Create() Job { return new(Bottleneck) } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
173 |
3219
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3164
diff
changeset
|
174 func (bnJobCreator) Depends() [2][]string { |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3164
diff
changeset
|
175 return [2][]string{ |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3164
diff
changeset
|
176 {"bottlenecks", "bottlenecks_riverbed_materials"}, |
3224
bfde4f8dd323
Add missing table dependencies of imports with area generation
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
177 {"gauges", "distance_marks_virtual", "waterway_axis", "waterway_area"}, |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
178 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
179 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
180 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
181 // StageDone moves the imported bottleneck out of the staging area. |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
182 func (bnJobCreator) StageDone( |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
183 ctx context.Context, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
184 tx *sql.Tx, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
185 id int64, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
186 ) error { |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
187 _, err := tx.ExecContext(ctx, bnStageDoneDeleteSQL, id) |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
188 if err == nil { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
189 _, err = tx.ExecContext(ctx, bnStageDoneSQL, id) |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
190 } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
191 return err |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
192 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
193 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
194 // CleanUp of a bottleneck import is a NOP. |
1676
4407ecaa2192
Imports: Cosmetics.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1664
diff
changeset
|
195 func (*Bottleneck) CleanUp() error { return nil } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
196 |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
197 var rblbRe = regexp.MustCompile(`(..)_(..)`) |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
198 |
2765
41f5ac76d642
BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents:
2752
diff
changeset
|
199 func splitRBLB(s string) (*string, *string) { |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
200 m := rblbRe.FindStringSubmatch(s) |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
201 if len(m) == 0 { |
2765
41f5ac76d642
BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents:
2752
diff
changeset
|
202 return nil, nil |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
203 } |
2765
41f5ac76d642
BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents:
2752
diff
changeset
|
204 return &m[1], &m[2] |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
205 } |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
206 |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
207 // Do executes the actual bottleneck import. |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
208 func (bn *Bottleneck) Do( |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
209 ctx context.Context, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
210 importID int64, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
211 conn *sql.Conn, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
212 feedback Feedback, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
213 ) (interface{}, error) { |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
214 |
2253
cfc523c70e90
Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2248
diff
changeset
|
215 fetch := func() ([]*ifbn.BottleNeckType, error) { |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
216 client := ifbn.NewIBottleneckService(bn.URL, bn.Insecure, nil) |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
217 |
4102
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
218 req := &ifbn.Export_bn_by_isrs{ |
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
219 Period: &ifbn.RequestedPeriod{Date_start: &time.Time{}}, |
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
220 } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
221 |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
222 resp, err := client.Export_bn_by_isrs(req) |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
223 if err != nil { |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
224 return nil, err |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
225 } |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
226 |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
227 if resp.Export_bn_by_isrsResult == nil { |
3562 | 228 return nil, errors.New( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4147
diff
changeset
|
229 "the requested service returned no bottlenecks") |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
230 } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
231 |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
232 return resp.Export_bn_by_isrsResult.BottleNeckType, nil |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
233 } |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
234 |
2563
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
235 return storeBottlenecks(ctx, fetch, importID, conn, feedback, bn.Tolerance) |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
236 } |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
237 |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
238 type bnStmts struct { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
239 insert *sql.Stmt |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
240 findExactMatch *sql.Stmt |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
241 findIntersecting *sql.Stmt |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
242 insertMaterial *sql.Stmt |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
243 track *sql.Stmt |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
244 } |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
245 |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
246 func (bs *bnStmts) close() { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
247 for _, s := range []**sql.Stmt{ |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
248 &bs.insert, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
249 &bs.findExactMatch, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
250 &bs.findIntersecting, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
251 &bs.insertMaterial, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
252 &bs.track, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
253 } { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
254 if *s != nil { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
255 (*s).Close() |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
256 *s = nil |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
257 } |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
258 } |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
259 } |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
260 |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
261 func (bs *bnStmts) prepare(ctx context.Context, conn *sql.Conn) error { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
262 for _, x := range []struct { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
263 sql string |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
264 stmt **sql.Stmt |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
265 }{ |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
266 {insertBottleneckSQL, &bs.insert}, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
267 {findExactMatchBottleneckSQL, &bs.findExactMatch}, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
268 {findIntersectingBottleneckSQL, &bs.findIntersecting}, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
269 {insertBottleneckMaterialSQL, &bs.insertMaterial}, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
270 {trackImportDeletionSQL, &bs.track}, |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
271 } { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
272 var err error |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
273 if *x.stmt, err = conn.PrepareContext(ctx, x.sql); err != nil { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
274 return err |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
275 } |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
276 } |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
277 return nil |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
278 } |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
279 |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
280 func storeBottlenecks( |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
281 ctx context.Context, |
2253
cfc523c70e90
Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2248
diff
changeset
|
282 fetch func() ([]*ifbn.BottleNeckType, error), |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
283 importID int64, |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
284 conn *sql.Conn, |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
285 feedback Feedback, |
2563
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
286 tolerance float64, |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
287 ) (interface{}, error) { |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
288 start := time.Now() |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
289 |
2253
cfc523c70e90
Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2248
diff
changeset
|
290 bns, err := fetch() |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
291 if err != nil { |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
292 return nil, err |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
293 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
294 |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
295 feedback.Info("Found %d bottlenecks for import", len(bns)) |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
296 |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
297 var bs bnStmts |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
298 defer bs.close() |
1580
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
299 |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
300 if err := bs.prepare(ctx, conn); err != nil { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
301 return nil, err |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
302 } |
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
303 |
1611
1835d1c9eb9b
Store bottleneck_id's instead of id's in bottleneck import summary.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1595
diff
changeset
|
304 var nids []string |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
305 seenOldBnIDs := make(map[int64]bool) |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
306 |
2693 | 307 feedback.Info("Tolerance used to snap waterway axis: %g", tolerance) |
308 | |
4119
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
309 tx, err := conn.BeginTx(ctx, nil) |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
310 if err != nil { |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
311 return nil, err |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
312 } |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
313 defer tx.Rollback() |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
314 |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
315 savepoint := Savepoint(ctx, tx, "insert_bottlenck") |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
316 |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
317 for _, bn := range bns { |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
318 if err := storeBottleneck( |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
319 ctx, tx, |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
320 importID, feedback, bn, |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
321 &nids, seenOldBnIDs, tolerance, |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
322 &bs, |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
323 savepoint, |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
324 ); err != nil { |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
325 return nil, err |
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
326 } |
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
327 } |
4119
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
328 if err = tx.Commit(); err != nil { |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
329 return nil, err |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
330 } |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
331 |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
332 if len(nids) == 0 { |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
333 return nil, UnchangedError("No new bottlenecks inserted") |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
334 } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
335 |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
336 feedback.Info("Storing %d bottlenecks took %s", len(nids), time.Since(start)) |
1906
32c56e6c089a
Stretch import: Added forgotten source file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1902
diff
changeset
|
337 feedback.Info("Import of bottlenecks was successful") |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
338 summary := struct { |
1611
1835d1c9eb9b
Store bottleneck_id's instead of id's in bottleneck import summary.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1595
diff
changeset
|
339 Bottlenecks []string `json:"bottlenecks"` |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
340 }{ |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
341 Bottlenecks: nids, |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
342 } |
1906
32c56e6c089a
Stretch import: Added forgotten source file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1902
diff
changeset
|
343 return &summary, nil |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
344 } |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
345 |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
346 func storeBottleneck( |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
347 ctx context.Context, |
4119
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
348 tx *sql.Tx, |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
349 importID int64, |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
350 feedback Feedback, |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
351 bn *ifbn.BottleNeckType, |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
352 nids *[]string, |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
353 seenOldBnIDs map[int64]bool, |
2563
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
354 tolerance float64, |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
355 bs *bnStmts, |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
356 savepoint func(func() error) error, |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
357 ) error { |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
358 feedback.Info("Processing %s (%s)", bn.OBJNAM, bn.Bottleneck_id) |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
359 |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
360 var ( |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
361 tfrom, tto pgtype.Timestamptz |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
362 uBound pgtype.BoundType |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
363 ) |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
364 |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
365 if bn.AdditionalData == nil || bn.AdditionalData.KeyValuePair == nil { |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
366 // This is a workaround for the fact that most BN data does not |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
367 // contain the optional validity information. |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
368 // |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
369 // The current solution makes every change in the data an |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
370 // update, which might be not the best solution. Alternative |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
371 // approaches could include usingthe Date Info of the data as |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
372 // start date, and maybe even inspecting the details of changed |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
373 // data for hints wether an update or historisation with a new |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
374 // version is advisable. |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
375 // |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
376 // Never the less, the current solution "just works" for the |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
377 // time being and reflects the upstream systems... -- sw |
4134
9864d682ab47
bn-import: Downgraded info on missing validity from Warning to Info.
Sascha Wilde <wilde@intevation.de>
parents:
4133
diff
changeset
|
378 feedback.Info("No validity information, assuming infinite validity.") |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
379 tfrom.Set(time.Date(1970, time.January, 1, 0, 0, 0, 0, time.UTC)) |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
380 uBound = pgtype.Unbounded |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
381 } else { |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
382 const ( |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
383 fromKey = "Valid_from_date" |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
384 toKey = "Valid_to_date" |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
385 ) |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
386 fromTo := make(map[string]time.Time) |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
387 for _, kv := range bn.AdditionalData.KeyValuePair { |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
388 if kv.Key == fromKey || kv.Key == toKey { |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
389 t, err := time.Parse(time.RFC3339, kv.Value) |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
390 if err != nil { |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
391 return err |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
392 } |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
393 fromTo[kv.Key] = t |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
394 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
395 } |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
396 |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
397 if t, ok := fromTo[fromKey]; ok { |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
398 tfrom.Set(t) |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
399 } else { |
4197
5d7ce7f926eb
Log fatal errors as errors, not warnings, in imports
Tom Gottfried <tom@intevation.de>
parents:
4177
diff
changeset
|
400 feedback.Error("Missing start date") |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
401 return nil |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
402 } |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
403 |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
404 if t, ok := fromTo[toKey]; ok { |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
405 tto.Set(t) |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
406 uBound = pgtype.Exclusive |
4102
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
407 feedback.Info("Valid from %s to %s", |
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
408 fromTo[fromKey].Format(common.TimeFormat), |
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
409 fromTo[toKey].Format(common.TimeFormat)) |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
410 } else { |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
411 uBound = pgtype.Unbounded |
4102
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
412 feedback.Info("Valid from %s", |
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
413 fromTo[fromKey].Format(common.TimeFormat)) |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
414 } |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
415 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
416 |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
417 validity := pgtype.Tstzrange{ |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
418 Lower: tfrom, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
419 Upper: tto, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
420 LowerType: pgtype.Inclusive, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
421 UpperType: uBound, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
422 Status: pgtype.Present, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
423 } |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
424 |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
425 rb, lb := splitRBLB(bn.Rb_lb) |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
426 |
3423
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
427 var revisitingTime *int |
3528
b9e331c1e616
Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents:
3423
diff
changeset
|
428 if bn.Revisiting_time != nil && |
b9e331c1e616
Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents:
3423
diff
changeset
|
429 len(strings.TrimSpace(*bn.Revisiting_time)) > 0 { |
3423
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
430 i, err := strconv.Atoi(*bn.Revisiting_time) |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
431 if err != nil { |
3530 | 432 feedback.Warn( |
433 "Cannot convert given revisiting time '%s' to number of months", | |
3423
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
434 *bn.Revisiting_time) |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
435 } else { |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
436 revisitingTime = &i |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
437 } |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
438 } |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
439 |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
440 var limiting, country string |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
441 |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
442 if bn.Limiting_factor != nil { |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
443 limiting = string(*bn.Limiting_factor) |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
444 } |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
445 |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
446 if bn.Responsible_country != nil { |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
447 country = string(*bn.Responsible_country) |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
448 } |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
449 |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
450 var materials []string |
2999
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
451 if bn.Riverbed != nil { |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
452 for _, material := range bn.Riverbed.Material { |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
453 if material != nil { |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
454 materials = append(materials, string(*material)) |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
455 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
456 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
457 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
458 |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
459 // Check if a bottleneck identical to the one we would insert already |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
460 // exists: |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
461 var old int64 |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
462 err := tx.StmtContext(ctx, bs.findExactMatch).QueryRowContext( |
4119
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
463 ctx, |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
464 bn.Bottleneck_id, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
465 &validity, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
466 bn.Fk_g_fid, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
467 bn.From_ISRS, bn.To_ISRS, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
468 country, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
469 limiting, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
470 bn.Date_Info, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
471 bn.Source, |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
472 ).Scan(&old) |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
473 switch { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
474 case err == sql.ErrNoRows: |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
475 // We dont have a matching old. |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
476 case err != nil: |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
477 return err |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
478 default: |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
479 // We could check if the materials are also matching -- but per |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
480 // specification the Date_Info would hvae to change on that kind of |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
481 // change anyway. So actually we are already checking more in depth than |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
482 // required. |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
483 feedback.Info("unchanged") |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
484 return nil |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
485 } |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
486 |
3815
aa7bede70b96
Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents:
3705
diff
changeset
|
487 // Additional Information, only of interest when we change something, so |
aa7bede70b96
Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents:
3705
diff
changeset
|
488 // it can be used for debugging if something goes wrong... |
aa7bede70b96
Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents:
3705
diff
changeset
|
489 feedback.Info("Range: from %s to %s", bn.From_ISRS, bn.To_ISRS) |
aa7bede70b96
Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents:
3705
diff
changeset
|
490 |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
491 // Check if the new bottleneck intersects with the validity of existing |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
492 // for the same bottleneck_id we consider this an update and mark the |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
493 // old data for deletion. |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
494 bns, err := tx.StmtContext(ctx, bs.findIntersecting).QueryContext( |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
495 ctx, bn.Bottleneck_id, &validity, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
496 ) |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
497 if err != nil { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
498 return err |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
499 } |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
500 defer bns.Close() |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
501 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
502 // Mark old intersecting bottleneck data for deletion. Don't worry about |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
503 // materials, they will be deleted via cascading. |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
504 var oldBnIDs []int64 |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
505 for bns.Next() { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
506 var oldID int64 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
507 err := bns.Scan(&oldID) |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
508 if err != nil { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
509 return err |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
510 } |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
511 oldBnIDs = append(oldBnIDs, oldID) |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
512 } |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
513 |
4128
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
514 if err := bns.Err(); err != nil { |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
515 return err |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
516 } |
14706384a464
Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents:
4119
diff
changeset
|
517 |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
518 switch { |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
519 case len(oldBnIDs) == 1: |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
520 feedback.Info("Bottleneck '%s' "+ |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
521 "with intersecting validity already exists: "+ |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
522 "UPDATING", bn.Bottleneck_id) |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
523 case len(oldBnIDs) > 1: |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
524 // This case is unexpected and should only happen when historic |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
525 // data in the bottleneck service was changed subsequently... |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
526 // We handle it gracefully anyway, but warn. |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
527 feedback.Warn("More than one Bottelneck '%s' "+ |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
528 "with intersecting validity already exists: "+ |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
529 "REPLACING all of them!", bn.Bottleneck_id) |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
530 } |
4133
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
531 // We write the actual tracking information for deletion of superseded |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
532 // bottlenecks later to the database -- AFTER the new bottleneck was |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
533 // created successfully. That way, we don't change the database, when |
4133
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
534 // an error arises during inserting the new data. |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
535 |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
536 var bnID int64 |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
537 // Add new BN data: |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
538 if err := savepoint(func() error { |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
539 if err := tx.StmtContext(ctx, bs.insert).QueryRowContext( |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
540 ctx, |
4119
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
541 bn.Bottleneck_id, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
542 &validity, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
543 bn.Fk_g_fid, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
544 bn.OBJNAM, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
545 bn.NOBJNM, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
546 bn.From_ISRS, bn.To_ISRS, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
547 rb, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
548 lb, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
549 country, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
550 revisitingTime, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
551 limiting, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
552 bn.Date_Info, |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
553 bn.Source, |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
554 tolerance, |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
555 ).Scan(&bnID); err != nil { |
4119
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
556 return err |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
557 } |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
558 |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
559 // Add new materials |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
560 for _, material := range materials { |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
561 if _, err := tx.StmtContext(ctx, bs.insertMaterial).ExecContext( |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
562 ctx, |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
563 bnID, |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
564 material, |
4119
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
565 ); err != nil { |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
566 return err |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
567 } |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
568 } |
0cf0008070db
bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents:
4113
diff
changeset
|
569 return nil |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
570 }); err != nil { |
4197
5d7ce7f926eb
Log fatal errors as errors, not warnings, in imports
Tom Gottfried <tom@intevation.de>
parents:
4177
diff
changeset
|
571 feedback.Error(pgxutils.ReadableError{Err: err}.Error()) |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
572 return nil |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
573 } |
2999
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
574 |
4133
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
575 // Now that adding BNs to staging was successful, write import tracking |
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
576 // information to database: |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
577 for _, oldID := range oldBnIDs { |
4133
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
578 // It is possible, that two new bottlenecks intersect with the |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
579 // same old one, therefore we have to handle duplicates in |
4133
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
580 // oldBnIds. |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
581 if !seenOldBnIDs[oldID] { |
4133
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
582 if _, err := tx.StmtContext(ctx, bs.track).ExecContext( |
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
583 ctx, importID, "waterway.bottlenecks", oldID, true, |
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
584 ); err != nil { |
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
585 return err |
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
586 } |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
587 seenOldBnIDs[oldID] = true |
4133
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
588 } |
eb08fbe33074
Fixed bn-import: never write tracking info to the db if import fails.
Sascha Wilde <wilde@intevation.de>
parents:
4130
diff
changeset
|
589 } |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
590 |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
591 if _, err := tx.StmtContext(ctx, bs.track).ExecContext( |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
592 ctx, importID, "waterway.bottlenecks", bnID, false, |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
593 ); err != nil { |
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4134
diff
changeset
|
594 return err |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
595 } |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
596 |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
597 *nids = append(*nids, bn.Bottleneck_id) |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
598 return nil |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
599 } |