Mercurial > gemma
annotate pkg/imports/bn.go @ 4113:f39d20427e89 request_hist_bns
WIP: Reworked bottleneck import:
- all available bottlenecks from the service are loaded (including
historic data) - kudos to Tom Gottfried
- Bottlenecks which are identical to an exiting (non staged) one are
recognized as "unchanged" and ignored.
- All remaining Bottlenecks are inserted into Staging area
- On "staging done" all existing bottlenecks with conflicting
(bottleneck_id,validity) are deleted (replaced by the new
bottlenecks).
Known Issues:
- Sometimes the identification of unchanged bottlenecks does not work,
needs debugging.
- The front end does not handle historic data correctly (we need to
define what "correctly" should be).
- Save points should be used to distinguish ignored errors from fatal
ones.
author | Sascha Wilde <wilde@intevation.de> |
---|---|
date | Wed, 31 Jul 2019 17:04:31 +0200 |
parents | 861760675497 |
children | 0cf0008070db |
rev | line source |
---|---|
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
7 // Copyright (C) 2018, 2019 by via donau |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
13 // * Tom Gottfried <tom.gottfried@intevation.de> |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
14 // * Sascha Wilde <sascha.wilde@intevation.de> |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 package imports |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 import ( |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "context" |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "database/sql" |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
21 "errors" |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
22 "regexp" |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
23 "strconv" |
3528
b9e331c1e616
Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents:
3423
diff
changeset
|
24 "strings" |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
25 "time" |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
26 |
4102
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
27 "gemma.intevation.de/gemma/pkg/common" |
4062
6c760abcff0e
Move handling of PostgreSQL errors to own package
Tom Gottfried <tom@intevation.de>
parents:
4058
diff
changeset
|
28 "gemma.intevation.de/gemma/pkg/pgxutils" |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
29 "gemma.intevation.de/gemma/pkg/soap/ifbn" |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
30 "github.com/jackc/pgx/pgtype" |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
31 ) |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
32 |
1695
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
33 // Bottleneck is an import job to import |
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
34 // bottlenecks from an IFBN SOAP service. |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
35 type Bottleneck struct { |
1695
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
36 // URL is the URL of the SOAP service. |
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
37 URL string `json:"url"` |
2563
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
38 // Tolerance used for axis snapping |
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
39 Tolerance float64 `json:"tolerance"` |
1695
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
40 // Insecure indicates if HTTPS traffic |
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
41 // should validate certificates or not. |
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
42 Insecure bool `json:"insecure"` |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
43 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
44 |
1696
ad5e1cddaa09
Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1695
diff
changeset
|
45 // BNJobKind is the import queue type identifier. |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
46 const BNJobKind JobKind = "bn" |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
47 |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
48 const ( |
1658
07f996b0e14a
Bottleneck import: Cleaned up source a bit (removed outdated TODOs).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1646
diff
changeset
|
49 insertBottleneckSQL = ` |
3566
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
50 WITH |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
51 bounds (b) AS (VALUES (isrs_fromText($6)), (isrs_fromText($7))), |
3566
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
52 r AS (SELECT isrsrange( |
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
53 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY), |
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
54 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r) |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
55 INSERT INTO waterway.bottlenecks ( |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
56 bottleneck_id, |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
57 validity, |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3224
diff
changeset
|
58 gauge_location, |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
59 objnam, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
60 nobjnm, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
61 stretch, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
62 area, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
63 rb, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
64 lb, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
65 responsible_country, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
66 revisiting_time, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
67 limiting, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
68 date_info, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
69 source_organization |
4069
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
70 ) VALUES ( |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
71 $1, |
4069
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
72 $2::tstzrange, |
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
73 isrs_fromText($3), |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
74 $4, |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
75 $5, |
3566
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
76 (SELECT r FROM r), |
1631
6ee06eb37fa9
Use new function to calculate bn area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1611
diff
changeset
|
77 ISRSrange_area( |
3566
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
78 ISRSrange_axis((SELECT r FROM r), |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
79 $15), |
2248
cce158db02b0
Input area as multipolygons to generate area from stretch
Tom Gottfried <tom@intevation.de>
parents:
2195
diff
changeset
|
80 (SELECT ST_Collect(CAST(area AS geometry)) |
1902
c4af342be999
Use waterway area as basis for bottleneck area generation.
Sascha Wilde <wilde@intevation.de>
parents:
1754
diff
changeset
|
81 FROM waterway.waterway_area)), |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
82 $8, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
83 $9, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
84 $10, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
85 $11, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
86 $12, |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
87 $13, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
88 $14 |
4069
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
89 ) |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
90 RETURNING id |
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
91 ` |
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
92 |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
93 findExactMatchBottleneckSQL = ` |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
94 WITH |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
95 bounds (b) AS (VALUES (isrs_fromText($6)), (isrs_fromText($7))), |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
96 r AS (SELECT isrsrange( |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
97 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY), |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
98 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r) |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
99 SELECT id FROM waterway.bottlenecks |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
100 WHERE ( |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
101 bottleneck_id, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
102 validity, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
103 gauge_location, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
104 objnam, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
105 nobjnm, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
106 stretch, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
107 rb, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
108 lb, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
109 responsible_country, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
110 revisiting_time, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
111 limiting, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
112 date_info, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
113 source_organization, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
114 staging_done |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
115 ) = ( SELECT |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
116 $1, |
4069
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
117 $2::tstzrange, |
12f476e91c70
Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents:
3815
diff
changeset
|
118 isrs_fromText($3), |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
119 $4, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
120 $5, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
121 (SELECT r FROM r), |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
122 $8, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
123 $9, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
124 $10, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
125 $11::smallint, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
126 $12, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
127 $13::timestamptz, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
128 $14, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
129 true |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
130 ) |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
131 ` |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
132 |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
133 findIntersectingBottleneckSQL = ` |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
134 SELECT id FROM waterway.bottlenecks |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
135 WHERE (bottleneck_id, staging_done) = ($1, true) |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
136 AND $2::tstzrange && validity |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
137 ` |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
138 |
2999
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
139 insertBottleneckMaterialSQL = ` |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
140 INSERT INTO waterway.bottlenecks_riverbed_materials ( |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
141 bottleneck_id, |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
142 riverbed |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
143 ) SELECT * |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
144 FROM unnest(CAST($1 AS int[])) AS bns, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
145 unnest(CAST($2 AS varchar[])) AS materials |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
146 ON CONFLICT (bottleneck_id, riverbed) DO NOTHING |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
147 ` |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
148 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
149 bnStageDoneDeleteSQL = ` |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
150 DELETE FROM waterway.bottlenecks WHERE id IN ( |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
151 SELECT key |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
152 FROM import.track_imports |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
153 WHERE import_id = $1 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
154 AND relation = 'waterway.bottlenecks'::regclass |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
155 AND deletion |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
156 )` |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
157 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
158 bnStageDoneSQL = ` |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
159 UPDATE waterway.bottlenecks SET staging_done = true |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
160 WHERE id IN ( |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
161 SELECT key |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
162 FROM import.track_imports |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
163 WHERE import_id = $1 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
164 AND relation = 'waterway.bottlenecks'::regclass |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
165 AND NOT deletion |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
166 )` |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
167 ) |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
168 |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
169 type bnJobCreator struct{} |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
170 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
171 func init() { |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
172 RegisterJobCreator(BNJobKind, bnJobCreator{}) |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
173 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
174 |
1754
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
175 func (bnJobCreator) Description() string { return "bottlenecks" } |
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
176 |
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
177 func (bnJobCreator) AutoAccept() bool { return false } |
1646
a0982c38eac0
Import queue: Implemented email notifications.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1631
diff
changeset
|
178 |
2187
7c83b5277c1c
Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
179 func (bnJobCreator) Create() Job { return new(Bottleneck) } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
180 |
3219
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3164
diff
changeset
|
181 func (bnJobCreator) Depends() [2][]string { |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3164
diff
changeset
|
182 return [2][]string{ |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3164
diff
changeset
|
183 {"bottlenecks", "bottlenecks_riverbed_materials"}, |
3224
bfde4f8dd323
Add missing table dependencies of imports with area generation
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
184 {"gauges", "distance_marks_virtual", "waterway_axis", "waterway_area"}, |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
185 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
186 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
187 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
188 // StageDone moves the imported bottleneck out of the staging area. |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
189 func (bnJobCreator) StageDone( |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
190 ctx context.Context, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
191 tx *sql.Tx, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
192 id int64, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
193 ) error { |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
194 _, err := tx.ExecContext(ctx, bnStageDoneDeleteSQL, id) |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
195 if err == nil { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
196 _, err = tx.ExecContext(ctx, bnStageDoneSQL, id) |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
197 } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
198 return err |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
199 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
200 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
201 // CleanUp of a bottleneck import is a NOP. |
1676
4407ecaa2192
Imports: Cosmetics.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1664
diff
changeset
|
202 func (*Bottleneck) CleanUp() error { return nil } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
203 |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
204 var rblbRe = regexp.MustCompile(`(..)_(..)`) |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
205 |
2765
41f5ac76d642
BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents:
2752
diff
changeset
|
206 func splitRBLB(s string) (*string, *string) { |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
207 m := rblbRe.FindStringSubmatch(s) |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
208 if len(m) == 0 { |
2765
41f5ac76d642
BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents:
2752
diff
changeset
|
209 return nil, nil |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
210 } |
2765
41f5ac76d642
BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents:
2752
diff
changeset
|
211 return &m[1], &m[2] |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
212 } |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
213 |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
214 // Do executes the actual bottleneck import. |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
215 func (bn *Bottleneck) Do( |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
216 ctx context.Context, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
217 importID int64, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
218 conn *sql.Conn, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
219 feedback Feedback, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
220 ) (interface{}, error) { |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
221 |
2253
cfc523c70e90
Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2248
diff
changeset
|
222 fetch := func() ([]*ifbn.BottleNeckType, error) { |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
223 client := ifbn.NewIBottleneckService(bn.URL, bn.Insecure, nil) |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
224 |
4102
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
225 req := &ifbn.Export_bn_by_isrs{ |
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
226 Period: &ifbn.RequestedPeriod{Date_start: &time.Time{}}, |
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
227 } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
228 |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
229 resp, err := client.Export_bn_by_isrs(req) |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
230 if err != nil { |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
231 return nil, err |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
232 } |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
233 |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
234 if resp.Export_bn_by_isrsResult == nil { |
3562 | 235 return nil, errors.New( |
236 "The requested service returned no bottlenecks") | |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
237 } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
238 |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
239 return resp.Export_bn_by_isrsResult.BottleNeckType, nil |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
240 } |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
241 |
2563
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
242 return storeBottlenecks(ctx, fetch, importID, conn, feedback, bn.Tolerance) |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
243 } |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
244 |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
245 func storeBottlenecks( |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
246 ctx context.Context, |
2253
cfc523c70e90
Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2248
diff
changeset
|
247 fetch func() ([]*ifbn.BottleNeckType, error), |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
248 importID int64, |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
249 conn *sql.Conn, |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
250 feedback Feedback, |
2563
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
251 tolerance float64, |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
252 ) (interface{}, error) { |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
253 start := time.Now() |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
254 |
2253
cfc523c70e90
Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2248
diff
changeset
|
255 bns, err := fetch() |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
256 if err != nil { |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
257 return nil, err |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
258 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
259 |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
260 feedback.Info("Found %d bottlenecks for import", len(bns)) |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
261 |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
262 var insertStmt, findExactMatchingBNStmt, findIntersectingBNStmt, |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
263 insertMaterialStmt, trackStmt *sql.Stmt |
1580
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
264 |
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
265 for _, x := range []struct { |
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
266 sql string |
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
267 stmt **sql.Stmt |
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
268 }{ |
1658
07f996b0e14a
Bottleneck import: Cleaned up source a bit (removed outdated TODOs).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1646
diff
changeset
|
269 {insertBottleneckSQL, &insertStmt}, |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
270 {findExactMatchBottleneckSQL, &findExactMatchingBNStmt}, |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
271 {findIntersectingBottleneckSQL, &findIntersectingBNStmt}, |
2999
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
272 {insertBottleneckMaterialSQL, &insertMaterialStmt}, |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
273 {trackImportDeletionSQL, &trackStmt}, |
1580
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
274 } { |
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
275 var err error |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
276 if *x.stmt, err = conn.PrepareContext(ctx, x.sql); err != nil { |
1580
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
277 return nil, err |
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
278 } |
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
279 defer (*x.stmt).Close() |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
280 } |
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
281 |
1611
1835d1c9eb9b
Store bottleneck_id's instead of id's in bottleneck import summary.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1595
diff
changeset
|
282 var nids []string |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
283 seenOldBnIds := make(map[int64]bool) |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
284 |
2693 | 285 feedback.Info("Tolerance used to snap waterway axis: %g", tolerance) |
286 | |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
287 for _, bn := range bns { |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
288 if err := storeBottleneck( |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
289 ctx, importID, conn, feedback, bn, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
290 &nids, &seenOldBnIds, tolerance, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
291 insertStmt, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
292 findExactMatchingBNStmt, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
293 findIntersectingBNStmt, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
294 insertMaterialStmt, |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
295 trackStmt); err != nil { |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
296 return nil, err |
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
297 } |
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
298 } |
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
299 if len(nids) == 0 { |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
300 return nil, UnchangedError("No new bottlenecks inserted") |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
301 } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
302 |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
303 feedback.Info("Storing %d bottlenecks took %s", len(nids), time.Since(start)) |
1906
32c56e6c089a
Stretch import: Added forgotten source file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1902
diff
changeset
|
304 feedback.Info("Import of bottlenecks was successful") |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
305 summary := struct { |
1611
1835d1c9eb9b
Store bottleneck_id's instead of id's in bottleneck import summary.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1595
diff
changeset
|
306 Bottlenecks []string `json:"bottlenecks"` |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
307 }{ |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
308 Bottlenecks: nids, |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
309 } |
1906
32c56e6c089a
Stretch import: Added forgotten source file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1902
diff
changeset
|
310 return &summary, nil |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
311 } |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
312 |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
313 func storeBottleneck( |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
314 ctx context.Context, |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
315 importID int64, |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
316 conn *sql.Conn, |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
317 feedback Feedback, |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
318 bn *ifbn.BottleNeckType, |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
319 nids *[]string, |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
320 seenOldBnIds *map[int64]bool, |
2563
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
321 tolerance float64, |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
322 insertStmt, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
323 findExactMatchingBNStmt, findIntersectingBNStmt, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
324 insertMaterialStmt, |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
325 trackStmt *sql.Stmt, |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
326 ) error { |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
327 feedback.Info("Processing %s (%s)", bn.OBJNAM, bn.Bottleneck_id) |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
328 |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
329 var tfrom, tto pgtype.Timestamptz |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
330 var uBound pgtype.BoundType |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
331 |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
332 if bn.AdditionalData == nil || bn.AdditionalData.KeyValuePair == nil { |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
333 // This is a workaround for the fact that most BN data does not |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
334 // contain the optional validity information. |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
335 // |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
336 // The current solution makes every change in the data an |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
337 // update, which might be not the best solution. Alternative |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
338 // approaches could include usingthe Date Info of the data as |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
339 // start date, and maybe even inspecting the details of changed |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
340 // data for hints wether an update or historisation with a new |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
341 // version is advisable. |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
342 // |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
343 // Never the less, the current solution "just works" for the |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
344 // time being and reflects the upstream systems... -- sw |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
345 feedback.Warn("No validity information, assuming infinite validity.") |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
346 tfrom.Set(time.Date(1970, time.January, 1, 0, 0, 0, 0, time.UTC)) |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
347 uBound = pgtype.Unbounded |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
348 } else { |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
349 const ( |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
350 fromKey = "Valid_from_date" |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
351 toKey = "Valid_to_date" |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
352 ) |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
353 fromTo := make(map[string]time.Time) |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
354 for _, kv := range bn.AdditionalData.KeyValuePair { |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
355 k := string(kv.Key) |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
356 if k == fromKey || k == toKey { |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
357 if t, err := time.Parse(time.RFC3339, kv.Value); err != nil { |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
358 return err |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
359 } else { |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
360 fromTo[k] = t |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
361 } |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
362 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
363 } |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
364 |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
365 if t, ok := fromTo[fromKey]; ok { |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
366 tfrom.Set(t) |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
367 } else { |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
368 feedback.Warn("Missing start date") |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
369 return nil |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
370 } |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
371 |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
372 if t, ok := fromTo[toKey]; ok { |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
373 tto.Set(t) |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
374 uBound = pgtype.Exclusive |
4102
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
375 feedback.Info("Valid from %s to %s", |
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
376 fromTo[fromKey].Format(common.TimeFormat), |
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
377 fromTo[toKey].Format(common.TimeFormat)) |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
378 } else { |
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
379 uBound = pgtype.Unbounded |
4102
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
380 feedback.Info("Valid from %s", |
3d2f02c16765
Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents:
4071
diff
changeset
|
381 fromTo[fromKey].Format(common.TimeFormat)) |
3679
0227670dedd5
Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents:
3666
diff
changeset
|
382 } |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
383 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
384 |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
385 validity := pgtype.Tstzrange{ |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
386 Lower: tfrom, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
387 Upper: tto, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
388 LowerType: pgtype.Inclusive, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
389 UpperType: uBound, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
390 Status: pgtype.Present, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
391 } |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
392 |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
393 rb, lb := splitRBLB(bn.Rb_lb) |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
394 |
3423
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
395 var revisitingTime *int |
3528
b9e331c1e616
Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents:
3423
diff
changeset
|
396 if bn.Revisiting_time != nil && |
b9e331c1e616
Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents:
3423
diff
changeset
|
397 len(strings.TrimSpace(*bn.Revisiting_time)) > 0 { |
3423
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
398 i, err := strconv.Atoi(*bn.Revisiting_time) |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
399 if err != nil { |
3530 | 400 feedback.Warn( |
401 "Cannot convert given revisiting time '%s' to number of months", | |
3423
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
402 *bn.Revisiting_time) |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
403 } else { |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
404 revisitingTime = &i |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
405 } |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
406 } |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
407 |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
408 var limiting, country string |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
409 |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
410 if bn.Limiting_factor != nil { |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
411 limiting = string(*bn.Limiting_factor) |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
412 } |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
413 |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
414 if bn.Responsible_country != nil { |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
415 country = string(*bn.Responsible_country) |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
416 } |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
417 |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
418 var materials []string |
2999
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
419 if bn.Riverbed != nil { |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
420 for _, material := range bn.Riverbed.Material { |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
421 if material != nil { |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
422 materials = append(materials, string(*material)) |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
423 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
424 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
425 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
426 |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
427 // Check if an bottleneck identical to the one we would insert already |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
428 // exists: |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
429 bns, err := findExactMatchingBNStmt.QueryContext(ctx, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
430 bn.Bottleneck_id, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
431 &validity, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
432 bn.Fk_g_fid, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
433 bn.OBJNAM, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
434 bn.NOBJNM, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
435 bn.From_ISRS, bn.To_ISRS, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
436 rb, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
437 lb, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
438 country, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
439 revisitingTime, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
440 limiting, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
441 bn.Date_Info, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
442 bn.Source, |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
443 ) |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
444 if err != nil { |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
445 return err |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
446 } |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
447 defer bns.Close() |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
448 // We could check if the materials are also matching -- but per |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
449 // specification the Date_Info would hvae to change on that kind of |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
450 // change anyway. So actualy we ar alreayd checking more in dpth than |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
451 // required. |
3698
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
452 if bns.Next() { |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
453 feedback.Info("unchanged") |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
454 return nil |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
455 } |
063a1883b5cb
Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents:
3681
diff
changeset
|
456 |
3815
aa7bede70b96
Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents:
3705
diff
changeset
|
457 // Additional Information, only of interest when we change something, so |
aa7bede70b96
Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents:
3705
diff
changeset
|
458 // it can be used for debugging if something goes wrong... |
aa7bede70b96
Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents:
3705
diff
changeset
|
459 feedback.Info("Range: from %s to %s", bn.From_ISRS, bn.To_ISRS) |
aa7bede70b96
Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents:
3705
diff
changeset
|
460 |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
461 tx, err := conn.BeginTx(ctx, nil) |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
462 if err != nil { |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
463 return err |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
464 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
465 defer tx.Rollback() |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
466 |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
467 // Check if the new bottleneck intersects with the validity of existing |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
468 // for the same bottleneck_id we consider this an update and mark the |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
469 // old data for deletion. |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
470 bns, err = tx.StmtContext(ctx, findIntersectingBNStmt).QueryContext( |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
471 ctx, bn.Bottleneck_id, &validity, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
472 ) |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
473 if err != nil { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
474 return err |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
475 } |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
476 defer bns.Close() |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
477 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
478 // Mark old intersecting bottleneck data for deletion. Don't worry about |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
479 // materials, they will be deleted via cascading. |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
480 var oldBnIds []int64 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
481 for bns.Next() { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
482 var oldID int64 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
483 err := bns.Scan(&oldID) |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
484 if err != nil { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
485 return err |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
486 } |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
487 oldBnIds = append(oldBnIds, oldID) |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
488 } |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
489 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
490 switch { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
491 case len(oldBnIds) == 1: |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
492 feedback.Info("Bottelneck '%s' "+ |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
493 "with intersecting validity already exists: "+ |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
494 "UPDATING", bn.Bottleneck_id) |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
495 case len(oldBnIds) > 1: |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
496 // This case is unexpected and should only happen when historic |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
497 // data in the bottleneck service was changed subsequently... |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
498 // We handle it gracefully anyway, but warn. |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
499 feedback.Warn("More than one Bottelneck '%s' "+ |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
500 "with intersecting validity already exists: "+ |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
501 "REPLACING all of them!", bn.Bottleneck_id) |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
502 } |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
503 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
504 for _, oldID := range oldBnIds { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
505 // It is possible, that two new bottlenecks intersect with the |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
506 // same old noe, therefor we have to handle duplicates in |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
507 // oldBnIds. |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
508 if !(*seenOldBnIds)[oldID] { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
509 if _, err := tx.StmtContext(ctx, trackStmt).ExecContext( |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
510 ctx, importID, "waterway.bottlenecks", oldID, true, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
511 ); err != nil { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
512 return err |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
513 } |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
514 (*seenOldBnIds)[oldID] = true |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
515 } |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
516 } |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
517 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
518 var bnIds []int64 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
519 // Add new BN data: |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
520 bns, err = tx.StmtContext(ctx, insertStmt).QueryContext(ctx, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
521 bn.Bottleneck_id, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
522 &validity, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
523 bn.Fk_g_fid, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
524 bn.OBJNAM, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
525 bn.NOBJNM, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
526 bn.From_ISRS, bn.To_ISRS, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
527 rb, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
528 lb, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
529 country, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
530 revisitingTime, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
531 limiting, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
532 bn.Date_Info, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
533 bn.Source, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
534 tolerance) |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
535 if err != nil { |
4071
5867dcf8e93c
Introduced a new ReadableError type for better readable error messages of pgx.PgErrors.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4063
diff
changeset
|
536 feedback.Warn(pgxutils.ReadableError{err}.Error()) |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
537 return nil |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
538 } |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
539 defer bns.Close() |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
540 for bns.Next() { |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
541 var nid int64 |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
542 if err := bns.Scan(&nid); err != nil { |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
543 return err |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
544 } |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
545 bnIds = append(bnIds, nid) |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
546 } |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
547 if err := bns.Err(); err != nil { |
4071
5867dcf8e93c
Introduced a new ReadableError type for better readable error messages of pgx.PgErrors.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4063
diff
changeset
|
548 feedback.Warn(pgxutils.ReadableError{err}.Error()) |
3681
c9e1848a516a
Handle violation of responsibility areas graceful for BN import.
Sascha Wilde <wilde@intevation.de>
parents:
3679
diff
changeset
|
549 return nil |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
550 } |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
551 if len(bnIds) == 0 { |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
552 feedback.Warn( |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
553 "No gauge matching '%s' or given time available", bn.Fk_g_fid) |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
554 return nil |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
555 } |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
556 |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
557 // Add new materials |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
558 var ( |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
559 pgBnIds pgtype.Int8Array |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
560 pgMaterials pgtype.VarcharArray |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
561 ) |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
562 pgBnIds.Set(bnIds) |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
563 pgMaterials.Set(materials) |
3648
0ec5c8ec1e44
Avoid empty validity time ranges
Tom Gottfried <tom@intevation.de>
parents:
3645
diff
changeset
|
564 |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
565 if materials != nil { |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
566 // Insert riverbed materials |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
567 if _, err := tx.StmtContext(ctx, insertMaterialStmt).ExecContext(ctx, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
568 &pgBnIds, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
569 &pgMaterials, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
570 ); err != nil { |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
571 feedback.Warn("Failed to insert riverbed materials") |
4071
5867dcf8e93c
Introduced a new ReadableError type for better readable error messages of pgx.PgErrors.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4063
diff
changeset
|
572 feedback.Warn(pgxutils.ReadableError{err}.Error()) |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
573 return nil |
2999
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
574 } |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
575 } |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
576 |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
577 // Only add new BN data to tracking for staging review. |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
578 for _, nid := range bnIds { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
579 if _, err := tx.StmtContext(ctx, trackStmt).ExecContext( |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
580 ctx, importID, "waterway.bottlenecks", nid, false, |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
581 ); err != nil { |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
582 return err |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
583 } |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
584 } |
3705
7006b92c0334
Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents:
3703
diff
changeset
|
585 |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
586 if err = tx.Commit(); err != nil { |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
587 return err |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
588 } |
4113
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
589 |
f39d20427e89
WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents:
4110
diff
changeset
|
590 *nids = append(*nids, bn.Bottleneck_id) |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
591 return nil |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
592 } |