Mercurial > gemma
annotate pkg/imports/bn.go @ 3666:db87f34805fb
Align bottleneck validity at gauges
Ensuring the validity of a bottleneck version is always contained
by the validity of the referenced gauge version allows to reliably
determine matching reference values of the gauge at a point in time.
Since this implies that a bottleneck version might be cut into more
than one time ranges, the concept of having only one non-erased
version is no longer applicable and replaced by using the 'current'
version of a bottleneck.
Fairway availability data are always kept with the 'current'
bottleneck version to have them at hand alltogether for analyses
over longer time ranges.
author | Tom Gottfried <tom@intevation.de> |
---|---|
date | Sat, 15 Jun 2019 14:36:50 +0200 |
parents | 29ef6d41e4af |
children | 0227670dedd5 |
rev | line source |
---|---|
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
7 // Copyright (C) 2018, 2019 by via donau |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
13 // * Tom Gottfried <tom.gottfried@intevation.de> |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 package imports |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 import ( |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 "context" |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "database/sql" |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "errors" |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
21 "regexp" |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
22 "strconv" |
3528
b9e331c1e616
Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents:
3423
diff
changeset
|
23 "strings" |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
24 "time" |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
26 "gemma.intevation.de/gemma/pkg/soap/ifbn" |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
27 "github.com/jackc/pgx/pgtype" |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
28 ) |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
29 |
1695
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
30 // Bottleneck is an import job to import |
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
31 // bottlenecks from an IFBN SOAP service. |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
32 type Bottleneck struct { |
1695
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
33 // URL is the URL of the SOAP service. |
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
34 URL string `json:"url"` |
2563
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
35 // Tolerance used for axis snapping |
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
36 Tolerance float64 `json:"tolerance"` |
1695
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
37 // Insecure indicates if HTTPS traffic |
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
38 // should validate certificates or not. |
be78b429ef6e
Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1676
diff
changeset
|
39 Insecure bool `json:"insecure"` |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
40 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
41 |
1696
ad5e1cddaa09
Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1695
diff
changeset
|
42 // BNJobKind is the import queue type identifier. |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
43 const BNJobKind JobKind = "bn" |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
44 |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
45 const ( |
1658
07f996b0e14a
Bottleneck import: Cleaned up source a bit (removed outdated TODOs).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1646
diff
changeset
|
46 insertBottleneckSQL = ` |
3566
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
47 WITH |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
48 bounds (b) AS (VALUES (isrs_fromText($6)), (isrs_fromText($7))), |
3566
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
49 r AS (SELECT isrsrange( |
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
50 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY), |
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
51 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r) |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
52 INSERT INTO waterway.bottlenecks ( |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
53 bottleneck_id, |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
54 validity, |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3224
diff
changeset
|
55 gauge_location, |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3224
diff
changeset
|
56 gauge_validity, |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
57 objnam, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
58 nobjnm, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
59 stretch, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
60 area, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
61 rb, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
62 lb, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
63 responsible_country, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
64 revisiting_time, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
65 limiting, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
66 date_info, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
67 source_organization |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
68 ) SELECT |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
69 $1, |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
70 validity * $2, -- intersections with gauge validity ranges |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
71 location, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
72 validity, |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
73 $4, |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
74 $5, |
3566
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
75 (SELECT r FROM r), |
1631
6ee06eb37fa9
Use new function to calculate bn area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1611
diff
changeset
|
76 ISRSrange_area( |
3566
4c585b5d4fe8
Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents:
3562
diff
changeset
|
77 ISRSrange_axis((SELECT r FROM r), |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
78 $15), |
2248
cce158db02b0
Input area as multipolygons to generate area from stretch
Tom Gottfried <tom@intevation.de>
parents:
2195
diff
changeset
|
79 (SELECT ST_Collect(CAST(area AS geometry)) |
1902
c4af342be999
Use waterway area as basis for bottleneck area generation.
Sascha Wilde <wilde@intevation.de>
parents:
1754
diff
changeset
|
80 FROM waterway.waterway_area)), |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
81 $8, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
82 $9, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
83 $10, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
84 $11, |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
85 $12, |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
86 $13, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
87 $14 |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
88 FROM waterway.gauges |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
89 WHERE location = isrs_fromText($3) AND validity && $2 |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
90 ON CONFLICT (bottleneck_id, validity) DO UPDATE SET |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
91 gauge_location = EXCLUDED.gauge_location, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
92 gauge_validity = EXCLUDED.gauge_validity, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
93 objnam = EXCLUDED.objnam, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
94 nobjnm = EXCLUDED.nobjnm, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
95 stretch = EXCLUDED.stretch, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
96 area = EXCLUDED.area, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
97 rb = EXCLUDED.rb, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
98 lb = EXCLUDED.lb, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
99 responsible_country = EXCLUDED.responsible_country, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
100 revisiting_time = EXCLUDED.revisiting_time, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
101 limiting = EXCLUDED.limiting, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
102 date_info = EXCLUDED.date_info, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
103 source_organization = EXCLUDED.source_organization |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
104 RETURNING id |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
105 ` |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
106 |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
107 // Alignment with gauge validity might have generated new entries |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
108 // for the same time range. Thus, remove the old ones |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
109 deleteObsoleteBNSQL = ` |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
110 DELETE FROM waterway.bottlenecks |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
111 WHERE bottleneck_id = $1 AND validity <@ $2 AND id <> ALL($3) |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
112 ` |
2999
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
113 |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
114 fixBNValiditySQL = ` |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
115 UPDATE waterway.bottlenecks SET |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
116 -- Set enddate of old entry to new startdate in case of overlap: |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
117 validity = validity - $2 |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
118 WHERE bottleneck_id = $1 |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
119 AND validity && $2 AND NOT validity <@ $2 |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
120 ` |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
121 |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
122 deleteBottleneckMaterialSQL = ` |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
123 WITH del AS ( |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
124 DELETE FROM waterway.bottlenecks_riverbed_materials |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
125 WHERE bottleneck_id = ANY($1) |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
126 AND riverbed <> ALL($2) |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
127 RETURNING riverbed) |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
128 SELECT DISTINCT riverbed FROM del |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
129 ` |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
130 |
2999
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
131 insertBottleneckMaterialSQL = ` |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
132 INSERT INTO waterway.bottlenecks_riverbed_materials ( |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
133 bottleneck_id, |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
134 riverbed |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
135 ) SELECT * |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
136 FROM unnest(CAST($1 AS int[])) AS bns, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
137 unnest(CAST($2 AS varchar[])) AS materials |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
138 ON CONFLICT (bottleneck_id, riverbed) DO NOTHING |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
139 ` |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
140 ) |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
141 |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
142 type bnJobCreator struct{} |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
143 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
144 func init() { |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
145 RegisterJobCreator(BNJobKind, bnJobCreator{}) |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
146 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
147 |
1754
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
148 func (bnJobCreator) Description() string { return "bottlenecks" } |
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
149 |
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
150 func (bnJobCreator) AutoAccept() bool { return false } |
1646
a0982c38eac0
Import queue: Implemented email notifications.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1631
diff
changeset
|
151 |
2187
7c83b5277c1c
Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
152 func (bnJobCreator) Create() Job { return new(Bottleneck) } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
153 |
3219
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3164
diff
changeset
|
154 func (bnJobCreator) Depends() [2][]string { |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3164
diff
changeset
|
155 return [2][]string{ |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3164
diff
changeset
|
156 {"bottlenecks", "bottlenecks_riverbed_materials"}, |
3224
bfde4f8dd323
Add missing table dependencies of imports with area generation
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
157 {"gauges", "distance_marks_virtual", "waterway_axis", "waterway_area"}, |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
158 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
159 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
160 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
161 const ( |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
162 bnStageDoneSQL = ` |
1533
8fc1a2298acb
Pseudo fixed unstaging of bottlenecks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1531
diff
changeset
|
163 UPDATE waterway.bottlenecks SET staging_done = true |
8fc1a2298acb
Pseudo fixed unstaging of bottlenecks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1531
diff
changeset
|
164 WHERE id IN ( |
1995
59055c8301df
Move import queue to its own database namespace
Tom Gottfried <tom@intevation.de>
parents:
1975
diff
changeset
|
165 SELECT key from import.track_imports |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
166 WHERE import_id = $1 AND |
1533
8fc1a2298acb
Pseudo fixed unstaging of bottlenecks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1531
diff
changeset
|
167 relation = 'waterway.bottlenecks'::regclass)` |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
168 ) |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
169 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
170 // StageDone moves the imported bottleneck out of the staging area. |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
171 func (bnJobCreator) StageDone( |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
172 ctx context.Context, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
173 tx *sql.Tx, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
174 id int64, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
175 ) error { |
1594
e3abd3b3ebc6
Fixed stage done SQL for bottleneck imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1580
diff
changeset
|
176 _, err := tx.ExecContext(ctx, bnStageDoneSQL, id) |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
177 return err |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
178 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
179 |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
180 // CleanUp of a bottleneck import is a NOP. |
1676
4407ecaa2192
Imports: Cosmetics.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1664
diff
changeset
|
181 func (*Bottleneck) CleanUp() error { return nil } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
182 |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
183 var rblbRe = regexp.MustCompile(`(..)_(..)`) |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
184 |
2765
41f5ac76d642
BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents:
2752
diff
changeset
|
185 func splitRBLB(s string) (*string, *string) { |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
186 m := rblbRe.FindStringSubmatch(s) |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
187 if len(m) == 0 { |
2765
41f5ac76d642
BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents:
2752
diff
changeset
|
188 return nil, nil |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
189 } |
2765
41f5ac76d642
BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents:
2752
diff
changeset
|
190 return &m[1], &m[2] |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
191 } |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
192 |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
193 // Do executes the actual bottleneck import. |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
194 func (bn *Bottleneck) Do( |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
195 ctx context.Context, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
196 importID int64, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
197 conn *sql.Conn, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
198 feedback Feedback, |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
199 ) (interface{}, error) { |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
200 |
2253
cfc523c70e90
Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2248
diff
changeset
|
201 fetch := func() ([]*ifbn.BottleNeckType, error) { |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
202 client := ifbn.NewIBottleneckService(bn.URL, bn.Insecure, nil) |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
203 |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
204 req := &ifbn.Export_bn_by_isrs{} |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
205 |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
206 resp, err := client.Export_bn_by_isrs(req) |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
207 if err != nil { |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
208 return nil, err |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
209 } |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
210 |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
211 if resp.Export_bn_by_isrsResult == nil { |
3562 | 212 return nil, errors.New( |
213 "The requested service returned no bottlenecks") | |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
214 } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
215 |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
216 return resp.Export_bn_by_isrsResult.BottleNeckType, nil |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
217 } |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
218 |
2563
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
219 return storeBottlenecks(ctx, fetch, importID, conn, feedback, bn.Tolerance) |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
220 } |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
221 |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
222 func storeBottlenecks( |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
223 ctx context.Context, |
2253
cfc523c70e90
Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2248
diff
changeset
|
224 fetch func() ([]*ifbn.BottleNeckType, error), |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
225 importID int64, |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
226 conn *sql.Conn, |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
227 feedback Feedback, |
2563
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
228 tolerance float64, |
2195
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
229 ) (interface{}, error) { |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
230 start := time.Now() |
3437ceee1529
Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
231 |
2253
cfc523c70e90
Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2248
diff
changeset
|
232 bns, err := fetch() |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
233 if err != nil { |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
234 return nil, err |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
235 } |
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
236 |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
237 feedback.Info("Found %d bottlenecks for import", len(bns)) |
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
238 |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
239 var insertStmt, deleteObsoleteBNStmt, fixValidityStmt, |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
240 deleteMaterialStmt, insertMaterialStmt, trackStmt *sql.Stmt |
1580
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
241 |
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
242 for _, x := range []struct { |
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
243 sql string |
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
244 stmt **sql.Stmt |
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
245 }{ |
1658
07f996b0e14a
Bottleneck import: Cleaned up source a bit (removed outdated TODOs).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1646
diff
changeset
|
246 {insertBottleneckSQL, &insertStmt}, |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
247 {deleteObsoleteBNSQL, &deleteObsoleteBNStmt}, |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
248 {fixBNValiditySQL, &fixValidityStmt}, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
249 {deleteBottleneckMaterialSQL, &deleteMaterialStmt}, |
2999
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
250 {insertBottleneckMaterialSQL, &insertMaterialStmt}, |
1580
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
251 {trackImportSQL, &trackStmt}, |
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
252 } { |
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
253 var err error |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
254 if *x.stmt, err = conn.PrepareContext(ctx, x.sql); err != nil { |
1580
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
255 return nil, err |
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
256 } |
c503d2fa9776
Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1572
diff
changeset
|
257 defer (*x.stmt).Close() |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
258 } |
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
259 |
1611
1835d1c9eb9b
Store bottleneck_id's instead of id's in bottleneck import summary.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1595
diff
changeset
|
260 var nids []string |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
261 |
2693 | 262 feedback.Info("Tolerance used to snap waterway axis: %g", tolerance) |
263 | |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
264 for _, bn := range bns { |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
265 if err := storeBottleneck( |
2563
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
266 ctx, importID, conn, feedback, bn, &nids, tolerance, |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
267 insertStmt, deleteObsoleteBNStmt, fixValidityStmt, |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
268 deleteMaterialStmt, insertMaterialStmt, trackStmt); err != nil { |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
269 return nil, err |
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
270 } |
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
271 } |
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
272 if len(nids) == 0 { |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
273 return nil, UnchangedError("No new bottlenecks inserted") |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
274 } |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
275 |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
276 feedback.Info("Storing %d bottlenecks took %s", len(nids), time.Since(start)) |
1906
32c56e6c089a
Stretch import: Added forgotten source file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1902
diff
changeset
|
277 feedback.Info("Import of bottlenecks was successful") |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
278 summary := struct { |
1611
1835d1c9eb9b
Store bottleneck_id's instead of id's in bottleneck import summary.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1595
diff
changeset
|
279 Bottlenecks []string `json:"bottlenecks"` |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
280 }{ |
1572
056a86b24be2
Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1569
diff
changeset
|
281 Bottlenecks: nids, |
1569
cf0ca4a9812a
Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents:
1533
diff
changeset
|
282 } |
1906
32c56e6c089a
Stretch import: Added forgotten source file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1902
diff
changeset
|
283 return &summary, nil |
1531
24445a618513
Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
284 } |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
285 |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
286 func storeBottleneck( |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
287 ctx context.Context, |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
288 importID int64, |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
289 conn *sql.Conn, |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
290 feedback Feedback, |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
291 bn *ifbn.BottleNeckType, |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
292 nids *[]string, |
2563
dc4fae4bdb8f
Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents:
2548
diff
changeset
|
293 tolerance float64, |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
294 insertStmt, deleteObsoleteBNStmt, fixValidityStmt, |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
295 deleteMaterialStmt, insertMaterialStmt, trackStmt *sql.Stmt, |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
296 ) error { |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
297 feedback.Info("Processing %s (%s)", bn.OBJNAM, bn.Bottleneck_id) |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
298 |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
299 if bn.AdditionalData == nil || bn.AdditionalData.KeyValuePair == nil { |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
300 feedback.Warn("Missing validity information") |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
301 return nil |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
302 } |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
303 const ( |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
304 fromKey = "Valid_from_date" |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
305 toKey = "Valid_to_date" |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
306 ) |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
307 fromTo := make(map[string]time.Time) |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
308 for _, kv := range bn.AdditionalData.KeyValuePair { |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
309 k := string(kv.Key) |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
310 if k == fromKey || k == toKey { |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
311 if t, err := time.Parse(time.RFC3339, kv.Value); err != nil { |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
312 return err |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
313 } else { |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
314 fromTo[k] = t |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
315 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
316 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
317 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
318 |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
319 var tfrom, tto pgtype.Timestamptz |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
320 if t, ok := fromTo[fromKey]; ok { |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
321 tfrom.Set(t) |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
322 } else { |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
323 feedback.Warn("Missing start date") |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
324 return nil |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
325 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
326 var uBound pgtype.BoundType |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
327 if t, ok := fromTo[toKey]; ok { |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
328 tto.Set(t) |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
329 uBound = pgtype.Exclusive |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
330 } else { |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
331 uBound = pgtype.Unbounded |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
332 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
333 validity := pgtype.Tstzrange{ |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
334 Lower: tfrom, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
335 Upper: tto, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
336 LowerType: pgtype.Inclusive, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
337 UpperType: uBound, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
338 Status: pgtype.Present, |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
339 } |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
340 |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
341 rb, lb := splitRBLB(bn.Rb_lb) |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
342 |
3423
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
343 var revisitingTime *int |
3528
b9e331c1e616
Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents:
3423
diff
changeset
|
344 if bn.Revisiting_time != nil && |
b9e331c1e616
Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents:
3423
diff
changeset
|
345 len(strings.TrimSpace(*bn.Revisiting_time)) > 0 { |
3423
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
346 i, err := strconv.Atoi(*bn.Revisiting_time) |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
347 if err != nil { |
3530 | 348 feedback.Warn( |
349 "Cannot convert given revisiting time '%s' to number of months", | |
3423
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
350 *bn.Revisiting_time) |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
351 } else { |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
352 revisitingTime = &i |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
353 } |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
354 } |
6592396f5061
Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents:
3302
diff
changeset
|
355 |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
356 var limiting, country string |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
357 |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
358 if bn.Limiting_factor != nil { |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
359 limiting = string(*bn.Limiting_factor) |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
360 } |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
361 |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
362 if bn.Responsible_country != nil { |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
363 country = string(*bn.Responsible_country) |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
364 } |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
365 |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
366 var materials []string |
2999
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
367 if bn.Riverbed != nil { |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
368 for _, material := range bn.Riverbed.Material { |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
369 if material != nil { |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
370 materials = append(materials, string(*material)) |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
371 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
372 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
373 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
374 |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
375 tx, err := conn.BeginTx(ctx, nil) |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
376 if err != nil { |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
377 return err |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
378 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
379 defer tx.Rollback() |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
380 |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
381 var bnIds []int64 |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
382 bns, err := tx.StmtContext(ctx, insertStmt).QueryContext(ctx, |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
383 bn.Bottleneck_id, |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
384 &validity, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
385 bn.Fk_g_fid, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
386 bn.OBJNAM, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
387 bn.NOBJNM, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
388 bn.From_ISRS, bn.To_ISRS, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
389 rb, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
390 lb, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
391 country, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
392 revisitingTime, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
393 limiting, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
394 bn.Date_Info, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
395 bn.Source, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
396 tolerance, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
397 ) |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
398 if err != nil { |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
399 feedback.Warn(handleError(err).Error()) |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
400 return nil |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
401 } |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
402 defer bns.Close() |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
403 for bns.Next() { |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
404 var nid int64 |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
405 if err := bns.Scan(&nid); err != nil { |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
406 return err |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
407 } |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
408 bnIds = append(bnIds, nid) |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
409 } |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
410 if err := bns.Err(); err != nil { |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
411 return err |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
412 } |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
413 if len(bnIds) == 0 { |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
414 feedback.Warn( |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
415 "No gauge matching '%s' or given time available", bn.Fk_g_fid) |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
416 return nil |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
417 } |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
418 |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
419 // Remove obsolete bottleneck version entries |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
420 var pgBnIds pgtype.Int8Array |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
421 pgBnIds.Set(bnIds) |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
422 if _, err = tx.StmtContext(ctx, deleteObsoleteBNStmt).ExecContext(ctx, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
423 bn.Bottleneck_id, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
424 &validity, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
425 &pgBnIds, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
426 ); err != nil { |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
427 feedback.Warn(handleError(err).Error()) |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
428 if err2 := tx.Rollback(); err2 != nil { |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
429 return err2 |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
430 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
431 return nil |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
432 } |
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
433 |
3648
0ec5c8ec1e44
Avoid empty validity time ranges
Tom Gottfried <tom@intevation.de>
parents:
3645
diff
changeset
|
434 // Set end of validity of old version to start of new version |
0ec5c8ec1e44
Avoid empty validity time ranges
Tom Gottfried <tom@intevation.de>
parents:
3645
diff
changeset
|
435 // in case of overlap |
0ec5c8ec1e44
Avoid empty validity time ranges
Tom Gottfried <tom@intevation.de>
parents:
3645
diff
changeset
|
436 if _, err = tx.StmtContext(ctx, fixValidityStmt).ExecContext(ctx, |
0ec5c8ec1e44
Avoid empty validity time ranges
Tom Gottfried <tom@intevation.de>
parents:
3645
diff
changeset
|
437 bn.Bottleneck_id, |
0ec5c8ec1e44
Avoid empty validity time ranges
Tom Gottfried <tom@intevation.de>
parents:
3645
diff
changeset
|
438 validity, |
0ec5c8ec1e44
Avoid empty validity time ranges
Tom Gottfried <tom@intevation.de>
parents:
3645
diff
changeset
|
439 ); err != nil { |
0ec5c8ec1e44
Avoid empty validity time ranges
Tom Gottfried <tom@intevation.de>
parents:
3645
diff
changeset
|
440 feedback.Warn(handleError(err).Error()) |
0ec5c8ec1e44
Avoid empty validity time ranges
Tom Gottfried <tom@intevation.de>
parents:
3645
diff
changeset
|
441 if err2 := tx.Rollback(); err2 != nil { |
0ec5c8ec1e44
Avoid empty validity time ranges
Tom Gottfried <tom@intevation.de>
parents:
3645
diff
changeset
|
442 return err2 |
0ec5c8ec1e44
Avoid empty validity time ranges
Tom Gottfried <tom@intevation.de>
parents:
3645
diff
changeset
|
443 } |
0ec5c8ec1e44
Avoid empty validity time ranges
Tom Gottfried <tom@intevation.de>
parents:
3645
diff
changeset
|
444 return nil |
0ec5c8ec1e44
Avoid empty validity time ranges
Tom Gottfried <tom@intevation.de>
parents:
3645
diff
changeset
|
445 } |
0ec5c8ec1e44
Avoid empty validity time ranges
Tom Gottfried <tom@intevation.de>
parents:
3645
diff
changeset
|
446 |
3645
02951a62e8c6
'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents:
3566
diff
changeset
|
447 if materials != nil { |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
448 // Remove obsolete riverbed materials |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
449 var pgMaterials pgtype.VarcharArray |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
450 pgMaterials.Set(materials) |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
451 mtls, err := tx.StmtContext(ctx, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
452 deleteMaterialStmt).QueryContext(ctx, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
453 &pgBnIds, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
454 &pgMaterials, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
455 ) |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
456 if err != nil { |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
457 return err |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
458 } |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
459 defer mtls.Close() |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
460 for mtls.Next() { |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
461 var delMat string |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
462 if err := mtls.Scan(&delMat); err != nil { |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
463 return err |
2999
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
464 } |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
465 feedback.Warn("Removed riverbed material %s", delMat) |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
466 } |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
467 if err := mtls.Err(); err != nil { |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
468 return err |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
469 } |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
470 |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
471 // Insert riverbed materials |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
472 if _, err := tx.StmtContext(ctx, insertMaterialStmt).ExecContext(ctx, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
473 &pgBnIds, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
474 &pgMaterials, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
475 ); err != nil { |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
476 feedback.Warn("Failed to insert riverbed materials") |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
477 feedback.Warn(handleError(err).Error()) |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
478 return nil |
2999
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
479 } |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
480 } |
b3c3c5b5b7c1
Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2765
diff
changeset
|
481 |
3666
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
482 for _, nid := range bnIds { |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
483 if _, err := tx.StmtContext(ctx, trackStmt).ExecContext( |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
484 ctx, importID, "waterway.bottlenecks", nid, |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
485 ); err != nil { |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
486 return err |
db87f34805fb
Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents:
3665
diff
changeset
|
487 } |
2532
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
488 } |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
489 if err = tx.Commit(); err != nil { |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
490 return err |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
491 } |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
492 *nids = append(*nids, bn.Bottleneck_id) |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
493 return nil |
452bc714bfd9
Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents:
2253
diff
changeset
|
494 } |