Mercurial > gemma

annotate pkg/imports/bn.go @ 4128:14706384a464 request_hist_bns

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Cleanup by Sascha L. Teichmann applied.
author Sascha Wilde <wilde@intevation.de>
date Thu, 01 Aug 2019 17:27:40 +0200
parents 0cf0008070db
children 980f12d3c766
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
7 // Copyright (C) 2018, 2019 by via donau
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
13 // * Tom Gottfried <tom.gottfried@intevation.de>
3705
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
14 // * Sascha Wilde <sascha.wilde@intevation.de>
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 package imports
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
17
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
18 import (
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
19 "context"
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
20 "database/sql"
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
21 "errors"
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
22 "regexp"
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
23 "strconv"
3528
b9e331c1e616 Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents: 3423
diff changeset
24 "strings"
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
25 "time"
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
26
4102
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
27 "gemma.intevation.de/gemma/pkg/common"
4062
6c760abcff0e Move handling of PostgreSQL errors to own package
Tom Gottfried <tom@intevation.de>
parents: 4058
diff changeset
28 "gemma.intevation.de/gemma/pkg/pgxutils"
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
29 "gemma.intevation.de/gemma/pkg/soap/ifbn"
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
30 "github.com/jackc/pgx/pgtype"
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
31 )
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
32
1695
be78b429ef6e Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1676
diff changeset
33 // Bottleneck is an import job to import
be78b429ef6e Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1676
diff changeset
34 // bottlenecks from an IFBN SOAP service.
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
35 type Bottleneck struct {
1695
be78b429ef6e Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1676
diff changeset
36 // URL is the URL of the SOAP service.
be78b429ef6e Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1676
diff changeset
37 URL string `json:"url"`
2563
dc4fae4bdb8f Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents: 2548
diff changeset
38 // Tolerance used for axis snapping
dc4fae4bdb8f Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents: 2548
diff changeset
39 Tolerance float64 `json:"tolerance"`
1695
be78b429ef6e Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1676
diff changeset
40 // Insecure indicates if HTTPS traffic
be78b429ef6e Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1676
diff changeset
41 // should validate certificates or not.
be78b429ef6e Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1676
diff changeset
42 Insecure bool `json:"insecure"`
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
43 }
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
44
1696
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1695
diff changeset
45 // BNJobKind is the import queue type identifier.
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
46 const BNJobKind JobKind = "bn"
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
47
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
48 const (
1658
07f996b0e14a Bottleneck import: Cleaned up source a bit (removed outdated TODOs).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1646
diff changeset
49 insertBottleneckSQL = `
3566
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents: 3562
diff changeset
50 WITH
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
51 bounds (b) AS (VALUES (isrs_fromText($6)), (isrs_fromText($7))),
3566
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents: 3562
diff changeset
52 r AS (SELECT isrsrange(
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents: 3562
diff changeset
53 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY),
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents: 3562
diff changeset
54 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r)
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
55 INSERT INTO waterway.bottlenecks (
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
56 bottleneck_id,
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
57 validity,
3302
ec6163c6687d 'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents: 3224
diff changeset
58 gauge_location,
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
59 objnam,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
60 nobjnm,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
61 stretch,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
62 area,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
63 rb,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
64 lb,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
65 responsible_country,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
66 revisiting_time,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
67 limiting,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
68 date_info,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
69 source_organization
4069
12f476e91c70 Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents: 3815
diff changeset
70 ) VALUES (
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
71 $1,
4069
12f476e91c70 Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents: 3815
diff changeset
72 $2::tstzrange,
12f476e91c70 Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents: 3815
diff changeset
73 isrs_fromText($3),
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
74 $4,
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
75 $5,
3566
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents: 3562
diff changeset
76 (SELECT r FROM r),
1631
6ee06eb37fa9 Use new function to calculate bn area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1611
diff changeset
77 ISRSrange_area(
3566
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents: 3562
diff changeset
78 ISRSrange_axis((SELECT r FROM r),
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
79 $15),
2248
cce158db02b0 Input area as multipolygons to generate area from stretch
Tom Gottfried <tom@intevation.de>
parents: 2195
diff changeset
80 (SELECT ST_Collect(CAST(area AS geometry))
1902
c4af342be999 Use waterway area as basis for bottleneck area generation.
Sascha Wilde <wilde@intevation.de>
parents: 1754
diff changeset
81 FROM waterway.waterway_area)),
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
82 $8,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
83 $9,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
84 $10,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
85 $11,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
86 $12,
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
87 $13,
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
88 $14
4069
12f476e91c70 Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents: 3815
diff changeset
89 )
3705
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
90 RETURNING id
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
91 `
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
92
3698
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
93 findExactMatchBottleneckSQL = `
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
94 WITH
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
95 bounds (b) AS (VALUES (isrs_fromText($6)), (isrs_fromText($7))),
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
96 r AS (SELECT isrsrange(
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
97 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY),
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
98 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r)
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
99 SELECT id FROM waterway.bottlenecks
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
100 WHERE (
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
101 bottleneck_id,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
102 validity,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
103 gauge_location,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
104 objnam,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
105 nobjnm,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
106 stretch,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
107 rb,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
108 lb,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
109 responsible_country,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
110 revisiting_time,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
111 limiting,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
112 date_info,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
113 source_organization,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
114 staging_done
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
115 ) = ( SELECT
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
116 $1,
4069
12f476e91c70 Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents: 3815
diff changeset
117 $2::tstzrange,
12f476e91c70 Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents: 3815
diff changeset
118 isrs_fromText($3),
3698
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
119 $4,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
120 $5,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
121 (SELECT r FROM r),
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
122 $8,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
123 $9,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
124 $10,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
125 $11::smallint,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
126 $12,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
127 $13::timestamptz,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
128 $14,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
129 true
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
130 )
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
131 `
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
132
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
133 findIntersectingBottleneckSQL = `
3705
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
134 SELECT id FROM waterway.bottlenecks
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
135 WHERE (bottleneck_id, staging_done) = ($1, true)
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
136 AND $2::tstzrange && validity
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
137 `
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
138
2999
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2765
diff changeset
139 insertBottleneckMaterialSQL = `
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2765
diff changeset
140 INSERT INTO waterway.bottlenecks_riverbed_materials (
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2765
diff changeset
141 bottleneck_id,
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2765
diff changeset
142 riverbed
3666
db87f34805fb Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents: 3665
diff changeset
143 ) SELECT *
db87f34805fb Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents: 3665
diff changeset
144 FROM unnest(CAST($1 AS int[])) AS bns,
db87f34805fb Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents: 3665
diff changeset
145 unnest(CAST($2 AS varchar[])) AS materials
db87f34805fb Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents: 3665
diff changeset
146 ON CONFLICT (bottleneck_id, riverbed) DO NOTHING
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
147 `
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
148
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
149 bnStageDoneDeleteSQL = `
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
150 DELETE FROM waterway.bottlenecks WHERE id IN (
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
151 SELECT key
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
152 FROM import.track_imports
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
153 WHERE import_id = $1
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
154 AND relation = 'waterway.bottlenecks'::regclass
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
155 AND deletion
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
156 )`
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
157
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
158 bnStageDoneSQL = `
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
159 UPDATE waterway.bottlenecks SET staging_done = true
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
160 WHERE id IN (
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
161 SELECT key
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
162 FROM import.track_imports
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
163 WHERE import_id = $1
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
164 AND relation = 'waterway.bottlenecks'::regclass
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
165 AND NOT deletion
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
166 )`
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
167 )
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
168
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
169 type bnJobCreator struct{}
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
170
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
171 func init() {
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
172 RegisterJobCreator(BNJobKind, bnJobCreator{})
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
173 }
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
174
1754
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1696
diff changeset
175 func (bnJobCreator) Description() string { return "bottlenecks" }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1696
diff changeset
176
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1696
diff changeset
177 func (bnJobCreator) AutoAccept() bool { return false }
1646
a0982c38eac0 Import queue: Implemented email notifications.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1631
diff changeset
178
2187
7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
179 func (bnJobCreator) Create() Job { return new(Bottleneck) }
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
180
3219
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3164
diff changeset
181 func (bnJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3164
diff changeset
182 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3164
diff changeset
183 {"bottlenecks", "bottlenecks_riverbed_materials"},
3224
bfde4f8dd323 Add missing table dependencies of imports with area generation
Tom Gottfried <tom@intevation.de>
parents: 3219
diff changeset
184 {"gauges", "distance_marks_virtual", "waterway_axis", "waterway_area"},
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
185 }
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
186 }
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
187
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
188 // StageDone moves the imported bottleneck out of the staging area.
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
189 func (bnJobCreator) StageDone(
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
190 ctx context.Context,
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
191 tx *sql.Tx,
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
192 id int64,
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
193 ) error {
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
194 _, err := tx.ExecContext(ctx, bnStageDoneDeleteSQL, id)
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
195 if err == nil {
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
196 _, err = tx.ExecContext(ctx, bnStageDoneSQL, id)
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
197 }
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
198 return err
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
199 }
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
200
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
201 // CleanUp of a bottleneck import is a NOP.
1676
4407ecaa2192 Imports: Cosmetics.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1664
diff changeset
202 func (*Bottleneck) CleanUp() error { return nil }
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
203
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
204 var rblbRe = regexp.MustCompile(`(..)_(..)`)
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
205
2765
41f5ac76d642 BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents: 2752
diff changeset
206 func splitRBLB(s string) (*string, *string) {
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
207 m := rblbRe.FindStringSubmatch(s)
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
208 if len(m) == 0 {
2765
41f5ac76d642 BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents: 2752
diff changeset
209 return nil, nil
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
210 }
2765
41f5ac76d642 BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents: 2752
diff changeset
211 return &m[1], &m[2]
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
212 }
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
213
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
214 // Do executes the actual bottleneck import.
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
215 func (bn *Bottleneck) Do(
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
216 ctx context.Context,
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
217 importID int64,
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
218 conn *sql.Conn,
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
219 feedback Feedback,
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
220 ) (interface{}, error) {
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
221
2253
cfc523c70e90 Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2248
diff changeset
222 fetch := func() ([]*ifbn.BottleNeckType, error) {
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
223 client := ifbn.NewIBottleneckService(bn.URL, bn.Insecure, nil)
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
224
4102
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
225 req := &ifbn.Export_bn_by_isrs{
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
226 Period: &ifbn.RequestedPeriod{Date_start: &time.Time{}},
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
227 }
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
228
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
229 resp, err := client.Export_bn_by_isrs(req)
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
230 if err != nil {
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
231 return nil, err
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
232 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
233
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
234 if resp.Export_bn_by_isrsResult == nil {
3562
c03224d33e78 Improve error message
Tom Gottfried <tom@intevation.de>
parents: 3533
diff changeset
235 return nil, errors.New(
c03224d33e78 Improve error message
Tom Gottfried <tom@intevation.de>
parents: 3533
diff changeset
236 "The requested service returned no bottlenecks")
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
237 }
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
238
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
239 return resp.Export_bn_by_isrsResult.BottleNeckType, nil
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
240 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
241
2563
dc4fae4bdb8f Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents: 2548
diff changeset
242 return storeBottlenecks(ctx, fetch, importID, conn, feedback, bn.Tolerance)
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
243 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
244
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
245 type bnStmts struct {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
246 insert *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
247 findExactMatch *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
248 findIntersecting *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
249 insertMaterial *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
250 track *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
251 }
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
252
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
253 func (bs *bnStmts) close() {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
254 for _, s := range []**sql.Stmt{
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
255 &bs.insert,
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
256 &bs.findExactMatch,
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
257 &bs.findIntersecting,
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
258 &bs.insertMaterial,
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
259 &bs.track,
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
260 } {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
261 if *s != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
262 (*s).Close()
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
263 *s = nil
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
264 }
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
265 }
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
266 }
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
267
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
268 func (bs *bnStmts) prepare(ctx context.Context, conn *sql.Conn) error {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
269 for _, x := range []struct {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
270 sql string
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
271 stmt **sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
272 }{
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
273 {insertBottleneckSQL, &bs.insert},
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
274 {findExactMatchBottleneckSQL, &bs.findExactMatch},
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
275 {findIntersectingBottleneckSQL, &bs.findIntersecting},
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
276 {insertBottleneckMaterialSQL, &bs.insertMaterial},
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
277 {trackImportDeletionSQL, &bs.track},
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
278 } {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
279 var err error
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
280 if *x.stmt, err = conn.PrepareContext(ctx, x.sql); err != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
281 return err
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
282 }
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
283 }
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
284 return nil
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
285 }
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
286
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
287 func storeBottlenecks(
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
288 ctx context.Context,
2253
cfc523c70e90 Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2248
diff changeset
289 fetch func() ([]*ifbn.BottleNeckType, error),
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
290 importID int64,
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
291 conn *sql.Conn,
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
292 feedback Feedback,
2563
dc4fae4bdb8f Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents: 2548
diff changeset
293 tolerance float64,
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
294 ) (interface{}, error) {
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
295 start := time.Now()
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
296
2253
cfc523c70e90 Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2248
diff changeset
297 bns, err := fetch()
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
298 if err != nil {
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
299 return nil, err
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
300 }
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
301
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
302 feedback.Info("Found %d bottlenecks for import", len(bns))
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
303
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
304 var bs bnStmts
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
305 defer bs.close()
1580
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1572
diff changeset
306
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
307 if err := bs.prepare(ctx, conn); err != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
308 return nil, err
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
309 }
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
310
1611
1835d1c9eb9b Store bottleneck_id's instead of id's in bottleneck import summary.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1595
diff changeset
311 var nids []string
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
312 seenOldBnIds := make(map[int64]bool)
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
313
2693
02d012272dd4 Log snapping tolerance
Tom Gottfried <tom@intevation.de>
parents: 2563
diff changeset
314 feedback.Info("Tolerance used to snap waterway axis: %g", tolerance)
02d012272dd4 Log snapping tolerance
Tom Gottfried <tom@intevation.de>
parents: 2563
diff changeset
315
4119
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
316 tx, err := conn.BeginTx(ctx, nil)
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
317 if err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
318 return nil, err
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
319 }
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
320 defer tx.Rollback()
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
321
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
322 for _, bn := range bns {
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
323 if err := storeBottleneck(
4119
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
324 tx, ctx, importID, feedback, bn,
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
325 &nids, seenOldBnIds, tolerance,
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
326 &bs,
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
327 ); err != nil {
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
328 return nil, err
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
329 }
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
330 }
4119
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
331 if err = tx.Commit(); err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
332 return nil, err
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
333 }
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
334
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
335 if len(nids) == 0 {
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
336 return nil, UnchangedError("No new bottlenecks inserted")
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
337 }
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
338
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
339 feedback.Info("Storing %d bottlenecks took %s", len(nids), time.Since(start))
1906
32c56e6c089a Stretch import: Added forgotten source file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1902
diff changeset
340 feedback.Info("Import of bottlenecks was successful")
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
341 summary := struct {
1611
1835d1c9eb9b Store bottleneck_id's instead of id's in bottleneck import summary.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1595
diff changeset
342 Bottlenecks []string `json:"bottlenecks"`
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
343 }{
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
344 Bottlenecks: nids,
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
345 }
1906
32c56e6c089a Stretch import: Added forgotten source file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1902
diff changeset
346 return &summary, nil
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
347 }
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
348
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
349 func storeBottleneck(
4119
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
350 tx *sql.Tx,
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
351 ctx context.Context,
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
352 importID int64,
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
353 feedback Feedback,
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
354 bn *ifbn.BottleNeckType,
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
355 nids *[]string,
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
356 seenOldBnIds map[int64]bool,
2563
dc4fae4bdb8f Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents: 2548
diff changeset
357 tolerance float64,
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
358 bs *bnStmts,
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
359 ) error {
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
360 feedback.Info("Processing %s (%s)", bn.OBJNAM, bn.Bottleneck_id)
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
361
3679
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
362 var tfrom, tto pgtype.Timestamptz
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
363 var uBound pgtype.BoundType
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
364
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
365 if bn.AdditionalData == nil || bn.AdditionalData.KeyValuePair == nil {
3679
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
366 // This is a workaround for the fact that most BN data does not
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
367 // contain the optional validity information.
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
368 //
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
369 // The current solution makes every change in the data an
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
370 // update, which might be not the best solution. Alternative
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
371 // approaches could include usingthe Date Info of the data as
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
372 // start date, and maybe even inspecting the details of changed
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
373 // data for hints wether an update or historisation with a new
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
374 // version is advisable.
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
375 //
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
376 // Never the less, the current solution "just works" for the
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
377 // time being and reflects the upstream systems... -- sw
3679
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
378 feedback.Warn("No validity information, assuming infinite validity.")
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
379 tfrom.Set(time.Date(1970, time.January, 1, 0, 0, 0, 0, time.UTC))
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
380 uBound = pgtype.Unbounded
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
381 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
382 const (
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
383 fromKey = "Valid_from_date"
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
384 toKey = "Valid_to_date"
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
385 )
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
386 fromTo := make(map[string]time.Time)
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
387 for _, kv := range bn.AdditionalData.KeyValuePair {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
388 k := string(kv.Key)
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
389 if k == fromKey || k == toKey {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
390 if t, err := time.Parse(time.RFC3339, kv.Value); err != nil {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
391 return err
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
392 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
393 fromTo[k] = t
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
394 }
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
395 }
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
396 }
3679
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
397
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
398 if t, ok := fromTo[fromKey]; ok {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
399 tfrom.Set(t)
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
400 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
401 feedback.Warn("Missing start date")
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
402 return nil
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
403 }
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
404
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
405 if t, ok := fromTo[toKey]; ok {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
406 tto.Set(t)
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
407 uBound = pgtype.Exclusive
4102
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
408 feedback.Info("Valid from %s to %s",
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
409 fromTo[fromKey].Format(common.TimeFormat),
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
410 fromTo[toKey].Format(common.TimeFormat))
3679
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
411 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
412 uBound = pgtype.Unbounded
4102
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
413 feedback.Info("Valid from %s",
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
414 fromTo[fromKey].Format(common.TimeFormat))
3679
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
415 }
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
416 }
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
417
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
418 validity := pgtype.Tstzrange{
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
419 Lower: tfrom,
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
420 Upper: tto,
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
421 LowerType: pgtype.Inclusive,
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
422 UpperType: uBound,
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
423 Status: pgtype.Present,
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
424 }
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
425
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
426 rb, lb := splitRBLB(bn.Rb_lb)
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
427
3423
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
428 var revisitingTime *int
3528
b9e331c1e616 Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents: 3423
diff changeset
429 if bn.Revisiting_time != nil &&
b9e331c1e616 Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents: 3423
diff changeset
430 len(strings.TrimSpace(*bn.Revisiting_time)) > 0 {
3423
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
431 i, err := strconv.Atoi(*bn.Revisiting_time)
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
432 if err != nil {
3530
e98ea184538e Improve warning message
Tom Gottfried <tom@intevation.de>
parents: 3528
diff changeset
433 feedback.Warn(
e98ea184538e Improve warning message
Tom Gottfried <tom@intevation.de>
parents: 3528
diff changeset
434 "Cannot convert given revisiting time '%s' to number of months",
3423
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
435 *bn.Revisiting_time)
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
436 } else {
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
437 revisitingTime = &i
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
438 }
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
439 }
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
440
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
441 var limiting, country string
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
442
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
443 if bn.Limiting_factor != nil {
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
444 limiting = string(*bn.Limiting_factor)
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
445 }
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
446
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
447 if bn.Responsible_country != nil {
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
448 country = string(*bn.Responsible_country)
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
449 }
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
450
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
451 var materials []string
2999
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2765
diff changeset
452 if bn.Riverbed != nil {
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2765
diff changeset
453 for _, material := range bn.Riverbed.Material {
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2765
diff changeset
454 if material != nil {
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
455 materials = append(materials, string(*material))
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
456 }
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
457 }
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
458 }
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
459
3698
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
460 // Check if an bottleneck identical to the one we would insert already
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
461 // exists:
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
462 var old int64
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
463 err := tx.StmtContext(ctx, bs.findExactMatch).QueryRowContext(
4119
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
464 ctx,
3698
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
465 bn.Bottleneck_id,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
466 &validity,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
467 bn.Fk_g_fid,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
468 bn.OBJNAM,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
469 bn.NOBJNM,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
470 bn.From_ISRS, bn.To_ISRS,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
471 rb,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
472 lb,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
473 country,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
474 revisitingTime,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
475 limiting,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
476 bn.Date_Info,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
477 bn.Source,
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
478 ).Scan(&old)
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
479 switch {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
480 case err == sql.ErrNoRows:
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
481 // We dont have a matching old.
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
482 case err != nil:
3698
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
483 return err
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
484 default:
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
485 // We could check if the materials are also matching -- but per
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
486 // specification the Date_Info would hvae to change on that kind of
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
487 // change anyway. So actualy we ar alreayd checking more in dpth than
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
488 // required.
3698
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
489 feedback.Info("unchanged")
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
490 return nil
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
491 }
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
492
3815
aa7bede70b96 Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents: 3705
diff changeset
493 // Additional Information, only of interest when we change something, so
aa7bede70b96 Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents: 3705
diff changeset
494 // it can be used for debugging if something goes wrong...
aa7bede70b96 Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents: 3705
diff changeset
495 feedback.Info("Range: from %s to %s", bn.From_ISRS, bn.To_ISRS)
aa7bede70b96 Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents: 3705
diff changeset
496
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
497 // Check if the new bottleneck intersects with the validity of existing
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
498 // for the same bottleneck_id we consider this an update and mark the
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
499 // old data for deletion.
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
500 bns, err := tx.StmtContext(ctx, bs.findIntersecting).QueryContext(
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
501 ctx, bn.Bottleneck_id, &validity,
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
502 )
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
503 if err != nil {
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
504 return err
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
505 }
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
506 defer bns.Close()
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
507
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
508 // Mark old intersecting bottleneck data for deletion. Don't worry about
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
509 // materials, they will be deleted via cascading.
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
510 var oldBnIds []int64
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
511 for bns.Next() {
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
512 var oldID int64
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
513 err := bns.Scan(&oldID)
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
514 if err != nil {
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
515 return err
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
516 }
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
517 oldBnIds = append(oldBnIds, oldID)
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
518 }
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
519
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
520 if err := bns.Err(); err != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
521 return err
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
522 }
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
523
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
524 switch {
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
525 case len(oldBnIds) == 1:
3705
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
526 feedback.Info("Bottelneck '%s' "+
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
527 "with intersecting validity already exists: "+
3705
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
528 "UPDATING", bn.Bottleneck_id)
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
529 case len(oldBnIds) > 1:
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
530 // This case is unexpected and should only happen when historic
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
531 // data in the bottleneck service was changed subsequently...
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
532 // We handle it gracefully anyway, but warn.
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
533 feedback.Warn("More than one Bottelneck '%s' "+
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
534 "with intersecting validity already exists: "+
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
535 "REPLACING all of them!", bn.Bottleneck_id)
3705
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
536 }
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
537
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
538 for _, oldID := range oldBnIds {
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
539 // It is possible, that two new bottlenecks intersect with the
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
540 // same old noe, therefor we have to handle duplicates in
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
541 // oldBnIds.
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
542 if !seenOldBnIds[oldID] {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
543 if _, err := tx.StmtContext(ctx, bs.track).ExecContext(
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
544 ctx, importID, "waterway.bottlenecks", oldID, true,
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
545 ); err != nil {
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
546 return err
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
547 }
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
548 seenOldBnIds[oldID] = true
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
549 }
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
550 }
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
551
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
552 var bnIds []int64
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
553 // Add new BN data:
4119
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
554 savepoint := Savepoint(ctx, tx, "insert_bottlenck")
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
555
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
556 err = savepoint(func() error {
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
557 bns, err := tx.StmtContext(ctx, bs.insert).QueryContext(ctx,
4119
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
558 bn.Bottleneck_id,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
559 &validity,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
560 bn.Fk_g_fid,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
561 bn.OBJNAM,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
562 bn.NOBJNM,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
563 bn.From_ISRS, bn.To_ISRS,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
564 rb,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
565 lb,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
566 country,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
567 revisitingTime,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
568 limiting,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
569 bn.Date_Info,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
570 bn.Source,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
571 tolerance)
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
572 if err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
573 return err
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
574 }
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
575 defer bns.Close()
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
576 for bns.Next() {
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
577 var nid int64
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
578 if err := bns.Scan(&nid); err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
579 return err
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
580 }
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
581 bnIds = append(bnIds, nid)
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
582 }
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
583 if err := bns.Err(); err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
584 return err
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
585 }
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
586
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
587 // Add new materials
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
588 if len(bnIds) > 0 && materials != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
589 var (
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
590 pgBnIds pgtype.Int8Array
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
591 pgMaterials pgtype.VarcharArray
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
592 )
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
593 pgBnIds.Set(bnIds)
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
594 pgMaterials.Set(materials)
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
595
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
596 // Insert riverbed materials
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
597 if _, err := tx.StmtContext(ctx, bs.insertMaterial).ExecContext(ctx,
4119
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
598
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
599 &pgBnIds,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
600 &pgMaterials,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
601 ); err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
602 return err
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
603 }
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
604 }
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
605 return nil
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
606 })
3666
db87f34805fb Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents: 3665
diff changeset
607 if err != nil {
4071
5867dcf8e93c Introduced a new ReadableError type for better readable error messages of pgx.PgErrors.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4063
diff changeset
608 feedback.Warn(pgxutils.ReadableError{err}.Error())
3666
db87f34805fb Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents: 3665
diff changeset
609 return nil
db87f34805fb Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents: 3665
diff changeset
610 }
2999
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2765
diff changeset
611
3705
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
612 // Only add new BN data to tracking for staging review.
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
613 for _, nid := range bnIds {
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
614 if _, err := tx.StmtContext(ctx, bs.track).ExecContext(
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
615 ctx, importID, "waterway.bottlenecks", nid, false,
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
616 ); err != nil {
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
617 return err
3666
db87f34805fb Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents: 3665
diff changeset
618 }
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
619 }
3705
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
620
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
621 *nids = append(*nids, bn.Bottleneck_id)
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
622 return nil
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
623 }