Mercurial > gemma

annotate pkg/imports/bn.go @ 4130:980f12d3c766 request_hist_bns

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bottleneck import: Fixed detection of identical bottlenecks.
author Sascha Wilde <wilde@intevation.de>
date Thu, 01 Aug 2019 18:23:32 +0200
parents 14706384a464
children eb08fbe33074
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
7 // Copyright (C) 2018, 2019 by via donau
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
13 // * Tom Gottfried <tom.gottfried@intevation.de>
3705
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
14 // * Sascha Wilde <sascha.wilde@intevation.de>
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 package imports
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
17
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
18 import (
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
19 "context"
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
20 "database/sql"
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
21 "errors"
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
22 "regexp"
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
23 "strconv"
3528
b9e331c1e616 Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents: 3423
diff changeset
24 "strings"
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
25 "time"
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
26
4102
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
27 "gemma.intevation.de/gemma/pkg/common"
4062
6c760abcff0e Move handling of PostgreSQL errors to own package
Tom Gottfried <tom@intevation.de>
parents: 4058
diff changeset
28 "gemma.intevation.de/gemma/pkg/pgxutils"
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
29 "gemma.intevation.de/gemma/pkg/soap/ifbn"
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
30 "github.com/jackc/pgx/pgtype"
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
31 )
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
32
1695
be78b429ef6e Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1676
diff changeset
33 // Bottleneck is an import job to import
be78b429ef6e Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1676
diff changeset
34 // bottlenecks from an IFBN SOAP service.
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
35 type Bottleneck struct {
1695
be78b429ef6e Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1676
diff changeset
36 // URL is the URL of the SOAP service.
be78b429ef6e Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1676
diff changeset
37 URL string `json:"url"`
2563
dc4fae4bdb8f Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents: 2548
diff changeset
38 // Tolerance used for axis snapping
dc4fae4bdb8f Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents: 2548
diff changeset
39 Tolerance float64 `json:"tolerance"`
1695
be78b429ef6e Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1676
diff changeset
40 // Insecure indicates if HTTPS traffic
be78b429ef6e Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1676
diff changeset
41 // should validate certificates or not.
be78b429ef6e Bottleneck import: Resolved remaining golint issues.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1676
diff changeset
42 Insecure bool `json:"insecure"`
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
43 }
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
44
1696
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1695
diff changeset
45 // BNJobKind is the import queue type identifier.
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
46 const BNJobKind JobKind = "bn"
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
47
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
48 const (
1658
07f996b0e14a Bottleneck import: Cleaned up source a bit (removed outdated TODOs).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1646
diff changeset
49 insertBottleneckSQL = `
3566
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents: 3562
diff changeset
50 WITH
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
51 bounds (b) AS (VALUES (isrs_fromText($6)), (isrs_fromText($7))),
3566
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents: 3562
diff changeset
52 r AS (SELECT isrsrange(
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents: 3562
diff changeset
53 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY),
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents: 3562
diff changeset
54 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r)
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
55 INSERT INTO waterway.bottlenecks (
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
56 bottleneck_id,
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
57 validity,
3302
ec6163c6687d 'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents: 3224
diff changeset
58 gauge_location,
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
59 objnam,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
60 nobjnm,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
61 stretch,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
62 area,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
63 rb,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
64 lb,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
65 responsible_country,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
66 revisiting_time,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
67 limiting,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
68 date_info,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
69 source_organization
4069
12f476e91c70 Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents: 3815
diff changeset
70 ) VALUES (
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
71 $1,
4069
12f476e91c70 Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents: 3815
diff changeset
72 $2::tstzrange,
12f476e91c70 Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents: 3815
diff changeset
73 isrs_fromText($3),
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
74 $4,
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
75 $5,
3566
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents: 3562
diff changeset
76 (SELECT r FROM r),
1631
6ee06eb37fa9 Use new function to calculate bn area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1611
diff changeset
77 ISRSrange_area(
3566
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes
Tom Gottfried <tom@intevation.de>
parents: 3562
diff changeset
78 ISRSrange_axis((SELECT r FROM r),
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
79 $15),
2248
cce158db02b0 Input area as multipolygons to generate area from stretch
Tom Gottfried <tom@intevation.de>
parents: 2195
diff changeset
80 (SELECT ST_Collect(CAST(area AS geometry))
1902
c4af342be999 Use waterway area as basis for bottleneck area generation.
Sascha Wilde <wilde@intevation.de>
parents: 1754
diff changeset
81 FROM waterway.waterway_area)),
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
82 $8,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
83 $9,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
84 $10,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
85 $11,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
86 $12,
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
87 $13,
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
88 $14
4069
12f476e91c70 Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents: 3815
diff changeset
89 )
3705
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
90 RETURNING id
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
91 `
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
92
4130
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents: 4128
diff changeset
93 // We only check for NOT NULL values, for correct compairison with
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents: 4128
diff changeset
94 // values, which might be null (and then muyst not be compairt with `='
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents: 4128
diff changeset
95 // but with `IS NULL' is comlicated and that we are checking more than
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents: 4128
diff changeset
96 // only (bottleneck_id, validity, date_info) is luxury already.
3698
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
97 findExactMatchBottleneckSQL = `
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
98 WITH
4130
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents: 4128
diff changeset
99 bounds (b) AS (VALUES (isrs_fromText($4)), (isrs_fromText($5))),
3698
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
100 r AS (SELECT isrsrange(
4130
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents: 4128
diff changeset
101 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY),
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents: 4128
diff changeset
102 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r)
3698
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
103 SELECT id FROM waterway.bottlenecks
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
104 WHERE (
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
105 bottleneck_id,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
106 validity,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
107 gauge_location,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
108 stretch,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
109 responsible_country,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
110 limiting,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
111 date_info,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
112 source_organization,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
113 staging_done
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
114 ) = ( SELECT
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
115 $1,
4069
12f476e91c70 Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents: 3815
diff changeset
116 $2::tstzrange,
12f476e91c70 Adapted bn-import to new historization model.
Sascha Wilde <wilde@intevation.de>
parents: 3815
diff changeset
117 isrs_fromText($3),
3698
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
118 (SELECT r FROM r),
4130
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents: 4128
diff changeset
119 $6,
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents: 4128
diff changeset
120 $7,
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks.
Sascha Wilde <wilde@intevation.de>
parents: 4128
diff changeset
121 $8::timestamptz,
3698
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
122 $9,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
123 true
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
124 )
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
125 `
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
126
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
127 findIntersectingBottleneckSQL = `
3705
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
128 SELECT id FROM waterway.bottlenecks
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
129 WHERE (bottleneck_id, staging_done) = ($1, true)
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
130 AND $2::tstzrange && validity
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
131 `
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
132
2999
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2765
diff changeset
133 insertBottleneckMaterialSQL = `
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2765
diff changeset
134 INSERT INTO waterway.bottlenecks_riverbed_materials (
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2765
diff changeset
135 bottleneck_id,
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2765
diff changeset
136 riverbed
3666
db87f34805fb Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents: 3665
diff changeset
137 ) SELECT *
db87f34805fb Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents: 3665
diff changeset
138 FROM unnest(CAST($1 AS int[])) AS bns,
db87f34805fb Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents: 3665
diff changeset
139 unnest(CAST($2 AS varchar[])) AS materials
db87f34805fb Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents: 3665
diff changeset
140 ON CONFLICT (bottleneck_id, riverbed) DO NOTHING
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
141 `
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
142
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
143 bnStageDoneDeleteSQL = `
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
144 DELETE FROM waterway.bottlenecks WHERE id IN (
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
145 SELECT key
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
146 FROM import.track_imports
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
147 WHERE import_id = $1
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
148 AND relation = 'waterway.bottlenecks'::regclass
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
149 AND deletion
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
150 )`
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
151
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
152 bnStageDoneSQL = `
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
153 UPDATE waterway.bottlenecks SET staging_done = true
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
154 WHERE id IN (
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
155 SELECT key
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
156 FROM import.track_imports
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
157 WHERE import_id = $1
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
158 AND relation = 'waterway.bottlenecks'::regclass
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
159 AND NOT deletion
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
160 )`
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
161 )
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
162
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
163 type bnJobCreator struct{}
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
164
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
165 func init() {
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
166 RegisterJobCreator(BNJobKind, bnJobCreator{})
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
167 }
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
168
1754
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1696
diff changeset
169 func (bnJobCreator) Description() string { return "bottlenecks" }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1696
diff changeset
170
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1696
diff changeset
171 func (bnJobCreator) AutoAccept() bool { return false }
1646
a0982c38eac0 Import queue: Implemented email notifications.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1631
diff changeset
172
2187
7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
173 func (bnJobCreator) Create() Job { return new(Bottleneck) }
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
174
3219
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3164
diff changeset
175 func (bnJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3164
diff changeset
176 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3164
diff changeset
177 {"bottlenecks", "bottlenecks_riverbed_materials"},
3224
bfde4f8dd323 Add missing table dependencies of imports with area generation
Tom Gottfried <tom@intevation.de>
parents: 3219
diff changeset
178 {"gauges", "distance_marks_virtual", "waterway_axis", "waterway_area"},
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
179 }
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
180 }
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
181
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
182 // StageDone moves the imported bottleneck out of the staging area.
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
183 func (bnJobCreator) StageDone(
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
184 ctx context.Context,
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
185 tx *sql.Tx,
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
186 id int64,
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
187 ) error {
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
188 _, err := tx.ExecContext(ctx, bnStageDoneDeleteSQL, id)
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
189 if err == nil {
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
190 _, err = tx.ExecContext(ctx, bnStageDoneSQL, id)
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
191 }
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
192 return err
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
193 }
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
194
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
195 // CleanUp of a bottleneck import is a NOP.
1676
4407ecaa2192 Imports: Cosmetics.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1664
diff changeset
196 func (*Bottleneck) CleanUp() error { return nil }
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
197
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
198 var rblbRe = regexp.MustCompile(`(..)_(..)`)
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
199
2765
41f5ac76d642 BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents: 2752
diff changeset
200 func splitRBLB(s string) (*string, *string) {
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
201 m := rblbRe.FindStringSubmatch(s)
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
202 if len(m) == 0 {
2765
41f5ac76d642 BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents: 2752
diff changeset
203 return nil, nil
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
204 }
2765
41f5ac76d642 BT import: allow for rb and lb to be null.
Sascha Wilde <wilde@intevation.de>
parents: 2752
diff changeset
205 return &m[1], &m[2]
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
206 }
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
207
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
208 // Do executes the actual bottleneck import.
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
209 func (bn *Bottleneck) Do(
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
210 ctx context.Context,
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
211 importID int64,
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
212 conn *sql.Conn,
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
213 feedback Feedback,
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
214 ) (interface{}, error) {
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
215
2253
cfc523c70e90 Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2248
diff changeset
216 fetch := func() ([]*ifbn.BottleNeckType, error) {
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
217 client := ifbn.NewIBottleneckService(bn.URL, bn.Insecure, nil)
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
218
4102
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
219 req := &ifbn.Export_bn_by_isrs{
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
220 Period: &ifbn.RequestedPeriod{Date_start: &time.Time{}},
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
221 }
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
222
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
223 resp, err := client.Export_bn_by_isrs(req)
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
224 if err != nil {
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
225 return nil, err
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
226 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
227
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
228 if resp.Export_bn_by_isrsResult == nil {
3562
c03224d33e78 Improve error message
Tom Gottfried <tom@intevation.de>
parents: 3533
diff changeset
229 return nil, errors.New(
c03224d33e78 Improve error message
Tom Gottfried <tom@intevation.de>
parents: 3533
diff changeset
230 "The requested service returned no bottlenecks")
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
231 }
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
232
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
233 return resp.Export_bn_by_isrsResult.BottleNeckType, nil
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
234 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
235
2563
dc4fae4bdb8f Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents: 2548
diff changeset
236 return storeBottlenecks(ctx, fetch, importID, conn, feedback, bn.Tolerance)
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
237 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
238
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
239 type bnStmts struct {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
240 insert *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
241 findExactMatch *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
242 findIntersecting *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
243 insertMaterial *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
244 track *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
245 }
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
246
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
247 func (bs *bnStmts) close() {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
248 for _, s := range []**sql.Stmt{
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
249 &bs.insert,
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
250 &bs.findExactMatch,
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
251 &bs.findIntersecting,
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
252 &bs.insertMaterial,
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
253 &bs.track,
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
254 } {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
255 if *s != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
256 (*s).Close()
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
257 *s = nil
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
258 }
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
259 }
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
260 }
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
261
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
262 func (bs *bnStmts) prepare(ctx context.Context, conn *sql.Conn) error {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
263 for _, x := range []struct {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
264 sql string
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
265 stmt **sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
266 }{
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
267 {insertBottleneckSQL, &bs.insert},
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
268 {findExactMatchBottleneckSQL, &bs.findExactMatch},
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
269 {findIntersectingBottleneckSQL, &bs.findIntersecting},
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
270 {insertBottleneckMaterialSQL, &bs.insertMaterial},
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
271 {trackImportDeletionSQL, &bs.track},
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
272 } {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
273 var err error
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
274 if *x.stmt, err = conn.PrepareContext(ctx, x.sql); err != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
275 return err
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
276 }
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
277 }
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
278 return nil
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
279 }
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
280
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
281 func storeBottlenecks(
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
282 ctx context.Context,
2253
cfc523c70e90 Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2248
diff changeset
283 fetch func() ([]*ifbn.BottleNeckType, error),
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
284 importID int64,
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
285 conn *sql.Conn,
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
286 feedback Feedback,
2563
dc4fae4bdb8f Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents: 2548
diff changeset
287 tolerance float64,
2195
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
288 ) (interface{}, error) {
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
289 start := time.Now()
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
290
2253
cfc523c70e90 Uploaded imports: Small code clean-ups.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2248
diff changeset
291 bns, err := fetch()
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
292 if err != nil {
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
293 return nil, err
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
294 }
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
295
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
296 feedback.Info("Found %d bottlenecks for import", len(bns))
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
297
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
298 var bs bnStmts
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
299 defer bs.close()
1580
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1572
diff changeset
300
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
301 if err := bs.prepare(ctx, conn); err != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
302 return nil, err
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
303 }
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
304
1611
1835d1c9eb9b Store bottleneck_id's instead of id's in bottleneck import summary.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1595
diff changeset
305 var nids []string
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
306 seenOldBnIds := make(map[int64]bool)
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
307
2693
02d012272dd4 Log snapping tolerance
Tom Gottfried <tom@intevation.de>
parents: 2563
diff changeset
308 feedback.Info("Tolerance used to snap waterway axis: %g", tolerance)
02d012272dd4 Log snapping tolerance
Tom Gottfried <tom@intevation.de>
parents: 2563
diff changeset
309
4119
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
310 tx, err := conn.BeginTx(ctx, nil)
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
311 if err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
312 return nil, err
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
313 }
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
314 defer tx.Rollback()
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
315
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
316 for _, bn := range bns {
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
317 if err := storeBottleneck(
4119
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
318 tx, ctx, importID, feedback, bn,
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
319 &nids, seenOldBnIds, tolerance,
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
320 &bs,
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
321 ); err != nil {
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
322 return nil, err
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
323 }
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
324 }
4119
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
325 if err = tx.Commit(); err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
326 return nil, err
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
327 }
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
328
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
329 if len(nids) == 0 {
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
330 return nil, UnchangedError("No new bottlenecks inserted")
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
331 }
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
332
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
333 feedback.Info("Storing %d bottlenecks took %s", len(nids), time.Since(start))
1906
32c56e6c089a Stretch import: Added forgotten source file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1902
diff changeset
334 feedback.Info("Import of bottlenecks was successful")
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
335 summary := struct {
1611
1835d1c9eb9b Store bottleneck_id's instead of id's in bottleneck import summary.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1595
diff changeset
336 Bottlenecks []string `json:"bottlenecks"`
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
337 }{
1572
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1569
diff changeset
338 Bottlenecks: nids,
1569
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 1533
diff changeset
339 }
1906
32c56e6c089a Stretch import: Added forgotten source file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1902
diff changeset
340 return &summary, nil
1531
24445a618513 Added stub for bottleneck importer.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
341 }
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
342
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
343 func storeBottleneck(
4119
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
344 tx *sql.Tx,
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
345 ctx context.Context,
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
346 importID int64,
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
347 feedback Feedback,
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
348 bn *ifbn.BottleNeckType,
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
349 nids *[]string,
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
350 seenOldBnIds map[int64]bool,
2563
dc4fae4bdb8f Expose axis snapping tolerance to users
Tom Gottfried <tom@intevation.de>
parents: 2548
diff changeset
351 tolerance float64,
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
352 bs *bnStmts,
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
353 ) error {
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
354 feedback.Info("Processing %s (%s)", bn.OBJNAM, bn.Bottleneck_id)
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
355
3679
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
356 var tfrom, tto pgtype.Timestamptz
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
357 var uBound pgtype.BoundType
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
358
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
359 if bn.AdditionalData == nil || bn.AdditionalData.KeyValuePair == nil {
3679
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
360 // This is a workaround for the fact that most BN data does not
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
361 // contain the optional validity information.
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
362 //
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
363 // The current solution makes every change in the data an
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
364 // update, which might be not the best solution. Alternative
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
365 // approaches could include usingthe Date Info of the data as
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
366 // start date, and maybe even inspecting the details of changed
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
367 // data for hints wether an update or historisation with a new
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
368 // version is advisable.
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
369 //
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
370 // Never the less, the current solution "just works" for the
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
371 // time being and reflects the upstream systems... -- sw
3679
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
372 feedback.Warn("No validity information, assuming infinite validity.")
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
373 tfrom.Set(time.Date(1970, time.January, 1, 0, 0, 0, 0, time.UTC))
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
374 uBound = pgtype.Unbounded
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
375 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
376 const (
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
377 fromKey = "Valid_from_date"
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
378 toKey = "Valid_to_date"
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
379 )
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
380 fromTo := make(map[string]time.Time)
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
381 for _, kv := range bn.AdditionalData.KeyValuePair {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
382 k := string(kv.Key)
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
383 if k == fromKey || k == toKey {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
384 if t, err := time.Parse(time.RFC3339, kv.Value); err != nil {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
385 return err
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
386 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
387 fromTo[k] = t
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
388 }
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
389 }
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
390 }
3679
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
391
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
392 if t, ok := fromTo[fromKey]; ok {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
393 tfrom.Set(t)
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
394 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
395 feedback.Warn("Missing start date")
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
396 return nil
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
397 }
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
398
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
399 if t, ok := fromTo[toKey]; ok {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
400 tto.Set(t)
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
401 uBound = pgtype.Exclusive
4102
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
402 feedback.Info("Valid from %s to %s",
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
403 fromTo[fromKey].Format(common.TimeFormat),
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
404 fromTo[toKey].Format(common.TimeFormat))
3679
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
405 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
406 uBound = pgtype.Unbounded
4102
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
407 feedback.Info("Valid from %s",
3d2f02c16765 Request historic bottleneck data from SOAP service
Tom Gottfried <tom@intevation.de>
parents: 4071
diff changeset
408 fromTo[fromKey].Format(common.TimeFormat))
3679
0227670dedd5 Use infinite validity for bottlenecks with no validity information.
Sascha Wilde <wilde@intevation.de>
parents: 3666
diff changeset
409 }
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
410 }
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
411
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
412 validity := pgtype.Tstzrange{
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
413 Lower: tfrom,
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
414 Upper: tto,
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
415 LowerType: pgtype.Inclusive,
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
416 UpperType: uBound,
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
417 Status: pgtype.Present,
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
418 }
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
419
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
420 rb, lb := splitRBLB(bn.Rb_lb)
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
421
3423
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
422 var revisitingTime *int
3528
b9e331c1e616 Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents: 3423
diff changeset
423 if bn.Revisiting_time != nil &&
b9e331c1e616 Ignore zero length and whitespace-only values
Tom Gottfried <tom@intevation.de>
parents: 3423
diff changeset
424 len(strings.TrimSpace(*bn.Revisiting_time)) > 0 {
3423
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
425 i, err := strconv.Atoi(*bn.Revisiting_time)
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
426 if err != nil {
3530
e98ea184538e Improve warning message
Tom Gottfried <tom@intevation.de>
parents: 3528
diff changeset
427 feedback.Warn(
e98ea184538e Improve warning message
Tom Gottfried <tom@intevation.de>
parents: 3528
diff changeset
428 "Cannot convert given revisiting time '%s' to number of months",
3423
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
429 *bn.Revisiting_time)
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
430 } else {
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
431 revisitingTime = &i
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
432 }
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
433 }
6592396f5061 Make revisiting time of a bottleneck optional
Tom Gottfried <tom@intevation.de>
parents: 3302
diff changeset
434
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
435 var limiting, country string
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
436
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
437 if bn.Limiting_factor != nil {
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
438 limiting = string(*bn.Limiting_factor)
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
439 }
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
440
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
441 if bn.Responsible_country != nil {
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
442 country = string(*bn.Responsible_country)
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
443 }
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
444
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
445 var materials []string
2999
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2765
diff changeset
446 if bn.Riverbed != nil {
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2765
diff changeset
447 for _, material := range bn.Riverbed.Material {
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2765
diff changeset
448 if material != nil {
3645
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
449 materials = append(materials, string(*material))
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
450 }
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
451 }
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
452 }
02951a62e8c6 'Historicise' bottlenecks on import
Tom Gottfried <tom@intevation.de>
parents: 3566
diff changeset
453
3698
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
454 // Check if an bottleneck identical to the one we would insert already
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
455 // exists:
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
456 var old int64
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
457 err := tx.StmtContext(ctx, bs.findExactMatch).QueryRowContext(
4119
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
458 ctx,
3698
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
459 bn.Bottleneck_id,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
460 &validity,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
461 bn.Fk_g_fid,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
462 bn.From_ISRS, bn.To_ISRS,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
463 country,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
464 limiting,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
465 bn.Date_Info,
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
466 bn.Source,
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
467 ).Scan(&old)
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
468 switch {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
469 case err == sql.ErrNoRows:
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
470 // We dont have a matching old.
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
471 case err != nil:
3698
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
472 return err
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
473 default:
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
474 // We could check if the materials are also matching -- but per
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
475 // specification the Date_Info would hvae to change on that kind of
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
476 // change anyway. So actualy we ar alreayd checking more in dpth than
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
477 // required.
3698
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
478 feedback.Info("unchanged")
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
479 return nil
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
480 }
063a1883b5cb Detect and handle unchanged BN during import.
Sascha Wilde <wilde@intevation.de>
parents: 3681
diff changeset
481
3815
aa7bede70b96 Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents: 3705
diff changeset
482 // Additional Information, only of interest when we change something, so
aa7bede70b96 Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents: 3705
diff changeset
483 // it can be used for debugging if something goes wrong...
aa7bede70b96 Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents: 3705
diff changeset
484 feedback.Info("Range: from %s to %s", bn.From_ISRS, bn.To_ISRS)
aa7bede70b96 Added range info to BN import log.
Sascha Wilde <wilde@intevation.de>
parents: 3705
diff changeset
485
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
486 // Check if the new bottleneck intersects with the validity of existing
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
487 // for the same bottleneck_id we consider this an update and mark the
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
488 // old data for deletion.
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
489 bns, err := tx.StmtContext(ctx, bs.findIntersecting).QueryContext(
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
490 ctx, bn.Bottleneck_id, &validity,
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
491 )
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
492 if err != nil {
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
493 return err
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
494 }
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
495 defer bns.Close()
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
496
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
497 // Mark old intersecting bottleneck data for deletion. Don't worry about
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
498 // materials, they will be deleted via cascading.
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
499 var oldBnIds []int64
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
500 for bns.Next() {
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
501 var oldID int64
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
502 err := bns.Scan(&oldID)
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
503 if err != nil {
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
504 return err
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
505 }
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
506 oldBnIds = append(oldBnIds, oldID)
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
507 }
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
508
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
509 if err := bns.Err(); err != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
510 return err
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
511 }
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
512
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
513 switch {
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
514 case len(oldBnIds) == 1:
3705
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
515 feedback.Info("Bottelneck '%s' "+
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
516 "with intersecting validity already exists: "+
3705
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
517 "UPDATING", bn.Bottleneck_id)
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
518 case len(oldBnIds) > 1:
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
519 // This case is unexpected and should only happen when historic
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
520 // data in the bottleneck service was changed subsequently...
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
521 // We handle it gracefully anyway, but warn.
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
522 feedback.Warn("More than one Bottelneck '%s' "+
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
523 "with intersecting validity already exists: "+
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
524 "REPLACING all of them!", bn.Bottleneck_id)
3705
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
525 }
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
526
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
527 for _, oldID := range oldBnIds {
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
528 // It is possible, that two new bottlenecks intersect with the
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
529 // same old noe, therefor we have to handle duplicates in
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
530 // oldBnIds.
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
531 if !seenOldBnIds[oldID] {
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
532 if _, err := tx.StmtContext(ctx, bs.track).ExecContext(
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
533 ctx, importID, "waterway.bottlenecks", oldID, true,
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
534 ); err != nil {
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
535 return err
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
536 }
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
537 seenOldBnIds[oldID] = true
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
538 }
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
539 }
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
540
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
541 var bnIds []int64
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
542 // Add new BN data:
4119
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
543 savepoint := Savepoint(ctx, tx, "insert_bottlenck")
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
544
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
545 err = savepoint(func() error {
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
546 bns, err := tx.StmtContext(ctx, bs.insert).QueryContext(ctx,
4119
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
547 bn.Bottleneck_id,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
548 &validity,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
549 bn.Fk_g_fid,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
550 bn.OBJNAM,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
551 bn.NOBJNM,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
552 bn.From_ISRS, bn.To_ISRS,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
553 rb,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
554 lb,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
555 country,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
556 revisitingTime,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
557 limiting,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
558 bn.Date_Info,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
559 bn.Source,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
560 tolerance)
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
561 if err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
562 return err
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
563 }
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
564 defer bns.Close()
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
565 for bns.Next() {
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
566 var nid int64
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
567 if err := bns.Scan(&nid); err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
568 return err
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
569 }
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
570 bnIds = append(bnIds, nid)
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
571 }
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
572 if err := bns.Err(); err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
573 return err
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
574 }
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
575
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
576 // Add new materials
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
577 if len(bnIds) > 0 && materials != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
578 var (
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
579 pgBnIds pgtype.Int8Array
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
580 pgMaterials pgtype.VarcharArray
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
581 )
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
582 pgBnIds.Set(bnIds)
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
583 pgMaterials.Set(materials)
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
584
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
585 // Insert riverbed materials
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
586 if _, err := tx.StmtContext(ctx, bs.insertMaterial).ExecContext(ctx,
4119
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
587
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
588 &pgBnIds,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
589 &pgMaterials,
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
590 ); err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
591 return err
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
592 }
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
593 }
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
594 return nil
0cf0008070db bottleneck import: Use a global transaction with savepoints.
Sascha Wilde <wilde@intevation.de>
parents: 4113
diff changeset
595 })
3666
db87f34805fb Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents: 3665
diff changeset
596 if err != nil {
4071
5867dcf8e93c Introduced a new ReadableError type for better readable error messages of pgx.PgErrors.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4063
diff changeset
597 feedback.Warn(pgxutils.ReadableError{err}.Error())
3666
db87f34805fb Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents: 3665
diff changeset
598 return nil
db87f34805fb Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents: 3665
diff changeset
599 }
2999
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2765
diff changeset
600
3705
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
601 // Only add new BN data to tracking for staging review.
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
602 for _, nid := range bnIds {
4128
14706384a464 Cleanup by Sascha L. Teichmann applied.
Sascha Wilde <wilde@intevation.de>
parents: 4119
diff changeset
603 if _, err := tx.StmtContext(ctx, bs.track).ExecContext(
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
604 ctx, importID, "waterway.bottlenecks", nid, false,
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
605 ); err != nil {
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
606 return err
3666
db87f34805fb Align bottleneck validity at gauges
Tom Gottfried <tom@intevation.de>
parents: 3665
diff changeset
607 }
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
608 }
3705
7006b92c0334 Handle updates (vs. historized and new versions) separately.
Sascha Wilde <wilde@intevation.de>
parents: 3703
diff changeset
609
4113
f39d20427e89 WIP: Reworked bottleneck import:
Sascha Wilde <wilde@intevation.de>
parents: 4110
diff changeset
610 *nids = append(*nids, bn.Bottleneck_id)
2532
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
611 return nil
452bc714bfd9 Continue importing bottlenecks if one item fails
Tom Gottfried <tom@intevation.de>
parents: 2253
diff changeset
612 }