gemma: pkg/imports/bn.go annotate

annotate pkg/imports/bn.go @ 4881:9f1c33ba3f0a

Report reference gauge on import of bottleneck Can be especially helpful if an errors follows, which says that no reference gauge with matching temporal validity is available, to track down the cause.

author	Tom Gottfried <tom@intevation.de>
date	Fri, 31 Jan 2020 13:38:08 +0100
parents	faabfed7f0e9
children	59a99655f34d

rev	line source
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	13 // * Tom Gottfried <tom.gottfried@intevation.de>
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	14 // * Sascha Wilde <sascha.wilde@intevation.de>
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 package imports
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 import (
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "context"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	20 "database/sql"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21 "errors"
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	22 "regexp"
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	23 "strconv"
3528 b9e331c1e616 Ignore zero length and whitespace-only values Tom Gottfried <tom@intevation.de> parents: 3423 diff changeset	24 "strings"
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	25 "time"
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	26
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	27 "github.com/jackc/pgx/pgtype"
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	28
4102 3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	29 "gemma.intevation.de/gemma/pkg/common"
4062 6c760abcff0e Move handling of PostgreSQL errors to own package Tom Gottfried <tom@intevation.de> parents: 4058 diff changeset	30 "gemma.intevation.de/gemma/pkg/pgxutils"
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	31 "gemma.intevation.de/gemma/pkg/soap/ifbn"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	32 )
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33
1695 be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	34 // Bottleneck is an import job to import
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	35 // bottlenecks from an IFBN SOAP service.
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36 type Bottleneck struct {
1695 be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	37 // URL is the URL of the SOAP service.
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	38 URL string `json:"url"`
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	39 // Tolerance used for axis snapping
dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	40 Tolerance float64 `json:"tolerance"`
1695 be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	41 // Insecure indicates if HTTPS traffic
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	42 // should validate certificates or not.
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	43 Insecure bool `json:"insecure"`
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1695 diff changeset	46 // BNJobKind is the import queue type identifier.
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47 const BNJobKind JobKind = "bn"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	49 const (
1658 07f996b0e14a Bottleneck import: Cleaned up source a bit (removed outdated TODOs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1646 diff changeset	50 insertBottleneckSQL = `
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	51 WITH
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	52 bounds (b) AS (VALUES (isrs_fromText($6)), (isrs_fromText($7))),
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	53 r AS (SELECT isrsrange(
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	54 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY),
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	55 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r)
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	56 INSERT INTO waterway.bottlenecks (
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	57 bottleneck_id,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	58 validity,
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3224 diff changeset	59 gauge_location,
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	60 objnam,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	61 nobjnm,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	62 stretch,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	63 area,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	64 rb,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	65 lb,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	66 responsible_country,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	67 revisiting_time,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	68 limiting,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	69 date_info,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	70 source_organization
4069 12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	71 ) VALUES (
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	72 $1,
4069 12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	73 $2::tstzrange,
12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	74 isrs_fromText($3),
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	75 $4,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	76 $5,
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	77 (SELECT r FROM r),
1631 6ee06eb37fa9 Use new function to calculate bn area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1611 diff changeset	78 ISRSrange_area(
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	79 ISRSrange_axis((SELECT r FROM r),
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	80 $15),
2248 cce158db02b0 Input area as multipolygons to generate area from stretch Tom Gottfried <tom@intevation.de> parents: 2195 diff changeset	81 (SELECT ST_Collect(CAST(area AS geometry))
1902 c4af342be999 Use waterway area as basis for bottleneck area generation. Sascha Wilde <wilde@intevation.de> parents: 1754 diff changeset	82 FROM waterway.waterway_area)),
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	83 $8,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	84 $9,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	85 $10,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	86 $11,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	87 $12,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	88 $13,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	89 $14
4069 12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	90 )
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	91 RETURNING id
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	92 `
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	93
4130 980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	94 // We only check for NOT NULL values, for correct compairison with
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	95 // values, which might be null (and then muyst not be compairt with `='
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	96 // but with `IS NULL' is comlicated and that we are checking more than
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	97 // only (bottleneck_id, validity, date_info) is luxury already.
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	98 findExactMatchBottleneckSQL = `
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	99 WITH
4130 980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	100 bounds (b) AS (VALUES (isrs_fromText($4)), (isrs_fromText($5))),
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	101 r AS (SELECT isrsrange(
4130 980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	102 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY),
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	103 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r)
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	104 SELECT id FROM waterway.bottlenecks
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	105 WHERE (
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	106 bottleneck_id,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	107 validity,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	108 gauge_location,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	109 stretch,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	110 responsible_country,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	111 limiting,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	112 date_info,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	113 source_organization,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	114 staging_done
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	115 ) = ( SELECT
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	116 $1,
4069 12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	117 $2::tstzrange,
12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	118 isrs_fromText($3),
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	119 (SELECT r FROM r),
4130 980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	120 $6,
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	121 $7,
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	122 $8::timestamptz,
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	123 $9,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	124 true
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	125 )
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	126 `
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	127
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	128 findIntersectingBottleneckSQL = `
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	129 SELECT id FROM waterway.bottlenecks
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	130 WHERE (bottleneck_id, staging_done) = ($1, true)
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	131 AND $2::tstzrange && validity
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	132 `
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	133
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	134 insertBottleneckMaterialSQL = `
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	135 INSERT INTO waterway.bottlenecks_riverbed_materials (
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	136 bottleneck_id,
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	137 riverbed
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	138 ) VALUES ($1, $2)
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	139 ON CONFLICT (bottleneck_id, riverbed) DO NOTHING
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	140 `
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	141
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	142 bnStageDoneDeleteSQL = `
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	143 DELETE FROM waterway.bottlenecks WHERE id IN (
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	144 SELECT key
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	145 FROM import.track_imports
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	146 WHERE import_id = $1
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	147 AND relation = 'waterway.bottlenecks'::regclass
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	148 AND deletion
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	149 )`
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	150
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	151 bnStageDoneSQL = `
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	152 UPDATE waterway.bottlenecks SET staging_done = true
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	153 WHERE id IN (
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	154 SELECT key
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	155 FROM import.track_imports
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	156 WHERE import_id = $1
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	157 AND relation = 'waterway.bottlenecks'::regclass
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	158 AND NOT deletion
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	159 )`
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	160 )
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	161
4851 faabfed7f0e9 Simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	162 // Description gives a short info about relevant facts of this import.
4798 ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4197 diff changeset	163 func (bn *Bottleneck) Description() (string, error) {
4851 faabfed7f0e9 Simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	164 return bn.URL, nil
4798 ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4197 diff changeset	165 }
ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4197 diff changeset	166
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	167 type bnJobCreator struct{}
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	168
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	169 func init() {
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	170 RegisterJobCreator(BNJobKind, bnJobCreator{})
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	171 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	172
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	173 func (bnJobCreator) Description() string { return "bottlenecks" }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	174
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	175 func (bnJobCreator) AutoAccept() bool { return false }
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1631 diff changeset	176
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	177 func (bnJobCreator) Create() Job { return new(Bottleneck) }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	178
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3164 diff changeset	179 func (bnJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3164 diff changeset	180 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3164 diff changeset	181 {"bottlenecks", "bottlenecks_riverbed_materials"},
3224 bfde4f8dd323 Add missing table dependencies of imports with area generation Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	182 {"gauges", "distance_marks_virtual", "waterway_axis", "waterway_area"},
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	183 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	184 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	185
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	186 // StageDone moves the imported bottleneck out of the staging area.
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	187 func (bnJobCreator) StageDone(
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	188 ctx context.Context,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	189 tx *sql.Tx,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	190 id int64,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	191 ) error {
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	192 _, err := tx.ExecContext(ctx, bnStageDoneDeleteSQL, id)
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	193 if err == nil {
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	194 _, err = tx.ExecContext(ctx, bnStageDoneSQL, id)
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	195 }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	196 return err
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	197 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	198
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	199 // CleanUp of a bottleneck import is a NOP.
1676 4407ecaa2192 Imports: Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	200 func (*Bottleneck) CleanUp() error { return nil }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	201
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	202 var rblbRe = regexp.MustCompile(`(..)_(..)`)
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	203
2765 41f5ac76d642 BT import: allow for rb and lb to be null. Sascha Wilde <wilde@intevation.de> parents: 2752 diff changeset	204 func splitRBLB(s string) (string, string) {
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	205 m := rblbRe.FindStringSubmatch(s)
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	206 if len(m) == 0 {
2765 41f5ac76d642 BT import: allow for rb and lb to be null. Sascha Wilde <wilde@intevation.de> parents: 2752 diff changeset	207 return nil, nil
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	208 }
2765 41f5ac76d642 BT import: allow for rb and lb to be null. Sascha Wilde <wilde@intevation.de> parents: 2752 diff changeset	209 return &m[1], &m[2]
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	210 }
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	211
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	212 // Do executes the actual bottleneck import.
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	213 func (bn *Bottleneck) Do(
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	214 ctx context.Context,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	215 importID int64,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	216 conn *sql.Conn,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	217 feedback Feedback,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	218 ) (interface{}, error) {
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	219
2253 cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2248 diff changeset	220 fetch := func() ([]*ifbn.BottleNeckType, error) {
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	221 client := ifbn.NewIBottleneckService(bn.URL, bn.Insecure, nil)
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	222
4102 3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	223 req := &ifbn.Export_bn_by_isrs{
3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	224 Period: &ifbn.RequestedPeriod{Date_start: &time.Time{}},
3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	225 }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	226
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	227 resp, err := client.Export_bn_by_isrs(req)
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	228 if err != nil {
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	229 return nil, err
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	230 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	231
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	232 if resp.Export_bn_by_isrsResult == nil {
3562 c03224d33e78 Improve error message Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	233 return nil, errors.New(
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4147 diff changeset	234 "the requested service returned no bottlenecks")
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	235 }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	236
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	237 return resp.Export_bn_by_isrsResult.BottleNeckType, nil
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	238 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	239
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	240 return storeBottlenecks(ctx, fetch, importID, conn, feedback, bn.Tolerance)
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	241 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	242
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	243 type bnStmts struct {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	244 insert *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	245 findExactMatch *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	246 findIntersecting *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	247 insertMaterial *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	248 track *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	249 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	250
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	251 func (bs *bnStmts) close() {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	252 for _, s := range []**sql.Stmt{
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	253 &bs.insert,
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	254 &bs.findExactMatch,
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	255 &bs.findIntersecting,
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	256 &bs.insertMaterial,
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	257 &bs.track,
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	258 } {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	259 if *s != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	260 (*s).Close()
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	261 *s = nil
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	262 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	263 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	264 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	265
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	266 func (bs bnStmts) prepare(ctx context.Context, conn sql.Conn) error {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	267 for _, x := range []struct {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	268 sql string
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	269 stmt **sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	270 }{
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	271 {insertBottleneckSQL, &bs.insert},
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	272 {findExactMatchBottleneckSQL, &bs.findExactMatch},
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	273 {findIntersectingBottleneckSQL, &bs.findIntersecting},
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	274 {insertBottleneckMaterialSQL, &bs.insertMaterial},
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	275 {trackImportDeletionSQL, &bs.track},
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	276 } {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	277 var err error
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	278 if *x.stmt, err = conn.PrepareContext(ctx, x.sql); err != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	279 return err
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	280 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	281 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	282 return nil
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	283 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	284
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	285 func storeBottlenecks(
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	286 ctx context.Context,
2253 cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2248 diff changeset	287 fetch func() ([]*ifbn.BottleNeckType, error),
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	288 importID int64,
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	289 conn *sql.Conn,
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	290 feedback Feedback,
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	291 tolerance float64,
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	292 ) (interface{}, error) {
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	293 start := time.Now()
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	294
2253 cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2248 diff changeset	295 bns, err := fetch()
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	296 if err != nil {
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	297 return nil, err
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	298 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	299
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	300 feedback.Info("Found %d bottlenecks for import", len(bns))
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	301
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	302 var bs bnStmts
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	303 defer bs.close()
1580 c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	304
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	305 if err := bs.prepare(ctx, conn); err != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	306 return nil, err
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	307 }
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	308
1611 1835d1c9eb9b Store bottleneck_id's instead of id's in bottleneck import summary. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1595 diff changeset	309 var nids []string
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	310 seenOldBnIDs := make(map[int64]bool)
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	311
2693 02d012272dd4 Log snapping tolerance Tom Gottfried <tom@intevation.de> parents: 2563 diff changeset	312 feedback.Info("Tolerance used to snap waterway axis: %g", tolerance)
02d012272dd4 Log snapping tolerance Tom Gottfried <tom@intevation.de> parents: 2563 diff changeset	313
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	314 tx, err := conn.BeginTx(ctx, nil)
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	315 if err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	316 return nil, err
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	317 }
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	318 defer tx.Rollback()
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	319
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	320 savepoint := Savepoint(ctx, tx, "insert_bottlenck")
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	321
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	322 for _, bn := range bns {
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	323 if err := storeBottleneck(
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	324 ctx, tx,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	325 importID, feedback, bn,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	326 &nids, seenOldBnIDs, tolerance,
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	327 &bs,
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	328 savepoint,
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	329 ); err != nil {
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	330 return nil, err
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	331 }
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	332 }
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	333 if err = tx.Commit(); err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	334 return nil, err
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	335 }
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	336
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	337 if len(nids) == 0 {
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	338 return nil, UnchangedError("No new bottlenecks inserted")
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	339 }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	340
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	341 feedback.Info("Storing %d bottlenecks took %s", len(nids), time.Since(start))
1906 32c56e6c089a Stretch import: Added forgotten source file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1902 diff changeset	342 feedback.Info("Import of bottlenecks was successful")
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	343 summary := struct {
1611 1835d1c9eb9b Store bottleneck_id's instead of id's in bottleneck import summary. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1595 diff changeset	344 Bottlenecks []string `json:"bottlenecks"`
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	345 }{
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	346 Bottlenecks: nids,
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	347 }
1906 32c56e6c089a Stretch import: Added forgotten source file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1902 diff changeset	348 return &summary, nil
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	349 }
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	350
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	351 func storeBottleneck(
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	352 ctx context.Context,
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	353 tx *sql.Tx,
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	354 importID int64,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	355 feedback Feedback,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	356 bn *ifbn.BottleNeckType,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	357 nids *[]string,
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	358 seenOldBnIDs map[int64]bool,
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	359 tolerance float64,
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	360 bs *bnStmts,
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	361 savepoint func(func() error) error,
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	362 ) error {
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	363 feedback.Info("Processing %s (%s)", bn.OBJNAM, bn.Bottleneck_id)
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	364
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	365 var (
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	366 tfrom, tto pgtype.Timestamptz
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	367 uBound pgtype.BoundType
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	368 )
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	369
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	370 if bn.AdditionalData == nil \|\| bn.AdditionalData.KeyValuePair == nil {
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	371 // This is a workaround for the fact that most BN data does not
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	372 // contain the optional validity information.
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	373 //
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	374 // The current solution makes every change in the data an
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	375 // update, which might be not the best solution. Alternative
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	376 // approaches could include usingthe Date Info of the data as
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	377 // start date, and maybe even inspecting the details of changed
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	378 // data for hints wether an update or historisation with a new
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	379 // version is advisable.
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	380 //
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	381 // Never the less, the current solution "just works" for the
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	382 // time being and reflects the upstream systems... -- sw
4134 9864d682ab47 bn-import: Downgraded info on missing validity from Warning to Info. Sascha Wilde <wilde@intevation.de> parents: 4133 diff changeset	383 feedback.Info("No validity information, assuming infinite validity.")
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	384 tfrom.Set(time.Date(1970, time.January, 1, 0, 0, 0, 0, time.UTC))
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	385 uBound = pgtype.Unbounded
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	386 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	387 const (
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	388 fromKey = "Valid_from_date"
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	389 toKey = "Valid_to_date"
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	390 )
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	391 fromTo := make(map[string]time.Time)
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	392 for _, kv := range bn.AdditionalData.KeyValuePair {
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	393 if kv.Key == fromKey \|\| kv.Key == toKey {
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	394 t, err := time.Parse(time.RFC3339, kv.Value)
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	395 if err != nil {
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	396 return err
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	397 }
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	398 fromTo[kv.Key] = t
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	399 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	400 }
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	401
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	402 if t, ok := fromTo[fromKey]; ok {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	403 tfrom.Set(t)
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	404 } else {
4197 5d7ce7f926eb Log fatal errors as errors, not warnings, in imports Tom Gottfried <tom@intevation.de> parents: 4177 diff changeset	405 feedback.Error("Missing start date")
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	406 return nil
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	407 }
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	408
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	409 if t, ok := fromTo[toKey]; ok {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	410 tto.Set(t)
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	411 uBound = pgtype.Exclusive
4102 3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	412 feedback.Info("Valid from %s to %s",
3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	413 fromTo[fromKey].Format(common.TimeFormat),
3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	414 fromTo[toKey].Format(common.TimeFormat))
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	415 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	416 uBound = pgtype.Unbounded
4102 3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	417 feedback.Info("Valid from %s",
3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	418 fromTo[fromKey].Format(common.TimeFormat))
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	419 }
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	420 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	421
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	422 validity := pgtype.Tstzrange{
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	423 Lower: tfrom,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	424 Upper: tto,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	425 LowerType: pgtype.Inclusive,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	426 UpperType: uBound,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	427 Status: pgtype.Present,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	428 }
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	429
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	430 rb, lb := splitRBLB(bn.Rb_lb)
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	431
3423 6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	432 var revisitingTime *int
3528 b9e331c1e616 Ignore zero length and whitespace-only values Tom Gottfried <tom@intevation.de> parents: 3423 diff changeset	433 if bn.Revisiting_time != nil &&
b9e331c1e616 Ignore zero length and whitespace-only values Tom Gottfried <tom@intevation.de> parents: 3423 diff changeset	434 len(strings.TrimSpace(*bn.Revisiting_time)) > 0 {
3423 6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	435 i, err := strconv.Atoi(*bn.Revisiting_time)
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	436 if err != nil {
3530 e98ea184538e Improve warning message Tom Gottfried <tom@intevation.de> parents: 3528 diff changeset	437 feedback.Warn(
e98ea184538e Improve warning message Tom Gottfried <tom@intevation.de> parents: 3528 diff changeset	438 "Cannot convert given revisiting time '%s' to number of months",
3423 6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	439 *bn.Revisiting_time)
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	440 } else {
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	441 revisitingTime = &i
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	442 }
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	443 }
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	444
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	445 var limiting, country string
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	446
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	447 if bn.Limiting_factor != nil {
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	448 limiting = string(*bn.Limiting_factor)
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	449 }
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	450
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	451 if bn.Responsible_country != nil {
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	452 country = string(*bn.Responsible_country)
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	453 }
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	454
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	455 var materials []string
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	456 if bn.Riverbed != nil {
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	457 for _, material := range bn.Riverbed.Material {
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	458 if material != nil {
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	459 materials = append(materials, string(*material))
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	460 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	461 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	462 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	463
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	464 // Check if a bottleneck identical to the one we would insert already
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	465 // exists:
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	466 var old int64
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	467 err := tx.StmtContext(ctx, bs.findExactMatch).QueryRowContext(
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	468 ctx,
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	469 bn.Bottleneck_id,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	470 &validity,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	471 bn.Fk_g_fid,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	472 bn.From_ISRS, bn.To_ISRS,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	473 country,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	474 limiting,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	475 bn.Date_Info,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	476 bn.Source,
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	477 ).Scan(&old)
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	478 switch {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	479 case err == sql.ErrNoRows:
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	480 // We dont have a matching old.
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	481 case err != nil:
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	482 return err
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	483 default:
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	484 // We could check if the materials are also matching -- but per
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	485 // specification the Date_Info would hvae to change on that kind of
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	486 // change anyway. So actually we are already checking more in depth than
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	487 // required.
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	488 feedback.Info("unchanged")
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	489 return nil
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	490 }
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	491
3815 aa7bede70b96 Added range info to BN import log. Sascha Wilde <wilde@intevation.de> parents: 3705 diff changeset	492 // Additional Information, only of interest when we change something, so
aa7bede70b96 Added range info to BN import log. Sascha Wilde <wilde@intevation.de> parents: 3705 diff changeset	493 // it can be used for debugging if something goes wrong...
aa7bede70b96 Added range info to BN import log. Sascha Wilde <wilde@intevation.de> parents: 3705 diff changeset	494 feedback.Info("Range: from %s to %s", bn.From_ISRS, bn.To_ISRS)
4881 9f1c33ba3f0a Report reference gauge on import of bottleneck Tom Gottfried <tom@intevation.de> parents: 4851 diff changeset	495 feedback.Info("Reference gauge: %s", bn.Fk_g_fid)
3815 aa7bede70b96 Added range info to BN import log. Sascha Wilde <wilde@intevation.de> parents: 3705 diff changeset	496
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	497 // Check if the new bottleneck intersects with the validity of existing
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	498 // for the same bottleneck_id we consider this an update and mark the
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	499 // old data for deletion.
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	500 bns, err := tx.StmtContext(ctx, bs.findIntersecting).QueryContext(
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	501 ctx, bn.Bottleneck_id, &validity,
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	502 )
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	503 if err != nil {
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	504 return err
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	505 }
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	506 defer bns.Close()
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	507
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	508 // Mark old intersecting bottleneck data for deletion. Don't worry about
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	509 // materials, they will be deleted via cascading.
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	510 var oldBnIDs []int64
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	511 for bns.Next() {
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	512 var oldID int64
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	513 err := bns.Scan(&oldID)
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	514 if err != nil {
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	515 return err
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	516 }
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	517 oldBnIDs = append(oldBnIDs, oldID)
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	518 }
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	519
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	520 if err := bns.Err(); err != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	521 return err
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	522 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	523
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	524 switch {
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	525 case len(oldBnIDs) == 1:
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	526 feedback.Info("Bottleneck '%s' "+
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	527 "with intersecting validity already exists: "+
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	528 "UPDATING", bn.Bottleneck_id)
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	529 case len(oldBnIDs) > 1:
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	530 // This case is unexpected and should only happen when historic
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	531 // data in the bottleneck service was changed subsequently...
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	532 // We handle it gracefully anyway, but warn.
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	533 feedback.Warn("More than one Bottelneck '%s' "+
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	534 "with intersecting validity already exists: "+
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	535 "REPLACING all of them!", bn.Bottleneck_id)
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	536 }
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	537 // We write the actual tracking information for deletion of superseded
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	538 // bottlenecks later to the database -- AFTER the new bottleneck was
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	539 // created successfully. That way, we don't change the database, when
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	540 // an error arises during inserting the new data.
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	541
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	542 var bnID int64
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	543 // Add new BN data:
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	544 if err := savepoint(func() error {
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	545 if err := tx.StmtContext(ctx, bs.insert).QueryRowContext(
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	546 ctx,
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	547 bn.Bottleneck_id,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	548 &validity,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	549 bn.Fk_g_fid,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	550 bn.OBJNAM,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	551 bn.NOBJNM,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	552 bn.From_ISRS, bn.To_ISRS,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	553 rb,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	554 lb,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	555 country,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	556 revisitingTime,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	557 limiting,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	558 bn.Date_Info,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	559 bn.Source,
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	560 tolerance,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	561 ).Scan(&bnID); err != nil {
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	562 return err
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	563 }
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	564
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	565 // Add new materials
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	566 for _, material := range materials {
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	567 if _, err := tx.StmtContext(ctx, bs.insertMaterial).ExecContext(
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	568 ctx,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	569 bnID,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	570 material,
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	571 ); err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	572 return err
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	573 }
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	574 }
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	575 return nil
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	576 }); err != nil {
4197 5d7ce7f926eb Log fatal errors as errors, not warnings, in imports Tom Gottfried <tom@intevation.de> parents: 4177 diff changeset	577 feedback.Error(pgxutils.ReadableError{Err: err}.Error())
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	578 return nil
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	579 }
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	580
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	581 // Now that adding BNs to staging was successful, write import tracking
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	582 // information to database:
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	583 for _, oldID := range oldBnIDs {
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	584 // It is possible, that two new bottlenecks intersect with the
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	585 // same old one, therefore we have to handle duplicates in
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	586 // oldBnIds.
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	587 if !seenOldBnIDs[oldID] {
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	588 if _, err := tx.StmtContext(ctx, bs.track).ExecContext(
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	589 ctx, importID, "waterway.bottlenecks", oldID, true,
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	590 ); err != nil {
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	591 return err
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	592 }
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	593 seenOldBnIDs[oldID] = true
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	594 }
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	595 }
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	596
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	597 if _, err := tx.StmtContext(ctx, bs.track).ExecContext(
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	598 ctx, importID, "waterway.bottlenecks", bnID, false,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	599 ); err != nil {
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	600 return err
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	601 }
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	602
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	603 nids = append(nids, bn.Bottleneck_id)
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	604 return nil
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	605 }

Mercurial > gemma

annotate pkg/imports/bn.go @ 4881:9f1c33ba3f0a