gemma: pkg/imports/bn.go annotate

annotate pkg/imports/bn.go @ 5712:6270951dda28 revive-cleanup

/interface{}/any/

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Tue, 20 Feb 2024 22:37:51 +0100
parents	f2204f91d286
children

rev	line source
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	13 // * Tom Gottfried <tom.gottfried@intevation.de>
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	14 // * Sascha Wilde <sascha.wilde@intevation.de>
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 package imports
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 import (
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "context"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	20 "database/sql"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21 "errors"
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	22 "regexp"
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	23 "strconv"
3528 b9e331c1e616 Ignore zero length and whitespace-only values Tom Gottfried <tom@intevation.de> parents: 3423 diff changeset	24 "strings"
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	25 "time"
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	26
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	27 "github.com/jackc/pgx/pgtype"
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	28
4102 3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	29 "gemma.intevation.de/gemma/pkg/common"
4062 6c760abcff0e Move handling of PostgreSQL errors to own package Tom Gottfried <tom@intevation.de> parents: 4058 diff changeset	30 "gemma.intevation.de/gemma/pkg/pgxutils"
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	31 "gemma.intevation.de/gemma/pkg/soap/ifbn"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	32 )
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33
1695 be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	34 // Bottleneck is an import job to import
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	35 // bottlenecks from an IFBN SOAP service.
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36 type Bottleneck struct {
1695 be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	37 // URL is the URL of the SOAP service.
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	38 URL string `json:"url"`
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	39 // Tolerance used for axis snapping
dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	40 Tolerance float64 `json:"tolerance"`
1695 be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	41 // Insecure indicates if HTTPS traffic
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	42 // should validate certificates or not.
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	43 Insecure bool `json:"insecure"`
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1695 diff changeset	46 // BNJobKind is the import queue type identifier.
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47 const BNJobKind JobKind = "bn"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	49 const (
1658 07f996b0e14a Bottleneck import: Cleaned up source a bit (removed outdated TODOs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1646 diff changeset	50 insertBottleneckSQL = `
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	51 WITH
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	52 bounds (b) AS (VALUES (isrs_fromText($6)), (isrs_fromText($7))),
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	53 r AS (SELECT isrsrange(
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	54 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY),
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	55 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r)
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	56 INSERT INTO waterway.bottlenecks (
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	57 bottleneck_id,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	58 validity,
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3224 diff changeset	59 gauge_location,
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	60 objnam,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	61 nobjnm,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	62 stretch,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	63 area,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	64 rb,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	65 lb,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	66 responsible_country,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	67 revisiting_time,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	68 limiting,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	69 date_info,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	70 source_organization
4069 12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	71 ) VALUES (
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	72 $1,
4069 12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	73 $2::tstzrange,
12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	74 isrs_fromText($3),
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	75 $4,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	76 $5,
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	77 (SELECT r FROM r),
1631 6ee06eb37fa9 Use new function to calculate bn area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1611 diff changeset	78 ISRSrange_area(
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	79 ISRSrange_axis((SELECT r FROM r),
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	80 $15),
2248 cce158db02b0 Input area as multipolygons to generate area from stretch Tom Gottfried <tom@intevation.de> parents: 2195 diff changeset	81 (SELECT ST_Collect(CAST(area AS geometry))
1902 c4af342be999 Use waterway area as basis for bottleneck area generation. Sascha Wilde <wilde@intevation.de> parents: 1754 diff changeset	82 FROM waterway.waterway_area)),
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	83 $8,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	84 $9,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	85 $10,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	86 $11,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	87 $12,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	88 $13,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	89 $14
4069 12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	90 )
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	91 RETURNING id
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	92 `
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	93
4130 980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	94 // We only check for NOT NULL values, for correct compairison with
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	95 // values, which might be null (and then muyst not be compairt with `='
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	96 // but with `IS NULL' is comlicated and that we are checking more than
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	97 // only (bottleneck_id, validity, date_info) is luxury already.
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	98 findExactMatchBottleneckSQL = `
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	99 WITH
4130 980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	100 bounds (b) AS (VALUES (isrs_fromText($4)), (isrs_fromText($5))),
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	101 r AS (SELECT isrsrange(
4130 980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	102 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY),
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	103 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r)
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	104 SELECT id FROM waterway.bottlenecks
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	105 WHERE (
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	106 bottleneck_id,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	107 validity,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	108 gauge_location,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	109 stretch,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	110 responsible_country,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	111 limiting,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	112 date_info,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	113 source_organization,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	114 staging_done
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	115 ) = ( SELECT
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	116 $1,
4069 12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	117 $2::tstzrange,
12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	118 isrs_fromText($3),
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	119 (SELECT r FROM r),
4130 980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	120 $6,
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	121 $7,
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	122 $8::timestamptz,
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	123 $9,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	124 true
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	125 )
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	126 `
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	127
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	128 findIntersectingBottleneckSQL = `
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	129 SELECT id FROM waterway.bottlenecks
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	130 WHERE (bottleneck_id, staging_done) = ($1, true)
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	131 AND $2::tstzrange && validity
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	132 `
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	133
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	134 insertBottleneckMaterialSQL = `
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	135 INSERT INTO waterway.bottlenecks_riverbed_materials (
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	136 bottleneck_id,
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	137 riverbed
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	138 ) VALUES ($1, $2)
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	139 ON CONFLICT (bottleneck_id, riverbed) DO NOTHING
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	140 `
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	141
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	142 bnStageDoneDeleteSQL = `
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	143 DELETE FROM waterway.bottlenecks WHERE id IN (
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	144 SELECT key
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	145 FROM import.track_imports
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	146 WHERE import_id = $1
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	147 AND relation = 'waterway.bottlenecks'::regclass
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	148 AND deletion
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	149 )`
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	150
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	151 bnStageDoneSQL = `
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	152 UPDATE waterway.bottlenecks SET staging_done = true
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	153 WHERE id IN (
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	154 SELECT key
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	155 FROM import.track_imports
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	156 WHERE import_id = $1
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	157 AND relation = 'waterway.bottlenecks'::regclass
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	158 AND NOT deletion
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	159 )`
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	160 )
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	161
4851 faabfed7f0e9 Simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	162 // Description gives a short info about relevant facts of this import.
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	163 func (bn *Bottleneck) Description([]string) (string, error) {
4851 faabfed7f0e9 Simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	164 return bn.URL, nil
4798 ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4197 diff changeset	165 }
ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4197 diff changeset	166
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	167 type bnJobCreator struct{}
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	168
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	169 func init() {
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	170 RegisterJobCreator(BNJobKind, bnJobCreator{})
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	171 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	172
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	173 func (bnJobCreator) Description() string { return "bottlenecks" }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	174
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	175 func (bnJobCreator) AutoAccept() bool { return false }
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1631 diff changeset	176
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	177 func (bnJobCreator) Create() Job { return new(Bottleneck) }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	178
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3164 diff changeset	179 func (bnJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3164 diff changeset	180 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3164 diff changeset	181 {"bottlenecks", "bottlenecks_riverbed_materials"},
3224 bfde4f8dd323 Add missing table dependencies of imports with area generation Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	182 {"gauges", "distance_marks_virtual", "waterway_axis", "waterway_area"},
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	183 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	184 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	185
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	186 // StageDone moves the imported bottleneck out of the staging area.
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	187 func (bnJobCreator) StageDone(
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	188 ctx context.Context,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	189 tx *sql.Tx,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	190 id int64,
5034 59a99655f34d Added feedback support for StageDone. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4881 diff changeset	191 _ Feedback,
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	192 ) error {
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	193 _, err := tx.ExecContext(ctx, bnStageDoneDeleteSQL, id)
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	194 if err == nil {
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	195 _, err = tx.ExecContext(ctx, bnStageDoneSQL, id)
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	196 }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	197 return err
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	198 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	199
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	200 // CleanUp of a bottleneck import is a NOP.
1676 4407ecaa2192 Imports: Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	201 func (*Bottleneck) CleanUp() error { return nil }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	202
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	203 var rblbRe = regexp.MustCompile(`(..)_(..)`)
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	204
2765 41f5ac76d642 BT import: allow for rb and lb to be null. Sascha Wilde <wilde@intevation.de> parents: 2752 diff changeset	205 func splitRBLB(s string) (string, string) {
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	206 m := rblbRe.FindStringSubmatch(s)
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	207 if len(m) == 0 {
2765 41f5ac76d642 BT import: allow for rb and lb to be null. Sascha Wilde <wilde@intevation.de> parents: 2752 diff changeset	208 return nil, nil
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	209 }
2765 41f5ac76d642 BT import: allow for rb and lb to be null. Sascha Wilde <wilde@intevation.de> parents: 2752 diff changeset	210 return &m[1], &m[2]
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	211 }
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	212
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	213 // Do executes the actual bottleneck import.
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	214 func (bn *Bottleneck) Do(
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	215 ctx context.Context,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	216 importID int64,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	217 conn *sql.Conn,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	218 feedback Feedback,
5712 6270951dda28 /interface{}/any/ Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5560 diff changeset	219 ) (any, error) {
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	220
2253 cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2248 diff changeset	221 fetch := func() ([]*ifbn.BottleNeckType, error) {
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	222 client := ifbn.NewIBottleneckService(bn.URL, bn.Insecure, nil)
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	223
4102 3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	224 req := &ifbn.Export_bn_by_isrs{
3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	225 Period: &ifbn.RequestedPeriod{Date_start: &time.Time{}},
3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	226 }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	227
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	228 resp, err := client.Export_bn_by_isrs(req)
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	229 if err != nil {
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	230 return nil, err
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	231 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	232
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	233 if resp.Export_bn_by_isrsResult == nil {
3562 c03224d33e78 Improve error message Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	234 return nil, errors.New(
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4147 diff changeset	235 "the requested service returned no bottlenecks")
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	236 }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	237
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	238 return resp.Export_bn_by_isrsResult.BottleNeckType, nil
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	239 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	240
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	241 return storeBottlenecks(ctx, fetch, importID, conn, feedback, bn.Tolerance)
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	242 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	243
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	244 type bnStmts struct {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	245 insert *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	246 findExactMatch *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	247 findIntersecting *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	248 insertMaterial *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	249 track *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	250 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	251
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	252 func (bs *bnStmts) close() {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	253 for _, s := range []**sql.Stmt{
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	254 &bs.insert,
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	255 &bs.findExactMatch,
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	256 &bs.findIntersecting,
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	257 &bs.insertMaterial,
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	258 &bs.track,
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	259 } {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	260 if *s != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	261 (*s).Close()
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	262 *s = nil
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	263 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	264 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	265 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	266
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	267 func (bs bnStmts) prepare(ctx context.Context, conn sql.Conn) error {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	268 for _, x := range []struct {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	269 sql string
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	270 stmt **sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	271 }{
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	272 {insertBottleneckSQL, &bs.insert},
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	273 {findExactMatchBottleneckSQL, &bs.findExactMatch},
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	274 {findIntersectingBottleneckSQL, &bs.findIntersecting},
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	275 {insertBottleneckMaterialSQL, &bs.insertMaterial},
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	276 {trackImportDeletionSQL, &bs.track},
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	277 } {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	278 var err error
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	279 if *x.stmt, err = conn.PrepareContext(ctx, x.sql); err != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	280 return err
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	281 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	282 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	283 return nil
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	284 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	285
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	286 func storeBottlenecks(
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	287 ctx context.Context,
2253 cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2248 diff changeset	288 fetch func() ([]*ifbn.BottleNeckType, error),
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	289 importID int64,
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	290 conn *sql.Conn,
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	291 feedback Feedback,
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	292 tolerance float64,
5712 6270951dda28 /interface{}/any/ Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5560 diff changeset	293 ) (any, error) {
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	294 start := time.Now()
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	295
2253 cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2248 diff changeset	296 bns, err := fetch()
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	297 if err != nil {
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	298 return nil, err
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	299 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	300
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	301 feedback.Info("Found %d bottlenecks for import", len(bns))
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	302
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	303 var bs bnStmts
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	304 defer bs.close()
1580 c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	305
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	306 if err := bs.prepare(ctx, conn); err != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	307 return nil, err
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	308 }
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	309
1611 1835d1c9eb9b Store bottleneck_id's instead of id's in bottleneck import summary. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1595 diff changeset	310 var nids []string
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	311 seenOldBnIDs := make(map[int64]bool)
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	312
2693 02d012272dd4 Log snapping tolerance Tom Gottfried <tom@intevation.de> parents: 2563 diff changeset	313 feedback.Info("Tolerance used to snap waterway axis: %g", tolerance)
02d012272dd4 Log snapping tolerance Tom Gottfried <tom@intevation.de> parents: 2563 diff changeset	314
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	315 tx, err := conn.BeginTx(ctx, nil)
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	316 if err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	317 return nil, err
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	318 }
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	319 defer tx.Rollback()
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	320
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	321 savepoint := Savepoint(ctx, tx, "insert_bottlenck")
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	322
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	323 for _, bn := range bns {
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	324 if err := storeBottleneck(
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	325 ctx, tx,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	326 importID, feedback, bn,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	327 &nids, seenOldBnIDs, tolerance,
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	328 &bs,
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	329 savepoint,
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	330 ); err != nil {
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	331 return nil, err
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	332 }
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	333 }
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	334 if err = tx.Commit(); err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	335 return nil, err
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	336 }
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	337
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	338 if len(nids) == 0 {
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	339 return nil, UnchangedError("No new bottlenecks inserted")
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	340 }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	341
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	342 feedback.Info("Storing %d bottlenecks took %s", len(nids), time.Since(start))
1906 32c56e6c089a Stretch import: Added forgotten source file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1902 diff changeset	343 feedback.Info("Import of bottlenecks was successful")
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	344 summary := struct {
1611 1835d1c9eb9b Store bottleneck_id's instead of id's in bottleneck import summary. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1595 diff changeset	345 Bottlenecks []string `json:"bottlenecks"`
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	346 }{
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	347 Bottlenecks: nids,
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	348 }
1906 32c56e6c089a Stretch import: Added forgotten source file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1902 diff changeset	349 return &summary, nil
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	350 }
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	351
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	352 func storeBottleneck(
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	353 ctx context.Context,
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	354 tx *sql.Tx,
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	355 importID int64,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	356 feedback Feedback,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	357 bn *ifbn.BottleNeckType,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	358 nids *[]string,
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	359 seenOldBnIDs map[int64]bool,
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	360 tolerance float64,
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	361 bs *bnStmts,
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	362 savepoint func(func() error) error,
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	363 ) error {
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	364 feedback.Info("Processing %s (%s)", bn.OBJNAM, bn.Bottleneck_id)
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	365
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	366 var (
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	367 tfrom, tto pgtype.Timestamptz
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	368 uBound pgtype.BoundType
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	369 )
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	370
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	371 if bn.AdditionalData == nil \|\| bn.AdditionalData.KeyValuePair == nil {
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	372 // This is a workaround for the fact that most BN data does not
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	373 // contain the optional validity information.
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	374 //
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	375 // The current solution makes every change in the data an
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	376 // update, which might be not the best solution. Alternative
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	377 // approaches could include usingthe Date Info of the data as
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	378 // start date, and maybe even inspecting the details of changed
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	379 // data for hints wether an update or historisation with a new
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	380 // version is advisable.
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	381 //
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	382 // Never the less, the current solution "just works" for the
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	383 // time being and reflects the upstream systems... -- sw
4134 9864d682ab47 bn-import: Downgraded info on missing validity from Warning to Info. Sascha Wilde <wilde@intevation.de> parents: 4133 diff changeset	384 feedback.Info("No validity information, assuming infinite validity.")
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	385 tfrom.Set(time.Date(1970, time.January, 1, 0, 0, 0, 0, time.UTC))
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	386 uBound = pgtype.Unbounded
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	387 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	388 const (
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	389 fromKey = "Valid_from_date"
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	390 toKey = "Valid_to_date"
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	391 )
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	392 fromTo := make(map[string]time.Time)
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	393 for _, kv := range bn.AdditionalData.KeyValuePair {
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	394 if kv.Key == fromKey \|\| kv.Key == toKey {
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	395 t, err := time.Parse(time.RFC3339, kv.Value)
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	396 if err != nil {
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	397 return err
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	398 }
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	399 fromTo[kv.Key] = t
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	400 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	401 }
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	402
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	403 if t, ok := fromTo[fromKey]; ok {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	404 tfrom.Set(t)
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	405 } else {
4197 5d7ce7f926eb Log fatal errors as errors, not warnings, in imports Tom Gottfried <tom@intevation.de> parents: 4177 diff changeset	406 feedback.Error("Missing start date")
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	407 return nil
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	408 }
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	409
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	410 if t, ok := fromTo[toKey]; ok {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	411 tto.Set(t)
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	412 uBound = pgtype.Exclusive
4102 3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	413 feedback.Info("Valid from %s to %s",
3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	414 fromTo[fromKey].Format(common.TimeFormat),
3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	415 fromTo[toKey].Format(common.TimeFormat))
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	416 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	417 uBound = pgtype.Unbounded
4102 3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	418 feedback.Info("Valid from %s",
3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	419 fromTo[fromKey].Format(common.TimeFormat))
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	420 }
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	421 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	422
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	423 validity := pgtype.Tstzrange{
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	424 Lower: tfrom,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	425 Upper: tto,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	426 LowerType: pgtype.Inclusive,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	427 UpperType: uBound,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	428 Status: pgtype.Present,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	429 }
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	430
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	431 rb, lb := splitRBLB(bn.Rb_lb)
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	432
3423 6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	433 var revisitingTime *int
3528 b9e331c1e616 Ignore zero length and whitespace-only values Tom Gottfried <tom@intevation.de> parents: 3423 diff changeset	434 if bn.Revisiting_time != nil &&
b9e331c1e616 Ignore zero length and whitespace-only values Tom Gottfried <tom@intevation.de> parents: 3423 diff changeset	435 len(strings.TrimSpace(*bn.Revisiting_time)) > 0 {
3423 6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	436 i, err := strconv.Atoi(*bn.Revisiting_time)
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	437 if err != nil {
3530 e98ea184538e Improve warning message Tom Gottfried <tom@intevation.de> parents: 3528 diff changeset	438 feedback.Warn(
e98ea184538e Improve warning message Tom Gottfried <tom@intevation.de> parents: 3528 diff changeset	439 "Cannot convert given revisiting time '%s' to number of months",
3423 6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	440 *bn.Revisiting_time)
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	441 } else {
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	442 revisitingTime = &i
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	443 }
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	444 }
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	445
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	446 var limiting, country string
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	447
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	448 if bn.Limiting_factor != nil {
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	449 limiting = string(*bn.Limiting_factor)
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	450 }
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	451
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	452 if bn.Responsible_country != nil {
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	453 country = string(*bn.Responsible_country)
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	454 }
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	455
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	456 var materials []string
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	457 if bn.Riverbed != nil {
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	458 for _, material := range bn.Riverbed.Material {
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	459 if material != nil {
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	460 materials = append(materials, string(*material))
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	461 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	462 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	463 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	464
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	465 // Check if a bottleneck identical to the one we would insert already
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	466 // exists:
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	467 var old int64
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	468 err := tx.StmtContext(ctx, bs.findExactMatch).QueryRowContext(
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	469 ctx,
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	470 bn.Bottleneck_id,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	471 &validity,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	472 bn.Fk_g_fid,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	473 bn.From_ISRS, bn.To_ISRS,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	474 country,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	475 limiting,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	476 bn.Date_Info,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	477 bn.Source,
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	478 ).Scan(&old)
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	479 switch {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	480 case err == sql.ErrNoRows:
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	481 // We dont have a matching old.
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	482 case err != nil:
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	483 return err
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	484 default:
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	485 // We could check if the materials are also matching -- but per
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	486 // specification the Date_Info would hvae to change on that kind of
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	487 // change anyway. So actually we are already checking more in depth than
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	488 // required.
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	489 feedback.Info("unchanged")
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	490 return nil
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	491 }
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	492
3815 aa7bede70b96 Added range info to BN import log. Sascha Wilde <wilde@intevation.de> parents: 3705 diff changeset	493 // Additional Information, only of interest when we change something, so
aa7bede70b96 Added range info to BN import log. Sascha Wilde <wilde@intevation.de> parents: 3705 diff changeset	494 // it can be used for debugging if something goes wrong...
aa7bede70b96 Added range info to BN import log. Sascha Wilde <wilde@intevation.de> parents: 3705 diff changeset	495 feedback.Info("Range: from %s to %s", bn.From_ISRS, bn.To_ISRS)
4881 9f1c33ba3f0a Report reference gauge on import of bottleneck Tom Gottfried <tom@intevation.de> parents: 4851 diff changeset	496 feedback.Info("Reference gauge: %s", bn.Fk_g_fid)
3815 aa7bede70b96 Added range info to BN import log. Sascha Wilde <wilde@intevation.de> parents: 3705 diff changeset	497
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	498 // Check if the new bottleneck intersects with the validity of existing
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	499 // for the same bottleneck_id we consider this an update and mark the
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	500 // old data for deletion.
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	501 bns, err := tx.StmtContext(ctx, bs.findIntersecting).QueryContext(
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	502 ctx, bn.Bottleneck_id, &validity,
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	503 )
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	504 if err != nil {
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	505 return err
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	506 }
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	507 defer bns.Close()
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	508
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	509 // Mark old intersecting bottleneck data for deletion. Don't worry about
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	510 // materials, they will be deleted via cascading.
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	511 var oldBnIDs []int64
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	512 for bns.Next() {
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	513 var oldID int64
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	514 err := bns.Scan(&oldID)
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	515 if err != nil {
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	516 return err
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	517 }
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	518 oldBnIDs = append(oldBnIDs, oldID)
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	519 }
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	520
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	521 if err := bns.Err(); err != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	522 return err
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	523 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	524
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	525 switch {
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	526 case len(oldBnIDs) == 1:
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	527 feedback.Info("Bottleneck '%s' "+
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	528 "with intersecting validity already exists: "+
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	529 "UPDATING", bn.Bottleneck_id)
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	530 case len(oldBnIDs) > 1:
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	531 // This case is unexpected and should only happen when historic
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	532 // data in the bottleneck service was changed subsequently...
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	533 // We handle it gracefully anyway, but warn.
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	534 feedback.Warn("More than one Bottelneck '%s' "+
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	535 "with intersecting validity already exists: "+
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	536 "REPLACING all of them!", bn.Bottleneck_id)
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	537 }
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	538 // We write the actual tracking information for deletion of superseded
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	539 // bottlenecks later to the database -- AFTER the new bottleneck was
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	540 // created successfully. That way, we don't change the database, when
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	541 // an error arises during inserting the new data.
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	542
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	543 var bnID int64
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	544 // Add new BN data:
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	545 if err := savepoint(func() error {
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	546 if err := tx.StmtContext(ctx, bs.insert).QueryRowContext(
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	547 ctx,
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	548 bn.Bottleneck_id,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	549 &validity,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	550 bn.Fk_g_fid,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	551 bn.OBJNAM,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	552 bn.NOBJNM,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	553 bn.From_ISRS, bn.To_ISRS,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	554 rb,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	555 lb,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	556 country,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	557 revisitingTime,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	558 limiting,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	559 bn.Date_Info,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	560 bn.Source,
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	561 tolerance,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	562 ).Scan(&bnID); err != nil {
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	563 return err
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	564 }
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	565
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	566 // Add new materials
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	567 for _, material := range materials {
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	568 if _, err := tx.StmtContext(ctx, bs.insertMaterial).ExecContext(
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	569 ctx,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	570 bnID,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	571 material,
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	572 ); err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	573 return err
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	574 }
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	575 }
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	576 return nil
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	577 }); err != nil {
4197 5d7ce7f926eb Log fatal errors as errors, not warnings, in imports Tom Gottfried <tom@intevation.de> parents: 4177 diff changeset	578 feedback.Error(pgxutils.ReadableError{Err: err}.Error())
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	579 return nil
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	580 }
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	581
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	582 // Now that adding BNs to staging was successful, write import tracking
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	583 // information to database:
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	584 for _, oldID := range oldBnIDs {
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	585 // It is possible, that two new bottlenecks intersect with the
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	586 // same old one, therefore we have to handle duplicates in
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	587 // oldBnIds.
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	588 if !seenOldBnIDs[oldID] {
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	589 if _, err := tx.StmtContext(ctx, bs.track).ExecContext(
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	590 ctx, importID, "waterway.bottlenecks", oldID, true,
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	591 ); err != nil {
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	592 return err
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	593 }
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	594 seenOldBnIDs[oldID] = true
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	595 }
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	596 }
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	597
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	598 if _, err := tx.StmtContext(ctx, bs.track).ExecContext(
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	599 ctx, importID, "waterway.bottlenecks", bnID, false,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	600 ); err != nil {
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	601 return err
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	602 }
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	603
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	604 nids = append(nids, bn.Bottleneck_id)
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	605 return nil
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	606 }

Mercurial > gemma

annotate pkg/imports/bn.go @ 5712:6270951dda28 revive-cleanup