gemma: pkg/imports/bn.go annotate

annotate pkg/imports/bn.go @ 5560:f2204f91d286

Join the log lines of imports to the log exports to recover data from them. Used in SR export to extract information that where in the meta json but now are only found in the log.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Wed, 09 Feb 2022 18:34:40 +0100
parents	59a99655f34d
children	6270951dda28

rev	line source
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	13 // * Tom Gottfried <tom.gottfried@intevation.de>
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	14 // * Sascha Wilde <sascha.wilde@intevation.de>
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 package imports
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 import (
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "context"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	20 "database/sql"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21 "errors"
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	22 "regexp"
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	23 "strconv"
3528 b9e331c1e616 Ignore zero length and whitespace-only values Tom Gottfried <tom@intevation.de> parents: 3423 diff changeset	24 "strings"
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	25 "time"
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	26
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	27 "github.com/jackc/pgx/pgtype"
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	28
4102 3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	29 "gemma.intevation.de/gemma/pkg/common"
4062 6c760abcff0e Move handling of PostgreSQL errors to own package Tom Gottfried <tom@intevation.de> parents: 4058 diff changeset	30 "gemma.intevation.de/gemma/pkg/pgxutils"
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	31 "gemma.intevation.de/gemma/pkg/soap/ifbn"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	32 )
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33
1695 be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	34 // Bottleneck is an import job to import
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	35 // bottlenecks from an IFBN SOAP service.
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36 type Bottleneck struct {
1695 be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	37 // URL is the URL of the SOAP service.
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	38 URL string `json:"url"`
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	39 // Tolerance used for axis snapping
dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	40 Tolerance float64 `json:"tolerance"`
1695 be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	41 // Insecure indicates if HTTPS traffic
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	42 // should validate certificates or not.
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	43 Insecure bool `json:"insecure"`
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1695 diff changeset	46 // BNJobKind is the import queue type identifier.
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47 const BNJobKind JobKind = "bn"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	49 const (
1658 07f996b0e14a Bottleneck import: Cleaned up source a bit (removed outdated TODOs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1646 diff changeset	50 insertBottleneckSQL = `
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	51 WITH
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	52 bounds (b) AS (VALUES (isrs_fromText($6)), (isrs_fromText($7))),
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	53 r AS (SELECT isrsrange(
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	54 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY),
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	55 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r)
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	56 INSERT INTO waterway.bottlenecks (
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	57 bottleneck_id,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	58 validity,
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3224 diff changeset	59 gauge_location,
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	60 objnam,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	61 nobjnm,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	62 stretch,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	63 area,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	64 rb,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	65 lb,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	66 responsible_country,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	67 revisiting_time,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	68 limiting,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	69 date_info,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	70 source_organization
4069 12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	71 ) VALUES (
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	72 $1,
4069 12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	73 $2::tstzrange,
12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	74 isrs_fromText($3),
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	75 $4,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	76 $5,
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	77 (SELECT r FROM r),
1631 6ee06eb37fa9 Use new function to calculate bn area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1611 diff changeset	78 ISRSrange_area(
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	79 ISRSrange_axis((SELECT r FROM r),
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	80 $15),
2248 cce158db02b0 Input area as multipolygons to generate area from stretch Tom Gottfried <tom@intevation.de> parents: 2195 diff changeset	81 (SELECT ST_Collect(CAST(area AS geometry))
1902 c4af342be999 Use waterway area as basis for bottleneck area generation. Sascha Wilde <wilde@intevation.de> parents: 1754 diff changeset	82 FROM waterway.waterway_area)),
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	83 $8,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	84 $9,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	85 $10,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	86 $11,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	87 $12,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	88 $13,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	89 $14
4069 12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	90 )
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	91 RETURNING id
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	92 `
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	93
4130 980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	94 // We only check for NOT NULL values, for correct compairison with
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	95 // values, which might be null (and then muyst not be compairt with `='
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	96 // but with `IS NULL' is comlicated and that we are checking more than
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	97 // only (bottleneck_id, validity, date_info) is luxury already.
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	98 findExactMatchBottleneckSQL = `
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	99 WITH
4130 980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	100 bounds (b) AS (VALUES (isrs_fromText($4)), (isrs_fromText($5))),
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	101 r AS (SELECT isrsrange(
4130 980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	102 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY),
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	103 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r)
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	104 SELECT id FROM waterway.bottlenecks
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	105 WHERE (
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	106 bottleneck_id,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	107 validity,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	108 gauge_location,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	109 stretch,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	110 responsible_country,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	111 limiting,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	112 date_info,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	113 source_organization,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	114 staging_done
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	115 ) = ( SELECT
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	116 $1,
4069 12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	117 $2::tstzrange,
12f476e91c70 Adapted bn-import to new historization model. Sascha Wilde <wilde@intevation.de> parents: 3815 diff changeset	118 isrs_fromText($3),
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	119 (SELECT r FROM r),
4130 980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	120 $6,
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	121 $7,
980f12d3c766 bottleneck import: Fixed detection of identical bottlenecks. Sascha Wilde <wilde@intevation.de> parents: 4128 diff changeset	122 $8::timestamptz,
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	123 $9,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	124 true
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	125 )
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	126 `
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	127
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	128 findIntersectingBottleneckSQL = `
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	129 SELECT id FROM waterway.bottlenecks
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	130 WHERE (bottleneck_id, staging_done) = ($1, true)
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	131 AND $2::tstzrange && validity
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	132 `
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	133
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	134 insertBottleneckMaterialSQL = `
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	135 INSERT INTO waterway.bottlenecks_riverbed_materials (
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	136 bottleneck_id,
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	137 riverbed
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	138 ) VALUES ($1, $2)
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	139 ON CONFLICT (bottleneck_id, riverbed) DO NOTHING
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	140 `
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	141
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	142 bnStageDoneDeleteSQL = `
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	143 DELETE FROM waterway.bottlenecks WHERE id IN (
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	144 SELECT key
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	145 FROM import.track_imports
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	146 WHERE import_id = $1
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	147 AND relation = 'waterway.bottlenecks'::regclass
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	148 AND deletion
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	149 )`
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	150
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	151 bnStageDoneSQL = `
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	152 UPDATE waterway.bottlenecks SET staging_done = true
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	153 WHERE id IN (
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	154 SELECT key
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	155 FROM import.track_imports
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	156 WHERE import_id = $1
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	157 AND relation = 'waterway.bottlenecks'::regclass
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	158 AND NOT deletion
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	159 )`
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	160 )
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	161
4851 faabfed7f0e9 Simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	162 // Description gives a short info about relevant facts of this import.
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	163 func (bn *Bottleneck) Description([]string) (string, error) {
4851 faabfed7f0e9 Simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	164 return bn.URL, nil
4798 ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4197 diff changeset	165 }
ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4197 diff changeset	166
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	167 type bnJobCreator struct{}
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	168
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	169 func init() {
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	170 RegisterJobCreator(BNJobKind, bnJobCreator{})
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	171 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	172
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	173 func (bnJobCreator) Description() string { return "bottlenecks" }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	174
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	175 func (bnJobCreator) AutoAccept() bool { return false }
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1631 diff changeset	176
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	177 func (bnJobCreator) Create() Job { return new(Bottleneck) }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	178
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3164 diff changeset	179 func (bnJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3164 diff changeset	180 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3164 diff changeset	181 {"bottlenecks", "bottlenecks_riverbed_materials"},
3224 bfde4f8dd323 Add missing table dependencies of imports with area generation Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	182 {"gauges", "distance_marks_virtual", "waterway_axis", "waterway_area"},
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	183 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	184 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	185
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	186 // StageDone moves the imported bottleneck out of the staging area.
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	187 func (bnJobCreator) StageDone(
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	188 ctx context.Context,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	189 tx *sql.Tx,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	190 id int64,
5034 59a99655f34d Added feedback support for StageDone. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4881 diff changeset	191 _ Feedback,
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	192 ) error {
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	193 _, err := tx.ExecContext(ctx, bnStageDoneDeleteSQL, id)
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	194 if err == nil {
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	195 _, err = tx.ExecContext(ctx, bnStageDoneSQL, id)
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	196 }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	197 return err
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	198 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	199
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	200 // CleanUp of a bottleneck import is a NOP.
1676 4407ecaa2192 Imports: Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	201 func (*Bottleneck) CleanUp() error { return nil }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	202
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	203 var rblbRe = regexp.MustCompile(`(..)_(..)`)
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	204
2765 41f5ac76d642 BT import: allow for rb and lb to be null. Sascha Wilde <wilde@intevation.de> parents: 2752 diff changeset	205 func splitRBLB(s string) (string, string) {
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	206 m := rblbRe.FindStringSubmatch(s)
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	207 if len(m) == 0 {
2765 41f5ac76d642 BT import: allow for rb and lb to be null. Sascha Wilde <wilde@intevation.de> parents: 2752 diff changeset	208 return nil, nil
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	209 }
2765 41f5ac76d642 BT import: allow for rb and lb to be null. Sascha Wilde <wilde@intevation.de> parents: 2752 diff changeset	210 return &m[1], &m[2]
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	211 }
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	212
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	213 // Do executes the actual bottleneck import.
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	214 func (bn *Bottleneck) Do(
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	215 ctx context.Context,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	216 importID int64,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	217 conn *sql.Conn,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	218 feedback Feedback,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	219 ) (interface{}, error) {
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	220
2253 cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2248 diff changeset	221 fetch := func() ([]*ifbn.BottleNeckType, error) {
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	222 client := ifbn.NewIBottleneckService(bn.URL, bn.Insecure, nil)
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	223
4102 3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	224 req := &ifbn.Export_bn_by_isrs{
3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	225 Period: &ifbn.RequestedPeriod{Date_start: &time.Time{}},
3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	226 }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	227
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	228 resp, err := client.Export_bn_by_isrs(req)
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	229 if err != nil {
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	230 return nil, err
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	231 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	232
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	233 if resp.Export_bn_by_isrsResult == nil {
3562 c03224d33e78 Improve error message Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	234 return nil, errors.New(
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4147 diff changeset	235 "the requested service returned no bottlenecks")
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	236 }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	237
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	238 return resp.Export_bn_by_isrsResult.BottleNeckType, nil
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	239 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	240
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	241 return storeBottlenecks(ctx, fetch, importID, conn, feedback, bn.Tolerance)
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	242 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	243
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	244 type bnStmts struct {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	245 insert *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	246 findExactMatch *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	247 findIntersecting *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	248 insertMaterial *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	249 track *sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	250 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	251
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	252 func (bs *bnStmts) close() {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	253 for _, s := range []**sql.Stmt{
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	254 &bs.insert,
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	255 &bs.findExactMatch,
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	256 &bs.findIntersecting,
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	257 &bs.insertMaterial,
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	258 &bs.track,
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	259 } {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	260 if *s != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	261 (*s).Close()
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	262 *s = nil
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	263 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	264 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	265 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	266
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	267 func (bs bnStmts) prepare(ctx context.Context, conn sql.Conn) error {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	268 for _, x := range []struct {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	269 sql string
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	270 stmt **sql.Stmt
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	271 }{
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	272 {insertBottleneckSQL, &bs.insert},
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	273 {findExactMatchBottleneckSQL, &bs.findExactMatch},
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	274 {findIntersectingBottleneckSQL, &bs.findIntersecting},
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	275 {insertBottleneckMaterialSQL, &bs.insertMaterial},
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	276 {trackImportDeletionSQL, &bs.track},
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	277 } {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	278 var err error
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	279 if *x.stmt, err = conn.PrepareContext(ctx, x.sql); err != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	280 return err
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	281 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	282 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	283 return nil
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	284 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	285
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	286 func storeBottlenecks(
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	287 ctx context.Context,
2253 cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2248 diff changeset	288 fetch func() ([]*ifbn.BottleNeckType, error),
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	289 importID int64,
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	290 conn *sql.Conn,
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	291 feedback Feedback,
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	292 tolerance float64,
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	293 ) (interface{}, error) {
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	294 start := time.Now()
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	295
2253 cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2248 diff changeset	296 bns, err := fetch()
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	297 if err != nil {
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	298 return nil, err
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	299 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	300
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	301 feedback.Info("Found %d bottlenecks for import", len(bns))
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	302
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	303 var bs bnStmts
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	304 defer bs.close()
1580 c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	305
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	306 if err := bs.prepare(ctx, conn); err != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	307 return nil, err
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	308 }
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	309
1611 1835d1c9eb9b Store bottleneck_id's instead of id's in bottleneck import summary. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1595 diff changeset	310 var nids []string
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	311 seenOldBnIDs := make(map[int64]bool)
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	312
2693 02d012272dd4 Log snapping tolerance Tom Gottfried <tom@intevation.de> parents: 2563 diff changeset	313 feedback.Info("Tolerance used to snap waterway axis: %g", tolerance)
02d012272dd4 Log snapping tolerance Tom Gottfried <tom@intevation.de> parents: 2563 diff changeset	314
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	315 tx, err := conn.BeginTx(ctx, nil)
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	316 if err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	317 return nil, err
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	318 }
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	319 defer tx.Rollback()
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	320
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	321 savepoint := Savepoint(ctx, tx, "insert_bottlenck")
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	322
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	323 for _, bn := range bns {
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	324 if err := storeBottleneck(
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	325 ctx, tx,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	326 importID, feedback, bn,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	327 &nids, seenOldBnIDs, tolerance,
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	328 &bs,
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	329 savepoint,
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	330 ); err != nil {
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	331 return nil, err
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	332 }
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	333 }
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	334 if err = tx.Commit(); err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	335 return nil, err
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	336 }
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	337
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	338 if len(nids) == 0 {
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	339 return nil, UnchangedError("No new bottlenecks inserted")
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	340 }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	341
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	342 feedback.Info("Storing %d bottlenecks took %s", len(nids), time.Since(start))
1906 32c56e6c089a Stretch import: Added forgotten source file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1902 diff changeset	343 feedback.Info("Import of bottlenecks was successful")
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	344 summary := struct {
1611 1835d1c9eb9b Store bottleneck_id's instead of id's in bottleneck import summary. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1595 diff changeset	345 Bottlenecks []string `json:"bottlenecks"`
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	346 }{
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	347 Bottlenecks: nids,
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	348 }
1906 32c56e6c089a Stretch import: Added forgotten source file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1902 diff changeset	349 return &summary, nil
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	350 }
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	351
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	352 func storeBottleneck(
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	353 ctx context.Context,
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	354 tx *sql.Tx,
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	355 importID int64,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	356 feedback Feedback,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	357 bn *ifbn.BottleNeckType,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	358 nids *[]string,
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	359 seenOldBnIDs map[int64]bool,
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	360 tolerance float64,
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	361 bs *bnStmts,
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	362 savepoint func(func() error) error,
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	363 ) error {
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	364 feedback.Info("Processing %s (%s)", bn.OBJNAM, bn.Bottleneck_id)
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	365
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	366 var (
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	367 tfrom, tto pgtype.Timestamptz
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	368 uBound pgtype.BoundType
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	369 )
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	370
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	371 if bn.AdditionalData == nil \|\| bn.AdditionalData.KeyValuePair == nil {
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	372 // This is a workaround for the fact that most BN data does not
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	373 // contain the optional validity information.
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	374 //
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	375 // The current solution makes every change in the data an
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	376 // update, which might be not the best solution. Alternative
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	377 // approaches could include usingthe Date Info of the data as
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	378 // start date, and maybe even inspecting the details of changed
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	379 // data for hints wether an update or historisation with a new
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	380 // version is advisable.
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	381 //
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	382 // Never the less, the current solution "just works" for the
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	383 // time being and reflects the upstream systems... -- sw
4134 9864d682ab47 bn-import: Downgraded info on missing validity from Warning to Info. Sascha Wilde <wilde@intevation.de> parents: 4133 diff changeset	384 feedback.Info("No validity information, assuming infinite validity.")
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	385 tfrom.Set(time.Date(1970, time.January, 1, 0, 0, 0, 0, time.UTC))
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	386 uBound = pgtype.Unbounded
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	387 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	388 const (
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	389 fromKey = "Valid_from_date"
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	390 toKey = "Valid_to_date"
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	391 )
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	392 fromTo := make(map[string]time.Time)
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	393 for _, kv := range bn.AdditionalData.KeyValuePair {
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	394 if kv.Key == fromKey \|\| kv.Key == toKey {
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	395 t, err := time.Parse(time.RFC3339, kv.Value)
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	396 if err != nil {
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	397 return err
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	398 }
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	399 fromTo[kv.Key] = t
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	400 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	401 }
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	402
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	403 if t, ok := fromTo[fromKey]; ok {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	404 tfrom.Set(t)
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	405 } else {
4197 5d7ce7f926eb Log fatal errors as errors, not warnings, in imports Tom Gottfried <tom@intevation.de> parents: 4177 diff changeset	406 feedback.Error("Missing start date")
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	407 return nil
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	408 }
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	409
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	410 if t, ok := fromTo[toKey]; ok {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	411 tto.Set(t)
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	412 uBound = pgtype.Exclusive
4102 3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	413 feedback.Info("Valid from %s to %s",
3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	414 fromTo[fromKey].Format(common.TimeFormat),
3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	415 fromTo[toKey].Format(common.TimeFormat))
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	416 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	417 uBound = pgtype.Unbounded
4102 3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	418 feedback.Info("Valid from %s",
3d2f02c16765 Request historic bottleneck data from SOAP service Tom Gottfried <tom@intevation.de> parents: 4071 diff changeset	419 fromTo[fromKey].Format(common.TimeFormat))
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	420 }
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	421 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	422
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	423 validity := pgtype.Tstzrange{
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	424 Lower: tfrom,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	425 Upper: tto,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	426 LowerType: pgtype.Inclusive,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	427 UpperType: uBound,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	428 Status: pgtype.Present,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	429 }
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	430
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	431 rb, lb := splitRBLB(bn.Rb_lb)
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	432
3423 6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	433 var revisitingTime *int
3528 b9e331c1e616 Ignore zero length and whitespace-only values Tom Gottfried <tom@intevation.de> parents: 3423 diff changeset	434 if bn.Revisiting_time != nil &&
b9e331c1e616 Ignore zero length and whitespace-only values Tom Gottfried <tom@intevation.de> parents: 3423 diff changeset	435 len(strings.TrimSpace(*bn.Revisiting_time)) > 0 {
3423 6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	436 i, err := strconv.Atoi(*bn.Revisiting_time)
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	437 if err != nil {
3530 e98ea184538e Improve warning message Tom Gottfried <tom@intevation.de> parents: 3528 diff changeset	438 feedback.Warn(
e98ea184538e Improve warning message Tom Gottfried <tom@intevation.de> parents: 3528 diff changeset	439 "Cannot convert given revisiting time '%s' to number of months",
3423 6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	440 *bn.Revisiting_time)
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	441 } else {
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	442 revisitingTime = &i
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	443 }
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	444 }
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	445
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	446 var limiting, country string
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	447
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	448 if bn.Limiting_factor != nil {
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	449 limiting = string(*bn.Limiting_factor)
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	450 }
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	451
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	452 if bn.Responsible_country != nil {
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	453 country = string(*bn.Responsible_country)
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	454 }
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	455
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	456 var materials []string
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	457 if bn.Riverbed != nil {
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	458 for _, material := range bn.Riverbed.Material {
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	459 if material != nil {
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	460 materials = append(materials, string(*material))
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	461 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	462 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	463 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	464
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	465 // Check if a bottleneck identical to the one we would insert already
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	466 // exists:
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	467 var old int64
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	468 err := tx.StmtContext(ctx, bs.findExactMatch).QueryRowContext(
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	469 ctx,
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	470 bn.Bottleneck_id,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	471 &validity,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	472 bn.Fk_g_fid,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	473 bn.From_ISRS, bn.To_ISRS,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	474 country,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	475 limiting,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	476 bn.Date_Info,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	477 bn.Source,
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	478 ).Scan(&old)
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	479 switch {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	480 case err == sql.ErrNoRows:
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	481 // We dont have a matching old.
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	482 case err != nil:
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	483 return err
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	484 default:
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	485 // We could check if the materials are also matching -- but per
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	486 // specification the Date_Info would hvae to change on that kind of
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	487 // change anyway. So actually we are already checking more in depth than
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	488 // required.
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	489 feedback.Info("unchanged")
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	490 return nil
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	491 }
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	492
3815 aa7bede70b96 Added range info to BN import log. Sascha Wilde <wilde@intevation.de> parents: 3705 diff changeset	493 // Additional Information, only of interest when we change something, so
aa7bede70b96 Added range info to BN import log. Sascha Wilde <wilde@intevation.de> parents: 3705 diff changeset	494 // it can be used for debugging if something goes wrong...
aa7bede70b96 Added range info to BN import log. Sascha Wilde <wilde@intevation.de> parents: 3705 diff changeset	495 feedback.Info("Range: from %s to %s", bn.From_ISRS, bn.To_ISRS)
4881 9f1c33ba3f0a Report reference gauge on import of bottleneck Tom Gottfried <tom@intevation.de> parents: 4851 diff changeset	496 feedback.Info("Reference gauge: %s", bn.Fk_g_fid)
3815 aa7bede70b96 Added range info to BN import log. Sascha Wilde <wilde@intevation.de> parents: 3705 diff changeset	497
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	498 // Check if the new bottleneck intersects with the validity of existing
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	499 // for the same bottleneck_id we consider this an update and mark the
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	500 // old data for deletion.
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	501 bns, err := tx.StmtContext(ctx, bs.findIntersecting).QueryContext(
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	502 ctx, bn.Bottleneck_id, &validity,
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	503 )
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	504 if err != nil {
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	505 return err
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	506 }
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	507 defer bns.Close()
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	508
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	509 // Mark old intersecting bottleneck data for deletion. Don't worry about
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	510 // materials, they will be deleted via cascading.
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	511 var oldBnIDs []int64
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	512 for bns.Next() {
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	513 var oldID int64
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	514 err := bns.Scan(&oldID)
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	515 if err != nil {
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	516 return err
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	517 }
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	518 oldBnIDs = append(oldBnIDs, oldID)
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	519 }
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	520
4128 14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	521 if err := bns.Err(); err != nil {
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	522 return err
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	523 }
14706384a464 Cleanup by Sascha L. Teichmann applied. Sascha Wilde <wilde@intevation.de> parents: 4119 diff changeset	524
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	525 switch {
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	526 case len(oldBnIDs) == 1:
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	527 feedback.Info("Bottleneck '%s' "+
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	528 "with intersecting validity already exists: "+
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	529 "UPDATING", bn.Bottleneck_id)
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	530 case len(oldBnIDs) > 1:
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	531 // This case is unexpected and should only happen when historic
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	532 // data in the bottleneck service was changed subsequently...
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	533 // We handle it gracefully anyway, but warn.
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	534 feedback.Warn("More than one Bottelneck '%s' "+
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	535 "with intersecting validity already exists: "+
f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	536 "REPLACING all of them!", bn.Bottleneck_id)
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	537 }
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	538 // We write the actual tracking information for deletion of superseded
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	539 // bottlenecks later to the database -- AFTER the new bottleneck was
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	540 // created successfully. That way, we don't change the database, when
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	541 // an error arises during inserting the new data.
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	542
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	543 var bnID int64
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	544 // Add new BN data:
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	545 if err := savepoint(func() error {
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	546 if err := tx.StmtContext(ctx, bs.insert).QueryRowContext(
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	547 ctx,
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	548 bn.Bottleneck_id,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	549 &validity,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	550 bn.Fk_g_fid,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	551 bn.OBJNAM,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	552 bn.NOBJNM,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	553 bn.From_ISRS, bn.To_ISRS,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	554 rb,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	555 lb,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	556 country,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	557 revisitingTime,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	558 limiting,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	559 bn.Date_Info,
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	560 bn.Source,
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	561 tolerance,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	562 ).Scan(&bnID); err != nil {
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	563 return err
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	564 }
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	565
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	566 // Add new materials
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	567 for _, material := range materials {
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	568 if _, err := tx.StmtContext(ctx, bs.insertMaterial).ExecContext(
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	569 ctx,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	570 bnID,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	571 material,
4119 0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	572 ); err != nil {
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	573 return err
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	574 }
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	575 }
0cf0008070db bottleneck import: Use a global transaction with savepoints. Sascha Wilde <wilde@intevation.de> parents: 4113 diff changeset	576 return nil
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	577 }); err != nil {
4197 5d7ce7f926eb Log fatal errors as errors, not warnings, in imports Tom Gottfried <tom@intevation.de> parents: 4177 diff changeset	578 feedback.Error(pgxutils.ReadableError{Err: err}.Error())
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	579 return nil
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	580 }
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	581
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	582 // Now that adding BNs to staging was successful, write import tracking
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	583 // information to database:
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	584 for _, oldID := range oldBnIDs {
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	585 // It is possible, that two new bottlenecks intersect with the
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	586 // same old one, therefore we have to handle duplicates in
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	587 // oldBnIds.
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	588 if !seenOldBnIDs[oldID] {
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	589 if _, err := tx.StmtContext(ctx, bs.track).ExecContext(
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	590 ctx, importID, "waterway.bottlenecks", oldID, true,
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	591 ); err != nil {
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	592 return err
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	593 }
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	594 seenOldBnIDs[oldID] = true
4133 eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	595 }
eb08fbe33074 Fixed bn-import: never write tracking info to the db if import fails. Sascha Wilde <wilde@intevation.de> parents: 4130 diff changeset	596 }
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	597
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	598 if _, err := tx.StmtContext(ctx, bs.track).ExecContext(
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	599 ctx, importID, "waterway.bottlenecks", bnID, false,
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	600 ); err != nil {
f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4134 diff changeset	601 return err
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	602 }
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	603
4113 f39d20427e89 WIP: Reworked bottleneck import: Sascha Wilde <wilde@intevation.de> parents: 4110 diff changeset	604 nids = append(nids, bn.Bottleneck_id)
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	605 return nil
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	606 }

Mercurial > gemma

annotate pkg/imports/bn.go @ 5560:f2204f91d286