gemma: pkg/imports/bn.go annotate

annotate pkg/imports/bn.go @ 3705:7006b92c0334

Handle updates (vs. historized and new versions) separately. We need this distinction as updated data currently can not be reviewed. More precisely: it can not be declined after review, as the old data is updated in place. The current exclusion from the review is a workaround and not meant to be the final solution. Note that there are additional minor problems, like the fact that the updated data is not counted as changed data for the import.

author	Sascha Wilde <wilde@intevation.de>
date	Wed, 19 Jun 2019 17:00:08 +0200
parents	b07511ff859e
children	aa7bede70b96

rev	line source
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	13 // * Tom Gottfried <tom.gottfried@intevation.de>
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	14 // * Sascha Wilde <sascha.wilde@intevation.de>
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 package imports
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 import (
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "context"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	20 "database/sql"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21 "errors"
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	22 "regexp"
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	23 "strconv"
3528 b9e331c1e616 Ignore zero length and whitespace-only values Tom Gottfried <tom@intevation.de> parents: 3423 diff changeset	24 "strings"
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	25 "time"
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	26
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	27 "gemma.intevation.de/gemma/pkg/soap/ifbn"
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	28 "github.com/jackc/pgx/pgtype"
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	29 )
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	30
1695 be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	31 // Bottleneck is an import job to import
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	32 // bottlenecks from an IFBN SOAP service.
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33 type Bottleneck struct {
1695 be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	34 // URL is the URL of the SOAP service.
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	35 URL string `json:"url"`
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	36 // Tolerance used for axis snapping
dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	37 Tolerance float64 `json:"tolerance"`
1695 be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	38 // Insecure indicates if HTTPS traffic
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	39 // should validate certificates or not.
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	40 Insecure bool `json:"insecure"`
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	41 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	42
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1695 diff changeset	43 // BNJobKind is the import queue type identifier.
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44 const BNJobKind JobKind = "bn"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	46 const (
1658 07f996b0e14a Bottleneck import: Cleaned up source a bit (removed outdated TODOs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1646 diff changeset	47 insertBottleneckSQL = `
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	48 WITH
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	49 bounds (b) AS (VALUES (isrs_fromText($6)), (isrs_fromText($7))),
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	50 r AS (SELECT isrsrange(
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	51 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY),
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	52 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r)
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	53 INSERT INTO waterway.bottlenecks (
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	54 bottleneck_id,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	55 validity,
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3224 diff changeset	56 gauge_location,
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3224 diff changeset	57 gauge_validity,
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	58 objnam,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	59 nobjnm,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	60 stretch,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	61 area,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	62 rb,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	63 lb,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	64 responsible_country,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	65 revisiting_time,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	66 limiting,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	67 date_info,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	68 source_organization
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	69 ) SELECT
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	70 $1,
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	71 validity * $2, -- intersections with gauge validity ranges
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	72 location,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	73 validity,
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	74 $4,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	75 $5,
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	76 (SELECT r FROM r),
1631 6ee06eb37fa9 Use new function to calculate bn area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1611 diff changeset	77 ISRSrange_area(
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	78 ISRSrange_axis((SELECT r FROM r),
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	79 $15),
2248 cce158db02b0 Input area as multipolygons to generate area from stretch Tom Gottfried <tom@intevation.de> parents: 2195 diff changeset	80 (SELECT ST_Collect(CAST(area AS geometry))
1902 c4af342be999 Use waterway area as basis for bottleneck area generation. Sascha Wilde <wilde@intevation.de> parents: 1754 diff changeset	81 FROM waterway.waterway_area)),
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	82 $8,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	83 $9,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	84 $10,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	85 $11,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	86 $12,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	87 $13,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	88 $14
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	89 FROM waterway.gauges
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	90 WHERE location = isrs_fromText($3) AND validity && $2
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	91 RETURNING id
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	92 `
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	93
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	94 updateBottleneckSQL = `
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	95 WITH
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	96 bounds (b) AS (VALUES (isrs_fromText($7)), (isrs_fromText($8))),
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	97 r AS (SELECT isrsrange(
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	98 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY),
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	99 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r)
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	100 UPDATE waterway.bottlenecks SET (
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	101 bottleneck_id,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	102 validity,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	103 gauge_location,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	104 gauge_validity,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	105 objnam,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	106 nobjnm,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	107 stretch,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	108 area,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	109 rb,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	110 lb,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	111 responsible_country,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	112 revisiting_time,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	113 limiting,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	114 date_info,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	115 source_organization
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	116 ) = ( SELECT
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	117 $2,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	118 validity * $3, -- intersections with gauge validity ranges
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	119 location,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	120 validity,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	121 $5,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	122 $6,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	123 (SELECT r FROM r),
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	124 ISRSrange_area(
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	125 ISRSrange_axis((SELECT r FROM r),
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	126 $16),
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	127 (SELECT ST_Collect(CAST(area AS geometry))
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	128 FROM waterway.waterway_area)),
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	129 $9,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	130 $10,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	131 $11,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	132 $12::smallint,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	133 $13,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	134 $14::timestamptz,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	135 $15
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	136 FROM waterway.gauges
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	137 WHERE location = isrs_fromText($4) AND validity && $3 )
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	138 WHERE id=$1
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	139 RETURNING id
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	140 `
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	141
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	142 findExactMatchBottleneckSQL = `
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	143 WITH
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	144 bounds (b) AS (VALUES (isrs_fromText($6)), (isrs_fromText($7))),
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	145 r AS (SELECT isrsrange(
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	146 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY),
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	147 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r)
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	148 SELECT id FROM waterway.bottlenecks
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	149 WHERE (
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	150 bottleneck_id,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	151 validity,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	152 gauge_location,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	153 gauge_validity,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	154 objnam,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	155 nobjnm,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	156 stretch,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	157 rb,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	158 lb,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	159 responsible_country,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	160 revisiting_time,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	161 limiting,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	162 date_info,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	163 source_organization,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	164 staging_done
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	165 ) = ( SELECT
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	166 $1,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	167 validity * $2, -- intersections with gauge validity ranges
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	168 location,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	169 validity,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	170 $4,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	171 $5,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	172 (SELECT r FROM r),
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	173 $8,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	174 $9,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	175 $10,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	176 $11::smallint,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	177 $12,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	178 $13::timestamptz,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	179 $14,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	180 true
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	181 FROM waterway.gauges
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	182 WHERE location = isrs_fromText($3) AND validity && $2
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	183 )
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	184 `
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	185
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	186 findMatchBottleneckSQL = `
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	187 SELECT id FROM waterway.bottlenecks
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	188 WHERE (
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	189 bottleneck_id,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	190 validity,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	191 staging_done
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	192 ) = ( SELECT
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	193 $1,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	194 validity * $2, -- intersections with gauge validity ranges
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	195 true
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	196 FROM waterway.gauges
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	197 WHERE location = isrs_fromText($3) AND validity && $2
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	198 )
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	199 `
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	200 // Alignment with gauge validity might have generated new entries
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	201 // for the same time range. Thus, remove the old ones
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	202 deleteObsoleteBNSQL = `
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	203 DELETE FROM waterway.bottlenecks
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	204 WHERE bottleneck_id = $1 AND validity <@ $2 AND id <> ALL($3)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	205 `
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	206
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	207 fixBNValiditySQL = `
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	208 UPDATE waterway.bottlenecks SET
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	209 -- Set enddate of old entry to new startdate in case of overlap:
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	210 validity = validity - $2
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	211 WHERE bottleneck_id = $1
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	212 AND validity && $2 AND NOT validity <@ $2
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	213 `
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	214
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	215 deleteBottleneckMaterialSQL = `
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	216 WITH del AS (
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	217 DELETE FROM waterway.bottlenecks_riverbed_materials
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	218 WHERE bottleneck_id = ANY($1)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	219 AND riverbed <> ALL($2)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	220 RETURNING riverbed)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	221 SELECT DISTINCT riverbed FROM del
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	222 `
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	223
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	224 insertBottleneckMaterialSQL = `
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	225 INSERT INTO waterway.bottlenecks_riverbed_materials (
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	226 bottleneck_id,
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	227 riverbed
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	228 ) SELECT *
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	229 FROM unnest(CAST($1 AS int[])) AS bns,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	230 unnest(CAST($2 AS varchar[])) AS materials
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	231 ON CONFLICT (bottleneck_id, riverbed) DO NOTHING
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	232 `
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	233 )
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	234
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	235 type bnJobCreator struct{}
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	236
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	237 func init() {
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	238 RegisterJobCreator(BNJobKind, bnJobCreator{})
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	239 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	240
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	241 func (bnJobCreator) Description() string { return "bottlenecks" }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	242
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	243 func (bnJobCreator) AutoAccept() bool { return false }
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1631 diff changeset	244
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	245 func (bnJobCreator) Create() Job { return new(Bottleneck) }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	246
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3164 diff changeset	247 func (bnJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3164 diff changeset	248 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3164 diff changeset	249 {"bottlenecks", "bottlenecks_riverbed_materials"},
3224 bfde4f8dd323 Add missing table dependencies of imports with area generation Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	250 {"gauges", "distance_marks_virtual", "waterway_axis", "waterway_area"},
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	251 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	252 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	253
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	254 const (
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	255 bnStageDoneSQL = `
1533 8fc1a2298acb Pseudo fixed unstaging of bottlenecks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1531 diff changeset	256 UPDATE waterway.bottlenecks SET staging_done = true
8fc1a2298acb Pseudo fixed unstaging of bottlenecks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1531 diff changeset	257 WHERE id IN (
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1975 diff changeset	258 SELECT key from import.track_imports
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	259 WHERE import_id = $1 AND
1533 8fc1a2298acb Pseudo fixed unstaging of bottlenecks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1531 diff changeset	260 relation = 'waterway.bottlenecks'::regclass)`
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	261 )
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	262
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	263 // StageDone moves the imported bottleneck out of the staging area.
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	264 func (bnJobCreator) StageDone(
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	265 ctx context.Context,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	266 tx *sql.Tx,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	267 id int64,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	268 ) error {
1594 e3abd3b3ebc6 Fixed stage done SQL for bottleneck imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1580 diff changeset	269 _, err := tx.ExecContext(ctx, bnStageDoneSQL, id)
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	270 return err
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	271 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	272
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	273 // CleanUp of a bottleneck import is a NOP.
1676 4407ecaa2192 Imports: Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	274 func (*Bottleneck) CleanUp() error { return nil }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	275
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	276 var rblbRe = regexp.MustCompile(`(..)_(..)`)
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	277
2765 41f5ac76d642 BT import: allow for rb and lb to be null. Sascha Wilde <wilde@intevation.de> parents: 2752 diff changeset	278 func splitRBLB(s string) (string, string) {
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	279 m := rblbRe.FindStringSubmatch(s)
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	280 if len(m) == 0 {
2765 41f5ac76d642 BT import: allow for rb and lb to be null. Sascha Wilde <wilde@intevation.de> parents: 2752 diff changeset	281 return nil, nil
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	282 }
2765 41f5ac76d642 BT import: allow for rb and lb to be null. Sascha Wilde <wilde@intevation.de> parents: 2752 diff changeset	283 return &m[1], &m[2]
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	284 }
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	285
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	286 // Do executes the actual bottleneck import.
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	287 func (bn *Bottleneck) Do(
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	288 ctx context.Context,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	289 importID int64,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	290 conn *sql.Conn,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	291 feedback Feedback,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	292 ) (interface{}, error) {
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	293
2253 cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2248 diff changeset	294 fetch := func() ([]*ifbn.BottleNeckType, error) {
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	295 client := ifbn.NewIBottleneckService(bn.URL, bn.Insecure, nil)
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	296
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	297 req := &ifbn.Export_bn_by_isrs{}
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	298
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	299 resp, err := client.Export_bn_by_isrs(req)
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	300 if err != nil {
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	301 return nil, err
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	302 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	303
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	304 if resp.Export_bn_by_isrsResult == nil {
3562 c03224d33e78 Improve error message Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	305 return nil, errors.New(
c03224d33e78 Improve error message Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	306 "The requested service returned no bottlenecks")
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	307 }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	308
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	309 return resp.Export_bn_by_isrsResult.BottleNeckType, nil
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	310 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	311
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	312 return storeBottlenecks(ctx, fetch, importID, conn, feedback, bn.Tolerance)
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	313 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	314
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	315 func storeBottlenecks(
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	316 ctx context.Context,
2253 cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2248 diff changeset	317 fetch func() ([]*ifbn.BottleNeckType, error),
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	318 importID int64,
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	319 conn *sql.Conn,
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	320 feedback Feedback,
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	321 tolerance float64,
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	322 ) (interface{}, error) {
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	323 start := time.Now()
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	324
2253 cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2248 diff changeset	325 bns, err := fetch()
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	326 if err != nil {
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	327 return nil, err
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	328 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	329
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	330 feedback.Info("Found %d bottlenecks for import", len(bns))
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	331
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	332 var insertStmt, updateStmt, findExactMatchingBNStmt, findMatchingBNStmt,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	333 deleteObsoleteBNStmt, fixValidityStmt, deleteMaterialStmt,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	334 insertMaterialStmt, trackStmt *sql.Stmt
1580 c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	335
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	336 for _, x := range []struct {
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	337 sql string
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	338 stmt **sql.Stmt
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	339 }{
1658 07f996b0e14a Bottleneck import: Cleaned up source a bit (removed outdated TODOs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1646 diff changeset	340 {insertBottleneckSQL, &insertStmt},
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	341 {updateBottleneckSQL, &updateStmt},
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	342 {findExactMatchBottleneckSQL, &findExactMatchingBNStmt},
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	343 {findMatchBottleneckSQL, &findMatchingBNStmt},
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	344 {deleteObsoleteBNSQL, &deleteObsoleteBNStmt},
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	345 {fixBNValiditySQL, &fixValidityStmt},
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	346 {deleteBottleneckMaterialSQL, &deleteMaterialStmt},
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	347 {insertBottleneckMaterialSQL, &insertMaterialStmt},
1580 c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	348 {trackImportSQL, &trackStmt},
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	349 } {
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	350 var err error
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	351 if *x.stmt, err = conn.PrepareContext(ctx, x.sql); err != nil {
1580 c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	352 return nil, err
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	353 }
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	354 defer (*x.stmt).Close()
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	355 }
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	356
1611 1835d1c9eb9b Store bottleneck_id's instead of id's in bottleneck import summary. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1595 diff changeset	357 var nids []string
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	358
2693 02d012272dd4 Log snapping tolerance Tom Gottfried <tom@intevation.de> parents: 2563 diff changeset	359 feedback.Info("Tolerance used to snap waterway axis: %g", tolerance)
02d012272dd4 Log snapping tolerance Tom Gottfried <tom@intevation.de> parents: 2563 diff changeset	360
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	361 for _, bn := range bns {
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	362 if err := storeBottleneck(
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	363 ctx, importID, conn, feedback, bn, &nids, tolerance,
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	364 insertStmt, updateStmt,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	365 findExactMatchingBNStmt, findMatchingBNStmt,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	366 deleteObsoleteBNStmt, fixValidityStmt,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	367 deleteMaterialStmt, insertMaterialStmt,
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	368 trackStmt); err != nil {
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	369 return nil, err
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	370 }
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	371 }
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	372 if len(nids) == 0 {
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	373 return nil, UnchangedError("No new bottlenecks inserted")
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	374 }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	375
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	376 feedback.Info("Storing %d bottlenecks took %s", len(nids), time.Since(start))
1906 32c56e6c089a Stretch import: Added forgotten source file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1902 diff changeset	377 feedback.Info("Import of bottlenecks was successful")
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	378 summary := struct {
1611 1835d1c9eb9b Store bottleneck_id's instead of id's in bottleneck import summary. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1595 diff changeset	379 Bottlenecks []string `json:"bottlenecks"`
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	380 }{
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	381 Bottlenecks: nids,
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	382 }
1906 32c56e6c089a Stretch import: Added forgotten source file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1902 diff changeset	383 return &summary, nil
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	384 }
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	385
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	386 func storeBottleneck(
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	387 ctx context.Context,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	388 importID int64,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	389 conn *sql.Conn,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	390 feedback Feedback,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	391 bn *ifbn.BottleNeckType,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	392 nids *[]string,
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	393 tolerance float64,
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	394 insertStmt, updateStmt,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	395 findExactMatchingBNStmt, findMatchingBNStmt,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	396 deleteObsoleteBNStmt, fixValidityStmt,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	397 deleteMaterialStmt, insertMaterialStmt,
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	398 trackStmt *sql.Stmt,
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	399 ) error {
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	400 feedback.Info("Processing %s (%s)", bn.OBJNAM, bn.Bottleneck_id)
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	401
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	402 var tfrom, tto pgtype.Timestamptz
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	403 var uBound pgtype.BoundType
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	404
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	405 if bn.AdditionalData == nil \|\| bn.AdditionalData.KeyValuePair == nil {
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	406 // This is a workaround for the fact that most BN data does not
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	407 // contain the optional validity information.
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	408 //
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	409 // The current solution makes every change in the data an
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	410 // update, which might be not the best solution. Alternative
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	411 // approaches could include usingthe Date Info of the data as
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	412 // start date, and maybe even inspecting the details of changed
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	413 // data for hints wether an update or historisation with a new
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	414 // version is advisable.
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	415 //
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	416 // Never the less, the current solution "just works" for the
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	417 // rtime being... -- sw
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	418 feedback.Warn("No validity information, assuming infinite validity.")
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	419 tfrom.Set(time.Date(1970, time.January, 1, 0, 0, 0, 0, time.UTC))
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	420 uBound = pgtype.Unbounded
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	421 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	422 const (
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	423 fromKey = "Valid_from_date"
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	424 toKey = "Valid_to_date"
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	425 )
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	426 fromTo := make(map[string]time.Time)
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	427 for _, kv := range bn.AdditionalData.KeyValuePair {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	428 k := string(kv.Key)
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	429 if k == fromKey \|\| k == toKey {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	430 if t, err := time.Parse(time.RFC3339, kv.Value); err != nil {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	431 return err
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	432 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	433 fromTo[k] = t
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	434 }
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	435 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	436 }
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	437
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	438 if t, ok := fromTo[fromKey]; ok {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	439 tfrom.Set(t)
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	440 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	441 feedback.Warn("Missing start date")
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	442 return nil
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	443 }
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	444
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	445 if t, ok := fromTo[toKey]; ok {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	446 tto.Set(t)
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	447 uBound = pgtype.Exclusive
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	448 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	449 uBound = pgtype.Unbounded
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	450 }
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	451 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	452
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	453 validity := pgtype.Tstzrange{
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	454 Lower: tfrom,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	455 Upper: tto,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	456 LowerType: pgtype.Inclusive,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	457 UpperType: uBound,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	458 Status: pgtype.Present,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	459 }
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	460
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	461 rb, lb := splitRBLB(bn.Rb_lb)
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	462
3423 6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	463 var revisitingTime *int
3528 b9e331c1e616 Ignore zero length and whitespace-only values Tom Gottfried <tom@intevation.de> parents: 3423 diff changeset	464 if bn.Revisiting_time != nil &&
b9e331c1e616 Ignore zero length and whitespace-only values Tom Gottfried <tom@intevation.de> parents: 3423 diff changeset	465 len(strings.TrimSpace(*bn.Revisiting_time)) > 0 {
3423 6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	466 i, err := strconv.Atoi(*bn.Revisiting_time)
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	467 if err != nil {
3530 e98ea184538e Improve warning message Tom Gottfried <tom@intevation.de> parents: 3528 diff changeset	468 feedback.Warn(
e98ea184538e Improve warning message Tom Gottfried <tom@intevation.de> parents: 3528 diff changeset	469 "Cannot convert given revisiting time '%s' to number of months",
3423 6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	470 *bn.Revisiting_time)
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	471 } else {
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	472 revisitingTime = &i
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	473 }
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	474 }
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	475
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	476 var limiting, country string
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	477
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	478 if bn.Limiting_factor != nil {
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	479 limiting = string(*bn.Limiting_factor)
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	480 }
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	481
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	482 if bn.Responsible_country != nil {
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	483 country = string(*bn.Responsible_country)
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	484 }
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	485
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	486 var materials []string
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	487 if bn.Riverbed != nil {
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	488 for _, material := range bn.Riverbed.Material {
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	489 if material != nil {
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	490 materials = append(materials, string(*material))
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	491 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	492 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	493 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	494
3698 063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	495 // Check if an bottleneck identical to the one we would insert already
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	496 // exists:
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	497 bns, err := findExactMatchingBNStmt.QueryContext(ctx,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	498 bn.Bottleneck_id,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	499 &validity,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	500 bn.Fk_g_fid,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	501 bn.OBJNAM,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	502 bn.NOBJNM,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	503 bn.From_ISRS, bn.To_ISRS,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	504 rb,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	505 lb,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	506 country,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	507 revisitingTime,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	508 limiting,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	509 bn.Date_Info,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	510 bn.Source,
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	511 )
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	512
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	513 if err != nil {
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	514 return err
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	515 }
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	516 defer bns.Close()
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	517 if bns.Next() {
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	518 feedback.Info("unchanged")
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	519 return nil
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	520 }
063a1883b5cb Detect and handle unchanged BN during import. Sascha Wilde <wilde@intevation.de> parents: 3681 diff changeset	521
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	522 // Check if an bottleneck with the same identity
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	523 // (bottleneck_id,validity) already exists:
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	524 // Check if an bottleneck identical to the one we would insert already
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	525 // exists:
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	526 var existing_bn_id *int64
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	527 err = findMatchingBNStmt.QueryRowContext(ctx,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	528 bn.Bottleneck_id,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	529 &validity,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	530 bn.Fk_g_fid,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	531 ).Scan(&existing_bn_id)
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	532 switch {
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	533 case err == sql.ErrNoRows:
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	534 existing_bn_id = nil
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	535 case err != nil:
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	536 // This is unexpected and propably a serious error
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	537 return err
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	538 }
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	539
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	540 tx, err := conn.BeginTx(ctx, nil)
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	541 if err != nil {
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	542 return err
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	543 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	544 defer tx.Rollback()
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	545
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	546 var bnIds []int64
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	547 if existing_bn_id != nil {
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	548 feedback.Info("Bottelneck '%s' "+
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	549 "with matching validity already exists:"+
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	550 "UPDATING", bn.Bottleneck_id)
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	551 // Updating existnig BN data:
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	552 bns, err = tx.StmtContext(ctx, updateStmt).QueryContext(ctx,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	553 existing_bn_id,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	554 bn.Bottleneck_id,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	555 &validity,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	556 bn.Fk_g_fid,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	557 bn.OBJNAM,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	558 bn.NOBJNM,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	559 bn.From_ISRS, bn.To_ISRS,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	560 rb,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	561 lb,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	562 country,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	563 revisitingTime,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	564 limiting,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	565 bn.Date_Info,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	566 bn.Source,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	567 tolerance,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	568 )
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	569 } else {
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	570 // New BN data:
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	571 bns, err = tx.StmtContext(ctx, insertStmt).QueryContext(ctx,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	572 bn.Bottleneck_id,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	573 &validity,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	574 bn.Fk_g_fid,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	575 bn.OBJNAM,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	576 bn.NOBJNM,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	577 bn.From_ISRS, bn.To_ISRS,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	578 rb,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	579 lb,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	580 country,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	581 revisitingTime,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	582 limiting,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	583 bn.Date_Info,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	584 bn.Source,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	585 tolerance,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	586 )
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	587 }
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	588 if err != nil {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	589 feedback.Warn(handleError(err).Error())
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	590 return nil
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	591 }
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	592 defer bns.Close()
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	593 for bns.Next() {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	594 var nid int64
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	595 if err := bns.Scan(&nid); err != nil {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	596 return err
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	597 }
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	598 bnIds = append(bnIds, nid)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	599 }
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	600 if err := bns.Err(); err != nil {
3681 c9e1848a516a Handle violation of responsibility areas graceful for BN import. Sascha Wilde <wilde@intevation.de> parents: 3679 diff changeset	601 feedback.Warn(handleError(err).Error())
c9e1848a516a Handle violation of responsibility areas graceful for BN import. Sascha Wilde <wilde@intevation.de> parents: 3679 diff changeset	602 return nil
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	603 }
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	604 if len(bnIds) == 0 {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	605 feedback.Warn(
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	606 "No gauge matching '%s' or given time available", bn.Fk_g_fid)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	607 return nil
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	608 }
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	609
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	610 // Remove obsolete bottleneck version entries
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	611 var pgBnIds pgtype.Int8Array
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	612 pgBnIds.Set(bnIds)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	613 if _, err = tx.StmtContext(ctx, deleteObsoleteBNStmt).ExecContext(ctx,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	614 bn.Bottleneck_id,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	615 &validity,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	616 &pgBnIds,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	617 ); err != nil {
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	618 feedback.Warn(handleError(err).Error())
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	619 if err2 := tx.Rollback(); err2 != nil {
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	620 return err2
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	621 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	622 return nil
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	623 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	624
3648 0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	625 // Set end of validity of old version to start of new version
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	626 // in case of overlap
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	627 if _, err = tx.StmtContext(ctx, fixValidityStmt).ExecContext(ctx,
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	628 bn.Bottleneck_id,
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	629 validity,
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	630 ); err != nil {
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	631 feedback.Warn(handleError(err).Error())
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	632 if err2 := tx.Rollback(); err2 != nil {
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	633 return err2
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	634 }
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	635 return nil
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	636 }
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	637
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	638 if materials != nil {
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	639 // Remove obsolete riverbed materials
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	640 var pgMaterials pgtype.VarcharArray
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	641 pgMaterials.Set(materials)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	642 mtls, err := tx.StmtContext(ctx,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	643 deleteMaterialStmt).QueryContext(ctx,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	644 &pgBnIds,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	645 &pgMaterials,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	646 )
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	647 if err != nil {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	648 return err
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	649 }
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	650 defer mtls.Close()
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	651 for mtls.Next() {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	652 var delMat string
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	653 if err := mtls.Scan(&delMat); err != nil {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	654 return err
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	655 }
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	656 feedback.Warn("Removed riverbed material %s", delMat)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	657 }
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	658 if err := mtls.Err(); err != nil {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	659 return err
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	660 }
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	661
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	662 // Insert riverbed materials
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	663 if _, err := tx.StmtContext(ctx, insertMaterialStmt).ExecContext(ctx,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	664 &pgBnIds,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	665 &pgMaterials,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	666 ); err != nil {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	667 feedback.Warn("Failed to insert riverbed materials")
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	668 feedback.Warn(handleError(err).Error())
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	669 return nil
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	670 }
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	671 }
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	672
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	673 // Only add new BN data to tracking for staging review.
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	674 //
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	675 // FIXME: Review for updated bottlenecks is currently not possible, as
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	676 // the update is done instantly in place.
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	677 if existing_bn_id == nil {
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	678 for _, nid := range bnIds {
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	679 if _, err := tx.StmtContext(ctx, trackStmt).ExecContext(
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	680 ctx, importID, "waterway.bottlenecks", nid,
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	681 ); err != nil {
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	682 return err
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	683 }
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	684 }
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	685 }
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	686
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	687 if err = tx.Commit(); err != nil {
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	688 return err
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	689 }
3705 7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	690 // See above...
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	691 if existing_bn_id == nil {
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	692 nids = append(nids, bn.Bottleneck_id)
7006b92c0334 Handle updates (vs. historized and new versions) separately. Sascha Wilde <wilde@intevation.de> parents: 3703 diff changeset	693 }
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	694 return nil
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	695 }

Mercurial > gemma

annotate pkg/imports/bn.go @ 3705:7006b92c0334