gemma: pkg/imports/bn.go annotate

annotate pkg/imports/bn.go @ 3681:c9e1848a516a

Handle violation of responsibility areas graceful for BN import. As many others this is an error, which should only affect the specific bottleneck failing and not the whole import.

author	Sascha Wilde <wilde@intevation.de>
date	Tue, 18 Jun 2019 12:43:01 +0200
parents	0227670dedd5
children	063a1883b5cb

rev	line source
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	13 // * Tom Gottfried <tom.gottfried@intevation.de>
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15 package imports
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17 import (
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 "context"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "database/sql"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	20 "errors"
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	21 "regexp"
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	22 "strconv"
3528 b9e331c1e616 Ignore zero length and whitespace-only values Tom Gottfried <tom@intevation.de> parents: 3423 diff changeset	23 "strings"
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	24 "time"
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	25
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	26 "gemma.intevation.de/gemma/pkg/soap/ifbn"
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	27 "github.com/jackc/pgx/pgtype"
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	28 )
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	29
1695 be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	30 // Bottleneck is an import job to import
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	31 // bottlenecks from an IFBN SOAP service.
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	32 type Bottleneck struct {
1695 be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	33 // URL is the URL of the SOAP service.
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	34 URL string `json:"url"`
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	35 // Tolerance used for axis snapping
dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	36 Tolerance float64 `json:"tolerance"`
1695 be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	37 // Insecure indicates if HTTPS traffic
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	38 // should validate certificates or not.
be78b429ef6e Bottleneck import: Resolved remaining golint issues. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1676 diff changeset	39 Insecure bool `json:"insecure"`
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	40 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	41
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1695 diff changeset	42 // BNJobKind is the import queue type identifier.
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	43 const BNJobKind JobKind = "bn"
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	45 const (
1658 07f996b0e14a Bottleneck import: Cleaned up source a bit (removed outdated TODOs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1646 diff changeset	46 insertBottleneckSQL = `
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	47 WITH
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	48 bounds (b) AS (VALUES (isrs_fromText($6)), (isrs_fromText($7))),
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	49 r AS (SELECT isrsrange(
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	50 (SELECT b FROM bounds ORDER BY b USING <~ FETCH FIRST ROW ONLY),
4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	51 (SELECT b FROM bounds ORDER BY b USING >~ FETCH FIRST ROW ONLY)) AS r)
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	52 INSERT INTO waterway.bottlenecks (
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	53 bottleneck_id,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	54 validity,
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3224 diff changeset	55 gauge_location,
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3224 diff changeset	56 gauge_validity,
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	57 objnam,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	58 nobjnm,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	59 stretch,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	60 area,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	61 rb,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	62 lb,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	63 responsible_country,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	64 revisiting_time,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	65 limiting,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	66 date_info,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	67 source_organization
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	68 ) SELECT
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	69 $1,
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	70 validity * $2, -- intersections with gauge validity ranges
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	71 location,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	72 validity,
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	73 $4,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	74 $5,
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	75 (SELECT r FROM r),
1631 6ee06eb37fa9 Use new function to calculate bn area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1611 diff changeset	76 ISRSrange_area(
3566 4c585b5d4fe8 Introduce hectometer based order for ranges of ISRS location codes Tom Gottfried <tom@intevation.de> parents: 3562 diff changeset	77 ISRSrange_axis((SELECT r FROM r),
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	78 $15),
2248 cce158db02b0 Input area as multipolygons to generate area from stretch Tom Gottfried <tom@intevation.de> parents: 2195 diff changeset	79 (SELECT ST_Collect(CAST(area AS geometry))
1902 c4af342be999 Use waterway area as basis for bottleneck area generation. Sascha Wilde <wilde@intevation.de> parents: 1754 diff changeset	80 FROM waterway.waterway_area)),
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	81 $8,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	82 $9,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	83 $10,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	84 $11,
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	85 $12,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	86 $13,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	87 $14
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	88 FROM waterway.gauges
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	89 WHERE location = isrs_fromText($3) AND validity && $2
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	90 ON CONFLICT (bottleneck_id, validity) DO UPDATE SET
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	91 gauge_location = EXCLUDED.gauge_location,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	92 gauge_validity = EXCLUDED.gauge_validity,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	93 objnam = EXCLUDED.objnam,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	94 nobjnm = EXCLUDED.nobjnm,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	95 stretch = EXCLUDED.stretch,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	96 area = EXCLUDED.area,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	97 rb = EXCLUDED.rb,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	98 lb = EXCLUDED.lb,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	99 responsible_country = EXCLUDED.responsible_country,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	100 revisiting_time = EXCLUDED.revisiting_time,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	101 limiting = EXCLUDED.limiting,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	102 date_info = EXCLUDED.date_info,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	103 source_organization = EXCLUDED.source_organization
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	104 RETURNING id
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	105 `
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	106
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	107 // Alignment with gauge validity might have generated new entries
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	108 // for the same time range. Thus, remove the old ones
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	109 deleteObsoleteBNSQL = `
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	110 DELETE FROM waterway.bottlenecks
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	111 WHERE bottleneck_id = $1 AND validity <@ $2 AND id <> ALL($3)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	112 `
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	113
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	114 fixBNValiditySQL = `
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	115 UPDATE waterway.bottlenecks SET
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	116 -- Set enddate of old entry to new startdate in case of overlap:
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	117 validity = validity - $2
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	118 WHERE bottleneck_id = $1
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	119 AND validity && $2 AND NOT validity <@ $2
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	120 `
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	121
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	122 deleteBottleneckMaterialSQL = `
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	123 WITH del AS (
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	124 DELETE FROM waterway.bottlenecks_riverbed_materials
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	125 WHERE bottleneck_id = ANY($1)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	126 AND riverbed <> ALL($2)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	127 RETURNING riverbed)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	128 SELECT DISTINCT riverbed FROM del
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	129 `
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	130
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	131 insertBottleneckMaterialSQL = `
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	132 INSERT INTO waterway.bottlenecks_riverbed_materials (
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	133 bottleneck_id,
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	134 riverbed
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	135 ) SELECT *
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	136 FROM unnest(CAST($1 AS int[])) AS bns,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	137 unnest(CAST($2 AS varchar[])) AS materials
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	138 ON CONFLICT (bottleneck_id, riverbed) DO NOTHING
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	139 `
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	140 )
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	141
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	142 type bnJobCreator struct{}
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	143
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	144 func init() {
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	145 RegisterJobCreator(BNJobKind, bnJobCreator{})
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	146 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	147
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	148 func (bnJobCreator) Description() string { return "bottlenecks" }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	149
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	150 func (bnJobCreator) AutoAccept() bool { return false }
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1631 diff changeset	151
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	152 func (bnJobCreator) Create() Job { return new(Bottleneck) }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	153
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3164 diff changeset	154 func (bnJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3164 diff changeset	155 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3164 diff changeset	156 {"bottlenecks", "bottlenecks_riverbed_materials"},
3224 bfde4f8dd323 Add missing table dependencies of imports with area generation Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	157 {"gauges", "distance_marks_virtual", "waterway_axis", "waterway_area"},
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	158 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	159 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	160
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	161 const (
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	162 bnStageDoneSQL = `
1533 8fc1a2298acb Pseudo fixed unstaging of bottlenecks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1531 diff changeset	163 UPDATE waterway.bottlenecks SET staging_done = true
8fc1a2298acb Pseudo fixed unstaging of bottlenecks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1531 diff changeset	164 WHERE id IN (
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1975 diff changeset	165 SELECT key from import.track_imports
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	166 WHERE import_id = $1 AND
1533 8fc1a2298acb Pseudo fixed unstaging of bottlenecks. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1531 diff changeset	167 relation = 'waterway.bottlenecks'::regclass)`
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	168 )
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	169
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	170 // StageDone moves the imported bottleneck out of the staging area.
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	171 func (bnJobCreator) StageDone(
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	172 ctx context.Context,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	173 tx *sql.Tx,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	174 id int64,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	175 ) error {
1594 e3abd3b3ebc6 Fixed stage done SQL for bottleneck imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1580 diff changeset	176 _, err := tx.ExecContext(ctx, bnStageDoneSQL, id)
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	177 return err
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	178 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	179
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	180 // CleanUp of a bottleneck import is a NOP.
1676 4407ecaa2192 Imports: Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	181 func (*Bottleneck) CleanUp() error { return nil }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	182
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	183 var rblbRe = regexp.MustCompile(`(..)_(..)`)
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	184
2765 41f5ac76d642 BT import: allow for rb and lb to be null. Sascha Wilde <wilde@intevation.de> parents: 2752 diff changeset	185 func splitRBLB(s string) (string, string) {
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	186 m := rblbRe.FindStringSubmatch(s)
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	187 if len(m) == 0 {
2765 41f5ac76d642 BT import: allow for rb and lb to be null. Sascha Wilde <wilde@intevation.de> parents: 2752 diff changeset	188 return nil, nil
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	189 }
2765 41f5ac76d642 BT import: allow for rb and lb to be null. Sascha Wilde <wilde@intevation.de> parents: 2752 diff changeset	190 return &m[1], &m[2]
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	191 }
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	192
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	193 // Do executes the actual bottleneck import.
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	194 func (bn *Bottleneck) Do(
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	195 ctx context.Context,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	196 importID int64,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	197 conn *sql.Conn,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	198 feedback Feedback,
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	199 ) (interface{}, error) {
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	200
2253 cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2248 diff changeset	201 fetch := func() ([]*ifbn.BottleNeckType, error) {
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	202 client := ifbn.NewIBottleneckService(bn.URL, bn.Insecure, nil)
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	203
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	204 req := &ifbn.Export_bn_by_isrs{}
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	205
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	206 resp, err := client.Export_bn_by_isrs(req)
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	207 if err != nil {
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	208 return nil, err
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	209 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	210
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	211 if resp.Export_bn_by_isrsResult == nil {
3562 c03224d33e78 Improve error message Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	212 return nil, errors.New(
c03224d33e78 Improve error message Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	213 "The requested service returned no bottlenecks")
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	214 }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	215
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	216 return resp.Export_bn_by_isrsResult.BottleNeckType, nil
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	217 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	218
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	219 return storeBottlenecks(ctx, fetch, importID, conn, feedback, bn.Tolerance)
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	220 }
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	221
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	222 func storeBottlenecks(
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	223 ctx context.Context,
2253 cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2248 diff changeset	224 fetch func() ([]*ifbn.BottleNeckType, error),
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	225 importID int64,
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	226 conn *sql.Conn,
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	227 feedback Feedback,
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	228 tolerance float64,
2195 3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	229 ) (interface{}, error) {
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	230 start := time.Now()
3437ceee1529 Uploaded bottleneck import: Implemented in terms of normal bottleneck import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	231
2253 cfc523c70e90 Uploaded imports: Small code clean-ups. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2248 diff changeset	232 bns, err := fetch()
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	233 if err != nil {
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	234 return nil, err
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	235 }
24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	236
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	237 feedback.Info("Found %d bottlenecks for import", len(bns))
cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	238
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	239 var insertStmt, deleteObsoleteBNStmt, fixValidityStmt,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	240 deleteMaterialStmt, insertMaterialStmt, trackStmt *sql.Stmt
1580 c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	241
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	242 for _, x := range []struct {
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	243 sql string
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	244 stmt **sql.Stmt
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	245 }{
1658 07f996b0e14a Bottleneck import: Cleaned up source a bit (removed outdated TODOs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1646 diff changeset	246 {insertBottleneckSQL, &insertStmt},
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	247 {deleteObsoleteBNSQL, &deleteObsoleteBNStmt},
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	248 {fixBNValiditySQL, &fixValidityStmt},
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	249 {deleteBottleneckMaterialSQL, &deleteMaterialStmt},
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	250 {insertBottleneckMaterialSQL, &insertMaterialStmt},
1580 c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	251 {trackImportSQL, &trackStmt},
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	252 } {
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	253 var err error
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	254 if *x.stmt, err = conn.PrepareContext(ctx, x.sql); err != nil {
1580 c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	255 return nil, err
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	256 }
c503d2fa9776 Bottleneck import: Use a prepared statement for import tracking, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	257 defer (*x.stmt).Close()
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	258 }
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	259
1611 1835d1c9eb9b Store bottleneck_id's instead of id's in bottleneck import summary. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1595 diff changeset	260 var nids []string
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	261
2693 02d012272dd4 Log snapping tolerance Tom Gottfried <tom@intevation.de> parents: 2563 diff changeset	262 feedback.Info("Tolerance used to snap waterway axis: %g", tolerance)
02d012272dd4 Log snapping tolerance Tom Gottfried <tom@intevation.de> parents: 2563 diff changeset	263
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	264 for _, bn := range bns {
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	265 if err := storeBottleneck(
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	266 ctx, importID, conn, feedback, bn, &nids, tolerance,
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	267 insertStmt, deleteObsoleteBNStmt, fixValidityStmt,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	268 deleteMaterialStmt, insertMaterialStmt, trackStmt); err != nil {
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	269 return nil, err
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	270 }
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	271 }
056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	272 if len(nids) == 0 {
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	273 return nil, UnchangedError("No new bottlenecks inserted")
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	274 }
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	275
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	276 feedback.Info("Storing %d bottlenecks took %s", len(nids), time.Since(start))
1906 32c56e6c089a Stretch import: Added forgotten source file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1902 diff changeset	277 feedback.Info("Import of bottlenecks was successful")
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	278 summary := struct {
1611 1835d1c9eb9b Store bottleneck_id's instead of id's in bottleneck import summary. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1595 diff changeset	279 Bottlenecks []string `json:"bottlenecks"`
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	280 }{
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1569 diff changeset	281 Bottlenecks: nids,
1569 cf0ca4a9812a Bottleneck import: Implemented backend to store bottlenecks in database. Raimund Renkert <raimund.renkert@intevation.de> parents: 1533 diff changeset	282 }
1906 32c56e6c089a Stretch import: Added forgotten source file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1902 diff changeset	283 return &summary, nil
1531 24445a618513 Added stub for bottleneck importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	284 }
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	285
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	286 func storeBottleneck(
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	287 ctx context.Context,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	288 importID int64,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	289 conn *sql.Conn,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	290 feedback Feedback,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	291 bn *ifbn.BottleNeckType,
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	292 nids *[]string,
2563 dc4fae4bdb8f Expose axis snapping tolerance to users Tom Gottfried <tom@intevation.de> parents: 2548 diff changeset	293 tolerance float64,
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	294 insertStmt, deleteObsoleteBNStmt, fixValidityStmt,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	295 deleteMaterialStmt, insertMaterialStmt, trackStmt *sql.Stmt,
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	296 ) error {
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	297 feedback.Info("Processing %s (%s)", bn.OBJNAM, bn.Bottleneck_id)
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	298
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	299 var tfrom, tto pgtype.Timestamptz
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	300 var uBound pgtype.BoundType
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	301
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	302 if bn.AdditionalData == nil \|\| bn.AdditionalData.KeyValuePair == nil {
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	303 // This is a workaround for the fact that most BN data does not
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	304 // contain the optional validity information.
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	305 //
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	306 // The current solution makes every change in the data an
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	307 // update, which might be not the best solution. Alternative
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	308 // approaches could include usingthe Date Info of the data as
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	309 // start date, and maybe even inspecting the details of changed
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	310 // data for hints wether an update or historisation with a new
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	311 // version is advisable.
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	312 //
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	313 // Never the less, the current solution "just works" for the
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	314 // rtime being... -- sw
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	315 feedback.Warn("No validity information, assuming infinite validity.")
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	316 tfrom.Set(time.Date(1970, time.January, 1, 0, 0, 0, 0, time.UTC))
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	317 uBound = pgtype.Unbounded
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	318 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	319 const (
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	320 fromKey = "Valid_from_date"
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	321 toKey = "Valid_to_date"
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	322 )
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	323 fromTo := make(map[string]time.Time)
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	324 for _, kv := range bn.AdditionalData.KeyValuePair {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	325 k := string(kv.Key)
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	326 if k == fromKey \|\| k == toKey {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	327 if t, err := time.Parse(time.RFC3339, kv.Value); err != nil {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	328 return err
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	329 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	330 fromTo[k] = t
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	331 }
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	332 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	333 }
3679 0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	334
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	335 if t, ok := fromTo[fromKey]; ok {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	336 tfrom.Set(t)
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	337 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	338 feedback.Warn("Missing start date")
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	339 return nil
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	340 }
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	341
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	342 if t, ok := fromTo[toKey]; ok {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	343 tto.Set(t)
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	344 uBound = pgtype.Exclusive
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	345 } else {
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	346 uBound = pgtype.Unbounded
0227670dedd5 Use infinite validity for bottlenecks with no validity information. Sascha Wilde <wilde@intevation.de> parents: 3666 diff changeset	347 }
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	348 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	349
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	350 validity := pgtype.Tstzrange{
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	351 Lower: tfrom,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	352 Upper: tto,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	353 LowerType: pgtype.Inclusive,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	354 UpperType: uBound,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	355 Status: pgtype.Present,
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	356 }
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	357
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	358 rb, lb := splitRBLB(bn.Rb_lb)
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	359
3423 6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	360 var revisitingTime *int
3528 b9e331c1e616 Ignore zero length and whitespace-only values Tom Gottfried <tom@intevation.de> parents: 3423 diff changeset	361 if bn.Revisiting_time != nil &&
b9e331c1e616 Ignore zero length and whitespace-only values Tom Gottfried <tom@intevation.de> parents: 3423 diff changeset	362 len(strings.TrimSpace(*bn.Revisiting_time)) > 0 {
3423 6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	363 i, err := strconv.Atoi(*bn.Revisiting_time)
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	364 if err != nil {
3530 e98ea184538e Improve warning message Tom Gottfried <tom@intevation.de> parents: 3528 diff changeset	365 feedback.Warn(
e98ea184538e Improve warning message Tom Gottfried <tom@intevation.de> parents: 3528 diff changeset	366 "Cannot convert given revisiting time '%s' to number of months",
3423 6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	367 *bn.Revisiting_time)
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	368 } else {
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	369 revisitingTime = &i
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	370 }
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	371 }
6592396f5061 Make revisiting time of a bottleneck optional Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	372
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	373 var limiting, country string
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	374
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	375 if bn.Limiting_factor != nil {
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	376 limiting = string(*bn.Limiting_factor)
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	377 }
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	378
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	379 if bn.Responsible_country != nil {
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	380 country = string(*bn.Responsible_country)
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	381 }
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	382
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	383 var materials []string
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	384 if bn.Riverbed != nil {
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	385 for _, material := range bn.Riverbed.Material {
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	386 if material != nil {
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	387 materials = append(materials, string(*material))
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	388 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	389 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	390 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	391
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	392 tx, err := conn.BeginTx(ctx, nil)
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	393 if err != nil {
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	394 return err
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	395 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	396 defer tx.Rollback()
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	397
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	398 var bnIds []int64
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	399 bns, err := tx.StmtContext(ctx, insertStmt).QueryContext(ctx,
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	400 bn.Bottleneck_id,
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	401 &validity,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	402 bn.Fk_g_fid,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	403 bn.OBJNAM,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	404 bn.NOBJNM,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	405 bn.From_ISRS, bn.To_ISRS,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	406 rb,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	407 lb,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	408 country,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	409 revisitingTime,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	410 limiting,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	411 bn.Date_Info,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	412 bn.Source,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	413 tolerance,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	414 )
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	415 if err != nil {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	416 feedback.Warn(handleError(err).Error())
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	417 return nil
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	418 }
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	419 defer bns.Close()
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	420 for bns.Next() {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	421 var nid int64
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	422 if err := bns.Scan(&nid); err != nil {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	423 return err
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	424 }
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	425 bnIds = append(bnIds, nid)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	426 }
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	427 if err := bns.Err(); err != nil {
3681 c9e1848a516a Handle violation of responsibility areas graceful for BN import. Sascha Wilde <wilde@intevation.de> parents: 3679 diff changeset	428 feedback.Warn(handleError(err).Error())
c9e1848a516a Handle violation of responsibility areas graceful for BN import. Sascha Wilde <wilde@intevation.de> parents: 3679 diff changeset	429 return nil
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	430 }
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	431 if len(bnIds) == 0 {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	432 feedback.Warn(
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	433 "No gauge matching '%s' or given time available", bn.Fk_g_fid)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	434 return nil
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	435 }
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	436
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	437 // Remove obsolete bottleneck version entries
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	438 var pgBnIds pgtype.Int8Array
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	439 pgBnIds.Set(bnIds)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	440 if _, err = tx.StmtContext(ctx, deleteObsoleteBNStmt).ExecContext(ctx,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	441 bn.Bottleneck_id,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	442 &validity,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	443 &pgBnIds,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	444 ); err != nil {
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	445 feedback.Warn(handleError(err).Error())
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	446 if err2 := tx.Rollback(); err2 != nil {
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	447 return err2
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	448 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	449 return nil
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	450 }
02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	451
3648 0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	452 // Set end of validity of old version to start of new version
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	453 // in case of overlap
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	454 if _, err = tx.StmtContext(ctx, fixValidityStmt).ExecContext(ctx,
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	455 bn.Bottleneck_id,
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	456 validity,
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	457 ); err != nil {
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	458 feedback.Warn(handleError(err).Error())
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	459 if err2 := tx.Rollback(); err2 != nil {
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	460 return err2
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	461 }
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	462 return nil
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	463 }
0ec5c8ec1e44 Avoid empty validity time ranges Tom Gottfried <tom@intevation.de> parents: 3645 diff changeset	464
3645 02951a62e8c6 'Historicise' bottlenecks on import Tom Gottfried <tom@intevation.de> parents: 3566 diff changeset	465 if materials != nil {
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	466 // Remove obsolete riverbed materials
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	467 var pgMaterials pgtype.VarcharArray
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	468 pgMaterials.Set(materials)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	469 mtls, err := tx.StmtContext(ctx,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	470 deleteMaterialStmt).QueryContext(ctx,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	471 &pgBnIds,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	472 &pgMaterials,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	473 )
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	474 if err != nil {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	475 return err
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	476 }
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	477 defer mtls.Close()
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	478 for mtls.Next() {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	479 var delMat string
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	480 if err := mtls.Scan(&delMat); err != nil {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	481 return err
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	482 }
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	483 feedback.Warn("Removed riverbed material %s", delMat)
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	484 }
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	485 if err := mtls.Err(); err != nil {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	486 return err
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	487 }
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	488
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	489 // Insert riverbed materials
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	490 if _, err := tx.StmtContext(ctx, insertMaterialStmt).ExecContext(ctx,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	491 &pgBnIds,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	492 &pgMaterials,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	493 ); err != nil {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	494 feedback.Warn("Failed to insert riverbed materials")
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	495 feedback.Warn(handleError(err).Error())
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	496 return nil
2999 b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	497 }
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	498 }
b3c3c5b5b7c1 Bottleneck import: Import riverbed materials, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2765 diff changeset	499
3666 db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	500 for _, nid := range bnIds {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	501 if _, err := tx.StmtContext(ctx, trackStmt).ExecContext(
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	502 ctx, importID, "waterway.bottlenecks", nid,
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	503 ); err != nil {
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	504 return err
db87f34805fb Align bottleneck validity at gauges Tom Gottfried <tom@intevation.de> parents: 3665 diff changeset	505 }
2532 452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	506 }
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	507 if err = tx.Commit(); err != nil {
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	508 return err
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	509 }
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	510 nids = append(nids, bn.Bottleneck_id)
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	511 return nil
452bc714bfd9 Continue importing bottlenecks if one item fails Tom Gottfried <tom@intevation.de> parents: 2253 diff changeset	512 }

Mercurial > gemma

annotate pkg/imports/bn.go @ 3681:c9e1848a516a